Comments (10)
Hi,
first you should determine if it's a problem of overfitting or underfitting. You can use the same data for training and validation to see if the error rate on the training set goes towards 0. If this happens, then you have overfitting, which probably means that your trainset is too small. If you don't even fit the training data well, then something is broken. In this case, maybe try to use the normal split of the dataset and the provided config file and see if you can reach a good result.
from returnn.
@pvoigtlaender
Thanks! I will try that.
What about LEARNING rates?
Any justification for using 0.0005, 0.0003, 0.0001 ?
I see there is code that calculates error rates from epoch!
Thanks again for help out!
from returnn.
I think you are talking about learning rates, not error rates, right? We determined them empirically, so basically trial and error. For a different setup it might be useful to adapt it.
from returnn.
Sure, how long does it take your research group to train the entire IAM dataset (with config_real) ?
And roughly how many epochs?
Is gtx 1080 ti + i7 machine up to the task?
from returnn.
GTX 1080 TI + i7 is definitely enough! I don't remember the exact values anymore, but I think we trained for ~80 epochs and 1 epoch might take around half an hour (very roughly).
from returnn.
Thanks!
Great Help!
from returnn.
If for some reason, I had to interrupt the training, I could always continue training from where I stopped by setting the "start_epoch" parameter in the configure file to "last saved epoch +1".
This should have no effect on the whole training result, right!
I hope not!
Thanks!
from returnn.
I think you can just rerun without any changes to the config and RETURNN will load the most recent model.
from returnn.
Got it!
from returnn.
Not sure that this is about a bug in Returnn, so closing.
from returnn.
Related Issues (20)
- Support `torch.compile` for RF
- RF backend: PyTorch code
- Different effective learning rate reported over gpus HOT 11
- CUDA error: initialization error HOT 3
- MultiProcDataset inside PyTorch DataLoader with num_workers>0, multiple issues HOT 4
- RuntimeError: CUDA error: unspecified launch failure HOT 2
- NonDaemonicSpawnProcess hangs at exit HOT 2
- High memory usage with datasets (specifically when multi procs are used)
- Hang at exit in TDL worker in multiprocessing `_run_finalizers`, deadlock in `_wait_for_tstate_lock`? HOT 6
- Hang HOT 2
- Returnn Native after using different apptainer uses old compilation HOT 6
- MetaDataset with sequence list filter file
- HDFDataset (or generic dataset) post processing HOT 15
- Dataset batching like ESPnet support
- torch.nn.functional.conv2d: RuntimeError: GET was unable to find an engine to execute this computation HOT 1
- TensorFlow 2.14 degradation in WER HOT 2
- Updates for recent TensorFlow version
- Hang in dataset iterator HOT 5
- Log GPU device for torch backend HOT 2
- torch.onnx.export requires input_names and output_names to be in order HOT 12
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from returnn.