Comments (7)
Hi,
I am not sure about 10, but something like 1.0 will be okay.
Nevertheless, it depends, so for best approach you can checkout original paper (I think so) - A disciplined approach to neural network hyper-parameters: Part 1--learning rate, batch size, momentum, and weight decay
from catalyst.
-
After checking Smith's reports I'm not sure about necessity default value. Moreover, after running some tests on my problem I observed expected convergence-divergence but then convergence again with large lr > 1. Such insights will be missed if I use default value. And, probably, it would be better if user would ask himself about final_lr.
-
However, implementing 'terminating on nan' as a standalone callback, outside LRFinder, seems to be better option since it reusability. Such callback would be like a 'early stopping', which, in general, may occurs when overfitting detected, etc. I think, it must be taken into consideration for better code structure.
-
Also I see, doc for LRFinder mentions 'log' and 'linear' options, however these are not realized yet.
-
And what about tests? I don't see examples to follow in catalyst, except test_main.py. How should be they organized?
-
And one more thing. It is more convenient to look at (learning_rate - loss) plot, not to separate lr and loss. But the only way I see to realize it is generate such plot inside LRFinder and translate it to tensorboard or to file.
-
Probably it is worth to add some usage examples for LRFinder.
from catalyst.
So, speaking about LRFinder and 'log' and 'linear' options (as I understand final_lr
is okay for everyone). There was some experiments with difference lr scheduling options, but in the end log
one was found most appropriate to find optimal LR.
Nevertheless, different LRFinder options or (learning_rate - loss) plot contribution is always welcome.
Except (learning_rate - loss) plot implementation should look something like make_report
script to make it really simple and reusable.
'terminating on nan' or as we call it DebugCallback/ExceptionHandler - is really good idea and it must be done soon. The main problem here is that it can't be done like a callback, because it need to wrap all of them and handle their exceptions. One possible solution for ExceptionHandler implementation is another method in Runner and wrap all callbacks with it. Nevertheless, I am still looking for more flexible solution.
And finally, tests, the most tricky part :)
Currently we are using only integration tests during PR. All other tests and their structure are WIP now, and will be the main goal for 19.02 release.
from catalyst.
After some investigation I see that the only way to get (lr-loss) plot is parse logs.txt, isn't it? The easiest way, of course, just use regular expressions in script. But if the log format changes, it will stop working. logging
module doesn't provide any parsing method, moreover message
has custom format. However, parsing logs may be required in some other places in perspective. May be, it is worth to add static method for parsing directly in Logger
callback?
from catalyst.
So, I think you can modify LoggerCallback to write all metrics in some logs.json
also. It should be quite easy, because all batch metrics are dicts.
PS. btw, speaking about the reports I still hope to implement the idea from Scitator/catalyst-examples#1 (comment), because this grid search plots are amazing.
from catalyst.
Hi!
Turning back to (learning rate - loss) plot, where it would be better to place script? contrib/sctipts? Or, may be, dl/scripts, as it provide basic functionality?
from catalyst.
Hi,
Speaking about LrFinder and LR-metric plot, dl-scripts are the best place to use. At least, if you don't need some heave libs like tensorflow or nmslib (current contrib ones).
from catalyst.
Related Issues (20)
- Dependency `packaging` not specified — ModuleNotFoundError HOT 2
- os.environ["CUDA_VISIBLE_DEVICES"] = "" does not use CPU HOT 5
- comprehensive classification example HOT 3
- `runner.evaluate_loader` does not work with DataParallelEngine HOT 4
- How to enforce WandbLogger::log_artifact to be invoked? HOT 2
- Idea: offload all launch-related code Accelerate HOT 2
- Custom loader stages HOT 1
- Crashes on 2xT4 GPUs HOT 3
- utils.process_model_params HOT 2
- Multi Criterion Training HOT 4
- No utils.initialization file HOT 3
- DynamicBalanceClassSampler example does not use sampler HOT 1
- KWArg fp16 does not exist HOT 6
- Replace getters with properties HOT 1
- Bug in catalyst/callbacks/backward.py if the grad_clip_fn value is set. HOT 2
- Bug in catalyst/callbacks/backward.py if the grad_clip_fn value is set. HOT 1
- Bug in catalyst/callbacks/backward.py if the grad_clip_fn value is set. HOT 1
- `DataParallelEngine.prepare_model` missing `device_placement` kwarg. HOT 2
- Columns and DataType Not Explicitly Set on line 96 of report.py HOT 2
- Importing DistributedSamplerWrapper will invalidate the setting CUDA_VISIBLE_DEVICE. HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from catalyst.