continualai / avalanche Goto Github PK
View Code? Open in Web Editor NEWAvalanche: an End-to-End Library for Continual Learning based on PyTorch.
Home Page: http://avalanche.continualai.org
License: MIT License
Avalanche: an End-to-End Library for Continual Learning based on PyTorch.
Home Page: http://avalanche.continualai.org
License: MIT License
I tried to run getting_started.py
with Split MNIST instead of Permuted MNIST but EvalProtocol
is crashing.
It is probably some problem caused by the missing classes in each task but I didn't look deeply at the code yet.
---------------------------------------------------------------------------
ValueError Traceback (most recent call last)
~/avalanche/examples/getting_started.py in <module>
59
60 # testing
---> 61 results.append(clmodel.test(test_full))
~/avalanche/avalanche/training/strategies/strategy.py in test(self, test_set)
183 self.after_task_test()
184
--> 185 self.eval_protocol.update_tb_test(res, self.batch_processed)
186
187 self.after_test()
~/avalanche/avalanche/evaluation/eval_protocol.py in update_tb_test(self, res, step)
100 in_out_scalars = {
101 "in_class": np.average(in_class_diff),
--> 102 "out_class": np.average(out_class_diff)
103 }
104
<__array_function__ internals> in average(*args, **kwargs)
~/anaconda3/lib/python3.7/site-packages/numpy/lib/function_base.py in average(a, axis, weights, returned)
391
392 if weights is None:
--> 393 avg = a.mean(axis)
394 scl = avg.dtype.type(a.size/avg.size)
395 else:
~/anaconda3/lib/python3.7/site-packages/numpy/core/_methods.py in _mean(a, axis, dtype, out, keepdims)
149 is_float16_result = True
150
--> 151 ret = umr_sum(arr, axis, dtype, out, keepdims)
152 if isinstance(ret, mu.ndarray):
153 ret = um.true_divide(
ValueError: operands could not be broadcast together with shapes (4,) (6,)
We need to add the CCA metric (described here). Adding it only in Tensorboard would be fine I think.
We should add a simple metric that keeps track of the elapsed time of the experiment.
Unlike other datasets like MNIST, CORe50 must be explicitly downloaded from the user. It would be better to have the same interface for all datasets.
Moreover, this behavior also breaks the script examples/simple_core50.py
, since the provided data folder does not exist.
CWR* and AR1 can either use an optimizer passed by the caller, or create one with the lr and momentum parameters.
I think the optimizer should not be an argument because right now they are silently overriding the user's choice. Let me know if there is any reason for this otherwise I will remove the parameter.
Check if it's possible to create a conda package (with dependencies) from GitHub.
This will be useful to create a "scenario" that follows the NCGenericScenario
API and works with any CL dataset based on filelists.
We need to add the metric related to the CPU/GPU consumption over time.
Add additional usage examples in the "examples" directory as a showcase of the avalanche functionalities.
Matplotlib complains that figures are left open when creating images of Confusion Matrices.
The exact warning is:
avalanche/evaluation/metrics.py:292: RuntimeWarning: More than 20 figures have been opened. Figures created through the pyplot interface (
matplotlib.pyplot.figure
) are retained until explicitly closed and may consume too much memory. (To control this warning, see the rcParamfigure.max_open_warning
).
We should consider closing them in a "finally" block.
We need to add the CIFAR-10 dataset (split of 2 classes each).
Up to now, CifarSplit and ICifar require the user to have the dataset stored locally. We should make the interface the same as all the other datasets and download them automatically if needed.
Add the ICaRL strategy to the main CL Baselines.
Add ImageNet as a benchmark based on the Pytorch data loader.
Add the EWC strategy as additional CL baseline.
We need to create the instructions to install the avalanche package directly from the master on GitHub, so we don't have to package it every time.
Could be an interesting dataset to have / explore.
paper here: https://arxiv.org/abs/1909.04951
what do you think ?
Hi @AntonioCarta, I've noticed this function in the LearningWithoutForgetting
class is never used, do we need it?
Add and test a basic rehearsal strategy!
Add Tiny-ImageNet to the benchmarks.
We need a custom Tensorboard logging class to pass to the evaluation protocol class. This class is important to give the user more control over tb logging regardless of the chosen metrics.
I am starting to play around with the Strategy class and I would like to propose some changes:
train
method with the entire training loop. This causes a lot of code duplication. I would like to have a compute_loss
method that gets called inside train
.after_train
, before_train
, ...) are all implemented as abstract methods. This means that each new strategy must define the method itself. Most of the strategies will implement these as empty methods. Can't we give a default empty implementation?multi_head
is not used. What is it doing?Finally, I think that we should try to separate logging code (tensorboard, print statements) from training code. The EvalProtocol should be the only one doing the logging. However, this is less urgent right now.
I can do the changes, but first I wanted to discuss them with you.
Notice that already existing code is not affected by these changes.
Before releasing our project to the public, the API should be well documented and easy to understand.
Check if all data loaders have the same API: In particular, they should return PyTorch tensors, not numpy!
Add unit tests using unittest.
Add the AR1 strategy to the other available baselines.
Metrics and EvalProtocol are a little bit unclear to me.
compute
method. Each time we add a new metric, we also have to add a new if case inside EvalProtocol's get_results.I would prefer a generic EvalProtocol that controls printing and logging and only delegates the computations to the metrics (e.g. instead of printing inside compute
EvalProtocol calls the __str__
method). I would also prefer to be able to choose where to print the metrics (output file, tensorboard, stdout).
Add a basic LwF strategy.
When a pull request is opened, it is build by travis CI to evaluate the correctness of the code. By now travis only signal if a build is passed ot failed, without any further information (e.g. why is failed, what are the error ecc..). It would be nice to have more feedback from travis, in order to immediately know why a build is failed without the need to open travis and inspect the console.
Already implemented in my private codebase, working on porting it to Avalanche.
This class will allow the user to create a NC (New Classes) scenario given a couple of generic train and test Datasets.
The user will be able to create a manager instance that will be an iterable. This iterable will output the incremental "task"s or "batch"es (terminology to be defined) and will also allow the user to execute certain task/batch complex management operations.
This is very similar to the current loader being implemented in the Avalanche codebase, but will allow the user to plug in his/her own dataset. Also, due to being extremely generic, this will speed-up the integration of new datasets.
The code I've already implemented in my private codebase works fine, but is complex. I'm working on slimming it down a bit. Here is a list of already implemented features, please feel free to comment if you feel we need even more features!
Features (already implemented):
Side features (already implemented):
To be defined (even in future development phases):
Feel free to comment.
Keep up with the excellent work you've been doing!
Similar to the generic New Classes manager, a New Instance manager will allow us to streamline the creation of New Insances benchmarks.
The NI Manager should mainly focus on the SIT scenario.
Key features:
The NI manager should also include features found in the NCScenario. For instance:
We should release Avalanche with a proper Continuous Integration system.
The CI setup should include:
@ggraffieti is already working on #3 for docs: I think that it'll be a good starting point.
Major obstacles are:
I'm creating this issue as "low priority" but we should definitely consider releasing the 0.1.0 version of Avalanche not before a decent CI setup has been defined.
I created a new environment for the project.
Using the environment.yml
file I was not able to install pytorchcv through conda.
I used pip to install pytorchcv 0.0.58.
However, the example in examples/getting_started.py
is not working for me.
ImportError: cannot import name 'DwsConvBlock' from 'pytorchcv.models.mobilenet' (/home/carta/anaconda3/envs/avalanche-env/lib/python3.8/site-packages/pytorchcv/models/mobilenet.py)
Maybe I need a different version of the package?
@vlomonaco or anyone else that is able to run the examples, can you tell me you pytorchcv version?
you can use pip list | grep pytorchcv
.
I do not know the library but from a quick look at the implementation it seems that they refactored their code, changing the name and location of the convolutional blocks.
ICifar100 dataset is present both in benchmarks/cdata_loaders/cifar_split.py
and in benchmarks/cdata_loaders/icifar100.py
. The versions however are different.
Is there a reason to keep two different versions? Otherwise we should either keep the correct version or merge them into icifar100.py
.
Moreover, in cifar_split.py
there is a typo in get_grow_test_set
method, which should be get_growing_testset
in order to be compliant with the avalanche interface.
The Strategy
API should be changed to suit the new Datasets API, i.e. it should be able to be trained on a batch_info
object. Since all the stratgies will break based on this, we can create a new Strategy file for now!
It would be nice to have also a MAC metric. It's difficult to compute it in native pytorch for every possible layer, but it would be a nice, hardware independed metric.
Any idea on how to do this easily in PyTorch?
Create a usage tutorial for google colab using the conda environment.
Add additional tests for each continual data loader.
We need to add some features statistics in tensorboard.
The user should be able to specify them during the creation of the tensorboard object. For now the TensorboardLogging
object is created in eval_protocol.py
I think it could be useful to add this quite popular approach. It could highlight pros and cons when monitored through the Memory and CPU Usage metrics.
We need to add the metric related to the disk usage: it would be nice to have both I/O usage and a check the size of an additional directory to be used by each strategy to store things that are not in RAM.
Create the sphinx auto documentation on gh-pages to be build with Travis CI.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.