Comments (6)
Yes, that would be ideal!
from avalanche.
Actually, it was more of a question than a proposal. I still don't have a clear idea on how to solve this problem.
from avalanche.
I think the EvalProtocol
job should be exactly what you said:
I would prefer a generic EvalProtocol that controls printing and logging and only delegates the computations to the metrics (e.g. instead of printing inside compute EvalProtocol calls the str method). I would also prefer to be able to choose where to print the metrics (output file, tensorboard, stdout).
As for the metrics, as soon as we have all the the metrics implemented we can define an unique signature for the compute
method... let's wait a little on this!
from avalanche.
@vlomonaco Can I work on this and also it would be helpful if you could suggest some initial metrics that are needed and also is issue #51 solved?
from avalanche.
#51 is closed.
Regarding the Metrics, maybe we can use Flows to define how and when to compute each metric? The problem that we have right now is that each metrics requires different computations and arguments. This could be easily solved with a callback system, like the flows used for training and test.
Take as an example the memory usage MU. It should be printed only once, instead it is printed everywhere because the EvaluationProtocol does not know how to print it:
Training completed
Computing accuracy on the whole test set
Task 0 - CF: 0.0000
Train Task 0 - MU: 0.259 GB
Confusion matrix, without normalization
[Evaluation] Task 0: Avg Loss 0.002227343698385049; Avg Acc 0.9242105484008789
Task 0 - CF: 0.9242
Train Task 0 - MU: 0.265 GB
Confusion matrix, without normalization
[Evaluation] Task 0: Avg Loss 0.06294100686298648; Avg Acc 0.0
Task 0 - CF: 0.9242
Train Task 0 - MU: 0.265 GB
Confusion matrix, without normalization
[Evaluation] Task 0: Avg Loss 0.0536595421147698; Avg Acc 0.0
Task 0 - CF: 0.9242
Train Task 0 - MU: 0.268 GB
Confusion matrix, without normalization
[Evaluation] Task 0: Avg Loss 0.05200100937101282; Avg Acc 0.0
Task 0 - CF: 0.9242
Train Task 0 - MU: 0.270 GB
Confusion matrix, without normalization
[Evaluation] Task 0: Avg Loss 0.06933177224355726; Avg Acc 0.0
Start of step 1
Each metric will probably need to implement a couple of methods, while the others will stay empty, so the implementation of new metrics should not be more complex than it is right now.
from avalanche.
Hi @AntonioCarta you are totally right. Indeed now the metrics are called through the EvaluationPlugin!
Being it a plugin, it can implement all the callbacks independently from the main strategy (all the plugin methods will be called before the main strategy methods). This also means that the calls he make can be fine-tuned to specific metric needs. Does it make sense or am I missing something?
from avalanche.
Related Issues (20)
- GenericCLScenario donot support data for regression tasks. TypeError: unhashable type: 'list' HOT 6
- before_forward callback confusing naming
- Usage of function ni_benchmark HOT 1
- The most frequently used benchmark in time series classification/regression is the [UCR datasets](https://timeseriesclassification.com/), which consists of 128 time-series datasets. Both [sktime](https://www.sktime.net/en/stable/index.html) (a Tensorflow-based Python package) and [tsai](https://timeseriesai.github.io/tsai/) (a Pytorch-based Python package) have provided comprehensive solutions for data handling and model development, including the SoTA. HOT 1
- DistributedHelper CI is failing HOT 2
- Using avalanche.benchmarks.utils.data_loader for loading dataset
- Class Indices Overwritten During Continual Learning with Avalanche 0.4.0a HOT 2
- Constantly run into this error when installing HOT 7
- Creation of a new envirnment failed for python 3.9
- Creation of a new envirnment failed for python 3.1
- units of various matrices of avalanche.evaluation.metrices
- ADD plasticity metrics
- Can I use my own dataset? HOT 1
- test dataset path broken HOT 2
- Checkpoint error in AGEM
- tutorial missing docs HOT 2
- Cannot catch last subexp (Affects Online settings with access to task boundaries) HOT 5
- Issue with cpu_usage metric HOT 2
- ConcatDatasets HOT 1
- forgetting_metrics is 0 with filelist_benchmark
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from avalanche.