Comments (10)
ajratner
commented on August 29, 2024
@jason-fries thanks for catching this! I'll take a look later today
from metal.
ajratner
commented on August 29, 2024
Hey @jason-fries I'm assuming the example above is with the GPU, and the LSTM? Do you know if it occurs without the GPU? And could you let me know how you had the model configured?
I'm a bit behind but going to try to isolate and correct as suggested on this branch, starting with adding basic tests for model and scoring determinism on CPU
Thanks,
Alex
from metal.
jason-fries
commented on August 29, 2024
Just checked -- it's also non-deterministic on CPU.
from metal.
ajratner
commented on August 29, 2024
Thanks! And could you share the model class (LSTM?) and kwargs / config used?
from metal.
jason-fries
commented on August 29, 2024
Perhaps I don't follow how the RandomSearchTuner is configured but there are couple things that seem weird/buggy to me.
For random seeding, I'd expect:
- A user-provided seed for model weight initialization that's fixed during hyperparameter search.
- A fixed, separate seed for sampling hyperparameters.
If I pass a single seed into the model search grid (see below) and set a different fixed seed for the RandomSearchTuner initialization, then the same hyperparameters settings are sampled for each trained model. If you remove the seed from the search space, then every model config gets a new seed by default (which seems weird). Is that seed separate from the model seed?
The best model also seems to be chosen using accuracy not matter what is chosen as validation_metric. How do you specify that the best model should be selected based on validation_metric?
Model config dictionary
{'input_module': lstm,
'lstm_reduction': 'attention',
'middle_modules': [Dropout(p=0.5],
'seed': 123,
'show_plots': False,
'train_config': {'batch_size': 64,
'checkpoint': True,
'checkpoint_config': {'checkpoint_min': -1,
'checkpoint_runway': 0},
'data_loader_config': {'batch_size': 16, 'num_workers': 8},
'disable_prog_bar': True,
'l2': 0.00166,
'n_epochs': 5,
'optimizer_config': {'adam_config': {'betas': (0.9,
0.999)},
'optimizer': 'adam',
'optimizer_common': {'lr': 0.01},
'sgd_config': {'momentum': 0.9}},
'print_every': 1,
'scheduler_config': {'exponential_config': {'gamma': 0.9},
'lr_freeze': 0,
'plateau_config': {'factor': 0.5,
'min_lr': 1e-05,
'patience': 1,
'threshold': 0.0001},
'scheduler': 'reduce_on_plateau'},
'validation_freq': 1,
'validation_metric': 'f1'},
'use_cuda': False,
'verbose': True}
and
search_space = {
'lr': {'range': [0.001, 0.01], 'scale': 'log'},
'seed':[args.seed]
}
searcher = RandomSearchTuner(EndModel, seed=args.seed + 1, log_writer_class=TensorBoardWriter, **log_config)
end_model = searcher.search(search_space,
dev_loader,
train_args=[train_loader],
init_args=init_args,
init_kwargs=init_kwargs,
train_kwargs=model_config['train_config'],
max_search=args.n_model_search)
from metal.
ajratner
commented on August 29, 2024
@jason-fries looking into this now; one more question, has the issue only occurred when using the tuner, or also when just training the model with dev set checkpointing?
from metal.
ajratner
commented on August 29, 2024
Okay well either way, I now have a basic model determinism test that is failing just with the LSTM on CPU, so probably need to fix the seeding in that module; doing this first
from metal.
ajratner
commented on August 29, 2024
@jason-fries LSTM with randomly initialized embeddings just needs to be seeded for determinism; passing tests now on model-determinism branch. Moving on to random tuner
from metal.
jason-fries
commented on August 29, 2024
Some quick issues (all on CPU for now):
validation_metricdoesn't look like it's being respected. I'm using F1, which is printed during each epoch update, but definitely not being printed out in the "best model" summary at the end of each loop. Validation scores also still change.
[E:9] Train Loss: 0.033 Dev score: 0.522
Restoring best model from iteration 5 with score 0.551
Finished Training
Accuracy: 0.809
y=1 y=2
l=1 36 51
l=2 26 290
Accuracy: 0.824
y=1 y=2
l=1 38 47
l=2 24 294
- validation scoring during param search should disable gradient history (can't see if this is being done or not now), e.g., something like:
end_model.eval()
with torch.set_grad_enabled(False):
y_pred, y_true, y_proba = end_model._get_predictions(test, return_probs=True)
This gives deterministic scoring on CPU for me.
from metal.
ajratner
commented on August 29, 2024
Fixed by PR #105
from metal.
Related Issues (20)
- Generative model snorkel vs Generative model metal HOT 3
- KeyError: 'METALHOME' error from MMTL_Basics tutorial HOT 1
- [QUESTION] Mixing textual and numerical inputs with multi-branch models HOT 2
- MMTL training accuracies swinging between two datasets HOT 3
- issue keras or Tensorflow in the future? HOT 1
- multiple
- when applying metal, how to estimate noisy-label probability for multilabel problem of a single task HOT 2
- Keep track of changes as CHANGELOG HOT 2
- I think there are some errors in the mt_end_model.py HOT 1
- Potential bug for the predict_proba function in label_model.py
- How to use weak supervised learning with MMTL HOT 1
- links to basic snorkel tutorials are broken. HOT 1
- How does an end model improve upon Snorkel? HOT 1
- End model seems to always do better when assuming a 50/50 class balance HOT 1
- Supporting "default" class HOT 1
- Question on LF influence on Snorkel predictions based on coverage HOT 1
- Change LF stat "empirical accuracy" to "precision" HOT 1
- Problems with networkx when using LabelFunction HOT 1
- Singular error
- Not support Networkx 3.2.1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from metal.