Metrics of experiments with different tech implementation about cml HOT 9 CLOSED

@andronovhopf really great feedback!

Most of it is under development already in the core-dvc:

• the original issue iterative/dvc#3393
• the implementation iterative/dvc#3515
• diffs with the request to avoid deltas - exactly as you asked :) - iterative/dvc#3528 (in the process)

The spreadsheet of experiments is another great idea. We should think about that.

Re the spreadsheet... what would be your criteria to include an experiment into the table? How many of these would you expect to see here?

from cml.

diffs with the request to avoid deltas - exactly as you asked :) - iterative/dvc#3528 (in the process)

@andronovhopf how and where do you specify the max_depth parameter? Is train.json is actually a file with hyperparams in your case? Could you share both json files please? :)

from cml.

@shcheklein yes train.json is a file containing hyperparameters, and that's where max_depth is specified. I just invited you and @dmpetrov to the repository; the metric files are here.

@dmpetrov, re: spreadsheet. Two ways of selecting experiments to display in a table come to mind:

• If I'm doing a PR, compare PR to master. So only two experiments.
• A view of all commits on a branch compared. So as many experiments as there are commits (assuming CI was done after each commit)

Any other ideas?

from cml.

@andronovhopf did you run it like dvc run -M metrics/train.json -M metrics/eval.json ... and write all the params and metrics separately?

I like both the ways. If we do that:

• the current one and the baseline are must-have.
• it is convenient to see all from the current branch up to the master. However, some limits required due to the CI-reports limitation. Something like 10 or 30.

from cml.

@andronovhopf did you run it like dvc run -M metrics/train.json -M metrics/eval.json ... and write all the params and metrics separately?

Oh, I see that in the repo https://github.com/andronovhopf/cml_scratch

from cml.

@dmpetrov the pipeline has two stages (train.dvc and eval.dvc) and each stage writes a metric file. And yep!

from cml.

Another observation: my project has two branches; on master I am running a random forest classifier and on DNN a deep neural network. When I look at the report for the last commit on DNN, it looks like this:

Now, because the hyperparameters I'm collecting are not the same as on master (epochs & neurons vs. max_depth), comparing metrics from train.json doesn't make a lot of sense.

Also, I know we are planning to do this eventually- but here's a case where being able to compare two commits on the same branch, instead of the head of two branches, would be great (as an additional option, not instead of). Since I want to test a few different numbers of neurons/epochs in the neural network.

from cml.

@andronovhopf nice observation. We had that discussion also. That every branch might be different implementations of the same problem to be solved. Like here a DNN vs Random forest.

You can setup a different baseline and a baseline can be an specific commit sha. You can setup your baseline i.e to be HEAD~1 to compare your experiments with your previous one. And thats why the top five list came also in place to have a fast access to the same branch.

In my personal experience, to solve your problem in your DNN branch change the baseline to master/dnn (supposing its called that way) and work with branches of that branch to adjust new parameters.

from cml.

Closed this is not relevant anymore. Belongs to the CML-DVC incarnation of CML

from cml.