vladvelici / dbxlogger Goto Github PK

Content-addressable file storage in the experiment repo.

Up from this is a class that would allow experiment runners to fetch files from the experiment repo, by reference or path.

First what does this actually mean in the following:

• in the log: when you resume, you resume from last checkpoint, not last log - add mechanism to make log entries ignored
• in the logger library: make an API to easily lookup the logs backward to find the point to resume from easily (no path typing necessary on the command line)

Most importantly, things to add for the integration to be useful:

• saving checkpoints (configurable)
• batch-level logging (configurable)
• example on how to analyse the resulting logs
• documentation on what logs are saved
• save/resume support and example (depends on adding support for save/resume in dbx initially)

The issue mentioned in the readme file where experiments are currently defined by only one run. this shouldn't be the case and more runs must be supported.

this issue is for tracking the progress in making a repo spec change to add support for multiple runs per experiment

I think the best example use case is when you have separated training and evaluation scripts - all the git, host, env vars, os args tracking should happen per-run but parsed hyper-parameters should remain per-experiment.

I'm looking for a design that keeps the simplicity of a single meta.json, maybe with an extra file per run and moving all the env, args, git and host stuff out of meta? The final format to be decided and implemented.

Generate hashes of files (and more) for the purpose of logging accurate references to them.

When #7 is closed the repo definition will be OK to move to building most tooling well. Time to write the clear spec for how synch happens between different repos.

As part of this issue also come up with a test suite - i can't really say this works without having a decent test suite with a bunch of edge cases