chovanecm / sacredboard Goto Github PK

It would be useful to have a comments column(?) that is editable

There should be a button to manually update the table, so that I don't have to uncheck and check a status box. 😄

Currently only mongodb is supported as a backend.
However sacred is able to use other databases as well. This could be supported in sacredboard as well.

Everytime the frontend should display an object from the backend that looks like {"$date": 123456}, it should display it as a date.

As a user, I want to filter based on the metrics measured.
Improvements for #8
the last / minimal / maximal value of certain metric (such as accuracy) is bigger / lower than...

For users who store their repository information(sacred provides a handy utility for this, see here) for reproduction, it helps to show informations like: git url, commit hash etc, in sacredboard.

Hi,

When viewing captured output, the display panel keeps scrolling down to the bottom, no matter how many times I scroll it up. It is helpful when opening the captured output panel for the first time, since most of the important logging is at the end. But it becomes bothering when looking for more previous logging : (

The file is getting messy.

When in detail view, one should be able to browse the object stored in the database in an expandable tree

As a user, I want to see list of all experiments in my database, including their name, machine where they run, the result field from the database and the last line of the captured output.

• Ideally the status would be a colored dot: green for completed, yellow for interrupted, blue for running, red for failed, grey for queued and black for died (running but last activity is long ago).

The overview table needs an id column at the very left and a command column to the right of the experiment name.

I want to see more details about selected experiment, especially

• config parameters (as the first option)
• the full captured output of the programme (stored in the database)
• fields [define_them_here]
• an interactive browser of the object tree in the database ( issue #5 )
• a simple graph of [define_what like accuracy etc (issue #6 )
• a link to start tensorboard if the experiment used that framework (issue #7 )

Hi,

It appears that sacredboard can not be used with databases created by sacred v6.* since the runs are in db.prefix.runs instead of db.prefix.

prefix was usually default, but is now runs. So sacredboard is trying to look into db.runs whereas my runs are in db.default.runs.

This appears to be an important issue for many users like me who have older sacred databases that we would like to browse. Is there a plan to provide compatibility with older databases?

User Story

A researcher wants to launch TensorBoard for multiple experiment runs that were using the TensorFlow framework so that he can view the computation graphs and advanced charts of all the runs of his choice at once.

Requirement Description

Sacredboard shall provide the user with an option to launch TensorBoard for experiment runs that have an associated TensorFlow log directory specified.

An experiment run can have zero to multiple (probably a few) associated TensorFlow directories.
A problem may arise if the log directory is located on a path that is inaccessible from the location where Sacredboard is running on (e.g. it is on another machine).
TensorBoard can handle the problem by simply pretending the directory is empty,
but the user may not be aware of the issue and can misinterpret the situation.
It is advisable that the directory path is accessible from the computer where Sacredboard is running (e.g. using a file server and mounting the directory to the same path as when the experiment was running).

The info dictionary (document in MongoDB) has a key named tensorflow whose value embeds another dictionary.
The embedded dictionary contains an array of all the paths to the TensorFlow log directories associated with the run.
The array is accessible via the logdirs key.
(The metrics are not relevant to this issue)

The composition (filled black diamond) arrows represent relations that are essential to the child object; without the connection to the parent object, the child has no meaning. Furthermore, the child object is embedded in the parent as a nested document. The standard association arrow points to a separately stored object, referenced by a key. The aggregation (filled light diamond) arrows point to collections containing the records.

Use Case

Preconditions: Sacredboard is running and the user sees the experiment run list

Base Course of Events

1. The user indicates that he wants to launch TensorBoard for some of the experiment runs.
2. The software responds by requesting the user to choose the runs that should be inspected in TensorBoard.
3. The user inputs his choice and confirms it.
4. The software navigates the user to TensorBoard with the desired experiment runs loaded.

Alternative Paths

1. At any time, the user may decide to cancel the action. If so, TensorBoard will not be launched and the user does not get navigated anywhere. (The postcondition does not apply.)
2. If TensorBoard cannot be launched in the last step (for instance, the executable is missing on system PATH, a corresponding error message is displayed and the postcondition does not apply.

Postconditions

TensorBoard has started and the user has been navigated to it.

The time displayed for last activity and the system time don't match. I suspect in the UI everything is displayed as GMT or some such?

Might also be good to simply have a timestamp ("system time is currently $x") somewhere on the page, to avoid confusion.

I suspect that always using the local timezone for everything might be a good first approximation, assuming most people work locally.

Some backends (like file storage #17) don't support filtering and sorting.
The user interface has to correctly deal with it, ideally, by hiding or disabling the filters, table sorting and pagination.

It would be very helpful, if it was possible to add and remove custom columns from the overview.
E.g. a way of adding a config.learning_rate column.

TensorBoard 1.2 outputs something like
Starting TensorBoard b'54' at http://0.0.0.0:6006 when starting, which is not what Sacredboard expects to detect the port.

As a user, I want to search in experiments

It would be nice to be able to be connected to multiple databases. In case of MongoDB, users should be able to provide the collection name that contains the runs (see #20 ).

As a user, I want to see a simple graph of training progress when in detail view

As a Sacredboard developer, I want to have a convenient way to draw multiple time series onto a single canvas (Plotly) so that I can display e.g. data of the Metrics API

EDIT: Duplicate issue.

From time to time, an experiment run may become useless to be maintained in the database, for instance when it fails and provides no interesting information to the researcher.
Sacredboard shall provide an option to delete the run.

/// for history purposes:
" delete entries (with confirmation and ideally using checkboxes for multiple actions)"

@Qwlouse - Do you mean deleting the whole experiment? (And really deleting it or just marking it as deleted and not showing them, something like a trash bin?)

An open question (not needed to be answered now)
In "server / multi user mode" (where I expect that you will probably first need to log in somehow), should be this allowed to anyone? (Anyway, as long as you can directly connect to the database from your local machine, you can delete whatever you want, if you decide to. Any logic preventing it in the application wouldn't guarantee it).

Might be useful for others than me, too. E.g. I have my own host_info_getter that saves the resources and meta information provided by our cluster system here, but I can't see that info in sacredboard.
Filtering for that info already works fine, but it's a bit weird to filter for things that are not visible...

• Show something meaningful if sacredboard cannot connect to the database
• When database doesn't have the schema the application expects, show an error message or something

If I see correctly, connecting to mongodb at the moment can only be done with a very limited subset of pymongo's capabilities. Would be great if sacredboard's capabilities would match those of sacred itself, so I can use the same config for both 😛

As a user, I want to be able to open Tensorboard directly from the console to see all graphs.
Applies only to experiments that use tensorflow.

As a user, we want to compare, for example, the captured output or the configurations, between two or more selected experiments.

As a user, I want to see the captured output when using the FileStore backend to monitor my experiment.

Is it possible?

• Analysis of how the Sacred FileStore observer stores the data.
• Implementation

ATM if sorting by results that are lists sorts by the last element of that list. Should be the first instead.

Hi, is there a way to get the experiment records using experiment id?

As a user, I want to show only experiments that match (or don't match) certain criteria, such as:

• machine where an experiment runs on
• experiment name
• experiment state
• the result field of an experiment
• the last / minimal / maximal value of certain metric (such as accuracy) is bigger / lower than...
• arbitrary value in arbitrary field stored even somewhere deep in the record structure
• users shouldn't be forced to write "config.something" everytime, instead: write simply "seed" for searching in "config.seed" or ".host.hostname" for "host.hostname".

Precondition: Sacredboard is running and the user sees the experiment run list

Base Course of Events

1. The user indicates that the software is to perform filtering of the run list.
2. The software responds by requesting the name of the experiment config property to be filtered on, the comparison operator and the searched value.
3. The user inputs the configuration property name and the comparison operator and indicates the type of the searched value (number/string/date).
4. The software applies the selected filter additionally to other filters that have been possibly applied before and indicates that a new filter condition has been added.

Alternative Paths

1. In Step 3 the user indicates he does not wish to perform the search based on an experiment run configuration property but based on another property of the experiment (not necessarily in the run config section). The software will allow that.
   In Step 1, the user indicates that he wants to instantly filter out experiments that have a certain state (e.g. FINISHED and INTERRUPTED). The software must support it without requesting any details about property names.
2. In Step 3, the user indicates he wants to filter based on an invalid combination of operator and value (e.g. number and "contains"). In this case, the software indicates that the choice is invalid and prevents the user from continuing to the next step.
3. In Step 1, the user indicates he wants to remove one of the existing filters instead. The software removes the filter of user's choice. All the postconditions apply.

Postconditions

1. The run list displays only results matching criteria of all the filters applied so far.
2. The software indicates all the filters applied.

I have the recommended option implemented to include linefeeds but in sacredboard I still see very verbose logging for progress bars in the captured output.

I'm using tqdm if anyone else has run into this problem before.

As a user, I want to see the source code of the experiment.

The source code is stored in "fs.files" collection using gridfs extension of mongodb.

Hi,

I noticed there is a Result column in sacredboard. How can I set the value for my experiments?

Here's a wishlist for improving the way the configuration is shown:

• Lists are shown as numbered subentries (e.g. a=['foo', 'bar'] would become two rows with a.0 foo and a.0 bar).
• Entries should be sorted alphabetically
• Bonus: It would be cool to group nested subentries (sub-dicts) visually and make them collapsible
• Bonus: if items are grouped, then sorting such that the groups come first and then the individual entries

The same for "info"

Similar to the config, the elements of a list are currently shown as individual lines which is inconvenient.
The dictionary structure could also use indentation for a better visual experience.

A researcher wants to limit the list of experiment runs to
runs that have a certain property on, before, after or on another date than the given date so that he can filter the results e.g. by the experiment start time.

Hi,

I've changed the _id field in the new version of sacred form ObjectId to int. Unfortunately that breaks your current webapp:

Traceback (most recent call last):
  File "/home/greff/venv/sacredboard/lib/python3.4/site-packages/flask/app.py", line 1982, in wsgi_app
    response = self.full_dispatch_request()
  File "/home/greff/venv/sacredboard/lib/python3.4/site-packages/flask/app.py", line 1614, in full_dispatch_request
    rv = self.handle_user_exception(e)
  File "/home/greff/venv/sacredboard/lib/python3.4/site-packages/flask/app.py", line 1517, in handle_user_exception
    reraise(exc_type, exc_value, tb)
  File "/home/greff/venv/sacredboard/lib/python3.4/site-packages/flask/_compat.py", line 33, in reraise
    raise value
  File "/home/greff/venv/sacredboard/lib/python3.4/site-packages/flask/app.py", line 1612, in full_dispatch_request
    rv = self.dispatch_request()
  File "/home/greff/venv/sacredboard/lib/python3.4/site-packages/flask/app.py", line 1598, in dispatch_request
    return self.view_functions[rule.endpoint](**req.view_args)
  File "/home/greff/Programming/sacredboard/sacredboard/app/config/routes.py", line 53, in api_run
    runs = data.get_run(run_id)
  File "/home/greff/Programming/sacredboard/sacredboard/app/data/mongodb.py", line 26, in get_run
    cursor = self._db.runs.find({"_id": bson.ObjectId(run_id)})
  File "/home/greff/venv/sacredboard/lib/python3.4/site-packages/bson/objectid.py", line 110, in __init__
    self.__validate(oid)
  File "/home/greff/venv/sacredboard/lib/python3.4/site-packages/bson/objectid.py", line 206, in __validate
    _raise_invalid_id(oid)
  File "/home/greff/venv/sacredboard/lib/python3.4/site-packages/bson/objectid.py", line 52, in _raise_invalid_id
    " or a 24-character hex string" % oid)
bson.errors.InvalidId: '1' is not a valid ObjectId, it must be a 12-byte input or a 24-character hex string

A question to @Qwlouse: Do you think sacredboard could also run on a server, so that one instance would be accessible to multiple users? I can imagine you (or other teams) having one database used by more researchers using one database which could be accessed via one single user interface. It could be useful e.g. where you're on the go and want to check your experiments via mobile or laptop just by visiting something like sacredboard.idsia.ch

This could affect some architectonic decisions, thanks!

For testing that the Sacredboard backend returns what is expected, tests should be created that send requests to the Backend and check the response.
The tester should be able to select the backend to test against but the tests should be performed on a known set of data.

What to test:

1. /api/run - simple fetch, sorting, filtering, pagination
2. /api/run/ - a single run including the full run object, including config and info.
3. TBD

The outputs in json may have different formatting (e.g. not pretty-printed).
Note: the content of "object" contains dates serialized as timestamps: there is currently no front-end functionality to convert it to back to a date as it is not needed (the start_time is already passed as a formatted string in the Run header):

        "start_time":{  
           "$date":1489763136401
        },

/api/run Example

{
    "draw": 1,
    "recordsTotal": 2,
    "recordsFiltered": 2,
    "data": [
    {
        "id": "3",
        "experiment_name": "iris_rbf_svm",
        "command": "run",
        "status": "COMPLETED",
        "is_alive": false,
        "start_time": "15:05:36 17.3.2017",
        "heartbeat": "15:05:36 17.3.2017",
        "heartbeat_diff": 6535250.874869,
        "hostname": "vmmint18",
        
        "result":0.95
    },
        
    {
        "id": "4",
        "experiment_name": "German nouns",
        "command": "runExperiment",
        "status": "RUNNING",
        "is_alive": false,
        "start_time": "09:49:56 30.3.2017",
        "heartbeat": "09:50:26 30.3.2017",
        "heartbeat_diff": 5430960.722107,
        "hostname": "ntbthinkpad",
        
        "result":""
    }]

/api/run/id Example where id=3

{  
   "draw":1,
   "recordsTotal":1,
   "recordsFiltered":1,
   "data":[  
      {  
         "id":"3",
         "experiment_name":"iris_rbf_svm",
         "command":"run",
         "status":"COMPLETED",
         "is_alive":false,
         "start_time":"15:05:36 17.3.2017",
         "heartbeat":"15:05:36 17.3.2017",
         "heartbeat_diff":6535396.954588,
         "hostname":"vmmint18",
         "result":0.95,
         "object":{  
            "host":{  
               "python_version":"3.5.2",
               "cpu":"Intel(R) Core(TM) i7-6600U CPU @ 2.60GHz",
               "os":[  
                  "Linux",
                  "Linux-4.4.0-64-generic-x86_64-with-LinuxMint-18.1-serena"
               ],
               "hostname":"vmmint18"
            },
            "info":{  

            },
            "result":0.95,
            "format":"MongoObserver-0.7.0",
            "status":"COMPLETED",
            "heartbeat":{  
               "$date":1489763136431
            },
            "config":{  
               "array":[  
                  "One",
                  "Two"
               ],
               "C":1.0,
               "seed":562201636,
               "gamma":0.7
            },
            "stop_time":{  
               "$date":1489763136430
            },
            "captured_out":"INFO - iris_rbf_svm - Running command 'run'\nINFO - iris_rbf_svm - Started run with ID \"3\"\nINFO - iris_rbf_svm - Result: 0.95\nINFO - iris_rbf_svm - Completed after 0:00:00\n",
            "start_time":{  
               "$date":1489763136401
            },
            "experiment":{  
               "dependencies":[  
                  "numpy==1.12.0",
                  "sacred==0.7b2",
                  "sklearn==0.18.1"
               ],
               "base_dir":"/media/sf_Martin/Documents/archiv/archiv/\u0161kola/\u010cVUT/Magistr/Diplomka/sandbox",
               "name":"iris_rbf_svm",
               "sources":[  
                  [  
                     "iris.py",
                     {  
                        "$oid":"58cbfb407ef9e60d73d66868"
                     }
                  ]
               ],
               "repositories":[  

               ]
            },
            "resources":[  

            ],
            "_id":3,
            "meta":{  
               "options":{  
                  "--print_config":false,
                  "COMMAND":null,
                  "--comment":null,
                  "--loglevel":null,
                  "--beat_interval":null,
                  "--priority":null,
                  "--tiny_db":null,
                  "--queue":false,
                  "UPDATE":[  

                  ],
                  "--file_storage":null,
                  "--unobserved":false,
                  "with":false,
                  "help":false,
                  "--mongo_db":"sacred07",
                  "--sql":null,
                  "--pdb":false,
                  "--help":false,
                  "--debug":false,
                  "--enforce_clean":false,
                  "--force":false
               },
               "command":"run"
            },
            "artifacts":[  

            ],
            "command":"run"
         }
      }
   ]
}

Users should be able to run the application on their local machine

The backend might render a response containing NaN, which is not a valid in JSON.

The user interface should be friendly for mobile devices

Right now, get_run() requires either an int or ObjectID type for id_. Would you accept a PR that adds support for string type id_ values?

As a developer, I want to see what data are returned by the backend in a pretty-formatted way so that I can debug the application.

It looks like in the Experiment Start and Last Activity field, the times are shown in UTC instead of local time

As a user, I want to quickly find an experiment by its ID without relying on database-specific field names (such as _id in MongoDB) so that I don't have to remember what field to write in the filter.

(Based on #50 )