Currently various users analyse the experiment data structure directly. It needs to be migrated to a Python class within jacquard.experiments.

Currently when experiments are loaded there is little to no validation on the JSON data entered. It should be gone through with a fine-toothed comb.

When launching experiments we should validate for any potential conflicts against current experiments.

Specifically:

• Site area conflicts
• Setting key conflicts

In the future it would also be good to check constraint conflicts for two experiments in the same site area, but that's not necessary for the first release.

This could be potentially useful for cleaning up experiment code.

This will set a default iff there is not an existing value. The use case is for setting new defaults automatically from CI.

We should be able to load experiments in YAML format, which is much more convenient for a human to write.

While being able to add plugins via the setuptools entry points mechanism is very general and good for plugins which can be installed by pip, it's not as good for simple customisation.

It would be good to have some kind of plugins section in the config file which specifies additional directories on the python path and entry points, something along the lines of:

[plugins]
pythonpath=/etc/jacquard/my_plugins

[plugins:storage_engines]
my_storage_engine=foo:StorageEngine

[plugins:directory_engines]
my_directory_engine=foo:DirectoryEngine

This probably involves a refactoring to abstract the current calls to pkg_resources.iter_entry_points.

Please add a licence so we can use this!

I added a couple of extra commands in #50, though I'm holding off suggesting that that be merged until we resolve the command naming issues it highlights. If we can't find a good solution soon-ish, it may be worth merging that ~now and fixing the names later, but I'm hoping that by discussing this now we can minimise the breakage needed.

Jacquard currently has the following commands:

• list (lists experiments)
• list-users
• show $USER_ID (shows the given user's settings)

#50 adds:

• list --detailed (which matches the current list behaviour, as #50 makes the default list output shorter)
• show-experiment $EXPERIMENT_ID (shows the same details as list currently does, but for a single experiment only)

This leaves us with inconsistency around what list and show operate on and they both end up with companion list-users & show-experiment commands which have the same issue in reverse.

I think we should move towards consistency sooner (while Jacquard is 0.x) so we don't end up with a UI as bad as git has.

There are a couple of ways I think we could go:

1. Replace the current commands with topic-based commands under new names:

   • experiments
   • experiment $ID
   • users
   • user $ID
     the advantages this has are:
   • (almost) completely new names minimise confusion
   • adding another class of thing can follow the same pattern without needing to become even more verbose
     A disadvantage is that this loses the imperative voice the commands currently all have. (I don't see this as a big deal since these commands are for display data only and make no changes to the system)

2. Adapt the current commands into unambiguous names:

   • list-experiments
   • show-experiment $ID
   • list-users
   • show-user $ID
     These feel quite verbose, though are much clearer and retain the imperative tone.
     There's a minor concern about users taking longer to remember that show is now show-user (the closer things are, the longer I expect users will take to adapt), though if we're going to break it anyway for a better command in the longer term this is probably doesn't matter too much.

There's also a debate to be had about whether the other (non-conflicting) commands such as launch, override etc. should be disambiguated in a similar manner (to be launch-experiment etc.).

Currently I find myself relying on storage-dump (a plumbing command) to find out what experiments exist.

There should be a porcelain CLI tool for this.

Would be useful to be able to run a commend which validates an experiment, possibly also asking it to highlight anything which would conflict with other known (and/or launched) experiments.

This could be useful in building tests for a repository of experiments.

These will skew analysis.

Union directories lack documentation in the RTD docs.

After an experiment (and even during) it would be useful to be able to query the branches a user has been in. This is particularly useful when closing a test early as it is found to be broken, but wanting to debug it by looking at the experiences that users got.

Currently the defaults assume the current working directory. They should assume /etc.

Ability to control what fraction of users goes into each branch.

Blocked on #19.

Where an experiment has, say, three branches control, test 1 and test 2, and test 2 is evidently awful, it would be useful to redistribute all buckets assigned to test 2 evenly between control and test 1. This is statistically valid to do, but currently isn't possible with Jacquard.

They may now be incorrect with the PyYAML dependency.

Blocked on #5.

Currently it's not discoverable how to add plugins.

#69 worked OK for the initial use-case, but isn't a properly general system.

storage-flush is currently a porcelain command which can immediately destroy the database. Especially after #66 this might be easy to do by accident. It's currently there so that you can empty the database and refill it with an import.

We should:

1. Add a --flush argument to storage-import,
2. Mark storage-flush as a plumbing command,
3. Add a mandatory --force argument to storage-flush.

An engine which keeps a local copy on all jacquard nodes.

The docs are pretty vague about actually using the library: there should be a simple example of a program which actually uses the API.

The current README does not make a good advert on PyPI.

Can we automatically generate this installed commands?

It would be useful, both generally and specifically when concluding an experiment, to be able to clear overrides.

This breaks down into what may be separate cases:

• clear all overrides for a given user
• clear overrides for settings from a given experiment (for all users)

This would help ensure that users who have been overridden to some value can be restored to the "normal" experience easily.

Had a chat with you about this on Slack - it'd be great to get an etcd backend.

This would be useful when editing the settings after loading an experiment in but before launching an experiment, particularly in environments where the loading of experiments is automated (for example by a Jenkins instance).

Currently there is little explanation of how to set up config.cfg.

As part of this, it makes sense to have --verbose actually work, and a flag to enable debug output too.

Depends on #11. Necessary to be able to use the poxy thing.

This needs some thought as to how to persuade argparse to allow this to happen.

May work well with lazy loading of subcommands.

It's common to stop an experiment due to bugs in the implementation and re-start it so as not to observe the behaviour of users encountering the bug. Usually we'd like to keep the grouping of users the same and not migrate users between conceptually what are the same experiments.

So as to preserve the requirement of keeping old experiments, one way to do this could be to designate an experiment as a successor to another, and to somehow preserve the bucketing of users between them.

Partition users into N buckets and split users between them. Keep things more stateful.

Flag experiments as being in different site areas. Two experiments in the same site area have to be checked for overlap.

This is more useful most of the time, and it's easy to point it at /etc/jacquard from a systemd unit file.

This is more immediately useful.