We use .nice() to make the y-axis always go from 0 to 100%. It works in Distribution Viewer, but not here in Experiments Viewer for some reason.

I'm going to be changing the metric types to return the same thing I'm getting upstream. Which will change from ['categorical', 'numerical'] to a more descriptive set that looks like ['EnumeratedHistogram', 'FlagHistogram', 'CountHistogram', 'BooleanHistogram', 'ExponentialHistogram'].

This will involve both front-end and back-end changes.

In the current data model we separate categorical and numerical data into separate tables. The only difference between the two is that the categorical data includes a rank field for the front-end to show certain buckets first for the CDF charts. I declare we no longer need this, and if we do we can do it at SQL query time.

Without this we can combine the tables and simplify a lot of the import logic and database logic and Django ORM code.

Hi 👊

This is my first visit to this fine repo, but it seems you have been working hard to keep all dependencies updated so far.

Once you have closed this issue, I'll create seperate pull requests for every update as soon as I find one.

That's it for now!

Happy merging! 🤖

The test data set currently reflects what dist viewer is using, we need to update that so that it contains multiple data sets, and so that the set of populations is defined per data set.

The experiments viewer will end up with many more metrics than the distribution viewer, most of which we don't have in our database.

On encountering new metrics the plan is to add them to our list of metrics in the database, so they will then have an id and a source_name. The fields for name, description, tooltip, and type would remain empty and should/could be populated in some other way at a future date. Without these fields the front-end can simply display the source_name as the name and default to a basic tooltip showing the values.

This bug is to figure out if there is a better upstream source for this metadata about the metrics. Or if we want to create one that various tools pull from. Or other thoughts that come up we can discuss here.

On stage, Exp 1 has a cohort named "group A" and Exp 2 has a cohort named "group B". If you start w Exp 1 and roll over the graphs, the text across the bottom bar shows you the values for "group A" and "control". If you switch to Exp 2, "group A" disappears and is replace by "group B", but "group A" still shows up in the rollover text across the bottom bar.

For example, the experiment with ID 1 would be shown first, 2 second, and so on. It might be easiest to do this on the back-end and have the API guarantee this order.

This is to allow viewer to submit bugs or see the code or contribute.

Gregg suggests putting it in the top nav area somewhere.

Currently when the front-end queries for a metric in a dataset that doesn't exist, Django throws a 500 with "DoesNotExist: Collection matching query does not exist." It would be preferable if we handled this more nicely and returned a 404 instead.

We should be comparing currentDataset vs the select value.

Many of these are good basic security settings and just require some config changes.

This corresponds to mozilla/distribution-viewer#227:

@gregglind said it would be great to have a button next to each chart that links to the JSON that was used to build the chart. Exposing the raw data would help users re-purpose it.

Subgroup data is now accessible through Redux thanks to the work Davor did recently. I'll refactor the front-end code to grab subgroups that way now, rather than the way it has done it in the past.

On staging right now, group B in experiment 2 looks like it should be showing up in the graphs, but it doesn't. It might be that the lines aren't rendering, but it also might be that cohort got assigned the color black and it's just invisible against the black background. In either case, it needs to be fixed so that cohorts are always visible.

Add a table at the same level as api_point to store statistics data coming from Sunah.

Suggested schema:

api_statistics
- id
- collection_id
- comparison_branch
- name
- p_value
- value

https://normandy.services.mozilla.com/api/v1/recipe/

Most modals I see out there offer an x button (or similar) to close it. I think it would be great for us to have one on the configuration modal, too.

Steps to reproduce

1. Load this page
2. Hover over a chart

Actual result

The hover dots stay at the bottom-left and the hover text doesn't update

Expected result

The hover dots move with the mouse and the hover description is updated along the way

1. Hit staging site, see "Exp 1" selected w 'group A' and 'control' both displayed, and they're both showing up in the color legend.

2. Select "Exp 2" from drop-down, hit 'apply' button. 'group B' shows up in the cohort list, but the color legend doesn't update, it still shows 'group A' from before.

The cohorts should always be listed with "control" as the first cohort in the list, and the rest in alphabetical order following.

Do something like this to add GA tracking:
https://github.com/mozilla/distribution-viewer/pull/180/files

I've asked our GA admin for a tracking code and I'm waiting to hear back.

We need a list (preferably less than like, 10) of OSes/versions to facet on

Steps to reproduce

1. Load Experiments Viewer
2. De-select all cohorts. The No data message will be shown
3. Select one cohort

Actual result

The No data message is still shown

Expected result

The cohort that was selected in step 3 is shown

We use the name subgroup on the front-end to refer to cohorts. But because we also want to use this platform elsewhere, where there may not be cohorts per se, we ideally want to use a word that would make sense for both. In other words, a term that means (cohort|population).

What would be a good name for this? Is population a generic term? @robhudson @spasovski

This likely belongs above the chart but below the chart title as plain/stylized text.

We discussed not wanting this in the experiments viewer.

Quoting @robhudson from here:

Can we add ESC when the modal is open? With a lack of a button to close or apply the modal that was my default key to try. When that didn't work I finally figured out clicking anywhere closed the modal but it made me nervous. :)

This may go away in the future if we pick up an auth0 lib. Having is separated would be good preparation and also good modularity for our code base.

We shouldn't hardcode a [cohort name] -> [cohort color] mapping in this project since we won't know the cohort names ahead of time.

For now, we can assign each cohort a random color except for control, which should always be the same color. Down the road, we can open an issue to make sure all cohorts have consistent colors (for example, group A would always be red and group B would always be blue) but to do that without knowing the names ahead of time we would have to have some kind of repeatable hash from name to color value. But that's outside the scope of this issue.

This involves iterating over all subgroups for each branch and creating a new api_collection with population being a concatenation of branch + subgroup.

The home page is showing all of the metrics for all of the experiments on each experiment, so the metrics that don't apply to the current experiment are showing up broken or with irrelevant and misleading data.

Currently we have:

{
  "datasets": [
    {
      "id": 1,
      "name": "Experiment 1",
      "populations": ["control", "group A"]
    },
    ...
  ]
}

I propose we update to have a subgroup key with list of subgroups included:

{
  "datasets": [
    {
      "id": 1,
      "name": "Experiment 1",
      "populations": ["control", "group A"],
      "subgroups": ["Windows", "Mac", "Linux", "Other"]
    },
    ....
  ]
}

http://python-dockerflow.readthedocs.io/en/latest/index.html and
https://github.com/mozilla-services/python-dockerflow

It should be possible to configure experiments and cohorts on the main page. Only one experiment can be chosen at a time. One or many cohorts can be chosen at a time. By default all cohorts will be selected. The list of possible cohorts that the user will be able to choose from will depend on the experiment that is chosen.

Experiments are essentially datasets. Each experiment has one or more metrics associated with it. Cohorts are essentially populations. For each cohort, another line will appear in the chart and another item will appear in the legend.

These options could be presented in the existing Configuration box or as separate widgets: for example, Experiment could be a simple drop-down and Cohorts could be a multi-select box.

When switching from experiment A to experiment B, it looks like a) any cohort in exp B that matches a cohort name from exp A will retain the "on/off" state that was set in exp A, and b) any cohort in exp B that does not exist in exp A will default to "off". Ideally, when switching to a new experiment, all cohorts for that new experiment will always default to "on".

Chart detail pages aren't working right now (see #63) so for now we should make sure the links aren't there so as not to confuse folks.

Currently on staging when I use the drop down to choose an experiment and then hit the "apply" button nothing seems to happen. Only when I hit "apply" a second time does the page update and show the new experiment as expected.

The chart detail page does not load when navigated to directly, for example by clicking this link.

When the chart detail page is navigated to from within the app (that is, by loading the homepage and then clicking on a chart), the detail page itself loads but the chart within it does not.

I think the former, at least, is due to the new default URL handling.

At the moment, the front-end always tries to load dataset 1 on first load. But there's no guarantee that dataset 1 will exist. Instead, it should load the [0]th dataset listed in /datasets.

Right now you have to hit apply to see the cohorts update.

This work corresponds to this Distribution Viewer PR which never got merged:
mozilla/distribution-viewer#226