Break down the output by release channel about probe-scraper HOT 3 CLOSED

The probe scraper currently produces 3 JSON files:

• revisions.json
• general.json
• probes.json

The general.json only contains the lastUpdate property, which is the date the files were generated.

The revisions.json file contains a list of revision hash -> Firefox version mapping for each channel.

Finally, the probes.json file contains the data for all the probes. It has the following format:

{
  probe_id: {
    type: 'histogram',
    name: 'some name',
    history: {
      "nightly": [
        {
          optout: True,
          ... other probe details ...
          revisions: {first: ..., last: ...}
        },
        ...
      ],
      ... other channels ...
    }
  }
}

My proposal is to keep the current format, writing a separate probes.json file for each channel, and only list the relevant channel data in the history section of the probe.

@sunahsuh and @fbertsch, since you are the main consumers of the probe data I can think of right now, what do you think of this proposal?

from probe-scraper.

Quick recap from discussion in IRC:

• This will include historic probes. Yay!
• We can remove revisions.json and just sub in the FF version
• Long-term we may need to rethink how we deal with historic revisions

from probe-scraper.

This happened, the output is already being written to S3 and the documentation changes landed. Closing this off!

from probe-scraper.