pypi / linehaul Goto Github PK

In linehaul, there are 3 states that any particular event can be in:

1. The user agent is parseable for data.
2. The user agent is unknown.
3. The user agent is known, but it's not parseable for data.

For (1) the correct outcome is obvious, we have data so we want to save it in BigQuery.

For (2) the current thing we do is record a download, but with all of the data that typically comes from the user agent missing. Thus the BigQuery table more accurately reflects all of the downloads, but projects querying the data needs to be more careful about how it queries the data (it's easy to do something like py3_downloads / total_downloads, however that would incorrectly give a smaller percentage, since it would count unknown as py2). Prior to Linehaul v3, the behavior was to throw away this event and not log anything for it.

For (3) Linehaul v3 and previous throw away the event (we implement this as "ignored" user agents). The list of these can be found at:

https://github.com/pypa/linehaul/blob/420354cf789b064f0d38ce02573f6af51aa0306a/linehaul/ua/parser.py#L260-L294

So ultimately the question is, do these behaviors make sense? Which boils down to whether we want BigQuery to most accurately reflect every download, or whether we want to filter the data to data that is more usable for specific questions (but of course, less usable for other questions that are likely to be more of an edge case).

Thoughts?

I ran a script today from 01:07 to 09:39 UTC, downloading a file every six minutes. It made a total of 86 downloads, but only 40 of them are in the database:

SELECT *
FROM
  TABLE_DATE_RANGE(
    [the-psf:pypi.downloads],
    TIMESTAMP("2018-05-25"),
    TIMESTAMP("2018-05-25")
  )
  where file.project = "six" and file.version = "0.9.0" 
  and country_code = "DE" and details.installer.version = "10.0.1"
  order by timestamp desc

Looking at the data more generally, there are 1440 minutes in a day, but on 05-23 only 748 of them had any data recorded:

#standardSQL
SELECT timestamp_trunc(timestamp, MINUTE) minute, count(*)
FROM `the-psf.pypi.downloads*`
where _table_suffix between "20180523" and "20180523"
  group by minute order by minute desc

Browsing through those results shows many extended gaps in the data, e.g. from 22:45 to 22:54, and 21:36 to 22:16.

I know this service isn't widely-known, but isn't it intended to be a complete record of all PyPI downloads? If so, something is clearly going wrong.

Hi!

It might be interesting to be able to query through failed installation data, if such a thing is possible.

I.e. -- to view pip install foo by volume, for foo a non-existing package, or existing package with no uploaded files.

i'd like to use this library!

• 😍

As mentioned by @ncoghlan in pypa/pip#5424 (comment):

submit a PR to update linehaul's UserAgent schema to include an extra optional wrapper = pyrsistent.field(type=Installer, factory=Installer.create) field

The motivation is to allow wrappers, like pipenv, to measure adoption of their tools.

The project has since switched to attrs, but the idea is the same. The change should be somewhere around here.