drexer006 / data Goto Github PK

There are different ways to access OONI data, wether that is via: OONI Explorer, the OONI API or clickhouse table dumps.

The OONI API is meant for developers and researches and allows searching for measurement metadata, fetching single measurements, and generating statistics.

Hovever the OONI API, is not designed for large data transfers (i.e. extracting tens of thousands of measurements or many GB of data) and implements rate limiting API. If you are interested in a dump of the clickhouse tables, please reach out to us instead of scraping our API.

Researchers can access the raw measurement data from an S3 bucket. The specifications of the OONI data formats can be found in ooni/spec.

"Raw measurement data" refers to data structures uploaded by OONI Probes (run by volunteers worldwide) to the processing pipeline.

Thanks to the Amazon Open Data program, the whole OONI dataset can be fetched from the ooni-data-eu-fra Amazon S3 bucket.

A single chunk of data is called "a measurement" and its uncompressed size can vary between 1KB to 1MB, roughly.

Probes usually upload multiple measurements on each execution. Measurements are stored temporarily and then batched together, compressed and uploaded to the S3 bucket once every hour. To ensure transparency, incoming measurements go through basic content validation and the API returns success or error; once a measurement is accepted it will be published on S3.

OONI measurements are also processed by the fastpath and made immediately available on OONI Explorer. See the "receive_measurement" function in the probe_services.py file in the API codebase for details.

The commands which follow will be using the aws s3 cli tool. See their documentation on how to install it.

Since OONI data is part of the AWS Open Data program, you don't have to pay for access and you can use the --no-sign-request flag to access it for free.

Contains a JSON document for each measurement, separated by newline and compressed, for easy processing. The path structure allows to easily select, identify and download data based on the researcher's needs.

In the path template:

• cc is an uppercase 2 letter country code
• testname is a test name where underscores are removed
• timestamp is a YYYYMMDD timestamp
• name is a unique filename

The path structure is: s3://ooni-data-eu-fra/raw/<timestamp>/<hour>/<cc>/<testname>/<ts2>_<cc>_<testname>.<host_id>.<counter>.jsonl.gz

Example: s3://ooni-data-eu-fra/raw/20210817/15/US/webconnectivity/2021081715_US_webconnectivity.n0.0.jsonl.gz

Note: The path will be updated in the future to live under /jsonl/

Listing JSONL files:

aws s3 --no-sign-request ls \
    s3://ooni-data-eu-fra/raw/20210817/15/US/webconnectivity/

If you would like to download the raw measurements for a particular country, you can use the aws s3 sync command.

For example to download all JSONL measurements from Italy on the 1st of February 2024, you can run:

aws s3 --no-sign-request sync \
    s3://ooni-data-eu-fra/raw/20240201/ ./ \
    --exclude "*" --include "*/IT/*.jsonl.gz"

Note: the difference in paths compared to older data

The path structure is: s3://ooni-data-eu-fra/jsonl/<testname>/<cc>/<timestamp>/00/<name>.jsonl.gz

Example: s3://ooni-data-eu-fra/jsonl/webconnectivity/IT/20200921/00/20200921_IT_webconnectivity.l.0.jsonl.gz

Listing JSONL files:

aws s3 --no-sign-request ls s3://ooni-data-eu-fra/jsonl/
aws s3 --no-sign-request ls \
    s3://ooni-data-eu-fra/jsonl/webconnectivity/US/20201021/00/

If you would like to download the raw measurements for a particular country, you can use the aws s3 sync command.

For example to download webconnectivity measurements from Italy on the 1st of February 2024, you can run:

aws s3 --no-sign-request sync \
    s3://ooni-data-eu-fra/jsonl/webconnectivity/IT/20200201/ ./ \
    --exclude "*" \
    --include "*"

Note: the difference in paths compared to newer data

A "postcan" is tarball containing measurements as they are uploaded by the probes, optionally compressed. Each HTTP POST is stored in the tarball as <timestamp>_<cc>_<testname>/<timestamp>_<cc>_<testname>_<hash>.post

Example: s3://ooni-data-eu-fra/raw/20210817/11/GB/webconnectivity/2021081711_GB_webconnectivity.n0.0.tar.gz

Listing postcan files:

aws s3 --no-sign-request ls s3://ooni-data-eu-fra/raw/20210817/
aws s3 --no-sign-request ls \
    s3://ooni-data-eu-fra/raw/20210817/11/GB/webconnectivity/