We should define a determinisc pattern for id generation and apply to all datasets
leaving this here for further discussion

• Patent and registration office: https://avoindata.prh.fi/index_en.html
• Income tax data for companies: https://www.vero.fi/tietoa-verohallinnosta/tilastot/avoin_dat/

I think we should go for: s3://data.opensanctions.org/graph/DATASET/index.json etc. and put the main catalog at s3://data.opensanctions.org/graph/index.json. (Remove from contrib/ where it is now).

SQL dumps are available here: https://ekosystem.slovensko.digital/

Fix memory problems with json parsing in ee_ariregister.
try: https://pypi.org/project/ijson/

• sources/
  • cy_companies/
  • dk_cvr
• Dockerfile
• .github/workflows/build-cy-companies.yml
• setup.py
• common
  • init.py
  • bods

Affects:

• ICIJ
• Cyprus

https://bods-data.openownership.org/

• Slovakia & Denmark

Some parsers log progress (e.g. every X parsed rows) and information about parsed files, some others do nothing. We should define a pattern and apply to all parsers.

Suggestion (as seen in the corpwatch parser):

• log file name on start of parsing a specific file
• then log every 100.000 rows/records
• log finished count of records for this file
• repeat
  This allows to keep track how source file sizes develop over time and to spot data source errors.

From alephcommons now

Right now, we're using this command invocation to upload the resulting artifacts:

aws s3 sync --no-progress --cache-control "public, max-age=64600" --metadata-directive REPLACE --acl public-read data/export s3://data.opensanctions.org/graph/icij_offshoreleaks

I would like to replace this with a call to a common shell script, like this:

bash ../../scripts/upload.sh icij_offshoreleaks data/export 

In that script we can then add cloudfront cache invalidation and content-disposition headers for the data