openspending / openspending-migrate Goto Github PK

In the bosnia-herzegovina dataset, cofog1, cofog2, and cofog3 are stored as three separate dimensions, but form a classification hierarchy. In FDP, this can be better represented as one dimension.

In the current OS API, each dimension has a taxonomy field, see for example here (https://openspending.org/api/2/aggregate?drilldown=region%7Ccofog3%7Ccofog2%7Ccofog1&cut=year%3A2010&dataset=ukgov-finances-cra)

{
  "drilldown": [
    {
      "region": {
        "taxonomy": "cra-region", 
        "html_url": "https://openspending.org/ukgov-finances-cra/region/not-identifiable", 
        "id": 14, 
        "name": "not-identifiable", 
        "label": "NOT IDENTIFIABLE"
      }, 
      "amount": 42529187000.0, 
      "cofog2": {
        "name": "01.7", 
        "color": "#9900cc", 
        "taxonomy": "cofog-2", 

both cra-region and cofog-2 are values which are not present in the migrated datapackage.json and seem to be needed by users of the data.

A few (8) public datasets have no owner. To whom can we assign these datasets?

• admin
• nobody
• (some prefixed version [e.g. '_admin', '_nobody'] of above)

Derechos reconocidos netos (Andalucía) lists its measure as "Total euros por habitante" (amount) which is reflected in visualizations, etc. However, the model.json lists amount as having "type": "attribute" instead of "type": "measure". In mapping to FDP, it seems we can't guarantee that a measure in the old OS will have "type": "measure", so in sorting the measures from dimensions, perhaps we fall back to columns that are called "amount"?

I am inclined to duplicate datasets that have multiple owners, however, 4.4 GB of datasets have 2 owners, meaning that this method would result in at least that much space being occupied by duplicate data.

Numbers

Private datasets: 586 (981 datasets have empty or incomplete "data" model)
Public datasets: 1094

Sizes

Public Datasets

without archived sources: 9.86 GB
with archived sources: 17.32 GB

Private Datasets

without archived sources: 5.1 GB
with archived sources: 10.61 GB

Sources

Private Datasets

valid_sources: 456 (78% of total sources)
invalid_sources: 128

Public Datasets

valid_sources: 1396 (92% of total sources)
invalid_sources: 120

Usernames

private

1 owner: 557 (2.7 GB)
2 owners: 19 (768.1 MB)
3 owners: 7 (1.6 GB)
4 owners: 1 (0)
5 owners: 2 (382.2 kB)
6 owners: 1 (3.0 MB)
8 owners: 1 (0)

public

0 owners: 8 (134.5 MB)
1 owner: 987 (4.2 GB)
2 owners: 75 (4.4 GB)
3 owners: 13 (557.3 MB)
4 owners: 8 (83.8 MB)
5 owners: 3 (461.3 MB)

In the initial pass at this, @trickvi mentioned:

Should we infer the schema for each resource (i.e. should we use the fields.type etc. in DP's schema), it might lead to false positives, I propose we don't use anything except fields.name

In creating the datapackage.json, I'm currently inferring schemas of the generated dataset.csv. With the large clean corpus, this could serve as a good test case for the JTS libraries. However, this also could obviously lead to errors. As an alternative, we can pull the types from the mapping object in the generated dataset.json.