ocha-dap / cod-migration Goto Github PK

View Code? Open in Web Editor NEW

0.0 0.0 0.0 120 KB

cod-migration's People

Contributors

Watchers

cod-migration's Issues

Implement checks to see if download of google docs succeeds.

When someone changes the permissions of a config, intent or taxonomy google doc file, it is possible that downloading the csv version of it is impossible without login.

We need to detect this situation and display nicely to the user that is running the script.

Use a separate resource folder

Some HTML tags need to be removed from HR.info dataset metadata

See below.
The big space between words on Instructions are < div > tags.

Have the source csv files in a separate folder

source csv files means:

config files
intent files
taxonomy files

Often unable to create resources

Trying to run gospawn for the first time since Friday. Getting this error on many resources (but not all):

Uploading resource resources/afg_ppl_settlements_pnt_cso.zip
FAILED.
{"help": "http://test-data.hdx.rwlabs.org/api/3/action/help_show?name=resource_create", "success": false, "error": {"url": ["Missing value"], "__type": "Validation Error"}}
Error occured when create : False

However even with the dataset registers as complete, there are no resources. For example: https://test-data.hdx.rwlabs.org/dataset/afghanistan-elevation-model

Missing "Summary", "Abstract", "Instructions" keyword on Notes

In log file, HDX URL is missing "/dataset"

Do not put the file name in resource description

Leave it empty for now.

paragraph tags appearing in HDX SOURCE

Long data set names

Date slippage

Dataset date is off by one day in node 4536:
https://test-data.hdx.rwlabs.org/dataset/afghanistan-water-courses
https://www.humanitarianresponse.info/en/operations/afghanistan/dataset/afghanistan-water-courses

Change maintainer to be the user specified in the config file

Dataset migrated success message appears after every resource creation.

It is only supposed to happen when ALL resources have been successfully migrated.
Please check log as well (to correct it if needed).

hrefs getting lost

The source field contains an href (both in interface and as delivered by api). We should be capturing these in HDX.

Resolve issue with default CFR license

host license somewhere
set license_other text for HDX

Could we pass authentication to HR to get unpublished datasets from the API

Unpublished datasets on CFR will fail (logged as failure). To be auto migrated, IMO would need to publish them at some point before the moment of migration and unpublish before HR.info scripts are run.

This could occur in the case of "publish only after a disaster" datasets or datasets that are previous versions than an office wants to keep around for archival purposes.

We will explore passing authentication to HR.info to give access to unpublished datasets that require migration (presumably to private status).

"Test Short" failing on node 4537 during dataset update step

Terminal output:

cj@ubuntu-vm:~/coding/hdx-gospawn$ python3.4 gospawn.py test short
Trying to get configs/intent.csv
SUCCESS.
Trying to get configs/intent.csv
SUCCESS.
Start processing dataset with Node ID = 22243
Dataset exists.
Identifying dataset: afghanistan-admin-level-1-boundaries
Updating identified existing dataset: afghanistan-admin-level-1-boundaries
Trying to get resources/afg_admbnd_reg_UNAMA.zip
Uploading resource resources/afg_admbnd_reg_UNAMA.zip
Success.
Dataset exists.
Dataset with Node ID = 22243 has been MIGRATED.
Start processing dataset with Node ID = 4246
Dataset exists.
Identifying dataset: yemen-admin-boundaries-settlements
Updating identified existing dataset: yemen-admin-boundaries-settlements
Trying to get resources/yemen_p-codes_27032013.xlsx
Uploading resource resources/yemen_p-codes_27032013.xlsx
Success.
Trying to get resources/yemen_pcode_final_010715.xlsx
Uploading resource resources/yemen_pcode_final_010715.xlsx
Success.
Dataset exists.
Dataset with Node ID = 4246 has been MIGRATED.
Start processing dataset with Node ID = 4537
Dataset exists.
Identifying dataset: afghanistan-population-statistics
Updating identified existing dataset: afghanistan-population-statistics

usage: gospawn.py test (short|full)

Full log output:

2015-08-03T11:40:21+0200 | dataset | START | True | 22243 | None | https://www.humanitarianresponse.info/en/api/v1.0/datasets/22243 | None | None | None | None | None
2015-08-03T11:40:22+0200 | dataset | get | True | 22243 | https://www.humanitarianresponse.info/en/operations/afghanistan/dataset/afghanistan-admin-level-1-boundaries | https://www.humanitarianresponse.info/en/api/v1.0/datasets/22243 | None | None | None | None | None
2015-08-03T11:40:22+0200 | dataset | format | True | 22243 | https://www.humanitarianresponse.info/en/operations/afghanistan/dataset/afghanistan-admin-level-1-boundaries | https://www.humanitarianresponse.info/en/api/v1.0/datasets/22243 | None | Afghanistan - Admin Level 1 Boundaries | None | None | None
2015-08-03T11:40:28+0200 | dataset | update | True | 22243 | https://www.humanitarianresponse.info/en/operations/afghanistan/dataset/afghanistan-admin-level-1-boundaries | https://www.humanitarianresponse.info/en/api/v1.0/datasets/22243 | 635f15d9-40f3-4d15-b8f2-4611253028bc | Afghanistan - Admin Level 1 Boundaries | None | https://test-data.hdx.rwlabs.org/dataset/afghanistan-admin-level-1-boundaries | https://test-data.hdx.rwlabs.org/api/3/action/package_show?id=afghanistan-admin-level-1-boundaries
2015-08-03T11:40:28+0200 | resource | format | True | 18900 | https://www.humanitarianresponse.info/sites/www.humanitarianresponse.info/files/datasets/afg_admbnd_reg_UNAMA.zip | None | afg_admbnd_reg_UNAMA.zip | None | afghanistan-admin-level-1-boundaries | None | https://test-data.hdx.rwlabs.org/api/3/action/resource_show?id=afg_admbnd_reg_UNAMA.zip
2015-08-03T11:40:31+0200 | resource | get | True | 18900 | https://www.humanitarianresponse.info/sites/www.humanitarianresponse.info/files/datasets/afg_admbnd_reg_UNAMA.zip | None | afg_admbnd_reg_UNAMA.zip | None | afghanistan-admin-level-1-boundaries | None | https://test-data.hdx.rwlabs.org/api/3/action/resource_show?id=afg_admbnd_reg_UNAMA.zip
2015-08-03T11:40:45+0200 | resource | create | True | 18900 | https://www.humanitarianresponse.info/sites/www.humanitarianresponse.info/files/datasets/afg_admbnd_reg_UNAMA.zip | None | afg_admbnd_reg_UNAMA.zip | None | afghanistan-admin-level-1-boundaries | None | https://test-data.hdx.rwlabs.org/api/3/action/resource_show?id=afg_admbnd_reg_UNAMA.zip
2015-08-03T11:40:50+0200 | dataset | update | True | 22243 | https://www.humanitarianresponse.info/en/operations/afghanistan/dataset/afghanistan-admin-level-1-boundaries | https://www.humanitarianresponse.info/en/api/v1.0/datasets/22243 | 635f15d9-40f3-4d15-b8f2-4611253028bc | Afghanistan - Admin Level 1 Boundaries | None | https://test-data.hdx.rwlabs.org/dataset/afghanistan-admin-level-1-boundaries | https://test-data.hdx.rwlabs.org/api/3/action/package_show?id=afghanistan-admin-level-1-boundaries
2015-08-03T11:40:50+0200 | dataset | COMPLETE | True | 22243 | https://www.humanitarianresponse.info/en/operations/afghanistan/dataset/afghanistan-admin-level-1-boundaries | https://www.humanitarianresponse.info/en/api/v1.0/datasets/22243 | 635f15d9-40f3-4d15-b8f2-4611253028bc | Afghanistan - Admin Level 1 Boundaries | None | https://test-data.hdx.rwlabs.org/dataset/afghanistan-admin-level-1-boundaries | https://test-data.hdx.rwlabs.org/api/3/action/package_show?id=afghanistan-admin-level-1-boundaries
2015-08-03T11:40:50+0200 | dataset | START | True | 4246 | None | https://www.humanitarianresponse.info/en/api/v1.0/datasets/4246 | None | None | None | None | None
2015-08-03T11:40:51+0200 | dataset | get | True | 4246 | https://www.humanitarianresponse.info/en/operations/yemen/dataset/yemen-admin-level-1-boundaries-admin-level-2-boundaries | https://www.humanitarianresponse.info/en/api/v1.0/datasets/4246 | None | None | None | None | None
2015-08-03T11:40:51+0200 | dataset | format | True | 4246 | https://www.humanitarianresponse.info/en/operations/yemen/dataset/yemen-admin-level-1-boundaries-admin-level-2-boundaries | https://www.humanitarianresponse.info/en/api/v1.0/datasets/4246 | None | Yemen - Admin Boundaries, Settlements | None | None | None
2015-08-03T11:40:54+0200 | dataset | update | True | 4246 | https://www.humanitarianresponse.info/en/operations/yemen/dataset/yemen-admin-level-1-boundaries-admin-level-2-boundaries | https://www.humanitarianresponse.info/en/api/v1.0/datasets/4246 | 24730302-590d-4539-9914-a6ae37c5f490 | Yemen - Admin Boundaries, Settlements | None | https://test-data.hdx.rwlabs.org/dataset/yemen-admin-boundaries-settlements | https://test-data.hdx.rwlabs.org/api/3/action/package_show?id=yemen-admin-boundaries-settlements
2015-08-03T11:40:54+0200 | resource | format | True | 3774 | https://www.humanitarianresponse.info/sites/www.humanitarianresponse.info/files/yemen_p-codes_27032013.xlsx | None | yemen_p-codes_27032013.xlsx | None | yemen-admin-boundaries-settlements | None | https://test-data.hdx.rwlabs.org/api/3/action/resource_show?id=yemen_p-codes_27032013.xlsx
2015-08-03T11:40:59+0200 | resource | get | True | 3774 | https://www.humanitarianresponse.info/sites/www.humanitarianresponse.info/files/yemen_p-codes_27032013.xlsx | None | yemen_p-codes_27032013.xlsx | None | yemen-admin-boundaries-settlements | None | https://test-data.hdx.rwlabs.org/api/3/action/resource_show?id=yemen_p-codes_27032013.xlsx
2015-08-03T11:41:27+0200 | resource | create | True | 3774 | https://www.humanitarianresponse.info/sites/www.humanitarianresponse.info/files/yemen_p-codes_27032013.xlsx | None | yemen_p-codes_27032013.xlsx | None | yemen-admin-boundaries-settlements | None | https://test-data.hdx.rwlabs.org/api/3/action/resource_show?id=yemen_p-codes_27032013.xlsx
2015-08-03T11:41:27+0200 | resource | format | True | 97630 | https://www.humanitarianresponse.info/sites/www.humanitarianresponse.info/files/datasets/yemen_pcode_final_010715.xlsx | None | yemen_pcode_final_010715.xlsx | None | yemen-admin-boundaries-settlements | None | https://test-data.hdx.rwlabs.org/api/3/action/resource_show?id=yemen_pcode_final_010715.xlsx
2015-08-03T11:41:31+0200 | resource | get | True | 97630 | https://www.humanitarianresponse.info/sites/www.humanitarianresponse.info/files/datasets/yemen_pcode_final_010715.xlsx | None | yemen_pcode_final_010715.xlsx | None | yemen-admin-boundaries-settlements | None | https://test-data.hdx.rwlabs.org/api/3/action/resource_show?id=yemen_pcode_final_010715.xlsx
2015-08-03T11:42:01+0200 | resource | create | True | 97630 | https://www.humanitarianresponse.info/sites/www.humanitarianresponse.info/files/datasets/yemen_pcode_final_010715.xlsx | None | yemen_pcode_final_010715.xlsx | None | yemen-admin-boundaries-settlements | None | https://test-data.hdx.rwlabs.org/api/3/action/resource_show?id=yemen_pcode_final_010715.xlsx
2015-08-03T11:42:07+0200 | dataset | update | True | 4246 | https://www.humanitarianresponse.info/en/operations/yemen/dataset/yemen-admin-level-1-boundaries-admin-level-2-boundaries | https://www.humanitarianresponse.info/en/api/v1.0/datasets/4246 | 24730302-590d-4539-9914-a6ae37c5f490 | Yemen - Admin Boundaries, Settlements | None | https://test-data.hdx.rwlabs.org/dataset/yemen-admin-boundaries-settlements | https://test-data.hdx.rwlabs.org/api/3/action/package_show?id=yemen-admin-boundaries-settlements
2015-08-03T11:42:07+0200 | dataset | COMPLETE | True | 4246 | https://www.humanitarianresponse.info/en/operations/yemen/dataset/yemen-admin-level-1-boundaries-admin-level-2-boundaries | https://www.humanitarianresponse.info/en/api/v1.0/datasets/4246 | 24730302-590d-4539-9914-a6ae37c5f490 | Yemen - Admin Boundaries, Settlements | None | https://test-data.hdx.rwlabs.org/dataset/yemen-admin-boundaries-settlements | https://test-data.hdx.rwlabs.org/api/3/action/package_show?id=yemen-admin-boundaries-settlements
2015-08-03T11:42:07+0200 | dataset | START | True | 4537 | None | https://www.humanitarianresponse.info/en/api/v1.0/datasets/4537 | None | None | None | None | None
2015-08-03T11:42:08+0200 | dataset | get | True | 4537 | https://www.humanitarianresponse.info/en/operations/afghanistan/dataset/afghanistan-population-statistics | https://www.humanitarianresponse.info/en/api/v1.0/datasets/4537 | None | None | None | None | None
2015-08-03T11:42:08+0200 | dataset | format | True | 4537 | https://www.humanitarianresponse.info/en/operations/afghanistan/dataset/afghanistan-population-statistics | https://www.humanitarianresponse.info/en/api/v1.0/datasets/4537 | None | Afghanistan - Population Statistics | None | None | None
2015-08-03T11:43:09+0200 | dataset | update | False | 4537 | https://www.humanitarianresponse.info/en/operations/afghanistan/dataset/afghanistan-population-statistics | https://www.humanitarianresponse.info/en/api/v1.0/datasets/4537 | None | Afghanistan - Population Statistics | None | https://test-data.hdx.rwlabs.org/dataset/afghanistan-population-statistics | https://test-data.hdx.rwlabs.org/api/3/action/package_show?id=afghanistan-population-statistics

It is possible for a CRF dataset to have nothing for the HDX description field

There are 24 datasets in CFR with null for summary, abstract, and instructions (which leaves nothing for the HDX description field, which is required). In those cases, could we populate the description with a space? The result looks ok in the interface.

Uppercase the two letter language codes used in HDX CAVEATS

Losing the third resource

In the most recent run, the log shows 3 resources migrated for node 4537, but only two are visible in ckan (the zip is dropped not appearing): https://test-data.hdx.rwlabs.org/dataset/afghanistan-population-statistics#

Log entries:

2015-07-10T13:51:51+0200 | dataset | START | True | 4537 | None | https://www.humanitarianresponse.info/en/api/v1.0/datasets/4537 | None | None | None | None | None
2015-07-10T13:51:52+0200 | dataset | get | True | 4537 | https://www.humanitarianresponse.info/en/operations/afghanistan/dataset/afghanistan-population-statistics | https://www.humanitarianresponse.info/en/api/v1.0/datasets/4537 | None | None | None | None | None
2015-07-10T13:51:52+0200 | dataset | format | True | 4537 | https://www.humanitarianresponse.info/en/operations/afghanistan/dataset/afghanistan-population-statistics | https://www.humanitarianresponse.info/en/api/v1.0/datasets/4537 | None | Afghanistan - Population Statistics | None | None | None
2015-07-10T13:51:59+0200 | dataset | update | True | 4537 | https://www.humanitarianresponse.info/en/operations/afghanistan/dataset/afghanistan-population-statistics | https://www.humanitarianresponse.info/en/api/v1.0/datasets/4537 | 4c58fa87-3259-4cc8-a35c-48489cafb773 | Afghanistan - Population Statistics | None | https://test-data.hdx.rwlabs.org/dataset/afghanistan-population-statistics | https://test-data.hdx.rwlabs.org/api/3/action/package_show?id=afghanistan-population-statistics
2015-07-10T13:51:59+0200 | resource | format | True | 51842 | https://www.humanitarianresponse.info/sites/www.humanitarianresponse.info/files/datasets/AFG_CSO_POP_DISTRICT_2014_2015_OCHA_201409Sep08.xlsx | None | AFG_CSO_POP_DISTRICT_2014_2015_OCHA_201409Sep08.xlsx | None | afghanistan-population-statistics | None | https://test-data.hdx.rwlabs.org/api/3/action/resource_show?id=AFG_CSO_POP_DISTRICT_2014_2015_OCHA_201409Sep08.xlsx
2015-07-10T13:52:00+0200 | resource | get | True | 51842 | https://www.humanitarianresponse.info/sites/www.humanitarianresponse.info/files/datasets/AFG_CSO_POP_DISTRICT_2014_2015_OCHA_201409Sep08.xlsx | None | AFG_CSO_POP_DISTRICT_2014_2015_OCHA_201409Sep08.xlsx | None | afghanistan-population-statistics | None | https://test-data.hdx.rwlabs.org/api/3/action/resource_show?id=AFG_CSO_POP_DISTRICT_2014_2015_OCHA_201409Sep08.xlsx
2015-07-10T13:52:22+0200 | resource | create | True | 51842 | https://www.humanitarianresponse.info/sites/www.humanitarianresponse.info/files/datasets/AFG_CSO_POP_DISTRICT_2014_2015_OCHA_201409Sep08.xlsx | None | AFG_CSO_POP_DISTRICT_2014_2015_OCHA_201409Sep08.xlsx | None | afghanistan-population-statistics | None | https://test-data.hdx.rwlabs.org/api/3/action/resource_show?id=AFG_CSO_POP_DISTRICT_2014_2015_OCHA_201409Sep08.xlsx
2015-07-10T13:52:22+0200 | resource | format | True | 3860 | https://www.humanitarianresponse.info/sites/www.humanitarianresponse.info/files/AFG_CSO_POP_DISTRICT_2013_2014_OCHA_201307JUL31.xlsx | None | AFG_CSO_POP_DISTRICT_2013_2014_OCHA_201307JUL31.xlsx | None | afghanistan-population-statistics | None | https://test-data.hdx.rwlabs.org/api/3/action/resource_show?id=AFG_CSO_POP_DISTRICT_2013_2014_OCHA_201307JUL31.xlsx
2015-07-10T13:52:24+0200 | resource | get | True | 3860 | https://www.humanitarianresponse.info/sites/www.humanitarianresponse.info/files/AFG_CSO_POP_DISTRICT_2013_2014_OCHA_201307JUL31.xlsx | None | AFG_CSO_POP_DISTRICT_2013_2014_OCHA_201307JUL31.xlsx | None | afghanistan-population-statistics | None | https://test-data.hdx.rwlabs.org/api/3/action/resource_show?id=AFG_CSO_POP_DISTRICT_2013_2014_OCHA_201307JUL31.xlsx
2015-07-10T13:52:32+0200 | resource | create | True | 3860 | https://www.humanitarianresponse.info/sites/www.humanitarianresponse.info/files/AFG_CSO_POP_DISTRICT_2013_2014_OCHA_201307JUL31.xlsx | None | AFG_CSO_POP_DISTRICT_2013_2014_OCHA_201307JUL31.xlsx | None | afghanistan-population-statistics | None | https://test-data.hdx.rwlabs.org/api/3/action/resource_show?id=AFG_CSO_POP_DISTRICT_2013_2014_OCHA_201307JUL31.xlsx
2015-07-10T13:52:32+0200 | resource | format | True | 3861 | https://www.humanitarianresponse.info/sites/www.humanitarianresponse.info/files/CSO%20Estimated%20Population%20of%20Afghanistan%202013-14%20Report.zip | None | CSO Estimated Population of Afghanistan 2013-14 Report.zip | None | afghanistan-population-statistics | None | https://test-data.hdx.rwlabs.org/api/3/action/resource_show?id=CSO Estimated Population of Afghanistan 2013-14 Report.zip
2015-07-10T13:52:41+0200 | resource | get | True | 3861 | https://www.humanitarianresponse.info/sites/www.humanitarianresponse.info/files/CSO%20Estimated%20Population%20of%20Afghanistan%202013-14%20Report.zip | None | CSO Estimated Population of Afghanistan 2013-14 Report.zip | None | afghanistan-population-statistics | None | https://test-data.hdx.rwlabs.org/api/3/action/resource_show?id=CSO Estimated Population of Afghanistan 2013-14 Report.zip
2015-07-10T13:53:54+0200 | resource | create | True | 3861 | https://www.humanitarianresponse.info/sites/www.humanitarianresponse.info/files/CSO%20Estimated%20Population%20of%20Afghanistan%202013-14%20Report.zip | None | CSO Estimated Population of Afghanistan 2013-14 Report.zip | None | afghanistan-population-statistics | None | https://test-data.hdx.rwlabs.org/api/3/action/resource_show?id=CSO Estimated Population of Afghanistan 2013-14 Report.zip
zip
2015-07-10T13:54:00+0200 | dataset | update | True | 4537 | https://www.humanitarianresponse.info/en/operations/afghanistan/dataset/afghanistan-population-statistics | https://www.humanitarianresponse.info/en/api/v1.0/datasets/4537 | 4c58fa87-3259-4cc8-a35c-48489cafb773 | Afghanistan - Population Statistics | None | https://test-data.hdx.rwlabs.org/dataset/afghanistan-population-statistics | https://test-data.hdx.rwlabs.org/api/3/action/package_show?id=afghanistan-population-statistics
2015-07-10T13:54:00+0200 | dataset | COMPLETE | True | 4537 | https://www.humanitarianresponse.info/en/operations/afghanistan/dataset/afghanistan-population-statistics | https://www.humanitarianresponse.info/en/api/v1.0/datasets/4537 | 4c58fa87-3259-4cc8-a35c-48489cafb773 | Afghanistan - Population Statistics | None | https://test-data.hdx.rwlabs.org/dataset/afghanistan-population-statistics | https://test-data.hdx.rwlabs.org/api/3/action/package_show?id=afghanistan-population-statistics

Identify an existing hdx dataset

dataset with same name:

tag the dataset with Node ID and a prefix (CJ proposed the form "nodeid12345", "nodeid" being the prefix and the numbers being the actual node id)
when check if exists, check also if the there is a tag with the specific node id
- if it is, compare with current node id
  - if equal, update the dataset
  - else change the name (add "-0" at the end of the name) and try again.
- else create a new dataset

What to do about html formatting of text

This is a single field using html

tags. The html is retained, but not rendered. Ideas? Could we try converting some html to markup? Or maybe it's better to deal with these manually.

http://www.humanitarianresponse.info/en/operations/mali/dataset/mali-water-bodies-water-courses

https://test-data.hdx.rwlabs.org/dataset/mali-water-bodies-water-courses

Not catching raised error on multiple locations

migrate should catch that.

make a method to format gdocs downlaod urls from document key

Apparently gogle docs keeps changeing things around in the api, so there's no guarantee the same url will work tomorrow.

Dataset should be private

The intent file indicates private for node 4536 but it ends up as public.

https://test-data.hdx.rwlabs.org/dataset/afghanistan-water-courses
https://www.humanitarianresponse.info/en/operations/afghanistan/dataset/afghanistan-water-courses

Failing on this dataset, maybe because it has category = COD

https://www.humanitarianresponse.info/en/operations/central-african-republic/dataset/central-african-republic-other-cod

We could ignore category = COD or FOD when there is more than one category. And if category is only COD or FOD, we could NOMIGRATE-CATEGORY

nginx 413 status code (entity too large)

This needs to be treated correctly, aka log the failure and continue to the next resource.

header text "Most Recent Changes" should be skipped if the field is null

Same is true for all other header texts, but I haven't noticed other errors like this on other fields

After license issue is resolved, edit the URL in the default Terms of Use text

When gospawn detects the default terms of use text, it maintains the URL to the HR.info terms of use. However, after OCHA-DAP/hdx-ckan#3055 is resolved, we can change the URL to be the HDX url for this terms of use file.

Add a log line when the script execution is interrupted (by Ctrl+C for example)

Have the file size in resource description

@cjhendrix needs to get approval for this.

Bug introduced by migration-incomplete tag

Please test it, identify it and correct it.

We are altering the names of files

The resource title and the name of the actual downloaded file do not match. Happy to discuss options.

Resource file name modified

The file name of the resources get modified (for example underscores are stripped).

Missing source in HDX

For node 4537, the HDX Source is "Unknown" but should be "The population data was sourced from the Islamic Republic of Afghanistan Central Statistics Organization (CSO)."
https://test-data.hdx.rwlabs.org/dataset/afghanistan-population-statistics

Other datasets have a value in Source. For example: https://test-data.hdx.rwlabs.org/dataset/mali-settlements

Multilingual terms of use

There are different language versions of the terms of use (depending on the language setting of the hr.info space). French and Spanish are the only other ones as far as I know.

Just as we do with the english version of CFR's default terms of use, we need to transform the French and Spanish as well. We will transform them to english.

Issue #10 and #43 have more info about how we are handling for english.

For this issue
Add a handler for the string below with the same output as for english. I'll add the spanish string here as soon as I find an example.

terms_of_use: "<p>Prier de voir les<a href="/applications/data/page/terms-use">Conditions d’utilisation</a> de ce site. Cela ne remplace pas les conditions d’utilisation liées à ces données.</p> "

Value for HDX CONTRIBUTOR should be set from config file

Contributor = HDX, but it should be ocha-afghanistan (from the config file). I think you've already fixed this one, but documenting it here for completeness.

Migration success / failure tagging

What to do on migration process failure? (meaning for example failure of one resource of a multi-resource dataset)

As discussed with @cjhendrix:

upon failure:
- tag it as "migration incomplete" - ONLY IF error does occur in resource part of the dataset
- make it private
- try to upload as much as possible (go to next resource) - this was harder to implement so it will just stop the current dataset migration process
upon success:
- remove tag "migration incomplete" if exists
- make it public if the intent says "public"

nomigrate-multiple locations error

We need to properly treat this error, aka log it and move to the next dataset in the intent file.
Right now it just raises an exception and exit.

requests logger cluttering log file

See below.

Either make a separate log file for our use or shut off the requests logger.

2015-06-21T01:07:44+0300 Starting new HTTP connection (1): test-data.hdx.rwlabs.org
2015-06-21T01:08:04+0300 Setting read timeout to None
2015-06-21T01:08:07+0300 "POST /api/action/resource_create HTTP/1.1" 200 2262
2015-06-21T01:08:07+0300 | resource | create | True | 3348 | https://www.humanitarianresponse.info/sites/www.humanitarianresponse.info/files/mli_dem_0.zip | None | mli_dem_0.zip | None | mali-elevation-model | None | http://test-data.hdx.rwlabs.org/api/3/action/resource_show?id=mli_dem_0.zip
2015-06-21T01:08:07+0300 Starting new HTTPS connection (1): www.humanitarianresponse.info
2015-06-21T01:08:08+0300 Setting read timeout to None
2015-06-21T01:08:08+0300 "GET /en/api/v1.0/datasets/26432 HTTP/1.1" 200 730
2015-06-21T01:08:08+0300 | dataset | get | True | 26432 | https://www.humanitarianresponse.info/en/operations/mali/dataset/mali-affected-per...

ocha-dap / cod-migration Goto Github PK

cod-migration's People

Contributors

Watchers

cod-migration's Issues

Recommend Projects

Recommend Topics

Recommend Org