german1608 / hdx-scraper-template Goto Github PK

Replace scrapername everywhere with your scraper's name eg. worldbank Replace ScraperName everywhere with your scraper's name eg. World Bank Look for xxx and ... and replace add text accordingly.

Scrapers can be installed on Jenkins and set up to run on a schedule.

Collector designed to collect ScraperName datasets from the ScraperName website and to automatically register datasets on the Humanitarian Data Exchange project.

For full scrapers following this template see: ACLED, FTS, WHO, World Bank, WorldPop

For a scraper that also creates datasets disaggregated by indicator (not just country) and reads metadata from a Google spreadsheet exported as csv, see: IDMC

python run.py

For the script to run, you will need to either pass in your HDX API key as a parameter or have a file called .hdx_configuration.yml in your home directory containing your HDX key eg.

hdx_key: "XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX"
hdx_read_only: false
hdx_site: test

You will also need to pass in your user agent as a parameter or pass a parameter user_agent_config_yaml specifying where your user agent file is located. It should be of the form:

user_agent: MY_USER_AGENT

If you have many user agents, you can create a file of this form, put its location in user_agent_config_yaml and specify the lookup in user_agent_lookup:

myscraper:
    user_agent: MY_USER_AGENT
myscraper2:
    user_agent: MY_USER_AGENT2

Note for HDX scrapers: there is a universal .useragents.yml file you should use.

Alternatively, you can set up environment variables eg. for production runs: USER_AGENT, HDX_KEY, HDX_SITE, BASIC_AUTH, EXTRA_PARAMS, TEMP_DIR, LOG_FILE_ONLY