Giter VIP home page Giter VIP logo

h3-population-helium's Introduction

H3 Population Helium

A global population dataset based on Uber's H3 mapping system and seeded from the open source Kontur Population dataset.

Optimized for easy consumption in Helium mapping projects.

Using the data

Download the latest kontur_population_20211109.csv.zip from this repository and extract it to your working directory.

The CSV file contains no header and two columns:

  • hex_res8 (string): H3 index at resolution 8 truncated to first 10 chars. For example, the hex 8828309565fffff will be 8828309565. If you need the full H3 index for any hex, simply append fffff.
  • population (integer): Total human population in this hex

The file is hundreds of megabytes, so it's recommended to import it into a database like Postgres for use. You can find an example of how to import a CSV into Postgres in this StackOverflow answer.

Preparing Kontur Population dataset in PostgreSQL from scratch

Although the Kontur Population dataset uses H3 resolution 8 to segment population data, it doesn't provide the H3 index of each hex. Instead, it provides a set of points that make up each hex.

We need the H3 index of each hex to easily join and compare with data in the Helium ecosystem. Let's calculate the H3 indices ourselves!

Requirements

  • PostgreSQL 9 or higher with PostGIS extension installed
  • ogr2ogr CLI (instructions below)

1. Download and extract

Download the latest dataset from this page.

The file will be of the format .gpkg.gz. Extract the .gpkg file to your working directory.

2. Use ogr2ogr to import gpkg into Postgres

You'll need ogr2ogr CLI installed for this.

If you're using Homebrew on macOS, the following command will accomplish this:

brew install gdal

Then, run the following command to import into your Postgres DB. This assumes you're running Postgres on localhost and that your DB name is heliumpop.

ogr2ogr -f PostgreSQL "PG:dbname=heliumpop" kontur_population_20211109.gpkg

This will import the data into a table called kontur_population. NOTE: If the import is stopped partway through, make sure to TRUNCATE this table before restarting the import or you will get duplicate data.

3. Convert geometry to SRID 4326 and find centers

The dataset includes hexagons in the format of PostGIS Geometry objects with SRID 3857 (Spherical Mercator). Since we're looking to get the H3 index of each hex, we need to convert the hex geometry to lat/long (SRID 4326) and find the center.

Open psql or your favorite Postgres client and run the following queries:

alter table kontur_population
add column center geometry(Geometry,4326);

update kontur_population
set center = st_centroid(st_transform(geom,4326));

4. Add H3 index to dataset

We're almost there! Add a new column for the H3 index of each hex and create a unique index to ensure consistency.

alter table kontur_population
add column hex_res8 text;

create unique index on kontur_population (hex_res8);

Clone this repo, then install required dependencies for the next step:

npm install

Copy .env.sample to a new file called .env and add your Postgres database credentials.

Then, run the H3 index update script. This will add H3 indexes, truncated to the first 10 significant characters, to the hex_res8 column of the kontur_population table:

npm run update-h3-index

5. Use or export data

And that's it! You can create the final dataset using the following query and export to CSV using your favorite tool:

select hex_res8, population
from kontur_population

h3-population-helium's People

Contributors

rawrmaan avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.