Giter VIP home page Giter VIP logo

cosmopolitan's Introduction

Cosmopolitan

Gitter Build Status Coverage Status

An API server for core data on places of the world.

http://cosmopolitan.openspending.org/

This API provides information about countries, cities and currencies all over the world and can be used as a cataloged data for various reasons.

Just a brief summary:

  • contains a list of all countries in the world with country code, country continent and list of countries on the same continent;
  • contains a list of all continents in the world with a list of countries for each continent;
  • contains a list of all regions in the world with a list of cities for each country;
  • contains a list of all cities in the world with population over 5000;
  • contains a list of all currencies in the world, each currency has a list of countries where this currency is being used.

Based on https://github.com/coderholic/django-cities

Using the API

Add param format=json in every request you want to perform to get JSON responses. For example: GET /v1/?format=json.

API has self-documented format, so when you request to / (root of the API without any params) you'll get descriptions of all endpoints and can follow them for more descriptions.

You could also use filters when using this API:

Filter: countries by continents

  • /v1/countries/?continents={id} - get countries of the particular continent;
  • /v1/countries/?continents={id},{id},{id} - get countries belongs to several continents.

Example: GET /v1/countries/?continents=an, GET /v1/countries/?continents=an,af

Filter: currencies by countries

  • /v1/currencies/?countries={id} - get currencies of the particular country;
  • /v1/currencies/?countries={id},{id},{id} - get currencies belongs to several countries.

Example: GET /v1/currencies/?countries=aq, GET /v1/currencies/?countries=aq,af

Filter: cities by countries

  • /v1/cities/?countries={id} - get cities of the particular country;
  • /v1/cities/?countries={id},{id},{id} - get cities belong to several countries.

Example: GET /v1/cities/?countries=gr, GET /v1/cities/?countries=gr,us

Filter: cities by regions

  • /v1/cities/?regions={id} - get cities of the particular region;
  • /v1/cities/?regions={id},{id},{id} - get cities belong to several regions.

Example: GET /v1/cities/?regions=tx, GET /v1/cities/?countries=tx,il

Filter: cities by continents

  • /v1/cities/?continents={id} - get cities of the particular continent;
  • /v1/cities/?continents={id},{id},{id} - get cities belong to several continents.

Example: GET /v1/cities/?continents=eu, GET /v1/cities/?continents=eu,af

Filter: cities by slug

  • /v1/cities/?slugs={slug} - get cities matching the particular slug, ;
  • /v1/cities/?slugs={slug},{slug},{slug} - get cities matching to several slugs.

Example: GET /v1/cities/?slugs=athens, GET /v1/cities/?slugs=athens,paris ** where slug means the name of the city in lowercase with no accents and instead of spaces, hyphens "-"

Filter: multiple filters

You can also define multiple filters on the city endpoint.

Example: GET /v1/cities/?slugs=athens,paris&countries=us&regions=tx

Filter: countries polygons by countries

  • /v1/countrypolygons/?countries={id},{id},{id}

Filter: get list of countries polygons for region(s)

  • /v1/countrypolygons/?regions={id},{id},{id}

Filter: all polygons by countries

  • /v1/polygons/?countries={id},{id},{id}

Filter: get list of all polygons for region(s)

  • /v1/polygons/?regions={id},{id},{id}

Running locally

Under virtual env (to create virtualenv, run python3 -m venv env):

source env/bin/activate

export LOCAL_DEV=1

export DEBUG=1

Postgres + PostGIS setup (OS X):

brew install postgis

createuser cosmopolitan

createdb cosmopolitan

psql cosmopolitan

alter user "cosmopolitan" with password '123456';

grant all privileges on database cosmopolitan to cosmopolitan;

CREATE EXTENSION postgis;

If you want your user to have super privileges:

ALTER USER cosmopolitan WITH SUPERUSER;

Tip for local development using (OS X)

Install GDAL binaries for OSX, and add the following to .bash_profile:

GDAL_LIBRARY_PATH="/Library/Frameworks/GDAL.framework/Versions/1.11/GDAL"
GEOS_LIBRARY_PATH="/Library/Frameworks/GEOS.framework/Versions/3/GEOS"

Then add the following to your local Django settings:

# GeoDjango fixes
GDAL_LIBRARY_PATH = os.environ.get('GDAL_LIBRARY_PATH')
GEOS_LIBRARY_PATH = os.environ.get('GEOS_LIBRARY_PATH')

(see https://www.alextomkins.com/2017/08/fixing-gdal-geos-django-macos/ for details)

Starting from scratch

Requirements

If you just pulled this repo, you need to:

  • run migrations ./manage.py migrate
  • fill in cities database with ./manage.py cities --import=all;
  • import data to cosmopolitan tables: ./manage.py import

License

Cosmopolitan is opensource, licensed under a standard MIT license (included in this repository as LICENSE).

Data sources licensies:

cosmopolitan's People

Contributors

akariv avatar brew avatar kiote avatar pwalsh avatar skarampatakis avatar

Stargazers

 avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar

cosmopolitan's Issues

Minimum viable implementation

Description

A simple API server that returns core data on the places of the world. Designed for use in OpenSpending.

Tasks

  • Setup a Django project using Django REST Framework (http://www.django-rest-framework.org)
  • Integrate django-cities (https://github.com/coderholic/django-cities)
  • Create the database for Django Cities and then, when we have that ready, Ask @pwalsh to add it to a remote server for use in production
  • Expose the following ListView API endpoints (using DRF browsable HTML, but JSON as only API format)
    • /v1/countries
    • /v1/regions (subregion? not sure, depends on data structure and the nesting of country, region, place, etc)
    • /v1/continents
    • /v1/currencies
  • Each object that is represented on a list view should be able to be looked up on a DetailView. (Example: /v1/countries/ru)
  • Any config options we need must be settable via env vars
  • Deploy app to heroku (@pwalsh will deploy the data elsewhere, on an Amazon RDS instance)
    • Use django database url extension which you'll need for heroku
  • Simple tests for each API endpoint using https://github.com/ydaniv/django-rest-assured

A detailed list of endpoints we need for this iteration:

We want all of the following endpoints.

  • Currency List /api/v1/currencies/ (/api/v1/currencies/?countries={code}, /api/v1/currencies/?countries={code},{code},{code})
    • Can filter by country slug, or list of country codes
    • Fields per object:
      • code
      • url
      • name
      • countries, being an array where each object is:
        • code
        • url
  • Currency Detail /api/v1/currencies/{code}
    • Fields:
      • code
      • url
      • name
      • countries, being an array where each object is:
        • name
        • code
        • url
      • continents, being an array where each object is:
        • name
        • code
        • url
      • Let @pwalsh know what other fields we have for each currency
  • Continent List /api/v1/continents/
    • Fields per object:
      • code
      • url
      • name
      • countries, being an array where each object is:
        • code
        • url
  • Continent Detail /api/v1/continents/{code}
    • Fields:
      • code
      • url
      • name
      • countries, being an array where each object is:
        • name
        • code
        • url
      • currencies, being an array where each object is:
        • name
        • code
        • url
      • Let @pwalsh know what other fields we have for each continent
  • Country List /api/v1/countries/ (/api/v1/countries/?continents={code}, /api/v1/countries/?continents={code},{code},{code})
    • Can filter by continent code, or list of continent codes
    • Fields per object:
      • code
      • url
      • name
      • continent (an object of code and url)
      • currencies (an array of objects of code and url)
  • Country Detail /api/v1/countries/{code}
    • Fields:
      • code
      • url
      • name
      • continent, being an object with:
        • name
        • code
        • url
        • related, being an array of countries in the continent as objects with:
          • code
          • url
      • currencies, being an array where each object is:
        • code
        • name
        • url
      • polygon
      • Let @pwalsh know what else we have on a countries

Add polygons data and endpoint

Description

We want to provide polygons for the continents, countries, regions and cities in the cosmopolitan data base. Our first data source for polygons is http://www.naturalearthdata.com/ which offers a range of downloads that together can provide coverage over pretty much everything we need.

At a high level, the task is to write a django command that ETLs the data from source into DB, and then to expose an API endpoint for polygon data.

Tasks

  • Write an ETL process that:
    • Grabs the data sources
    • Unpacks and transforms them to GeoJSON
    • Writes the data into a new table for geojsons, with appropriate identifiers
      • I'm imagining that we need a unique_together on the id + a type field with is an enum of (city or region or country or continent)
    • table needs these fields: id, type, geojson
  • Expose a new API end point called polygons
    • List and detail views as other endpoints

Regions, subregions and districts

The geoname database has regions, subregions and districts. What is the coverage like? For example, do we have data for ~70% and above of continents and countries?

Tasks

  • Give a general overview of the data we already have from Django Cities
  • Show some examples of important data:
    • Coverage in the USA?
    • Coverage across Europe?

Utie our models from django_cities models

For now we have some strong relationships between our models and django_cities models. For example this. Relationships like this should be eliminated since we going to have more than one datasource with information about cities, countries etc.

Iteration 2

Description

Next iteration on the API to expose more data, and refactor the model layer for adding additional data sources in near future.

Tasks

Model/Serialiser Refactor

  • Remove direct dependency on django-cities models
    • Data migration writes from Django Cities to our own tables/models
    • CosmopolitanContinent
    • CosmopolitanCountry
    • CosmopolitanCurrency
    • CosmopolitanCity
    • CosmopolitanRegion
    • CosmopolitanPostCode
  • Serialisers use these new models
  • The primary key for each model should be a field explicitly called id, so we expose a standard interface even though the "primary key" has various different names like "code" or "name" in our data sources
  • Update test suite to test our own models/serialisers

New endpoints

Cities

  • City List /api/v1/countries/ (/api/v1/cities/?countries={id}, /api/v1/cities/?countries={id},{id},{id})
    • Can filter by country ids, or list of country ids
    • Fields per object:
      • id - should be the ascii name?
      • name
      • name_std
      • url
      • kind
      • country (an object of code and url)
      • region (an object of code and url)
  • City Detail /api/v1/cities/{id}
    • Fields:
      • ALL the fields directly declared on the City model
      • country, being an object with the same serialiser as the country list object
      • region, being an object with the same serialiser as the region list object

Regions

  • Region List /api/v1/regions/ (/api/v1/regions/?countries={id}, /api/v1/regions/?countries={id},{id},{id})
    • Can filter by country ids, or list of country ids
    • Fields per object:
      • name
      • name_std
      • url
      • code
      • country (an object of code and url)
  • Region Detail /api/v1/regions/{id}
    • Fields:
      • ALL the fields directly declared on the City model
      • country, being an object with the same serialiser as the country list object
      • continent, being an object with the same serialiser as the region list object

Postcodes

  • Postcode List /api/v1/postcodes/ (/api/v1/postcodes/?countries={id}, /api/v1/postcodes/?countries={id},{id},{id})
    • Can filter by country ids, or list of country ids
    • Fields per object:
      • id
      • name_std
      • url
      • kind
      • country (an object of code and url)
      • region (an object of code and url)
  • Postcode Detail /api/v1/postcodes/{id}
    • Fields:
      • ALL the fields directly declared on the PostalCode model
      • country, being an object with the same serialiser as the country list object
      • region, being an object with the same serialiser as the region list object

@kiote do cities have AlternativeName references? If yes, we'll need to provide lookups based on that data, and also include it in serialisers. Please advise.

Postal code coverage

The geoname database has postal codes and associated geopoints. What is the coverage like? For example, do we have post code data for ~70% and above of countries?

Tasks

  • Give a general overview of the post code data we already have from Django Cities

Disambiguation issues

Hello,
I have set up a local instance of cosmopolitan to test some features for openbudgets/platform#32
This was because the live instance has some errors for the cities endpoint.
http://cosmopolitan.openspending.org/v1/cities/
I managed to populate the cities tables and have the cities endpoint work on my local instance. I was particularly interested for the Greek cities. But I noticed that our biggest city is missing, Athens. I have noticed also that each city has each own URL, like http://localhost:8000/v1/cities/pyrgos/. So my next thought was to look for http://localhost:8000/v1/cities/athens/. It does exist, but it is Athens in Alabama, USA! So my belief is that the IDs got overwritten because of using not unique names. Could this behavior change? And how would the URLs look like? My thought is that country and region codes could be used, or even better the IDs from the original Geonames database. So, to wrap it up:

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.