Data Pipeline and Active Projects

Our Development Folders (docs) and Built Output (dist)

Google UN Data Commons Nav for API - Widen your screen to explore root of 17 UN Goals.
Display 21 "GOALS" in our left navigation by turning on "Dev Mode" under Settings in the upper right.

IN PROGRESS - Javascript Timelines from Google Data Commons API - Mehul

TO DO: Load the UN Goals timeline nav tree from the GDC API. Display a clean hierarchy with only 2 or 3 levels. Narrow to data usable in visualizations. The Colab above might use this process.

Data Commons Timelines CoLab

Observable Data Loaders can be used to pre-save data, but python in a CoLab is easier to collaborate on.

CSV files in docs/innovation data with an Observable Data Loader. See our Weather Data Loader notes.

TO DO: Display our US Industry Timelines using common timeline javascript from javascript timelines above.

After creating your webroot and pulling down data-commons, edit pages and build your static site to update the local "dist" folder.

cd data-commons
yarn build	

Visit the following to view built: http://localhost:8887/data-commons/dist and source http://localhost:8887/data-commons/docs
The "dist" folder is only deployed live by Loren to avoid file conflicts.

UN Data Commons: SDG Data by Goal The UN Sustainable Development Goals (SDG) data resides in the Google Data Commons API.

Blog: Google Data Commons API for UN Global Goals

About the 17 Global Goals - Latest Goal News - World's Largest Lesson

UN Comtrade Data Pull and Intergovernmental Panel on Climate Change (IPCC) - The UN body for climate change data assessments.

TO DO: Replace the above image with linked images and interactivity by finding a GitHub repo containing UN Goal marketing material, including icons and related images.

This is an Observable Framework project. Run once daily when you start - updates dependencies defined in package.json:

yarn install

To build - this pulls from APIs and outputs from "docs" to static files in "dist":

yarn build

The following is an alternative to using yarn dev (which you'd use if only viewing dist)

This setup allows you to view multiple repos in one webroot. If you haven't yet, start an http server in your webroot, external to the data-commons folder.

cd ../ && python -m http.server 8887

If you haven't yet, pull down the localsite repo. This provides navigation when you view src files in the "docs" folder.

git clone https://github.com/modelearth/localsite localsite &&
cd data-commons

Then visit the following to view: http://localhost:8887/data-commons/dist http://localhost:8887/data-commons/docs

Turn on GitHub Pages for both repos so we can preview changes.

For more, see https://observablehq.com/framework/getting-started.

In our project, folders for components and data reside in multiple "goal" folders. Add the localsite repo external to the data-commons repo. Your webroot setup with 2 folders:

localsite
data-commons
├─ README.md
├─ docs
│  ├─ space
│  │  ├─ components
│  │  │ 	└─ timeline.js     # an importable module
│  │  ├─ data
│  │  │ 	└─ events.json     # a static data file
│  │  └─ launches.csv.js       # a data loader
│  ├─ jobs
│  ├─ transit
│  ├─ innovation
│  ├─ education
│  ├─ economy
│  ├─ index.html               # a localsite page visible in docs
│  └─ index.md                 # a dist page
├─ dist
├─ observablehq.config.ts      # the project config file
├─ package.json
└─ .gitignore

docs - This is the “source root” — where your source files live. Pages go here. Each page is a Markdown file. Observable Framework uses file-based routing, which means that the name of the file controls where the page is served. You can create as many pages as you like. Use folders to organize your pages.

docs/index.md - This is the home page for your site. You can have as many additional pages as you’d like, but you should always have a home page, too.

docs/data - You can put data loaders or static data files anywhere in your source root, but we recommend putting them here.

docs/components - You can put shared JavaScript modules anywhere in your source root, but we recommend putting them here. This helps you pull code out of Markdown files and into JavaScript modules, making it easier to reuse code across pages, write tests and run linters, and even share code with vanilla web applications.

observablehq.config.ts - This is the project configuration file, such as the pages and sections in the sidebar navigation, and the project’s title.