It could include:

• The raw table ID
• A URL leading to the Census Reporter page about the table
• A humanized version of the metadata found in the file name
• A URL leading to the raw data file

The two income columns for acs5_2017_poverty_zctas.csv both say that the column is counting the number of people below poverty level: income_past12months_below_poverty_level and income_past12months_at_or_below_poverty_level

5 year and 1 yr surveys, maybe even SF1

-5..., -3..., -2... --> margin of error

look for + or - --> estimate (not just single +/-)

"c" in estimate N/A in moe

DC, NE

Perhaps a YAML or JSON config input that is parsed by Python?

Bug/Issue

Census data downloader correctly downloads raw data but creates a CSV duplicated data in the processed directory.

Environment

• Python 3.8
• Pipenv version 2018.11.27.dev0
• Latest version of censusdatadownloader

Reproduce

Install the package and then try to download a data set.

pipenv install census-data-downloader
censusdatadownloader --data-dir data/census race states

Expected behavior

A 52 row CSV file with total population by race in the processed directory.

Actual behavior

A 52 CSV with the same data for each column processed directory.

Possible issues/solutions

It looks like the data is correctly downloaded in the raw directory which makes me think something's happening in the process step. I'm seeing this behavior specifically with the race [geography] arguments.

I noticed the same behavior for internet counties but did get the correct data when I used internet states.

I'll see if I can debug what's happening at the process step but in the meantime I'll rely on the raw data. Thanks for your work on this!

• By gender
• By age
• By age and gender

Sources available via the Census API include:

• Five-year ACS
• Three-year ACS
• One-year ACS
• Five-year ACS data profiles
• Three-year ACS data profiles
• One-year ACS data profiles
• Decennial Summary File 1
• Decennial Summary File 3

This package seems to depend on census-data-aggregator, which is not listed in the setup.py. In a fresh environment it therefore fails to import.

This could be automatically generated by our classes via a template.

I've had a set of scripts using census-data-downloader to pull tract data for the U.S. (that's for the awesome library btw).

I noticed today that DC is suddenly missing. I thought it as possibly related to the python-us's handling of DC. I added a DC_STATEHOOD=1 environment variable but I'm still not pulling any data for DC.

I swore the data was always but maybe I'm misremembering. Regardless, is there a way to debug how census-data-downloader pulls records by state? If so, I'm happy to fork and debug further. Thanks!

It would be nice to quickly be able to inspect a map of each dataset.

This could be accomplished by providing a ".csvt" companion for each processed CSV file that includes the data types in a format QGIS will respect. Then the files could be manually merged with the shapes.

Another approach would be to have the downloader, or some downstream module based on it, do the merger automatically. Such a system could create a companion shapefile for each set with the data already merged in.

I bet I busted it.

They are explained here

Look to see why this is happening

Bug spotted by @irisslee

Here's what the API now supports:

• Nationwide
• States
• Congressional districts
• Counties
• Places
• Tracts
• State legislative districts
• ZIP codes
• MSAs
• CSAs
• combined new england city and town area
• new england city and town area
• school district (elementary)
• school district (secondary)
• school district (unified)
• metropolitan division
• county subdivision
• tribal census tract
• alaska native regional corporation

Drop jam values from aggregation and report what is dropped

Hi there I'm unable to activate the API key on windows CMD.
I've tried

export CENSUS_API_KEY=......

Also to no avail below command. As is states API key not present.

censusdatadownloader --year 2018 --data-dir data/external/ race counties

Where should I add the API key in the query above?

add this in so it downloads with data types automatically

As we think about integrating the output of census-data-downloader to be an input into the census-data-aggregator, is it possible to include the jam value information in the headers of the processed files?

For example, in the household income table the column name "10_and_under" would be replaced with "2499_10." The convention of having the headers in units of 1000s makes the 2499 representation a bit awkward though.

For tables with custom groups (such as PovertyAge), the MOEs are not returned.

This is somewhat related to #2.

I find this project to be extremely useful and a great framework for a task that I have to do often. In my projects, I've found myself using the base classes and concepts from this project when I want to download and process data from other Census Bureau API sources.

However, for non-ACS sources, I find myself entirely reimplementing many of the methods on my geotype downloader classes because the changes in functionality aren't possible by just calling super() and then adding additional logic.

I think adding these methods to BaseGeoTypeDownloader could make adding additional data sources easier, both in this project, and for other users in their own projects:

• BaseGeoTypeDownloader.get_api_client(): This would be called from the constructor to set sefl.api and allow subclasses to specify a customized subclass of census.Census that supports additional API endpoints.
• BaseGeoTypeDownloader.get_field_type_map(): This would be similar to BaseGeoTypeDownloader.get_raw_field_map() except it would map from raw field names to types that would be passed to pd.Series.astype(). Like BaseGeoTypeDownloader.get_raw_field_map(), this would be called from BaseGeoTypeDownloader.process() when setting the column types after reading in the raw table. The implementation could check for the existence of a FIELD_TYPES attribute on the table configuration class, and if that doesn't exist, default to the existing logic for ACS tables that checks the field name suffix. Adding the ability to explicitly set type conversions allows supporting non-ACS tables that might have field names that don't have the same suffix convention as ACS tables.

Hi all,

Excited to use this tool! I noticed this error when I just tried installing census-data-downloader:

Traceback (most recent call last):
  File "/Users/williamsa/.local/share/virtualenvs/fema-X3N8u1-d/bin/censusdatadownloader", line 6, in <module>
    from census_data_downloader.cli import cmd
  File "/Users/williamsa/.local/share/virtualenvs/fema-X3N8u1-d/lib/python3.7/site-packages/census_data_downloader/__init__.py", line 5, in <module>
    from .tables import TABLE_LIST
ModuleNotFoundError: No module named 'census_data_downloader.tables'

I tried installing via pipx and pipenv and received the same error.

I could be wrong about this, but looking through the source code quickly, it seems like all of the tables are ACS tables. I love this tool, but always forget whether or not it supports other kinds of census data, e.g. population estimates. It would be selfishly helpful to have a note in the README that clarifies this.

If I'm wrong about this only covering ACS tables, please let me know.

If my request makes sense to you all, I'm happy to make a pull request for this little documentation change.