dssg / 411-on-311 Goto Github PK

• Split munging into utilities and actual munging
• Organize clustering to handle both open and private data

I now have a data extract from the Data Ferrett with both census tract and block group level data. Plan is to adapt Nick Mader's Stata code to R and provide useful variable labels and organization.

So that we don't have to choose cutoffs that are quite so arbitrary

• Code to visualize/analyze 3-1-1 requests vs. daily temperature (big database)

Directory Structure

• Clean, Well named set of directories. Examples include webapp, database, and models.
• No random files in the root.
• Explanation of each directory in README.
• Sub-README's in appropriate folders
• No directories named after DSSG specific info (ie, person names)
• Should your team have more than one project, each should have it its own repo.
• In your data or database folder, provide a way to re-create your database from scratch. .sql files are often appropriate for this.

README

• Links to appropriate sections in wiki. See wiki issue for more info.
• Answers: What have you built? In a few sentences.
• Answers: How do you install it?
• Answers: What needs to be done/How can I help?
• Has some sort of Contact Info
• Open source license
• Future Issues opened.

Config

• No public facing config info - Make sure never to hardcode in database url, password, etc.
• Description of how you hide config info, ie yaml, environment variables, etc
• config.example files
• Requirements.txt or similar file.
• Relative links in any html.

Make the existing code working on EC2 (upload open portal 311 tables and make paths absolute)

They're a mess.

Should be finished today.

REQUIRED WIKI SECTIONS

• Homepage

  Intro: have a sentence or two about the project: the problems its solving, the partner, and how you're solving it. Also say that "this wiki is the central place to learn about the social problem we worked on, the data we used, the methods we used to solve it, and our findings" so people know what they're looking at.
  List of pages in the wiki

• Problem

  An in-depth description of the problem your organization, the problem you're trying to solve, and any relevant domain knowledge. Feel free to copy from blog posts and posters, if relevant.

• Data

  Describe the dataset(s) you used in the project as well as your database. Walk people through the data model (tables are handy for this), and include a (fake) sample of each dataset.
  If you scraped data, this is the place to document that.

• Methodology

  An in-depth, technical write up of the method(s) you used on your projects. Use latex equation, walk people through algorithms and models, link out to relevant documentation when possible.

• Results

  Discuss what metrics you're using to evaluate performance (if applicable), and what your final findings where.

• Future work

  Discuss what you would like to do / what is in progress.

OPTIONAL WIKI SECTIONS - if its fits your project

• Analysis

  If you did exploratory data analysis, this is the place to put it and explain your findings. Explain each finding and what your learned from it / how it motivated the methods you used. Put this between the "Data" and "Methodology" sections. Feel free to lift content from relevant blog posts, if any.

• Resources

  Resources for domain knowledge, methods, and tech. Whatever pieces of paper you used to learn what you know.

• Tool

  • API Documentation
  • Web app Documenation