sel-columbia / bamboo Goto Github PK

Add new end point POST /aggregations/[ID]

request parameters:

• name is column name that results will be stored in, e.g. avg_num_students
• formula is the formula used to calculate the field values, e.g. avg(num_students)
• query is a selector query, e.g. where is_hs=True
• group is the column to group by, e.g. lga

the results are stored in a new dataset, the aggregation are stored in a aggregation model that includes the above, id of originating dataset, and id of resulting dataset.

aggregations with the same group value for the same originating dataset are store in the same resulting dataset.

start by support a count formula operation

[for the future ratio, min, max, sum, mean, etc]

see comments: #16

set up bamboo.modilabs.org to resolve to this instance

• on hitting summary stats cache the results
• maybe in the database
• maybe with redis or another tool
• this includes group bys

(in general cache as much as reasonable)

when you do a summary group on a numeric column you get a server error

needed so that I can complete static/bar_graphs.html, which is otherwise almost there.

test process:

• load bar_graphs.html
• open firebug
• on the right hand side, you see a select box, change it; the action should be to hit bamboo with a group url
• see if firebug output includes a json that contains a grouped json summary; doesn't right now.

• calculate branch implements twister
• celery branch uses celery but requires patching lib file, change pymongo => bson

Stats for

• insert
• summary statistics
• summary statistics with group
• calculations
• aggregations
• update dataset (and calcs/aggs accordingly)

Use date column to show changes in data over time, moving average, interpolation

see http://www.mongodb.org/display/DOCS/Retrieving+a+Subset+of+Fields

(would be useful for summarizr's new incarnation, which i can show tomorrow, and in general).

• explicit detect or set column types
• store column type with column and return in JSON, etc.
• in calculations store type from input type
  • maybe use input column types to validate formula

note: currently boolean calculations generate outputs as 1 or 0 integers, in the future type these columns as booleans

• formula with canonical name for arguments and results
  • e.g. student_teacher_ratio := num_teachers / num_students
• user defined mapping between their columns names and columns required by formula,
  • num_teachers: my_num_teachers
  • num_students: my_num_girl + my_num_boys
• or
  • num_teachers: my_num_teachers
  • num_students: my_num_students

Currently,

---------- coverage: platform linux2, python 2.7.3-final-0 -----------                                    
Name                                  Stmts   Miss  Cover
---------------------------------------------------------
bamboo                                    7      1    86%
celeryconfig                              6      0   100%
config/__init__                           0      0   100%
config/db                                15      0   100%
config/settings                           1      0   100%
controllers/__init__                      0      0   100%
controllers/calculations                 18      0   100%
controllers/datasets                     28      7    75%
controllers/root                          4      1    75%
lib/__init__                              0      0   100%
lib/constants                            13      0   100%
lib/decorators                           13      8    38%
lib/exceptions                            2      0   100%
lib/io                                   22      2    91%
lib/parser                               55      4    93%
lib/summary                              23      8    65%
lib/tasks/__init__                        0      0   100%
lib/tasks/calculator                      9      0   100%
lib/tasks/import_dataset                 12      2    83%
lib/utils                                40      1    98%
models/__init__                           0      0   100%
models/abstract_model                    11      0   100%
models/calculation                       23      2    91%
models/dataset                           19      0   100%
models/observation                       38      6    84%
tests/__init__                            0      0   100%
tests/controllers/__init__                0      0   100%
tests/controllers/test_calculations      21      0   100%
tests/controllers/test_datasets          46      8    83%
tests/decorators                         17      4    76%
tests/lib/test_calculator                30      0   100%
tests/lib/test_parser                    14      0   100%
tests/lib/test_summary                    7      0   100%
tests/models/__init__                     0      0   100%
tests/models/test_calculation            23      0   100%
tests/models/test_dataset                29      0   100%
tests/models/test_observation            40      0   100%
tests/test_base                          21      0   100%
tests/test_server                         4      0   100%
---------------------------------------------------------
TOTAL                                   611     54    91%

right now you can browse the production site at 184.73.235.244 but this should re-route to bamboo.io or whatever we decide to use

• add tests
• add fixtures from public data

• add new data to an existing dataset
• merge new data into an existing dataset

Depends on #23

• adding a calculation that refers to columns which do not exist fails

• figure out datastructure on bamboo for storing the schema
  • does it just define datatypes?
  • how to add type information to the schema

Right now it just ignores your group by request and gives you the all summary stats back. Do we want to let the end user know? But, maybe this doesn't matter since we'll support numeric group by in the future.

see, https://github.com/modilabs/bamboo/blob/master/bamboo/tests/core/test_parser.py#L35, we will add more examples to this

• when existing datasets are updated with a new column for calculations don't overwrite data added in the interim

• use bulk inserts chunked by the maximum number of rows on insert

right now nginx is set to run it under bamboo.modilabs.org, we should update to bamboo.io

< > >= <=

(put in 0.1 for the time being. based on thinking about fulfilling requests from nigeria.)

we should have something other than "Ohai World!"... preferably a link to the docs/src/etc. and possibly a description or api reference

In summary statistics for numeric types:

• calculate outliers
• remove outliers from summary statistics
• return outlier rows to user

• should we cache calulations?
  • tied to hash
• do we also prime them?
  • create on upload of data, asynchronously
  • otherwise you'll get a delay on first pull

These are aggregations which need to be calculate for and may be different for each row.

Example calculations:

• percentile
• standard_deviation
• variation
• pearson_correlation

for reference: #16