betanyc / budgetbuddy Goto Github PK

Taken from: https://github.com/BetaNYC/budgetBuddy/blob/master/apiCalls.txt

Some API Questions:

Our Hierarchy looks like:
year/operatingBudget/agency/unitOfAppropriation/responsibilityCenter/budgetCode/objectclass/object

• Should we include all subordinate data when a given level is queried?
• Should we have a flag for whether the results are roll-up for that level, or include subordinate data?

GET /:year/op - citywide totals with agency totals rolled up.
GET /:year/op/detail - full drill down - this is basically the whole database... no?

GET /:year/op/:agency - just has the totals for the agency and roll ups for the next lower element in the hierarchy
GET /:year/op/:agency/detail - full drill down

GET /:year/op/:agency/:uoa - just this UOA and its children (responsibility centers)
GET /:year/op/:agency/:uoa/detail - full drill down

...and so on...

Year to Year comparisons?

GET /:year/op/:agency/history?since=2012

Rollup Object Classes at the agency level:

GET /:year/op/:agency/?objectClass=01 This will return the sum of all object class 01 values in this agency
//how do we query multiples?

Hi, I've been using the scraper as I downloaded it June 29 to work with the adopted budget files, but I've been running into some problems where in some places the comma is misplaced and it looks like this in some places in the original csv:
This is from the adopted 2014 budget:
841,DEPARTMENT OF TRANSPORTATION,4000,"EXECUTIVE MANAGEMENT, TRAFFIC",Z030,PlaNYC Congestion Mitigation,10 SUPPLYS&MATL,1,00 S,UPPLIES + MATERIALS - GENERAL,ADOPTED BUDGEFY15,,AMOUNT,11881478,ss6_14.txt,128315
I was trying to compare the two years but there were mistakes because it didn't recognize that the value in one corresponded with the other.
Related, in another case it seems to have ignored a corresponding value in the modified budget column in that file, so for example it only listed this :
002,MAYORALTY,,,0244,Central Insurance Program,04 ADD GRS PAY,,042,LONGEVITY DIFFERENTIAL,ADOPTED BUDGET FY15,,AMOUNT,342,ss6_14.txt,295
when that same value also exists in the Modified section but it doesn't register for some reason.
This is only for a class (project due Thursday), so if this can't be fixed easily, I can do a more basic exercise with the existing converted data, but ideally I wanted to work with the adopted budget files and compare. My main objective at the moment is to get the files into CSV.format and then run some basic operations in Python/Pandas. Thanks!

Ideas are in /apiCalls.txt

Heroku app? What database? What backend framework?

Should we allow for CSV and JSON results?

FYI, the City will adopt a new budget by the end of this month (they had a handshake deal last Thursday. Should be adopted this week). So new data will be coming in shortly.

... one that works on "supporting schedules" budget PDFs for current and previous years.

I tried the scraper on converted to text versions of both this year's and last year's adopted budgets , and in each case got the error:Traceback (most recent call last):
File "parse.py", line 228, in scrape
'file_name': f.name
File "parse.py", line 188, in parse
data = self.line2dict(line)
File "parse.py", line 136, in line2dict
value = self.process_value(value)
File "parse.py", line 37, in process_value
return Decimal(v.replace(',', ''))
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/decimal.py", line 548, in new
"Invalid literal for Decimal: %r" % value)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/decimal.py", line 3872, in _raise_error
raise error(explanation)
InvalidOperation: Invalid literal for Decimal: '-'

Traceback (most recent call last):
File "parse.py", line 236, in
scrape(open(sys.argv[1], 'r'))
File "parse.py", line 232, in scrape
sys.stderr.write(e + u'\n')
TypeError: unsupported operand type(s) for +: 'InvalidOperation' and 'unicode'

Any thoughts?
-- Miranda N.

The response to an API call on the console is a great opportunity to explain the fields in the budget dataset. For example, what does value mean in this context?

We can use the Swagger API to give a more detailed explanation of these fields.

Volkan has suggested Swagger for API docs and console.

From an email thread:

The way I look at it, it's going to end up in PostgreSQL anyway when we deploy to Heroku. (We can swap it out later.) So we might as well keep the sqlite in the repo and consider that our development server. That would also make it easy for newcomers to get it up and running on their own.
(Volkan)

I meant to ask about that (we should have had this thread in an issue). What are the implications of leaving it in sqlite? What are the benefits of moving it to PostGres, especially since it's not that big and won't change at all except for adding more data a year from now.
(Chris)

We should do some work to get it on master without moving over the DB, as I'd prefer not to commit the sqlite binary (instead, we should have something that generates the DB from the pdf/text, most likely storing it in Postgres.)

(John)