Giter VIP home page Giter VIP logo

openstates's Introduction

The Open State Project collects and makes available data about state legislative activities, including bill summaries, votes, sponsorships and state legislator information. This data is gathered directly from the states and made available in a common format for interested developers, through a JSON API and data dumps.

Installation

see http://openstates.org/contributing/

Used Modules ========== Here are some of the important modules that play a major role in the project.

* Openstates, I have my own fork here https://github.com/h4ck3rm1k3/openstates which contains experimental code. Do not use it in production.

* billy The billy system is the main driver for openstates, openstates is driven by billy. main project :https://github.com/sunlightlabs/billy my fork : https://github.com/h4ck3rm1k3/billy

* scrapelib A thin layer of icing over the requests lib main project :https://github.com/sunlightlabs/scrapelib my fork : https://github.com/h4ck3rm1k3/scrapelib

* requests The user friendly agent lib main project :https://github.com/kennethreitz/requests my fork : https://github.com/h4ck3rm1k3/requests

  • urllib3

The underlying lib for managing http official : https://github.com/shazow/urllib3 my fork : https://github.com/h4ck3rm1k3/urllib3

* pymongo The python mongo lib, official : https://github.com/mongodb/mongo-python-driver my fork : https://github.com/h4ck3rm1k3/mongo-python-driver

The bson lib is in there and used to store the binary json data.

* Api Server : This server is my experimental server based on the kansas server. https://github.com/h4ck3rm1k3/openstates-api-dancer

Scraping single bills

My plan for scraping single bills : extract the list of data from the api and filter it.

Searching for bills

My plan for searching, be able to filter the API while downloading, or download only matching data. Related to single bills.

Debian Stable (python 2.6) ============= sudo apt-get install abiword

sudo pip-2.6 install -r requirements.txt --use-mirrors sudo pip-2.6 install -r requirements-site.txt --use-mirrors

Schema

The json from the states must match the schema files located in openstates/billy/schemas/, each scraper will provide a ._get_schema() method that loads the schema and passes it to the validator. see validator.validator.SchemaValidator.

Links

openstates's People

Contributors

bhitov avatar binaryechoes avatar brandonlewis avatar browning avatar coopernurse avatar cweber avatar dschep avatar dsummersl avatar empact avatar gabriel4649 avatar h4ck3rm1k3 avatar ianwhalen avatar jamesturk avatar jballanc avatar joegermuska avatar konklone avatar markolson avatar mattgrayson avatar mikejs avatar onyxfish avatar paultag avatar rhouse2 avatar rmanocha avatar rmd6502 avatar rshapiro avatar sarindipity avatar schneidy avatar tamilyn avatar twneale avatar zestyping avatar

Stargazers

avatar

Watchers

avatar avatar

Forkers

jmdupont

openstates's Issues

cow_jcow_317 Committee of the Whole - Motion to Amend

cow_jcow_317 Committee of the Whole - Handwritten Motion to Amend - Offered by Representative Burgess.
cow_jcow_317 Committee of the Whole - Handwritten Motion to Amend - Offered by Representative Carlin
cow_jcow_317 Committee of the Whole - Handwritten Motion to Amend - Offered by Representative Johnson.
cow_jcow_317 Committee of the Whole - Handwritten Motion to Amend - Offered by Representative Worley.
cow_jcow_317 Committee of the Whole - Motion to Amend - Offered by Representative O'Hara

http://www.kslegislature.org/li/b2011_12/measures/sb1/

more warnings

22:21:02 billy WARNING ks skipping blank committee DUI Commission
22:21:02 billy WARNING ks skipping blank committee Special Committee on Assessment & Taxation
22:21:02 billy WARNING ks skipping blank committee Special Committee on Education
22:21:02 billy WARNING ks skipping blank committee Special Committee on Eminent Domain in Condemnation of Water Rights
22:21:02 billy WARNING ks skipping blank committee Special Committee on Kansas Technology Enterprise Corporation
22:21:04 billy WARNING ks error fetching committee http://www.kslegislature.org/li/api/v5/rev-1/ctte/ctte_spc_special_committee_on_kan_ed_study_commission_1/
22:21:04 billy WARNING ks skipping blank committee Special Committee on Utilities
22:21:04 billy WARNING ks skipping blank committee Special Committee on Transportation
22:21:04 billy WARNING ks skipping blank committee Special Committee on New Comprehensive Transportation Plan
22:21:04 billy WARNING ks skipping blank committee Special Committee on Natural Gas Storage Fields and Facilities
22:21:04 billy WARNING ks skipping blank committee Special Committee on Legislative Streamlining
22:21:04 billy WARNING ks skipping blank committee Special Committee on Judiciary
22:21:04 billy WARNING ks skipping blank committee Special Committee on Gaming Oversight
22:21:04 billy WARNING ks skipping blank committee Special Committee on Federal & State Affairs
22:21:04 billy WARNING ks skipping blank committee Special Committee on Energy, Natural Resources & Environment
22:21:05 billy WARNING ks error fetching committee http://www.kslegislature.org/li/api/v5/rev-1/ctte/ctte_2011_redistricting_advisory_group_1/
22:21:05 billy WARNING ks skipping blank committee Kansas Criminal Code Recodification Commission
22:21:05 billy WARNING ks skipping blank committee Kansas Criminal Code Recodification Commission-Subcommittee
22:21:05 billy WARNING ks skipping blank committee Kansas Rural Policy
22:21:05 billy WARNING ks skipping blank committee 2010 Commission
22:21:05 billy WARNING ks skipping blank committee Capitol Restoration Commission
22:21:05 billy WARNING ks skipping blank committee Special Education Funding Task Force
22:21:05 billy WARNING ks skipping blank committee Health Care Stabilization Fund
22:21:06 billy WARNING ks error fetching committee http://www.kslegislature.org/li/api/v5/rev-1/ctte/ctte_s_phw_1/
22:21:07 billy WARNING ks error fetching committee http://www.kslegislature.org/li/api/v5/rev-1/ctte/ctte_h_corr_juv_jus_1/
22:23:16 billy WARNING ks unable to fetch HTML for bill SR1885
23:10:30 billy WARNING ks unable to fetch HTML for bill SR1884
23:10:31 billy WARNING ks unable to fetch HTML for bill SR1871
23:23:47 billy WARNING ks unable to fetch HTML for bill SR1883
23:23:50 billy WARNING ks unable to fetch HTML for bill SR1882
23:23:53 billy WARNING ks unable to fetch HTML for bill SR1879
23:23:54 billy WARNING ks unable to fetch HTML for bill SR1881
23:23:57 billy WARNING ks unable to fetch HTML for bill SR1880
23:23:58 billy WARNING ks unable to fetch HTML for bill SR1878
23:26:47 billy WARNING ks unable to fetch HTML for bill SR1876
23:26:50 billy WARNING ks unable to fetch HTML for bill SR1877
23:26:50 billy WARNING ks unable to fetch HTML for bill SR1874
23:26:51 billy WARNING ks unable to fetch HTML for bill SR1873
23:26:52 billy WARNING ks unable to fetch HTML for bill SR1872
23:26:52 billy WARNING ks unable to fetch HTML for bill SR1875
23:29:20 billy WARNING ks unable to fetch HTML for bill SR1870
02:37:47 billy WARNING ks unable to fetch HTML for bill HB2210
02:46:33 billy WARNING ks unable to fetch HTML for bill HR6034

unknown code misc_bs_100

18:32:22 billy WARNING ks unknown action code on HB2606: misc_bs_100 Senator Vratil motioned to reconsider action recommending bill be passed as amended. Motion adopted.

misc_bs_100 * random notices

These are random unsorted type
misc_bs_100 Call of the Senate.
misc_bs_100 Call of the Senate lifted.
misc_bs_100 Motion to adopt Conference Committee Report
misc_bs_100 Motion to suspend Joint Rule 4 (k) to allow consideration adopted;
misc_bs_100 Ruled dead, Joint Rule 4(f) May 19, 2012
misc_bs_100 Ruled materially changed and re-referred to Committee on Education.
misc_bs_100 Senator Bruce motioned to concur
misc_bs_100 Senator Taddiken motioned to adopt the conference committee report
misc_bs_100 Stricken from the Calendar (House Rule 2307)
misc_bs_100 Substitute motion to not adopt Conference Committee Report Yea
misc_bs_100 Substitute motion to not adopt the Conference Committee Report and appoint a new Conference Committee.

billy-scrape --import --fast ks failing 

mdupont@localhost:~/experiments/sunlight/openstates$ billy-scrape   --import --fast ks 2>&1| tee ks2.log 
22:00:54 billy INFO ks billy-update abbr=ks
    actions=import
    types=bills,legislators,votes,committees,alldata
    sessions=2011-2012
    terms=2011-2012
22:00:55 billy INFO ks imported 162 legislator files
22:00:55 billy INFO ks imported 0 vote files
Traceback (most recent call last):
  File "/usr/local/bin/billy-scrape", line 9, in <module>
    load_entry_point('billy==1.3.0', 'console_scripts', 'billy-scrape')()
  File "/usr/local/lib/python2.6/dist-packages/billy-1.3.0-py2.6.egg/billy/bin/update.py", line 414, in scrape_compat_main
    main(True)
  File "/usr/local/lib/python2.6/dist-packages/billy-1.3.0-py2.6.egg/billy/bin/update.py", line 398, in main
    import_report = _do_imports(abbrev, args)
  File "/usr/local/lib/python2.6/dist-packages/billy-1.3.0-py2.6.egg/billy/bin/update.py", line 155, in _do_imports
    report['bills'] = import_bills(abbrev, settings.BILLY_DATA_DIR)
  File "/usr/local/lib/python2.6/dist-packages/billy-1.3.0-py2.6.egg/billy/importers/bills.py", line 409, in import_bills
    ret = import_bill(data, votes, categorizer)
  File "/usr/local/lib/python2.6/dist-packages/billy-1.3.0-py2.6.egg/billy/importers/bills.py", line 267, in import_bill
    doc_matcher.set_ids(data['versions'] + data['documents'])
  File "/usr/local/lib/python2.6/dist-packages/billy-1.3.0-py2.6.egg/billy/importers/bills.py", line 499, in set_ids
    item[self.id_key] = self.ids.get(key) or self._get_next_id()
  File "/usr/local/lib/python2.6/dist-packages/billy-1.3.0-py2.6.egg/billy/importers/bills.py", line 476, in _get_next_id
    return next_big_id(self.abbr, self.id_letter, self.id_collection)
  File "/usr/local/lib/python2.6/dist-packages/billy-1.3.0-py2.6.egg/billy/importers/utils.py", line 288, in next_big_id
    ('upsert', True)]))['value']['seq']
  File "/usr/local/lib/python2.6/dist-packages/pymongo/database.py", line 355, in command
    msg, allowable_errors)
  File "/usr/local/lib/python2.6/dist-packages/pymongo/helpers.py", line 129, in _check_command_response
    raise OperationFailure(msg % response["errmsg"])
pymongo.errors.OperationFailure: command SON([('findandmodify', 'document_ids'), ('query', SON([('_id', u'ks')])), ('update', SON([('$inc', SON([('seq', 1)]))])), ('new', True), ('upsert', True)]) failed: No matching object found

unknown code ccal_faccl_900

18:28:28 billy WARNING ks unknown action code on HB2460: ccal_faccl_900 Consent Calendar Passed Yea: 40 Nay: 0

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.