palewire / django-calaccess-raw-data Goto Github PK

These already exist at the tail end of downloadcalaccess management command but could be prettified or take the form of a separate management command or a dashboard webpage.

rserver is a management command that runs python manage.py runserver and grunt at the same time so you can watch for changes and run the python server simultaneously.

See: https://github.com/cirlabs/django-project-template/blob/master/project_name/apps/core/management/commands/rserver.py

Need to move the campaign_finance.load stuff into a management command. I just did it that way while I figured it all out. The management command stuff is still pretty new to my feeble mind.

Probably need a load from scratch command as well as an update with the new stuff only command.

Problem

python manage.py downloadcalaccess throws a UnicodeEncodeError on #L124.

Reproduce the problem

1. Create a new virtualenv, install django and create a new django project
2. git clone this repo, cd into it and checkout the rewrite branch
3. build the django-calaccess-parser package with python setup.py sdist
4. Install the parser into your new virtualenv with pip install /path/to/django-calaccess-parser/dist/django-calaccess-parser-0.1.tar.gz
5. Add calaccess to your INSTALLED_APPS and python manage.py syncdb
6. Create a MySQL database and user and give them privileges
7. Set a CALACCESS_DOWNLOAD_DIR path in settings.py

Breathe.

1. Run python manage.py downloadcalaccess to see the error

Possible solutions

I don't think I adjusted any code when I started porting things over but we should look into that. Also, I don't think how well this command works now that it's been moved to package install of an app explicitly inside the package.

Stacktrace

Traceback (most recent call last):
  File "manage.py", line 10, in <module>
    execute_from_command_line(sys.argv)
  File "/home/aaron/.envs/test/local/lib/python2.7/site-packages/django/core/management/__init__.py", line 399, in execute_from_command_line
    utility.execute()
  File "/home/aaron/.envs/test/local/lib/python2.7/site-packages/django/core/management/__init__.py", line 392, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/home/aaron/.envs/test/local/lib/python2.7/site-packages/django/core/management/base.py", line 242, in run_from_argv
    self.execute(*args, **options.__dict__)
  File "/home/aaron/.envs/test/local/lib/python2.7/site-packages/django/core/management/base.py", line 285, in execute
    output = self.handle(*args, **options)
  File "/home/aaron/.envs/test/local/lib/python2.7/site-packages/calaccess/management/commands/downloadcalaccess.py", line 124, in handle
    confirm = input(self.prompt)
UnicodeEncodeError: 'ascii' codec can't encode character u'\xa0' in position 73: ordinal not in range(128)

Here is a running list of ideas I have for improving this repository at the upcoming code convening.

• The load method should have a more explict connection between CSV files and tables rather than making namespace-based connections. And it should throw errors if things don't link up.
• The clean method seems to be too slow. How can we make it go faster? Figuring out #30 would help us here.
• The database models need docstrings, verbose names or something to spell out what they are more clearly to newbies. They also need __unicode__ methods and other prettifying thing like ordering defaults.
• There are a number of source data files that do not yet have a corresponding Django model. We need to figure out which ones we want and get them into the system.
• LOAD DATA INFILE is great, but it only works for MySQL. Is there a way we could write the code so that it could work for other database backends supported by Django, particularly PostgreSQL?
• Row count checks now run at the tail end of the downloadcalaccess management command. It would be great if they were prettified, expanded and made available on demand -- maybe in a separate management command or on a dedicated dashboard webpage.
• We need unittests -- but we can't ask our testing suite to download and install this gigantic data dump. Any ideas about how to work that out?

Many of them are now constricted to DateField attributes

• CALACCESS_DOWNLOAD_DIR
• SITE_ROOT
• PROJECT_ROOT
• SETTINGS_ROOT
• location of CSVs and TSVs
• MySQL DATABASE tuple config

We can use python's getattr() function for this,

some_setting = getattr(settings, 'SOME_SETTING', DEFAULT_VALUE)

Why didn't this happen before. Does anybody else get it? cc @aboutaaron @mikejcorey

python example/manage.py loadcalaccessfile AcronymsCd
- Loading AcronymsCd
Traceback (most recent call last):
  File "example/manage.py", line 9, in <module>
    execute_from_command_line(sys.argv)
  File "/home/ben/Code/ccdc/django-calaccess-parser/local/lib/python2.7/site-packages/django/core/management/__init__.py", line 399, in execute_from_command_line
    utility.execute()
  File "/home/ben/Code/ccdc/django-calaccess-parser/local/lib/python2.7/site-packages/django/core/management/__init__.py", line 392, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/home/ben/Code/ccdc/django-calaccess-parser/local/lib/python2.7/site-packages/django/core/management/base.py", line 242, in run_from_argv
    self.execute(*args, **options.__dict__)
  File "/home/ben/Code/ccdc/django-calaccess-parser/local/lib/python2.7/site-packages/django/core/management/base.py", line 285, in execute
    output = self.handle(*args, **options)
  File "/home/ben/Code/ccdc/django-calaccess-parser/local/lib/python2.7/site-packages/django/core/management/base.py", line 385, in handle
    label_output = self.handle_label(label, **options)
  File "calaccess_raw/management/commands/loadcalaccessfile.py", line 16, in handle_label
    self.load(label)
  File "calaccess_raw/management/commands/loadcalaccessfile.py", line 68, in load
    cnt = c.execute(bulk_sql_load)
  File "/home/ben/Code/ccdc/django-calaccess-parser/local/lib/python2.7/site-packages/django/db/backends/util.py", line 69, in execute
    return super(CursorDebugWrapper, self).execute(sql, params)
  File "/home/ben/Code/ccdc/django-calaccess-parser/local/lib/python2.7/site-packages/django/db/backends/util.py", line 51, in execute
    return self.cursor.execute(sql)
  File "/home/ben/Code/ccdc/django-calaccess-parser/local/lib/python2.7/site-packages/django/db/backends/mysql/base.py", line 124, in execute
    return self.cursor.execute(query, args)
  File "/home/ben/Code/ccdc/django-calaccess-parser/local/lib/python2.7/site-packages/MySQLdb/cursors.py", line 207, in execute
    if not self._defer_warnings: self._warning_check()
  File "/home/ben/Code/ccdc/django-calaccess-parser/local/lib/python2.7/site-packages/MySQLdb/cursors.py", line 117, in _warning_check
    warn(w[-1], self.Warning, 3)
_mysql_exceptions.Warning: Field 'ACRONYM' doesn't have a default value
make: *** [loadtable] Error 1

Unfortunately the zip file got blown up so I don't have a copy. Will download and check if this is just a bad copy of the zip file.

  File "/home/michaelcorey/.Envs/calaccess/local/lib/python2.7/site-packages/django/core/management/base.py", line 285, in execute
    output = self.handle(*args, **options)
  File "/home/michaelcorey/.Envs/calaccess/local/lib/python2.7/site-packages/django/core/management/base.py", line 385, in handle
    label_output = self.handle_label(label, **options)
  File "/home/michaelcorey/.Envs/calaccess/local/lib/python2.7/site-packages/calaccess_raw/management/commands/cleancalaccessfile.py", line 20, in handle_label
    self.clean(label)
  File "/home/michaelcorey/.Envs/calaccess/local/lib/python2.7/site-packages/calaccess_raw/management/commands/cleancalaccessfile.py", line 40, in clean
    csv_file = open(csv_path, 'wb')
IOError: [Errno 2] No such file or directory: '/bulk/calaccess/csv/cvr_campaign_disclosure_cd.csv'

For some reason it bombs out at this line claiming the directory already exists and that ruins everything.

This might be a good excuse to develop a Python wrapper that is able to access MySQL's LOAD DATA INFILE as well as PostgreSQL's COPY.

routes:
No Data
/committee/calderon-legal-defense-fund-ron/1131/

Bad/missing data
committee/perez-for-assembly-2010-john-a/2848/

Errors:
For no data
In Cal Heatmap: Uncaught TypeError: Cannot call method 'getFullYear' of undefined

It might make sense to break these into separate commands and then have one mega command that runs them all in order when needed.

This could help us identify the bottlenecks that need refactoring to increase speed.

Add DEBUG to context_processor to enable a DEBUG template tag.

See: https://github.com/cirlabs/django-project-template/blob/master/project_name/apps/core/context_processors.py

So, I'm working to get django-calaccess-campaign-finance ready, but it looks like setup_tools can't find the parser on pypi.

Reproduce

$ git clone https://github.com/california-civic-data-coalition/django-calaccess-campaign-finance.git
$ cd django-calaccess-campaign-finance
$ python setup.py install

Output:

Installed /home/aaron/.envs/calaccess/lib/python2.7/site-packages/django_calaccess_campaign_finance-0.2-py2.7.egg
Processing dependencies for django-calaccess-campaign-finance==0.2
Searching for django-calaccess-parser>=0.2
Reading http://pypi.python.org/simple/django-calaccess-parser/
No local packages or download links found for django-calaccess-parser>=0.2
error: Could not find suitable distribution for Requirement.parse('django-calaccess-parser>=0.2')

Possible reason

It looks like setup_tools looks for the repo at http://pypi.python.org/simple/django-calaccess-parser/ rather than the actual Pypi index.

I'll drop the dependency for now just to check out some other stuff, but thought I'd let you know before we get started Wednesday

Today, @mikejcorey and I tried running downloadcalaccess but ran into an invalid zip error. I then tried download and open up the file from the browser and ran into the same error. Seems to me that sometimes CAL ACCESS may register a bad dump.

What should this app do in these instances?

http://www.sos.ca.gov/prd/cal-access/

In an effort to keep things simple, we're gonna move the campaign finance app and ui into a separate repo. This way, we can focus on making this parser simple and let individual newsrooms decide on the ui and app features they want.

We could consider sticking them in a table so they can be eyeballed

Could potentially log the counts in a dedicated database table