Do we want to add a standard citation for people who use this code as part of a significant contribution to a paper?

Codec can't decode byte 0xa0 error. Probably a unicode python 2 v. python 3 kinda thing.

Try a simple bag-of-words model for article tagging.

Could we easily and automatically generate documentation using something like sphinx?

Are there easy-to-use, pre-trained, cross-platform word vectorizers? Preferably with a python interface?

I've been struggling with how to do this because:

• I don't want to put trained models in version control
• In order to train a model you need the data
• We can't pull the 3GB of data down to the CI server each time we want to run the unit tests
• W/o a trained model there's not much to test

I think instead it may make sense to put a very small subset of data in version control, both of article data and word vectors. Just enough to train any model. The tests will then not be testing accuracy or anything, but we could test everything is hooked up correctly.

Things like removing the string "block of ", or appending "Chicago Illinois" to the geostring if it doesn't already have it.

This will be somewhat geotagging provider dependent, but there's probably a lot of overlap in what transformations are beneficial for geotagging.

Named Entity Recognition describes a superset of what we are trying to do w/ the location extraction. It is also a well studied field, so there are probably open corpora that have labeled data we could use to augment (or use exclusively as) our training data.

We should do some investigating to try and find such a corpus.

This caused an error in the batch processing.

KeyError: "None of [['[Take', 'Our', 'Poll][1]', '[1]:', 'http://polldaddy.com/poll/6009092']] are in the [index]"

This will facilitate an extremely useful sanity check on the data.

Preferably in Python, what seems to be a good geocoder library?

Perform word-vectorization as a pre-processing step.

• Simplify setup instructions, especially for code dependencies
• How to get the database exports
• Brief summary of directory structure
• Breakdown the crime tags model
  • Where it is and what it is right now
  • Ways in which it could be improved
  • How to measure performance
• Breakdown the geostring model
  • Where it is and what it is right now
  • Ways in which it could be improved
  • How to measure performance (https://github.com/chicago-justice-project/geo-extract-tester)
• Geocoding
  • What to do if it breaks (do we want a section in the README too?)
  • Ways in which it could be improved (confidence score)
• Tests
  • Travis CI
  • Running locally
  • Writing tests
• Documentation
  • How to write documentation
  • How to publish it?
• How to publish a new version
  • Add section on how to get the models in there!
• Other, non-coding ways to help
• Remove/consolidate GEOTAGGING_CONTRIB

After cloning article-tagging to the local, if the python setup.py install isn't run first, the df = ld.load_data() code line in the bag-of-words notebook will error that it can't find the news article being called.

Seems to work once the setup is run.

While trying to create the model locally in bag-of-words-count-stemmed-binary.ipynb at this code block:

fpr = pd.DataFrame(bench_results['fpr'], columns=crime_df.columns.values[9:]).T

tpr = pd.DataFrame(bench_results['tpr'], columns=crime_df.columns.values[9:]).T

ppv = pd.DataFrame(bench_results['ppv'], columns=crime_df.columns.values[9:]).T

I get the following error:

ValueError Traceback (most recent call last)
~/anaconda/envs/cjp-ap/lib/python3.6/site-packages/pandas/core/internals.py in create_block_manager_from_blocks(blocks, axes)
4293 blocks = [make_block(values=blocks[0],
-> 4294 placement=slice(0, len(axes[0])))]
4295

~/anaconda/envs/cjp-ap/lib/python3.6/site-packages/pandas/core/internals.py in make_block(values, placement, klass, ndim, dtype, fastpath)
2718
-> 2719 return klass(values, ndim=ndim, fastpath=fastpath, placement=placement)
2720

~/anaconda/envs/cjp-ap/lib/python3.6/site-packages/pandas/core/internals.py in init(self, values, placement, ndim, fastpath)
114 'implies %d' % (len(self.values),
--> 115 len(self.mgr_locs)))
116

ValueError: Wrong number of items passed 38, placement implies 41

During handling of the above exception, another exception occurred:

ValueError Traceback (most recent call last)
in ()
----> 1 fpr = pd.DataFrame(bench_results['fpr'], columns=crime_df.columns.values[9:]).T
2
3 tpr = pd.DataFrame(bench_results['tpr'], columns=crime_df.columns.values[9:]).T
4
5 ppv = pd.DataFrame(bench_results['ppv'], columns=crime_df.columns.values[9:]).T

~/anaconda/envs/cjp-ap/lib/python3.6/site-packages/pandas/core/frame.py in init(self, data, index, columns, dtype, copy)
304 else:
305 mgr = self._init_ndarray(data, index, columns, dtype=dtype,
--> 306 copy=copy)
307 elif isinstance(data, (list, types.GeneratorType)):
308 if isinstance(data, types.GeneratorType):

~/anaconda/envs/cjp-ap/lib/python3.6/site-packages/pandas/core/frame.py in _init_ndarray(self, values, index, columns, dtype, copy)
481 values = maybe_infer_to_datetimelike(values)
482
--> 483 return create_block_manager_from_blocks([values], [columns, index])
484
485 @Property

~/anaconda/envs/cjp-ap/lib/python3.6/site-packages/pandas/core/internals.py in create_block_manager_from_blocks(blocks, axes)
4301 blocks = [getattr(b, 'values', b) for b in blocks]
4302 tot_items = sum(b.shape[0] for b in blocks)
-> 4303 construction_error(tot_items, blocks[0].shape[1:], axes, e)
4304
4305

~/anaconda/envs/cjp-ap/lib/python3.6/site-packages/pandas/core/internals.py in construction_error(tot_items, block_shape, axes, e)
4278 raise ValueError("Empty data passed with indices specified.")
4279 raise ValueError("Shape of passed values is {0}, indices imply {1}".format(
-> 4280 passed, implied))
4281
4282

ValueError: Shape of passed values is (38, 4), indices imply (41, 4)

I did manage to run the model successfully before using the new database files. However, even when I try running the previous database subsequently, I get the exact same problem.

Any ideas as to how to resolve this?

Before submitting this issue, I have since deleted and re-cloned this repo, then setup, and re-run this notebook three times. I'm working on mac OSX. And finally, when I run the tagnews.test() from the command line that fails - if that helps.

Might be useful when trying to convert the highlighted text to the text as it appears in the bodytext column of the data dump.

Investigate Named Entity Recognition and the related field of Conditional Random Fields for the purpose of location extraction from articles.

Benchmarker should...

• accept pre-processed NxM matrix of numeric data
• a classifier that has a .fit and .predict_proba method
• perform k-fold cross validation (with a fixed random seed!), making sure the classifier is reset after each run

Open question: How to handle other languages?

The constant struggle to cover every possible path through your code with every possible state.

Try a n-gram model.

Perhaps a better description would blame windows for refusing to follow conventions, but it appears as if "ctrl-d" does not send an EOF signal in the command prompt as it (kind of) does in bash. This means it is impossible to gracefully exit from the stdin-based CLI:

>python -m tagnews.crimetype.cli
Go ahead and start typing. Hit ctrl-d when done.
This is a test article, attempting to get reading in from the stdin working.
^D^D^D

It would be great to either figure out how to send EOF from command prompt and document that (preferred), or modify the CLI to correctly handle the situation.

• installation
• tests
• usage

Try an RNN-based model.

• Script to train and save a model.
  • Documentation on how to run it.
• Simple API to extract geo-strings from article text.
  • Documentation on how to run it.
• Publish to pypi
  • Documentation on main README of how to run it.

Enumerate and test different pre-processing steps.

Would the project find any value in capturing incident times associated with location information in news articles? I'm working on address extraction and it would be a small effort to annotate related time data in the test corpus.

There's too many notebooks! Let's get rid of some of them.

The new data model contains extra information on the human-selected location information. This has caused a couple changes in the data format that will need to be resolved.

chicago-justice-project/chicago-justice#37
chicago-justice-project/chicago-justice@39b7eee

Right now the geocoding example notebook uses the NER dataset from kaggle. Current best results are achieved using only our own training data downloadable from https://geo-extract-tester.herokuapp.com. To make things as easy as possible, the example notebook should be updated to download our training.txt file and not kaggle's.

Notebook in question: https://github.com/chicago-justice-project/article-tagging/blob/master/lib/notebooks/extract-geostring-example.ipynb

Does it make sense to publish documentation to a website?

Ref #112

Currently, when no arguments are passed to the CLI then it reads from stdin. A message is printed out to guide the user. However, this is just annoying when using it by piping something else to it, e.g.

$ cat article.txt | python -m tagnews.crimetype.cli

I think it would be better to

• update CLI so that there's a flag for read from stdin, in which case no guidance is needed because the user already chose to use a flag, so they must know what they're doing
• when no arguments are passed print help and quit.

• Remove/update out-of-date info
• Simplify README.md to make landing page more manageable
• Clearly state the goals of the project
• Make explicit the limitations of the project/using machine learning for something like this
• Clarify the connections to the other CJP projects
• Investigate other projects' READMEs, are there other sections we're missing?

Any realistic machine learning project will not have 100% accuracy. We should report the expected performance levels as found on held-out data.

This will include updated dependency installation instructions (tf + keras at a minimum) and possibly updating data download/placement instructions.

Related to #72.

Function signature should approximate

>>> lat_longs = lat_longs_from_geo_strings(['1700 block of south halsted', 'merchandise mart'])
>>> print(lat_longs)
[[12.345, 67.890], [98.76, 54.321]]

Some installation instructions for whatever API is being used will also be needed.

keras seems to rely on h5py but doesn't install it as a required package when installing tagnews

So that a model can be trained on a subset of the full dataset. Useful for quickly iterating or for people on more resource-constrained systems.

https://github.com/chicago-justice-project/article-tagging/blob/master/lib/tagnews/crimetype/models/binary_stemmed_logistic/save_model.py

Probably as a section in CONTRIBUTING.md.

• Versions
• Where to extract it
• How to load it
• Memory usage and ways to mitigate it
• Column descriptions (including the new locations column, and the one-hot-encoded type-of-crime tags)

(These can be chunked off and handled one-by-one depending on contributor's comfortability with each bullet point.)