titipata / grant_database Goto Github PK

I want to create a bash script that download all grant zip files > unzip them > parse them (will add script for doing that later on). Right now, what I have is as following (see this post on Stack Overflow):

python download_award_links.py # load to nsf
while read p; do
  echo $p
done <nsf_awards.txt

# unzip all files and parse later

Would be great if we can have full workflow for this.

Cannot find dedupe_nsf.py anywhere, could you please check it?

We lost year in NSF grant data between 2000-2010. It might be a problem from parser.

just a thought to add this database to download also http://www.cihr-irsc.gc.ca/e/46295.html

We will use pubmed_parser in order to grab all PIs name from MEDLINE. This will be in format: KP Kording. Right now, in NIH, we have first_name, last_name columns. We will transform author string into same format as in MEDLINE. String matching would be nice for the first attempt. We will check affiliation later to make sure they are same person.

File from MEDLINE is located in S3, downloading by using,

aws s3 sync s3://science-of-science-bucket/medline/pmid_author_affil.csv/ pmid_author_affil/

We will modify bash or python later on if data is already existed, we will find new links > copy new zip file and parse that instead.

I use the script to scrape this page. However, it doesn't work right now.

import lxml
from lxml import html
import requests
url = 'http://exporter.nih.gov/ExPORTER_Catalog.aspx?sid=0&index=1'
page = requests.get(url)
tree = html.fromstring(page.content)
awards_links = tree.xpath('//tbody//tr[@class="row_bg"]//td//div//a/@href')

This aim to include deduplicated authors and affiliations across datasets also. Put some instruction on how to deduplicate the dataset. Related to issue #9 and #10

Need to launch computer for doing processing

@daniel-acuna put database on Amazon S3 and can be downloaded to dedupe_output folder as follows:

aws s3 cp s3://grant-dataset/dedupe/ dedupe_output/  --recursive

We want to create a script that merge these dataset all together.

@daniel-acuna, can you check dedupe_affiliation.py in deduplication part. I got error when running the script as follows:

Traceback (most recent call last):
  File "dedupe_affiliation.py", line 94, in <module>
    deduper.train(ppc=None, recall=0.95)
  File "/Users/titipat/anaconda3/lib/python3.5/site-packages/dedupe/api.py", line 659, in train
    index_predicates)
  File "/Users/titipat/anaconda3/lib/python3.5/site-packages/dedupe/api.py", line 679, in _trainBlocker
    recall)
  File "/Users/titipat/anaconda3/lib/python3.5/site-packages/dedupe/training.py", line 102, in learn
    raise ValueError(NO_PREDICATES_ERROR)
ValueError: No predicate found! We could not learn a single good predicate. Maybe give Dedupe more training data or increasing the `max_comparisons` argument to the train method

use dedupe to deduplicate affiliations within NIH and NSF data

see https://grid.ac/ for grid dataset.