Comments (6)
@nick-hahner Yes! I used the ftp://ftp.ncbi.nlm.nih.gov/pub/pmc/PMC-ids.csv.gz file for my local conversion.
from pubmed_parser.
I uploaded PMID-PMC pairs (size of 91 MB, not bad not bad) where we can download as follow:
wget https://s3-us-west-2.amazonaws.com/science-of-science-bucket/nih/pmid_pmc_pair.csv
For given file, you can convert PMID to PMC on your own. From here, we can modify parse_citation_web
function to receive just PMC be as below.
def parse_citation_web(pmc):
"""
Parse citations from given PMC
Parameters
----------
pmc: str, PMC of the document e.g. 'PMC1217341'
Returns
-------
dict_out: dict, contains following keys
pmc: Pubmed Central ID
n_citations: number of citations for given articles
pmc_cited: list of PMCs that cite the given PMC
"""
link = "http://www.ncbi.nlm.nih.gov/pmc/articles/%s/citedby/" % str(pmc)
page = requests.get(link)
tree = html.fromstring(page.content)
n_citations = extract_citations(tree)
n_pages = int(n_citations/30) + 1
pmc_cited_all = list() # all PMC cited
citations = tree.xpath('//div[@class="rprt"]/div[@class="title"]/a/@href')[1::]
pmc_cited = list(map(extract_pmc, citations))
pmc_cited_all.extend(pmc_cited)
if n_pages >= 2:
for i in range(2, n_pages+1):
link = "http://www.ncbi.nlm.nih.gov/pmc/articles/%s/citedby/?page=%s" % (pmc, str(i))
page = requests.get(link)
tree = html.fromstring(page.content)
citations = tree.xpath('//div[@class="rprt"]/div[@class="title"]/a/@href')[1::]
pmc_cited = list(map(extract_pmc, citations))
pmc_cited_all.extend(pmc_cited)
pmc_cited_all = [p for p in pmc_cited_all if p is not pmc]
dict_out = {'n_citations': n_citations,
'pmc': pmc,
'pmc_cited': pmc_cited_all}
return dict_out
from pubmed_parser.
Also, we also want to add Copyright Notice for scraping function so that users don't scrape too much and get blocked https://www.ncbi.nlm.nih.gov/pmc/about/copyright/#copy-PMC
from pubmed_parser.
What about ftp://ftp.ncbi.nlm.nih.gov/pub/pmc/oa_file_list.csv ?
Not every PMCID has a corresponding PMID though according to this list.
from pubmed_parser.
@nick-hahner, nice! It contains ~ 1.8M rows of PMID/ PMC pairs of Open Access Subset. I'm still thinking about how to update the list regularly by not hurting the repository. I mean, I could upload PMC-PMID pairs from MEDLINE somewhere as I mentioned. Do you have any preference or suggestions on how to make it available on the repository?
from pubmed_parser.
Actually this file is probably better with 4,892,265 rows:
ftp://ftp.ncbi.nlm.nih.gov/pub/pmc/PMC-ids.csv.gz
First rsync
or wget -c -N ...
the file to some directory like ~/.pp_data
Then you can use an sqlite3 db
# Create an indexed sqlite db
import pandas as pd
from sqlalchemy import create_engine
engine = create_engine('sqlite:///pmid_to_pmcid.db') # but some better location
df = pd.read_csv('~/.pp_data/PMC-ids.csv.gz', dtype=str)
df[['PMCID', 'PMID']].to_sql('pmc_pmid', engine, index=False, if_exists='replace')
engine.execute('create index pmc_idx on pmc_pmid(PMCID)')
engine.execute('create index pmid_idx on pmc_pmid(PMID)')
# then later you can fetch like so:
from sqlalchemy import create_engine, text as sqa_text
def get_pmcid_from_pmid(pmid):
engine = create_engine('sqlite:///pmid_to_pmcid.db')
ret = engine.execute(sqa_text('select pmcid from pmc_pmid where pmid = :pmid;'), pmcid=pmcid).fetchone()
return ret[0] if ret else None
How's that sound?
from pubmed_parser.
Related Issues (20)
- AttributeError: 'NoneType' object has no attribute 'find' HOT 2
- Table parsing issues with parse_pubmed_table HOT 2
- parse_pubmed_table() and parse_pubmed_references() returning None HOT 1
- Question for extracting text HOT 1
- Question: Abstract with Mesh Tag HOT 3
- question using pp.parese_medline HOT 2
- Question: parsing error first line expecting '<' not found
- parse_pubmed_caption() failing on some papers
- ValueError when attempting to parse OA XML HOT 3
- Bug report - XML citations from website HOT 1
- CALL FOR MAINTAINER/CONTRIBUTOR FOR PUBMED_PARSER HOT 7
- Parse language in `parse_xml_web()` HOT 1
- Update tests: Use current data
- Fix Medline parser tests
- Parse VersionID and VersionDate
- Inconsistent PMID and DOI when using parse_xml_web for an XML file. HOT 1
- Handling of non-ASCII / unicode characters
- Fix GitHub workflows HOT 3
- Missing paragraphs from parse_pubmed_paragraph output
- It couldn't recognize the xml file I downloaded from pubmed
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from pubmed_parser.