Presently the institution and country fields in Paper are represented using dicts. This is difficult to store in a database without a lot of overhead. There must be a simpler way!

Traceback (most recent call last):
File "<pyshell#6>", line 1, in
import tethne.writers as wr
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/tethne/writers/init.py", line 7, in
import tethne.writers.collection
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/tethne/writers/collection.py", line 88
for i in sorted(nodes[n].keys()):

Preferably, would include a subclass of Corpus that interfaces with the SQL database; to allow operations on really big corpora without tying up too much memory.

After opening Python from command line and then while trying to import corpus

Downloaded and unzipped tethne 0.6.2-beta, when running install from command line using pip unable to install

'''
Downloading/unpacking tethne
Could not find a version that satisfies the requirement tethne (from versions: 0.3.0-alpha, 0.3.1-alpha, 0.4.0-alpha, 0.4.2-alpha, 0.6.0-beta, 0.6.1-beta, 0.6.2-beta)
Cleaning up...
No distributions matching the version for tethne
Storing debug log for failure in /Users/Kenneth/.pip/pip.log
'''

Using TethneGUI, downloaded Feb 27 @ 1pm. MacOSX.

----------------------------------------
    Workflow step: Graph
----------------------------------------
Loading DataCollection with slices from /Users/Bianca/Desktop/PMS Tethne/tmp_DST1315262543/DST1315262543_DataCollection_sliced.pickle...done.
Using first slice in DataCollection: date.

Traceback (most recent call last):
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/runpy.py", line 162, in _run_module_as_main
    "__main__", fname, loader, pkg_name)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/runpy.py", line 72, in _run_code
    exec code in run_globals
  File "/Users/Bianca/Downloads/TethneGUI-2/configuration/org.eclipse.osgi/bundles/2/1/.cp/python/tethne/__main__.py", line 403, in <module>
    kwargs['threshold'] = int(options.threshold)
ValueError: invalid literal for int() with base 10: '2.0'

Many users build text collections in Zotero, which can download full-text PDFs from JSTOR, PubMed, and other sources. Zotero uses pdf2text to extract articulated plain text from PDFs. A user should be able to export their Zotero collection (in Zotero RDF, say) and load it directly into a Tethne a Corpus object.

From a .graphml document generated by tethne v0.6.1-beta:

<graphml xmlns="http://graphml.graphdrawing.org/xmlns" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://graphml.graphdrawing.org/xmlns http://graphml.graphdrawing.org/xmlns/1.0/graphml.xsd">
  <key attr.name="weight" attr.type="double" for="edge" id="weight" />
  <key attr.name="atitle" attr.type="string" for="node" id="atitle" />
  <key attr.name="jtitle" attr.type="string" for="node" id="jtitle" />
  <key attr.name="id" attr.type="string" for="node" id="id" />
  <key attr.name="date" attr.type="string" for="node" id="date" />

Right now we just rely on the NLTK stoplist and the NLTK Porter stemmer. Users may wish to use different stoplists or stemmers, and we should make that easy.

Talk about annoying...

Right now it just returns a string with newlines

This dataset has papers from 1999-2013

>>> dates_meta = {}
>>> for node in meta_list:
...    if node['date'] not in dates_meta.keys(): dates_meta[node['date']] = 1
...    else: dates_meta[node['date']] += 1
>>> dates_meta.keys()
[1999,
 2000,
 2001,
 2002,
 2003,
 2004,
 2005,
 2006,
 2007,
 2008,
 2009,
 2010,
 2011,
 2012,
 2013]

But if I run biblio_coupling, and then pull out the nodes, all nodes have date=1999:

>>> dates = {}
>>> for node in bc_nodes:
...     if node[1]['date'] not in dates.keys(): dates[node[1]['date']] = 1
...     else: dates[node[1]['date']] += 1
>>> dates.keys()
[1999]

Seems to do the same thing with titles.

Perhaps an optional boolean parameter?

E.g. only journals for which there is at least one year with a non-zero value.

Using sparse matrices from Scipy

Hi there,

I have been trying to follow the tutorial on topic modelling on the main tethne website. I installed anaconda, tethne, nltk, and also mallet. But when I run the line

MyLDAModel = MyManager.build(Z=50, max_iter=300, prep=True)

i get the following error

Traceback (most recent call last):
File "", line 1, in
File "//anaconda/lib/python2.7/site-packages/tethne/model/managers/init.py", line 108, in build
self.prep()
File "//anaconda/lib/python2.7/site-packages/tethne/model/managers/init.py", line 89, in prep
self._generate_corpus(meta)
File "//anaconda/lib/python2.7/site-packages/tethne/model/managers/mallet.py", line 152, in _generate_corpus
vocab=self.D.features[self.feature]['index'] )
File "//anaconda/lib/python2.7/site-packages/tethne/writers/corpora.py", line 59, in to_documents
meta += [ str(metadict[p][f]) for f in metakeys ]
KeyError: '10.1525/rac.2006.16.1.95'

I will appreciate all the help in this regard

Hi Erick,

In your tethne 0.6.1 beta documentation, the section of Load some data, there is a mistake. See below:

from tethne.readers import wos
C = dfr.read_corpus('/path/to/my/wosdata.txt') -------This one should be wos.read_corpus instead of dfr.read_corpus.

Thanks

Y

GraphCollection.build is currently overwriting method_kwargs, which means that the user has no control over node_id. This causes problems for networks.papers.cocitation, because WoS corpora are indexed by wosid by default; .build sets node_id by Corpus.indexed_by, which is inappropriate for cited references (which lack wosids).

'\n--------------------------------------------------------------------------\nTypeError Traceback (most recent call last)\n in ()\n----> 1 fig = C.plot_distribution('jtitle')\n\n/Users/Kenneth/anaconda/lib/python2.7/site-packages/tethne/classes/corpus.pyc in plot_distribution(self, x_axis, y_axis, type, aspect, step, fig, mode, fkwargs, *_kwargs)\n 1509 yvals = self.feature_distribution(featureset, feature, x_axis,\n 1510 mode=fmode, normed=fnormed)\n-> 1511 plt.dict[type](xkeys, yvals, **kwargs)\n 1512 plt.xlim(xkeys[0], xkeys[-1]) # Already sorted.\n 1513 else:\n\n/Users/Kenneth/anaconda/lib/python2.7/site-packages/matplotlib/pyplot.pyc in bar(left, height, width, bottom, hold, *_kwargs)\n 2513 ax.hold(hold)\n 2514 try:\n-> 2515 ret = ax.bar(left, height, width=width, bottom=bottom, *_kwargs)\n 2516 draw_if_interactive()\n 2517 finally:\n\n/Users/Kenneth/anaconda/lib/python2.7/site-packages/matplotlib/axes.pyc in bar(self, left, height, width, bottom, *_kwargs)\n 5051 r.get_path()._interpolation_steps = 100\n 5052 #print r.get_label(), label, 'label' in kwargs\n-> 5053 self.add_patch(r)\n 5054 patches.append(r)\n 5055 \n\n/Users/Kenneth/anaconda/lib/python2.7/site-packages/matplotlib/axes.pyc in add_patch(self, p)\n 1560 if p.get_clip_path() is None:\n 1561 p.set_clip_path(self.patch)\n-> 1562 self._update_patch_limits(p)\n 1563 self.patches.append(p)\n 1564 p._remove_method = lambda h: self.patches.remove(h)\n\n/Users/Kenneth/anaconda/lib/python2.7/site-packages/matplotlib/axes.pyc in _update_patch_limits(self, patch)\n 1578 vertices = patch.get_path().vertices\n 1579 if vertices.size > 0:\n-> 1580 xys = patch.get_patch_transform().transform(vertices)\n 1581 if patch.get_data_transform() != self.transData:\n 1582 patch_to_data = (patch.get_data_transform() -\n\n/Users/Kenneth/anaconda/lib/python2.7/site-packages/matplotlib/patches.pyc in get_patch_transform(self)\n 574 \n 575 def get_patch_transform(self):\n--> 576 self._update_patch_transform()\n 577 return self._rect_transform\n 578 \n\n/Users/Kenneth/anaconda/lib/python2.7/site-packages/matplotlib/patches.pyc in _update_patch_transform(self)\n 567 width = self.convert_xunits(self._width)\n 568 height = self.convert_yunits(self._height)\n--> 569 bbox = transforms.Bbox.from_bounds(x, y, width, height)\n 570 rot_trans = transforms.Affine2D()\n 571 rot_trans.rotate_deg_around(x, y, self._angle)\n\n/Users/Kenneth/anaconda/lib/python2.7/site-packages/matplotlib/transforms.pyc in from_bounds(x0, y0, width, height)\n 819 width and height may be negative.\n 820 """\n--> 821 return Bbox.from_extents(x0, y0, x0 + width, y0 + height)\n 822 \n 823 @staticmethod\n\nTypeError: cannot concatenate 'str' and 'float' objects\n'

test

Aaron, you may have already done this in your branch? In writers.to_sif():

• The first line in each attribute file should be the name of the attribute,
• For edge attributes, the relationship/predicate should be enclosed in parentheses.

E.g. if the interaction is:
Darwin cites Lamarck

An edge attribute file might look like:

date
Darwin (cites) Lamarck = 1859

I would like to be able to take two separate records, isolate the papers that cite both papers, and build networks and analyze just those co-citing documents.

e.g. TAPModelManager could indicate when modeling for a slice is complete

I'm using tethne to work with JSTOR DfR data. I'm using tethne to explicitly correlate wordcounts with DOIs. Working from the command line, when I correlated the wordcounts with the DOIs, I needed to export the information in a file, which I could then use in Mallet.

Problem: When I used the command: >>> wr.corpora.to_documents('./mycorpu1s', wordcounts),

I received the following error message: Traceback (most recent call last):
File "", line 1, in
File "/Applications/anaconda/lib/python2.7/site-packages/tethne/writers/corpora.py", line 36, in to_documents
for i in xrange(freq) ]) + '\n')
UnicodeEncodeError: 'ascii' codec can't encode character u'\u03c6' in position 17799: ordinal not in range(128)

Just like in the networks.topics methods; makes a bit more sense than a static threshold?

E.g. should be able to do a nPMI network with only nouns

When trying to create a corpus from using the command line:

$ >>>from tethne.readers import dfr

the following error occurs

Just like we can Papers (#41)

I received an error called ""NoneType" object is not utterable." I have attached a screenshot for your diagnosis. I used the same datadirpath for Notebook 2 and it works fine. Thanks

Author names are formatted as all caps or caps and lower case

>>> print [ i for i in range(len(meta_list)) if meta_list[i]['citations'] is None ]
[685]

Causes:

>>> bc = nt.nx_biblio_coupling(meta_list, 'ayjid', 2, 'ayjid', 'date')
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-97-51155c7d4843> in <module>()
----> 1 bc = nt.nx_biblio_coupling(meta_list, 'ayjid', 2, 'ayjid', 'date')

//anaconda/lib/python2.7/site-packages/tethne/networks.pyc in nx_biblio_coupling(doc_list, citation_id, threshold, node_id, *node_attribs)
    222                         for j in xrange(i+1, len(doc_list)):
    223                                 if doc_list[j]['citations'] is not None:
--> 224                                         #make a list of citation_id's for each document
    225                                         j_list = []
    226                                         for citation in doc_list[j]['citations']:

TypeError: 'NoneType' object is not iterable

Line 223 was my 10-second attempt to solve this problem, but it didn't work.

But if I...

meta_list[685]['citations'] = []

It proceeds without complaint.

Following the tutorial, I am running into problems with wos.corpus_from_dir(datapath).

The error message is as follows:

Traceback (most recent call last):
File "/Users/derycpainter/Desktop/WILLIAMS_57_networks/basic_coCitation", line 4, in
MyCorpus = wos.corpus_from_dir('datapath')
AttributeError: 'module' object has no attribute 'corpus_from_dir'

Here's a link to the page: http://bit.ly/1AwiZU9

Perhaps the module was renamed?

Following the quickstart guide for loading data, after creating a corpus using JSTOR Dfr dataset, when trying to run the command to load JSTOR Dfr corpus

OSError                                   Traceback (most recent call last)
<ipython-input-11-fdf5386772be> in <module>()
----> 1 Corpus = dfr.read_corpus('/Users/Kenneth/Downloads/Microbiome', 'uni')

/Users/Kenneth/anaconda/lib/python2.7/site-packages/tethne/readers/dfr.pyc in read_corpus(path, features, exclude, **kwargs)
    214     if features is not None:
    215         for feat in features:
--> 216             grams[feat+'grams'] = ngrams(path, feat, **kwargs)
    217 
    218     return Corpus(papers, features=grams, index_by='doi', exclude=exclude)

/Users/Kenneth/anaconda/lib/python2.7/site-packages/tethne/readers/dfr.pyc in ngrams(datapath, N, ignore_hash, mode)
    399         ngrams = {}
    400 
--> 401         for file in os.listdir(gram_path):
    402             if file.split('.')[-1] == 'XML':
    403                 root = ET.parse(gram_path + "/" + file).getroot()

OSError: [Errno 2] No such file or directory: '/Users/Kenneth/Downloads/Microbiome/ugrams'

Greetings all. Have been using Tethne as a means to parse the vast amounts of WoS exports I have for a particular project.

Out of 160 files (each containing the maximum 500 references), ~31 of them fail with the following error (although the actual string values vary):

In [19]: meta_list = rd.wos.convert(wos_list)
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-19-c13337f31f8c> in <module>()
----> 1 meta_list = rd.wos.convert(wos_list)

/usr/local/lib/python2.7/dist-packages/tethne/readers/wos.pyc in convert(wos_data)
    592                         #  as our mapping key to ensure consistency with older
    593                         #  datasets.
--> 594                         author_index = wos_dict['AF'].index(author)
    595                         #e.g."WU, ZD"
    596                         author_au = wos_dict['AU'][author_index].upper()

ValueError: 'Brazilian Aging Brain Study Grp' is not in list

For example, the only line "Brazilian Aging Brain Study Grp" appears in the file in question is:

data/ims/arrastra$ grep -H -r "Brazilian Aging Brain Study Grp" /data/arrastra/alzheimers_xph1_v4/Citations/
/data/arrastra/alzheimers_xph1_v4/Citations/sub1/savedrecs (12).txt.perror:CA Brazilian Aging Brain Study Grp
/data/arrastra/alzheimers_xph1_v4/Citations/sub1/savedrecs (12).txt.perror:   [Grinberg, L. T.; Alba, J. G.; Farfel, J. M.; Suemoto, C. K.; de Lucena Ferretti, R. E.; Leite, R. E. P.; de Andrade, M. P.; Pasqualucci, C. A.; Nitrini, R.; Jacob-Filho, W.; Brazilian Aging Brain Study Grp] Univ Sao Paulo, Sch Med, Brazilian Aging Brain Study Grp LIM22, Sao Paulo, Brazil.

I could easily add a check to ensure that the value is in fact present in the list, but that wouldn't really be solving the issue. Perhaps this is failing due to the fact that it is listed as a "Group Author" (WoS Field = CA)?

Thanks!

http://gexf.net/format/

It's redundant to have names like authors.author_coinstitution; should be authors.coinstitution.

Working with DfR data, the mycorpus_doc.txt file is created, but the mycorpus_meta.csv is not.

import tethne.readers as rd
datapath = '/Users/derycpainter/Desktop/Research/1-12-15/125_records_williams57'
wordcounts = rd.dfr.ngrams(datapath, N='uni')
import tethne.writers as wr
wr.corpora.to_documents('./mycorpus', wordcounts)

<abstract>Background. We hypothesized that the inflammatory response in the lungs of children with cystic fibrosis (CF) would vary with the type of infecting organism, being greatest with Pseudomonas aeruginosa and Staphylococcus aureus. Methods. A microbiological surveillance program based on annual bronchoalveolar lavage (BAL) collected fluid for culture and assessment of inflammation was conducted. Primary analyses compared inflammation in samples that grew a single organism with uninfected samples in cross-sectional and longitudinal analyses. Results. Results were available for 653 samples from 215 children with CF aged 24 days to 7 years. A single agent was associated with pulmonary infection (≥10 5 cfu/mL) in 67 BAL samples, with P. aeruginosa (n = 25), S. aureus (n = 17), and Aspergillus species (n = 19) being the most common. These microorganisms were associated with increased levels of inflammation, with P. aeruginosa being the most proinflammatory. Mixed oral flora (MOF) alone was isolated from 165 BAL samples from 112 patients, with 97 of these samples having a bacterial density ≥10 5 cfu/mL, and was associated with increased pulmonary inflammation (P <.001). For patients with current, but not past, infections there was an association with a greater inflammatory response, compared with those who were never infected (P <.05). However, previous infection with S. aureus was associated with a greater inflammatory response in subsequent BAL. Conclusions. Pulmonary infection with P. aeruginosa, S. aureus, or Aspergillus species and growth of MOF was associated with significant inflammatory responses in young children with CF. Our data support the use of specific surveillance and eradication programs for these organisms. The inflammatory response to MOF requires additional investigation.</abstract>

Python 2.7 installed, tethne 0.6.0-beta downloaded and unzipped, when installing from terminal I receive this error

labels should be str representations of words from the featureset index

Being able to create a featureset from phi posterior values in LDA might make it easier to visualize the results of topic modeling

While attempting to write a coCitation dynamic graph, I received this error:

Reader: tethne.readers.mallet, and networks from tethne.networks.papers.topic_coupling, tethne.networks.terms and tethne.networks.topics.