Possibility to ignore certain WoS-tags about tethne HOT 5 OPEN

@Epipremnum Yeah, that's a great idea. Just the last few days I have been working on "streaming" representations of corpora and papers (i.e. on disk, in a database) to cut down on memory overhead -- the logic is basically as you describe, to pass over the metadata records once and load into memory only the immediately-needed fields. So your suggestion is a logical continuation of that line of work. I'll keep this thread up to date as we work on it!

Thanks for using tethne -- it would be good to hear more about your use-case, if you're willing to share. :-)

from tethne.

Hi Erick,

thanks for the quick response. That would be great.

For now i wrote a script to preprocess my bibliography files. I copied all lines corresponding to tags i wanted to keep to a new file. Tethne then gets only these lines as input that i am interested in. It works. But there is still a lot of memory used.

Thank you.

from tethne.

This will be TETHNE-124.

from tethne.

@Epipremnum On the develop branch I have added a parameter called parse_only to the WoS and DfR readers. If you have a moment, it would be great to hear whether or not this addresses your need.

The relevant tests are here: https://github.com/diging/tethne/blob/develop/tethne/tests/test_readers_parseonly.py

from tethne.

This is now in v0.8.1.dev2, which can be installed via pip with --pre:

$ pip install -U tethne --pre

Example:

>>> from tethne.readers.wos import read
>>> corpus = read('/path/to/my/data', parse_only=['title', 'date'])

from tethne.