cligs / pyzeta Goto Github PK

Add howto separate from run_pytzeta.py.

Offer the option to run a comparison not on a meaningful partitioning of the data, but on a random one, for a better understanding of the level of differences to be expected if the partitioning is not meaningful.

This is an idea of AZ. Make sure that longer documents don't skew the results by sampling a fix number of segments from each document. In this way, each document will have an equal weight in the overall Zeta score of a group of texts. It does mean some loss of information, because more text is discarded.

Move basic parameter settings to a separate, plain text settings file to be loaded by run_pyzeta.

Add sample input (texts and metadata)

warnings.warn(DEPRECATION_MSG_1D, DeprecationWarning)
/usr/local/lib/python3.5/dist-packages/sklearn/preprocessing/data.py:321: DeprecationWarning: Passing 1d arrays as data is deprecated in 0.17 and will raise ValueError in 0.19. Reshape your data either using X.reshape(-1, 1) if your data has a single feature or X.reshape(1, -1) if it contains a single sample.

Require utf-8 input (in docs and ind code)

I tried running run_pyzeta with doyle and roman20 corpus and I get this error:

Traceback (most recent call last):
  File "run_pyzeta.py", line 101, in <module>
    calculate.main(datafolder, dtmfolder, metadatafile, separator, contrast, logaddition, resultsfolder, segmentlength, featuretype, absolutefreqs, relativefreqs, binaryfreqs)
  File "/Users/reiterns/Documents/Projects/fanfiction/pyzeta/scripts/calculate.py", line 375, in main
    idlists = make_idlists(metadatafile, separator, contrast)
  File "/Users/reiterns/Documents/Projects/fanfiction/pyzeta/scripts/calculate.py", line 36, in make_idlists
    metadata.set_index("idno", inplace=True)
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/pandas/core/frame.py", line 4391, in set_index
    raise KeyError("None of {} are in the columns".format(missing))
KeyError: "None of ['idno'] are in the columns"

Any ideas?