hmsm means-ghmm instability produces MemoryError and ValueError about msmbuilder HOT 4 CLOSED

A MemoryError means python is running out of memory. That may be an
artifact of the machine not having enough memory at the moment.

On Sun, Mar 2, 2014 at 12:45 AM, gkiss [email protected] wrote:

/g/g90/kiss2/Progs/Python2.7/bin/python2.7 /g/g90/kiss2/mixtape-master/scripts/hmsm means-ghmm -i ../hmms.jsonlines --featurizer ../features_rmsd.dat --n-states 7 --n-per-state 20 --lag-time 100 -o rmsd_HMM_7states_20struct.csv --dir "../../Trajectories" --ext lh5 --top ../../2g2z_TetInt_TIP3P_300Wat.pdb

Most often gives a MemoryError

Traceback (most recent call last):
File "/g/g90/kiss2/mixtape-master/scripts/hmsm", line 8, in
app.start()
File
"/g/g90/kiss2/Progs/Python2.7/lib/python2.7/site-packages/mixtape-0.1-py2.7-linux-x86_64.egg/mixtape/cmdline.py",
line 219, in start
instance.start()
File
"/g/g90/kiss2/Progs/Python2.7/lib/python2.7/site-packages/mixtape-0.1-py2.7-linux-x86_64.egg/mixtape/commands/pullmeans.py",
line 73, in start
features, ii, ff = mixtape.featurizer.featurize_all(self.filenames,
featurizer, self.topology)
File
"/g/g90/kiss2/Progs/Python2.7/lib/python2.7/site-packages/mixtape-0.1-py2.7-linux-x86_64.egg/mixtape/featurizer.py",
line 13, in featurize_all
x = featurizer.featurize(t)
File
"/g/g90/kiss2/Progs/Python2.7/lib/python2.7/site-packages/mixtape-0.1-py2.7-linux-x86_64.egg/mixtape/featurizer.py",
line 49, in featurize
traj.superpose(self.reference_traj, atom_indices=self.atom_indices)
File
"/g/g90/kiss2/Progs/Python2.7/lib/python2.7/site-packages/mdtraj-0.7.0-py2.7-linux-x86_64.egg/mdtraj/trajectory.py",
line 785, in superpose
self.xyz = self_displace_xyz + ref_offset
MemoryError
But on occasion also a ValueError:

Traceback (most recent call last):
File "/g/g90/kiss2/mixtape-master/scripts/hmsm", line 8, in
app.start()
File
"/g/g90/kiss2/Progs/Python2.7/lib/python2.7/site-packages/mixtape-0.1-py2.7-linux-x86_64.egg/mixtape/cmdline.py",
line 219, in start
instance.start()
File
"/g/g90/kiss2/Progs/Python2.7/lib/python2.7/site-packages/mixtape-0.1-py2.7-linux-x86_64.egg/mixtape/commands/pullmeans.py",
line 95, in start
df = pd.DataFrame(data)
File
"/g/g90/kiss2/Progs/Python2.7/lib/python2.7/site-packages/pandas-0.13.1-py2.7-linux-x86_64.egg/pandas/core/frame.py",
line 201, in init
mgr = self._init_dict(data, index, columns, dtype=dtype)
File
"/g/g90/kiss2/Progs/Python2.7/lib/python2.7/site-packages/pandas-0.13.1-py2.7-linux-x86_64.egg/pandas/core/frame.py",
line 323, in _init_dict
dtype=dtype)
File
"/g/g90/kiss2/Progs/Python2.7/lib/python2.7/site-packages/pandas-0.13.1-py2.7-linux-x86_64.egg/pandas/core/frame.py",
line 4463, in _arrays_to_mgr
index = extract_index(arrays)
File
"/g/g90/kiss2/Progs/Python2.7/lib/python2.7/site-packages/pandas-0.13.1-py2.7-linux-x86_64.egg/pandas/core/frame.py",
line 4511, in extract_index
raise ValueError('arrays must all be same length')
ValueError: arrays must all be same length

Reply to this email directly or view it on GitHubhttps://github.com//issues/64
.

from msmbuilder.

I know.
This was a problem in the past, but not in more than ~20% of the cases.
Now it's around 90% and it has become almost impossible to generate .csv files and to then pull structures.
Note that I'm running the same data on the same machine now than a few days ago (when the error rate was at 20%).
I'm raising this issue to elevate the importance of issue #37

from msmbuilder.

Update:
Additional poking around suggests that the issue is not with mixtape, but with the imaging protocol going from Anton's Desmond trajectories to the water-stripped input that I'm throwing at mixtape.
Let's put this issue on hold until I have ruled out any ambiguity.

from msmbuilder.

The issue was indeed with the Anton imaging protocol.

from msmbuilder.