out of memory issue about ssm HOT 9 CLOSED

Can you try using the version from this branch: https://github.com/slinderman/ssm/tree/lowmem_stationary_transitions ? It contains my first stab at a low memory implementation of the EM algorithm for the stationary transition model. Basically, it allows the transition model to return a (1,K,K) array for log_transition_matrices(...). If the returned value is 1d along the first axis, the downstream code should not instantiate any (T,K,K) arrays.

Note: you will need to checkout this branch and rerun python setup.py build_ext --inplace to recompile the message passing code.

from ssm.

Hi Scott,

I have the following error:

Traceback (most recent call last):
File "ssm.py", line 272, in
hmm, lps = _fit(hmm)
File "ssm.py", line 54, in func_wrapper
results = func(*args, **kwargs)
File "ssm.py", line 33, in fit_hmm
lps = hmm.fit(x_norm, method="em", num_iters=num_iters)
File "/tigress/hjang/ssm/ssm/util.py", line 109, in wrapper
return f(self, datas, inputs=inputs, masks=masks, tags=tags, **kwargs)
File "/tigress/hjang/ssm/ssm/core.py", line 371, in fit
return _fitting_methods[method](datas, inputs=inputs, masks=masks, tags=tags, **kwargs)
File "/tigress/hjang/ssm/ssm/core.py", line 332, in _fit_em
lls = [self.log_probability(datas, inputs, masks, tags)]
File "/tigress/hjang/ssm/ssm/util.py", line 109, in wrapper
return f(self, datas, inputs=inputs, masks=masks, tags=tags, **kwargs)
File "/tigress/hjang/ssm/ssm/core.py", line 215, in log_probability
return self.log_likelihood(datas, inputs, masks, tags) + self.log_prior()
File "/tigress/hjang/ssm/ssm/util.py", line 109, in wrapper
return f(self, datas, inputs=inputs, masks=masks, tags=tags, **kwargs)
File "/tigress/hjang/ssm/ssm/core.py", line 208, in log_likelihood
log_likes = self.observations.log_likelihoods(data, input, mask, tag)
File "/tigress/hjang/ssm/ssm/observations.py", line 381, in log_likelihoods
return stats.multivariate_studentst_logpdf(data[:, None, :], mus, Sigmas, nus)
File "/tigress/hjang/ssm/ssm/stats.py", line 346, in multivariate_studentst_logpdf
q = batch_mahalanobis(Ls, data - mus) / nus # (...,)
File "/tigress/hjang/ssm/ssm/stats.py", line 63, in batch_mahalanobis
return np.sum(np.sum(x[..., None] * L_inv, axis=-2)**2, axis=-1)
MemoryError

from ssm.

I see. I think that operation is performing a TxKxDx1 * KxDxD multiplication and then summing out the last two dimensions to get a TxK array. I'm guessing that the intermediate TxKxDxD array is too much to fit in memory.

Here's a simple thing to try: what if you artificially partition your data into a bunch of smaller chunks (e.g. T=1e4 or T=1e5) and pass the list of data chunks in your call to model.fit(datas)? That would mean smaller intermediate arrays in your likelihood calculations, etc. It comes at the cost of a minor loss of information at the artificial partitions of the data, but it should be negligible.

from ssm.

Thanks, it bypasses the memory issue. It's taking a while for the model fitting to complete, so I'll update again after I check whether the results look reasonable!

from ssm.

Hi Heejae,

Just checking back in on this issue. Any update? Re training time, I'm sure it could be considerably faster and nearly as accurate with a stochastic optimization method using minibatches of data. Let's discuss that possibility next, once your "batch mode" fitting finishes.

Scott

from ssm.

Yes, batch mode works! Log probability converged pretty fast so I only trained for 12 iterations.
So you'd suggest using stochastic optmization instead of em?

from ssm.

Hi Scott,

I tried splitting my sequence up into pieces and still had this problem, did I do something wrong?

a = data[0:500000,:]
b = data[500000:1000000,:]
c = data[1000000:1500000,:]
...
out = hmm.fit([a,b,c],method="em",num_em_iters=10)

Thanks for any advice!

from ssm.

Hi @heejaeyunajang and @wanderingGradient, I think this issue should have been fixed in PR #36. There I updated the message passing code to work with a single transition matrix rather than a TxKxK array. There's still room for improvement on very long datasets, where a stochastic EM update based on a small batch of data could suffice. Nevertheless, I think the core out of memory errors should be addressed.

from ssm.

I'm going to close this issue now that the data splitting code and the special-case message passing code has been merged to master.

from ssm.