Comments (9)
Can you try using the version from this branch: https://github.com/slinderman/ssm/tree/lowmem_stationary_transitions ? It contains my first stab at a low memory implementation of the EM algorithm for the stationary transition model. Basically, it allows the transition model to return a (1,K,K)
array for log_transition_matrices(...)
. If the returned value is 1d along the first axis, the downstream code should not instantiate any (T,K,K)
arrays.
Note: you will need to checkout this branch and rerun python setup.py build_ext --inplace
to recompile the message passing code.
from ssm.
Hi Scott,
I have the following error:
Traceback (most recent call last):
File "ssm.py", line 272, in
hmm, lps = _fit(hmm)
File "ssm.py", line 54, in func_wrapper
results = func(*args, **kwargs)
File "ssm.py", line 33, in fit_hmm
lps = hmm.fit(x_norm, method="em", num_iters=num_iters)
File "/tigress/hjang/ssm/ssm/util.py", line 109, in wrapper
return f(self, datas, inputs=inputs, masks=masks, tags=tags, **kwargs)
File "/tigress/hjang/ssm/ssm/core.py", line 371, in fit
return _fitting_methods[method](datas, inputs=inputs, masks=masks, tags=tags, **kwargs)
File "/tigress/hjang/ssm/ssm/core.py", line 332, in _fit_em
lls = [self.log_probability(datas, inputs, masks, tags)]
File "/tigress/hjang/ssm/ssm/util.py", line 109, in wrapper
return f(self, datas, inputs=inputs, masks=masks, tags=tags, **kwargs)
File "/tigress/hjang/ssm/ssm/core.py", line 215, in log_probability
return self.log_likelihood(datas, inputs, masks, tags) + self.log_prior()
File "/tigress/hjang/ssm/ssm/util.py", line 109, in wrapper
return f(self, datas, inputs=inputs, masks=masks, tags=tags, **kwargs)
File "/tigress/hjang/ssm/ssm/core.py", line 208, in log_likelihood
log_likes = self.observations.log_likelihoods(data, input, mask, tag)
File "/tigress/hjang/ssm/ssm/observations.py", line 381, in log_likelihoods
return stats.multivariate_studentst_logpdf(data[:, None, :], mus, Sigmas, nus)
File "/tigress/hjang/ssm/ssm/stats.py", line 346, in multivariate_studentst_logpdf
q = batch_mahalanobis(Ls, data - mus) / nus # (...,)
File "/tigress/hjang/ssm/ssm/stats.py", line 63, in batch_mahalanobis
return np.sum(np.sum(x[..., None] * L_inv, axis=-2)**2, axis=-1)
MemoryError
from ssm.
I see. I think that operation is performing a TxKxDx1 * KxDxD
multiplication and then summing out the last two dimensions to get a TxK
array. I'm guessing that the intermediate TxKxDxD
array is too much to fit in memory.
Here's a simple thing to try: what if you artificially partition your data into a bunch of smaller chunks (e.g. T=1e4
or T=1e5
) and pass the list of data chunks in your call to model.fit(datas)
? That would mean smaller intermediate arrays in your likelihood calculations, etc. It comes at the cost of a minor loss of information at the artificial partitions of the data, but it should be negligible.
from ssm.
Thanks, it bypasses the memory issue. It's taking a while for the model fitting to complete, so I'll update again after I check whether the results look reasonable!
from ssm.
Hi Heejae,
Just checking back in on this issue. Any update? Re training time, I'm sure it could be considerably faster and nearly as accurate with a stochastic optimization method using minibatches of data. Let's discuss that possibility next, once your "batch mode" fitting finishes.
Scott
from ssm.
Yes, batch mode works! Log probability converged pretty fast so I only trained for 12 iterations.
So you'd suggest using stochastic optmization instead of em?
from ssm.
Hi Scott,
I tried splitting my sequence up into pieces and still had this problem, did I do something wrong?
a = data[0:500000,:]
b = data[500000:1000000,:]
c = data[1000000:1500000,:]
...
out = hmm.fit([a,b,c],method="em",num_em_iters=10)
Thanks for any advice!
from ssm.
Hi @heejaeyunajang and @wanderingGradient, I think this issue should have been fixed in PR #36. There I updated the message passing code to work with a single transition matrix rather than a TxKxK
array. There's still room for improvement on very long datasets, where a stochastic EM update based on a small batch of data could suffice. Nevertheless, I think the core out of memory errors should be addressed.
from ssm.
I'm going to close this issue now that the data splitting code and the special-case message passing code has been merged to master.
from ssm.
Related Issues (20)
- Prediction of observations on a test set HOT 3
- installation error HOT 1
- State Transition Parameters HOT 1
- using GPU HOT 1
- documentation: add pointer to ssm-jax
- access lag-specific autoregressive A matrix HOT 1
- Regularization of GLM weights for input driven observations and transitions HOT 1
- Singularity Container for the code
- error during installation HOT 4
- ARHMM usage HOT 1
- Unit tests fail in Python 3.8 HOT 1
- Diagonal Gaussian emissions for LDS? HOT 1
- Constraining AR-HMM emission parameters HOT 4
- Explained variance for the rSLDS latents HOT 3
- How to use _vjp_solve_banded_A HOT 1
- Is there any documentation for the ssm package, or at least for GLM-HMM part? HOT 1
- Time-varying C matrix? HOT 1
- Finding the variance explained for # of dimensions HOT 4
- Question on method ssm.HMM (GLM weight sign) HOT 2
- Trouble with SSM installation HOT 7
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from ssm.