Comments (2)
Hi Anand,
I see two solutions here.
The 10000 limit is probably defined by a parameter somewhere in
pytables. I looked quickly in the pytables manual but did not find it,
but I'm relatively confident it is somewhere. Maybe worth asking on
their ML.
If the 10000 limit cannot be changed or if it causes performance
problem, then we'll need to find a way to store these
observed_stochastics elsewhere in the group, for instance in an array.
We'll need a try else statement both in the initialize function and in
Database.init to load the objects back in memory no matter whether
they are stored as table attributes or stand-alone arrays.
David
On Tue, May 17, 2011 at 8:41 AM, apatil
[email protected]
wrote:
This line: https://github.com/pymc-devs/pymc/blob/master/pymc/database/hdf5.py#L416 causes the following error if an observed stochastic has length of around 10k.
HDF5-DIAG: Error detected in HDF5 (1.8.5-patch1) thread 0:
#000: H5Adeprec.c line 165 in H5Acreate1(): unable to create attribute
major: Attribute
minor: Unable to initialize object
#1: H5A.c line 492 in H5A_create(): unable to create attribute in object header
major: Attribute
minor: Unable to insert object
#2: H5Oattribute.c line 346 in H5O_attr_create(): unable to create new attribute in header
major: Attribute
minor: Unable to insert object
#3: H5Omessage.c line 224 in H5O_msg_append_real(): unable to create new message
major: Object header
minor: No space available for allocation
#4: H5Omessage.c line 1925 in H5O_msg_alloc(): unable to allocate space for message
major: Object header
minor: Unable to initialize object
#5: H5Oalloc.c line 1135 in H5O_alloc(): object header message is too large
major: Object header
minor: Unable to initialize object
HDF5-DIAG: Error detected in HDF5 (1.8.5-patch1) thread 0:
#000: H5A.c line 916 in H5Awrite(): not an attribute
major: Invalid arguments to routine
minor: Inappropriate typeReply to this email directly or view it on GitHub:
#41
from pymc.
This is not a PyTables issue, it is an HDF5 one. HDF5 limits the size of data type headers to 64k, which in practice is about ~1000 columns. I am currently looking for a workaround myself...
from pymc.
Related Issues (20)
- Test `freeze_rv_and_dims` with `PartiallyObservedRV` HOT 1
- ENH: Customizable plate labels HOT 11
- First time congratulatory message images are huge HOT 8
- DOC: Plots squished in the documentation page HOT 2
- Problematic casting of integers in `convert_observed_data`
- ENH: Get sampling working using Apple Silicon GPU via jax backend HOT 32
- DOC: conflicting `fixed_params` type-hint in `find_constrained_prior` HOT 1
- ENH: Support foward sampling via JAX HOT 7
- ConnectionResetError from multiprocess sampling
- BUG: Repeated predictions from GP grow steadily slower (pytensor cache issue?) HOT 1
- BUG: Compiler error when calling pm.sample() HOT 2
- Remove `mutable=True` in test suite HOT 1
- Numpy deprecation warning issued from within `test_metropolis.py`
- ENH: Add an explicit dummy prior for predictive modeling HOT 1
- Add CustomDist as second level header below Distributions in the API HOT 1
- ENH: implement deterministic ADVI (DADVI)
- BUG: Using a length-only coord causes it to be volatile HOT 5
- ENH: Incorporated a download analytics badge that tracks all downloads from Conda. HOT 1
- RTD is broken HOT 3
- Missing static information for jax backend for models with missing values HOT 3
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from pymc.