Comments (1)
Thanks for bringing this up! I can see how this would be a problem.
I think your approach is a good solution, and Ill adopt that in an upcoming ncempy
release. I don't like the idea that the file object is lost, but h5py
documentation explicitly mentions this solution at this link. So, going out of scope with so called 'weak' closing is a suggested way to deal with hdf5 memmaps.
This is a tricky problem, actually. Numpy just totally ignores the issue! See numpy documentation here.
Currently there is no API to close the underlying mmap. It is tricky to ensure the resource is actually closed, since it may be shared between different memmap instances.
Alternatively, a user could keep the fileEMD
object in scope by returning the file object with the memmap. Then when you are done with the data you can del
the file object and the dataset object. Or just let all of them go out of scope and everything is garbage collected at the same time hopefully closing the object properly. I don't think this is the best option in py4DSTEM
since you just want to return data
by itself.
This is an example of what I suggested above in case anyone runs into a similar issue.
from ncempy.io.emd import fileEMD
fpath = "Sample_LFP_v0,12.h5"
def load_memmap(fpath, N):
f = fileEMD(fpath)
return f, *f.get_memmap(N)
f, data, dims = load_memmap(fpath,0)
print(data)
# <HDF5 dataset "data": shape (10, 10), type "<f8">
del f, data, dims
from openncem.
Related Issues (20)
- Example code HOT 2
- Matlab Code Location HOT 1
- ser.py and import ncempy.io.emd HOT 1
- SER array size backwards HOT 3
- ser tab complete infinite loop HOT 1
- Submodule name io clobbers built in python3 module io HOT 1
- 4D-STEM data reads incorrectly HOT 1
- Could not build ncempy. Cannot load backend 'Qt5Agg' which requires the 'qt5' HOT 34
- Change genfire import line HOT 1
- dm data not accessible after file closed using on_memory option HOT 2
- RingDiff tool and PyQT HOT 1
- module 'ncempy' has no attribute 'io' HOT 4
- DM import can't read newest-version files HOT 1
- DM reader fails in `__del__` on current Python 3.9 HOT 1
- `BufferError` caused by eager closing of DM file handles HOT 3
- emd import does not support default dim attributes HOT 3
- Unsupported tag breaks DM reading
- IMG and SMV not recognized
- SMV and IMG reader errors HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from openncem.