Comments (6)
krischer
commented on August 26, 2024
I'll work on improving the error reporting and handling!
from pyasdf.
wjlei1990
commented on August 26, 2024
Great.
This is the output using multiprocessing. What I am going to using mpi in the future :) I think for mpi case, it is probably hard to output this kind of things, right?
Can we have something like:
try:
process_blalbla(trace)
except Exception as err:
print("%s: %s" % (trace.id, err))
Very primitive thoughts...Ignore me if it doesn't make sense.
from pyasdf.
krischer
commented on August 26, 2024
Sure. Already at it. Should be done in the next half hour :-)
from pyasdf.
krischer
commented on August 26, 2024
This took a bit longer but I also wrote some more documentation: http://seismicdata.github.io/pyasdf/parallel_processing.html
The latest master has much better output and error reporting when run with multiprocessing. Additionally you can now set the number of processors and the length of the printed traceback for the multiprocessing processing function.
I think this solves your problem :-)
$ python process_test.py
Launching processing using multiprocessing on 8 cores ...
-> Processing approximately task 0 of 107 ...
-> Processing approximately task 8 of 107 ...
-> Processing approximately task 16 of 107 ...
-> Processing approximately task 24 of 107 ...
-> Processing approximately task 32 of 107 ...
-> Processing approximately task 40 of 107 ...
Error during the processing of station 'IC.LSA' and tag 'synthetic' on CPU 2:
Traceback (At max 3 levels - most recent call last):
File "/Users/lion/.miniconda3/envs/obspy_py34/lib/python3.4/threading.py", line 888, in _bootstrap
self._bootstrap_inner()
File "/Users/lion/.miniconda3/envs/obspy_py34/lib/python3.4/threading.py", line 920, in _bootstrap_inner
self.run()
File "/Users/lion/workspace/code/pyasdf/pyasdf/asdf_data_set.py", line 1785, in run
output_stream = self.processing_function(stream, inv)
File "process_test.py", line 9, in process
raise ValueError("Something went wrong!!!")
ValueError: Something went wrong!!!
-> Processing approximately task 48 of 107 ...
-> Processing approximately task 56 of 107 ...
-> Processing approximately task 64 of 107 ...
-> Processing approximately task 72 of 107 ...
Error during the processing of station 'IU.DAV' and tag 'synthetic' on CPU 0:
Traceback (At max 3 levels - most recent call last):
File "/Users/lion/.miniconda3/envs/obspy_py34/lib/python3.4/threading.py", line 888, in _bootstrap
self._bootstrap_inner()
File "/Users/lion/.miniconda3/envs/obspy_py34/lib/python3.4/threading.py", line 920, in _bootstrap_inner
self.run()
File "/Users/lion/workspace/code/pyasdf/pyasdf/asdf_data_set.py", line 1785, in run
output_stream = self.processing_function(stream, inv)
File "process_test.py", line 9, in process
raise ValueError("Something went wrong!!!")
ValueError: Something went wrong!!!
-> Processing approximately task 80 of 107 ...
-> Processing approximately task 88 of 107 ...
-> Processing approximately task 96 of 107 ...
-> Processing approximately task 104 of 107 ...
Finished processing in 3.13 seconds (0.029 sec/station).from pyasdf.
wjlei1990
commented on August 26, 2024
👍
Thanks, Lion. That is great!
from pyasdf.
krischer
commented on August 26, 2024
c75d65b does the same for the MPI based processing so it should be much easier to diagnose problems now.
from pyasdf.
Related Issues (20)
- Why are tag paths validated against r"^[a-zA-Z0-9][a-zA-Z0-9_]*[a-zA-Z0-9]$"? HOT 7
- Slow performance when adding gappy data to ASDF HOT 3
- adding an existing StationXML throws TypeError HOT 1
- setup broken between 0.5.1 and 0.6.1 HOT 4
- How to make pyasdf to support multi-date-range station inventory + custom tags? HOT 6
- Calling parallel process() for a very large file gets stuck HOT 7
- Why the differences between input station XML file to ASDF and the output station XML file HOT 5
- Problem with hdf5 1.10 and _add_trace_write_collective_information during process() HOT 2
- SUGGESTION: adding public APIs for parallel writing functions HOT 1
- pkg_resources deprecated? HOT 1
- newer versions h5py > 2.1 will break pyasdf events HOT 2
- Error @opening after copying pyasdf-archive HOT 2
- Renaming a tag HOT 1
- Increasingly slow to add events to ASDF dataset HOT 3
- use dense attribute storage for large attributes
- Citation?
- Rounding start times to nearest sample rather than previous sample
- Issue in tutorial
- Whether each event can be associated with part of the waveform in ASDFDataset.h5
- Failed to flush and close HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from pyasdf.