Comments (5)
KrisJanssen
commented on August 22, 2024
No ideas from the community?
from relion.
rahelwoldeyes
commented on August 22, 2024
Hi @KrisJanssen I am seeing the same error message. Were you able to fix it?
RELION version: 5.0-beta-3-commit-7a062e
from relion.
KrisJanssen
commented on August 22, 2024
@rahelwoldeyes : unfortunately not, really hoping the devs or other docs with knowledge of cuda and MPI might be able to shed some light…
from relion.
rahelwoldeyes
commented on August 22, 2024
I hope so too. It's interesting that the problem started with commit fa8dce3. Commit 90d239e works as expected. Unfortunately, I couldn't determine the root cause
from relion.
rahelwoldeyes
commented on August 22, 2024
To further support this issue, the error consistently appears on our cluster regardless of whether Relion is containerized or installed directly.
Describe your problem
Refine3D consistently crashes with a segmentation fault at the beginning of the first iteration, regardless of MPI or GPU use. This occurs for datasets larger than trivially small ones (~500 particles), indicating a potential memory management issue. The dataset appears to be read six times. The problem was introduced in commit fa8dce3, as commit 90d239e works correctly.
Environment:
OS: RHEL 8.6
MPI runtime: mpirun (Open MPI) 4.1.6rc4
RELION version: 5.0-beta-3-commit-d08e4d
Memory: 11264MiB
GPU: NVIDIA GeForce RTX 2080 Ti
GPU Driver: 535.161.07
CUDA: 12.2
Dataset:
Box size: 48 px
Pixel size: 8.53 Å/px #binned 4
Number of particles: 4,000
Job options:
Type of job: Refine3D
Number of MPI processes: 5
Number of threads: 4
Full command:
mpirun --mca btl self,vader,tcp --mca pml ob1 `which relion_refine_mpi` --o Refine3D/job001/run --auto_refine --split_random_halves --ios optimisation_set.star --ref init/intModel.mrc --firstiter_cc --trust_ref_size --ini_high 60 --dont_combine_weights_via_disc --pool 8 --pad 2 --ctf --particle_diameter 340 --flatten_solvent --zero_mask --oversampling 1 --healpix_order 2 --auto_local_healpix_order 4 --offset_range 5 --offset_step 2 --sym C1 --low_resol_join_halves 40 --norm --scale --j 4 --gpu "" --pipeline_control Refine3D/job001/ ```
Error message:
Please cite the full error message as the example below.
WARNING: tomogram S65-TS_002.tomostar has relion-4 definition of projection matrices; converting them now...
WARNING: tomogram S39-TS_001.tomostar has relion-4 definition of projection matrices; converting them now...
WARNING: tomogram S26-TS_004.tomostar has relion-4 definition of projection matrices; converting them now...
WARNING: tomogram S26-TS_002.tomostar has relion-4 definition of projection matrices; converting them now...
WARNING: tomogram S65-TS_002.tomostar has relion-4 definition of projection matrices; converting them now...
WARNING: tomogram S39-TS_001.tomostar has relion-4 definition of projection matrices; converting them now...
WARNING: tomogram S26-TS_006.tomostar has relion-4 definition of projection matrices; converting them now...
WARNING: tomogram S26-TS_004.tomostar has relion-4 definition of projection matrices; converting them now...
WARNING: tomogram S39-TS_005.tomostar has relion-4 definition of projection matrices; converting them now...
WARNING: tomogram S26-TS_002.tomostar has relion-4 definition of projection matrices; converting them now...
WARNING: tomogram S65-TS_002.tomostar has relion-4 definition of projection matrices; converting them now...
WARNING: tomogram S39-TS_001.tomostar has relion-4 definition of projection matrices; converting them now...
WARNING: tomogram S65-TS_002.tomostar has relion-4 definition of projection matrices; converting them now...
WARNING: tomogram S39-TS_001.tomostar has relion-4 definition of projection matrices; converting them now...
WARNING: tomogram S26-TS_006.tomostar has relion-4 definition of projection matrices; converting them now...
WARNING: tomogram S26-TS_004.tomostar has relion-4 definition of projection matrices; converting them now...
WARNING: tomogram S26-TS_004.tomostar has relion-4 definition of projection matrices; converting them now...
WARNING: tomogram S26-TS_002.tomostar has relion-4 definition of projection matrices; converting them now...
WARNING: tomogram S26-TS_003.tomostar has relion-4 definition of projection matrices; converting them now...
WARNING: tomogram S39-TS_005.tomostar has relion-4 definition of projection matrices; converting them now...
WARNING: tomogram S26-TS_002.tomostar has relion-4 definition of projection matrices; converting them now...
WARNING: tomogram S39-TS_002.tomostar has relion-4 definition of projection matrices; converting them now...
WARNING: tomogram S26-TS_006.tomostar has relion-4 definition of projection matrices; converting them now...
WARNING: tomogram S65-TS_001.tomostar has relion-4 definition of projection matrices; converting them now...
WARNING: tomogram S26-TS_006.tomostar has relion-4 definition of projection matrices; converting them now...
WARNING: tomogram S39-TS_006.tomostar has relion-4 definition of projection matrices; converting them now...
WARNING: tomogram S26-TS_001.tomostar has relion-4 definition of projection matrices; converting them now...
WARNING: tomogram S39-TS_005.tomostar has relion-4 definition of projection matrices; converting them now...
WARNING: tomogram S26-TS_003.tomostar has relion-4 definition of projection matrices; converting them now...
WARNING: tomogram S39-TS_005.tomostar has relion-4 definition of projection matrices; converting them now...
WARNING: tomogram S39-TS_002.tomostar has relion-4 definition of projection matrices; converting them now...
WARNING: tomogram S26-TS_005.tomostar has relion-4 definition of projection matrices; converting them now...
WARNING: tomogram S65-TS_001.tomostar has relion-4 definition of projection matrices; converting them now...
WARNING: tomogram S39-TS_006.tomostar has relion-4 definition of projection matrices; converting them now...
WARNING: tomogram S26-TS_003.tomostar has relion-4 definition of projection matrices; converting them now...
WARNING: tomogram S26-TS_001.tomostar has relion-4 definition of projection matrices; converting them now...
WARNING: tomogram S26-TS_003.tomostar has relion-4 definition of projection matrices; converting them now...
WARNING: tomogram S39-TS_002.tomostar has relion-4 definition of projection matrices; converting them now...
WARNING: tomogram S39-TS_002.tomostar has relion-4 definition of projection matrices; converting them now...
WARNING: tomogram S65-TS_001.tomostar has relion-4 definition of projection matrices; converting them now...
WARNING: tomogram S65-TS_001.tomostar has relion-4 definition of projection matrices; converting them now...
WARNING: tomogram S26-TS_005.tomostar has relion-4 definition of projection matrices; converting them now...
WARNING: tomogram S39-TS_006.tomostar has relion-4 definition of projection matrices; converting them now...
WARNING: tomogram S39-TS_006.tomostar has relion-4 definition of projection matrices; converting them now...
WARNING: tomogram S26-TS_001.tomostar has relion-4 definition of projection matrices; converting them now...
WARNING: tomogram S26-TS_001.tomostar has relion-4 definition of projection matrices; converting them now...
WARNING: tomogram S26-TS_005.tomostar has relion-4 definition of projection matrices; converting them now...
WARNING: tomogram S26-TS_005.tomostar has relion-4 definition of projection matrices; converting them now...
WARNING: tomogram S65-TS_002.tomostar has relion-4 definition of projection matrices; converting them now...
WARNING: tomogram S39-TS_001.tomostar has relion-4 definition of projection matrices; converting them now...
WARNING: tomogram S26-TS_004.tomostar has relion-4 definition of projection matrices; converting them now...
WARNING: tomogram S26-TS_002.tomostar has relion-4 definition of projection matrices; converting them now...
WARNING: tomogram S26-TS_006.tomostar has relion-4 definition of projection matrices; converting them now...
WARNING: tomogram S39-TS_005.tomostar has relion-4 definition of projection matrices; converting them now...
WARNING: tomogram S26-TS_003.tomostar has relion-4 definition of projection matrices; converting them now...
WARNING: tomogram S39-TS_002.tomostar has relion-4 definition of projection matrices; converting them now...
WARNING: tomogram S65-TS_001.tomostar has relion-4 definition of projection matrices; converting them now...
WARNING: tomogram S39-TS_006.tomostar has relion-4 definition of projection matrices; converting them now...
WARNING: tomogram S26-TS_001.tomostar has relion-4 definition of projection matrices; converting them now...
WARNING: tomogram S26-TS_005.tomostar has relion-4 definition of projection matrices; converting them now...
WARNING: tomogram S65-TS_002.tomostar has relion-4 definition of projection matrices; converting them now...
WARNING: tomogram S39-TS_001.tomostar has relion-4 definition of projection matrices; converting them now...
WARNING: tomogram S26-TS_004.tomostar has relion-4 definition of projection matrices; converting them now...
WARNING: tomogram S26-TS_002.tomostar has relion-4 definition of projection matrices; converting them now...
WARNING: tomogram S26-TS_006.tomostar has relion-4 definition of projection matrices; converting them now...
WARNING: tomogram S39-TS_005.tomostar has relion-4 definition of projection matrices; converting them now...
WARNING: tomogram S26-TS_003.tomostar has relion-4 definition of projection matrices; converting them now...
WARNING: tomogram S39-TS_002.tomostar has relion-4 definition of projection matrices; converting them now...
WARNING: tomogram S65-TS_001.tomostar has relion-4 definition of projection matrices; converting them now...
WARNING: tomogram S39-TS_006.tomostar has relion-4 definition of projection matrices; converting them now...
WARNING: tomogram S26-TS_001.tomostar has relion-4 definition of projection matrices; converting them now...
WARNING: tomogram S26-TS_005.tomostar has relion-4 definition of projection matrices; converting them now...
[turing023:2658793] *** Process received signal ***
[turing023:2658793] Signal: Segmentation fault (11)
[turing023:2658793] Signal code: Address not mapped (1)
[turing023:2658793] Failing at address: 0x80
[turing023:2658793] [ 0] /lib/x86_64-linux-gnu/libc.so.6(+0x42520)[0x7efe733c8520]
[turing023:2658793] [ 1] /opt/relion/bin/relion_refine_mpi(+0x4190d0)[0x55c8c15480d0]
[turing023:2658793] [ 2] /opt/relion/bin/relion_refine_mpi(+0x420ca5)[0x55c8c154fca5]
[turing023:2658793] [ 3] /opt/relion/bin/relion_refine_mpi(_ZN15MlOptimiserCuda32doThreadExpectationSomeParticlesEi+0xf6)[0x55c8c1551326]
[turing023:2658793] [ 4] /opt/relion/bin/relion_refine_mpi(_Z36globalThreadExpectationSomeParticlesPvi+0x43)[0x55c8c14d5ed3]
[turing023:2658793] [ 5] /opt/relion/bin/relion_refine_mpi(+0x3a6f5c)[0x55c8c14d5f5c]
[turing023:2658793] [ 6] /lib/x86_64-linux-gnu/libgomp.so.1(GOMP_parallel+0x46)[0x7efe735e2a16]
[turing023:2658793] [ 7] /opt/relion/bin/relion_refine_mpi(_ZN11MlOptimiser24expectationSomeParticlesEll+0xd45)[0x55c8c14b5525]
[turing023:2658793] [ 8] /opt/relion/bin/relion_refine_mpi(_ZN14MlOptimiserMpi11expectationEv+0x1eaa)[0x55c8c1278a2a]
[turing023:2658793] [ 9] /opt/relion/bin/relion_refine_mpi(_ZN14MlOptimiserMpi7iterateEv+0x1d3)[0x55c8c128bd03]
[turing023:2658793] [10] /opt/relion/bin/relion_refine_mpi(main+0x85)[0x55c8c1234565]
[turing023:2658793] [11] /lib/x86_64-linux-gnu/libc.so.6(+0x29d90)[0x7efe733afd90]
[turing023:2658793] [12] /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0x80)[0x7efe733afe40]
[turing023:2658793] [13] /opt/relion/bin/relion_refine_mpi(_start+0x25)[0x55c8c1238005]
[turing023:2658793] *** End of error message ***
--------------------------------------------------------------------------
Primary job terminated normally, but 1 process returned
a non-zero exit code. Per user-direction, the job has been aborted.
--------------------------------------------------------------------------
--------------------------------------------------------------------------
mpirun noticed that process rank 3 with PID 2658663 on node turing023 exited on signal 11 (Segmentation fault).
--------------------------------------------------------------------------
from relion.
Related Issues (20)
- AreTomo executable was not found HOT 3
- relion_tomo_reconstruct_tomogram_mpi requiring more memory than before and then crashing HOT 2
- Proxy error during build HOT 1
- Refine auto 3D error: Helical reconstruction HOT 1
- Denoise Tomograms with continue Flag HOT 1
- Align tilt-series additional arguments HOT 4
- Import Problem: no such option: -optics-group-name
- Relion 5 fails to reconstruct tomograms by asking for _rlnTomoMicrographNameEven label HOT 10
- BUG in RELION 5: Denoise fails with an assert command HOT 4
- Feature request: Color jobs in the GUI HOT 1
- Improve method finding CUDA libraries HOT 2
- corrupted size vs. prev_size
- Error in running blush regularisation in 3D classification HOT 1
- cannot import data with symbolic links HOT 1
- Import successfull but resulting star file contains only single movie for all tiit angles HOT 3
- Relion tomo segmentation fault in Refine3D
- Relion5 Ctf estimation： BUG: MDin should contain a rlnMicrographPreExposure label HOT 4
- AreTomo alignment and tomogram reconstruction faulty HOT 2
- RELION tomogram reconstruction fails (or produces weird contrast) when carbon is in images HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from relion.