Hello there.

I'm installing OpenFoam 8 on the above mentioned HW and am having trouble running tutorials/incompressible/pimpleFoam/laminar/planarPoiseuille. It's running way slower than any other case tested so far (I'm running it as of now and it's already 3 hours of CPU wall time and counting).
Is there something I can do to try to pinpoint the cause of this slowness?

I've installed it through spack from version-8 release tag.
OS: Centos 7.7
CC / CXX: GCC 9.2.0
MPI: Openmpi 3.1.6
SCOTCH: 6.0.8

TIA,
Fabrício.

Summary

OpenFOAM version 8 experiences what appears to be a communication deadlock in a scheduled send\receive operation.

The case in question attempts to solve a toy CFD problem evaluating airflow within a rectangular prism using parallel instances of MPPICFoam on decomposed input.

This behavior occurs on multiple machines. We have tested on multiple workstations and HPC Clusters.

Our research team has prepared a detailed report (attached).

Multiple versions of OpenMPI in terms of both release (e.g. 4.0.3, 2.1.6), compiler (e.g Intel, gcc), bit layer transport (e.g. ucx, openib) in conjunction with multiple builds of OpenFOAM 8, have been attempted. Blocking vs. nonblocking communication and a number of mpirun command line tuning parameters (including varied world sizes) have also been attempted with no resolution.

To determine if the file-system was a factor, the case was run on both local and parallel (GPFS) storage. No difference in runtime behavior was observed when running on local vs. parallel storage.

Additionally, a number of case configuration values (e.g. mesh sizing, simulation times, etc.) without any effect.

For debugging purposes the simulation deltaT was adjusted from 1e-3 to 1.0 which greatly reduces the time to failure.

Based on what we see from our debugging tools, we think we have isolated the issue to the applications/solvers/lagrangian/DPMFoam solver.

Our research team has indicated that the simulation runs successfully when particle injection is disabled.

Some key findings:

1.) The deadlock occurs with send\recv size = 1
2.) The deadlock occurs when the number of iterations = 1. This differs significantly from the previous iteration counts = 1000. This seems to indicate that there may be some miscalculation or an unexpected value passed somewhere.

Compiled_MPPICFoam_Parallel_Troublshooting.pptx

HW Environment

Architecture: x86_64
Processor: 2x Intel(R) Xeon(R) Gold 6248R CPU @ 3.00GHz
Cores \ Node: 48

Software Environment

OS: CentOS Linux release 7.9.2009
Application: OpenFOAM 8
MPI: OpenMPI 4.0.3

MPI Build

.. code-block:: bash

$ ompi_info 

                 Package: Open MPI ... Distribution
                Open MPI: 4.0.3
  Open MPI repo revision: v4.0.3
   Open MPI release date: Mar 03, 2020
                Open RTE: 4.0.3
  Open RTE repo revision: v4.0.3
   Open RTE release date: Mar 03, 2020
                    OPAL: 4.0.3
      OPAL repo revision: v4.0.3
       OPAL release date: Mar 03, 2020
                 MPI API: 3.1.0
            Ident string: 4.0.3
                  Prefix: /tools/openmpi-4.0.3/gcc/4.8.5/ucx
 Configured architecture: x86_64-unknown-linux-gnu
          Configure host: c20-login01
           Configured by: hpcuser
           Configured on: Wed Apr 14 10:40:09 CDT 2021
          Configure host: c20-login01
  Configure command line: '--prefix=/tools/openmpi-4.0.3/gcc/4.8.5/ucx'
                          '--with-slurm'
                Built by: hpcuser
                Built on: Wed Apr 14 10:50:39 CDT 2021
              Built host: c20-login01
              C bindings: yes
            C++ bindings: no
             Fort mpif.h: yes (all)
            Fort use mpi: yes (limited: overloading)
       Fort use mpi size: deprecated-ompi-info-value
        Fort use mpi_f08: no
 Fort mpi_f08 compliance: The mpi_f08 module was not built
  Fort mpi_f08 subarrays: no
           Java bindings: no
  Wrapper compiler rpath: runpath
              C compiler: /usr/bin/gcc
     C compiler absolute: 
  C compiler family name: GNU
      C compiler version: 4.8.5
            C++ compiler: /usr/bin/g++
   C++ compiler absolute: none
           Fort compiler: /usr/bin/gfortran
       Fort compiler abs: 
         Fort ignore TKR: no
   Fort 08 assumed shape: no
      Fort optional args: no
          Fort INTERFACE: yes
    Fort ISO_FORTRAN_ENV: yes
       Fort STORAGE_SIZE: no
      Fort BIND(C) (all): no
      Fort ISO_C_BINDING: yes
 Fort SUBROUTINE BIND(C): no
       Fort TYPE,BIND(C): no
 Fort T,BIND(C,name="a"): no
            Fort PRIVATE: no
          Fort PROTECTED: no
           Fort ABSTRACT: no
       Fort ASYNCHRONOUS: no
          Fort PROCEDURE: no
         Fort USE...ONLY: no
           Fort C_FUNLOC: no
 Fort f08 using wrappers: no
         Fort MPI_SIZEOF: no
             C profiling: yes
           C++ profiling: no
   Fort mpif.h profiling: yes
  Fort use mpi profiling: yes
   Fort use mpi_f08 prof: no
          C++ exceptions: no
          Thread support: posix (MPI_THREAD_MULTIPLE: yes, OPAL support: yes,
                          OMPI progress: no, ORTE progress: yes, Event lib:
                          yes)
           Sparse Groups: no
  Internal debug support: no
  MPI interface warnings: yes
     MPI parameter check: runtime
Memory profiling support: no
Memory debugging support: no
              dl support: yes
   Heterogeneous support: no
 mpirun default --prefix: no
       MPI_WTIME support: native
     Symbol vis. support: yes
   Host topology support: yes
            IPv6 support: no
      MPI1 compatibility: no
          MPI extensions: affinity, cuda, pcollreq
   FT Checkpoint support: no (checkpoint thread: no)
   C/R Enabled Debugging: no
  MPI_MAX_PROCESSOR_NAME: 256
    MPI_MAX_ERROR_STRING: 256
     MPI_MAX_OBJECT_NAME: 64
        MPI_MAX_INFO_KEY: 36
        MPI_MAX_INFO_VAL: 256
       MPI_MAX_PORT_NAME: 1024

OpenFOAM Case

/scratch/hpcadmn/djy0006/case5/B-3-22471/system/decomposeParDict

/--------------------------------- C++ -----------------------------------
========= |
\ / F ield | OpenFOAM: The Open Source CFD Toolbox
\ / O peration | Website: https://openfoam.org
\ / A nd | Version: 8
\/ M anipulation |
*---------------------------------------------------------------------------*/
FoamFile
{
version 2.0;
format ascii;
class dictionary;
location "system";
object decomposeParDict;
}
// * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * //

numberOfSubdomains 3;

method simple;

simpleCoeffs
{
n (3 1 1);
delta 0.001;
}

// ************************************************************************* //

/scratch/hpcadmn/djy0006/case5/B-3-22471/system/controlDict

/--------------------------------- C++ -----------------------------------
========= |
\ / F ield | OpenFOAM: The Open Source CFD Toolbox
\ / O peration | Website: https://openfoam.org
\ / A nd | Version: 8
\/ M anipulation |
*---------------------------------------------------------------------------*/
FoamFile
{
version 2.0;
format ascii;
class dictionary;
location "system";
object controlDict;
}
// * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * //

application MPPICFoam;
startFrom startTime;
startTime 0.0;
stopAt endTime;
endTime 6.5;
deltaT 1.0;
writeControl timeStep;
writeInterval 1;
purgeWrite 0;
writeFormat ascii;
writePrecision 6;
writeCompression off;
timeFormat general;
timePrecision 6;
runTimeModifiable no;

// ************************************************************************* //

OptimisationSwitches {

fileModificationSkew 60;
fileModificationChecking timeStampMaster;
fileHandler uncollated;
maxThreadFileBufferSize 2e9;
maxMasterFileBufferSize 2e9;
commsType       blocking; // nonBlocking; // scheduled; // blocking;
floatTransfer   0;
nProcsSimpleSum 0;
Force dumping (at next timestep) upon signal (-1 to disable)
writeNowSignal             -1; // 10;
stopAtWriteNowSignal       -1;
inputSyntax dot;
mpiBufferSize   200000000;
maxCommsSize    0;
trapFpe         1;
setNaN          0;

}

DebugSwitches {
UPstream 1;
Pstream 1;
processor 1;
IFstream 1;
OFstream 1;
}

Summary of Debug Output

The following debug output was generated using the above case configuration with an MPI_World size of 3...

$ srun -N1 -n3 --pty /bin/bash
...
$ module load openfoam/8-ompi2
$ source /tools/openfoam-8/mpich/OpenFOAM-8/etc/bashrc
$ decomposePar -force
$ mpirun -np $SLURM_NTASKS MPPICFoam -parallel

Following the process tree of ...

$ pstree -ac --show-parents -p -l 54148
systemd,1
└─slurmstepd,262636
└─bash,262643
└─mpirun,54148 -np 3 MPPICFoam -parallel
├─MPPICFoam,54152 -parallel
│ ├─{MPPICFoam},
│ ├─{MPPICFoam},
│ └─{MPPICFoam},
├─MPPICFoam,541523 -parallel
│ ├─{MPPICFoam},
│ ├─{MPPICFoam},
│ └─{MPPICFoam},
├─MPPICFoam,54154 -parallel
│ ├─{MPPICFoam},
│ ├─{MPPICFoam},
│ └─{MPPICFoam},
├─{mpirun},
├─{mpirun},
└─{mpirun},

The case output at the time of failure looks like ...

[0] UPstream::waitRequests : starting wait for 0 outstanding requests starting at 0
[0] UPstream::waitRequests : finished wait.
[0] UIPstream::read : starting read from:1 tag:1 comm:0 wanted size:1 commsType:scheduled
[0] UIPstream::read : finished read from:1 tag:1 read size:1 commsType:scheduled
[0] UIPstream::read : starting read from:2 tag:1 comm:0 wanted size:1 commsType:scheduled
[0] UIPstream::read : finished read from:2 tag:1 read size:1 commsType:scheduled
[0] UOPstream::write : starting write to:2 tag:1 comm:0 size:1 commsType:scheduled
[0] UOPstream::write : finished write to:2 tag:1 size:1 commsType:scheduled
[0] UOPstream::write : starting write to:1 tag:1 comm:0 size:1 commsType:scheduled
[0] UOPstream::write : finished write to:1 tag:1 size:1 commsType:scheduled
[2] UPstream::waitRequests : starting wait for 0 outstanding requests starting at 0
[2] UPstream::waitRequests : finished wait.
[2] UOPstream::write : starting write to:0 tag:1 comm:0 size:1 commsType:scheduled
[2] UOPstream::write : finished write to:0 tag:1 size:1 commsType:scheduled
[2] UIPstream::read : starting read from:0 tag:1 comm:0 wanted size:1 commsType:scheduled
[2] UIPstream::read : finished read from:0 tag:1 read size:1 commsType:scheduled
[1] UPstream::waitRequests : starting wait for 0 outstanding requests starting at 0
[1] UPstream::waitRequests : finished wait.
[1] UOPstream::write : starting write to:0 tag:1 comm:0 size:1 commsType:scheduled
[1] UOPstream::write : finished write to:0 tag:1 size:1 commsType:scheduled
[1] UIPstream::read : starting read from:0 tag:1 comm:0 wanted size:1 commsType:scheduled
[1] UIPstream::read : finished read from:0 tag:1 read size:1 commsType:scheduled

<... freeze ...>

Here, the communication schedule seems to be balanced with all matching send and receives (based on size and tag). However, the behavior indicates a blocking send or receive call.

The deadlock always seems to occur for size=1 send\recv operations.

*********** The remaining content consists of strace\gdb output from the MPI ranks. ***************

In the root mpirun process (54152) looks like it is stuck in a poll loop.

Rank 0 appears to be issuing a return from Foam::BarycentricTensor::BarycentricTensor().

Ranks 1 and 2 appear to be waiting on PMPI_Alltoall communication.

GDB

Root (MPI) Process

[node040 B-3-22471]$ mpirun -np $SLURM_NTASKS MPPICFoam -parallel > /dev/null 2>&1 &
[1] 54148

[node040 B-3-22471]$ ps -ef | grep hpcuser
hpcuser 47219 47212 0 09:08 pts/0 00:00:00 /bin/bash
hpcuser 54148 47219 1 09:54 pts/0 00:00:00 mpirun -np 3 MPPICFoam -parallel
hpcuser 54152 54148 72 09:54 pts/0 00:00:02 MPPICFoam -parallel
hpcuser 54153 54148 81 09:54 pts/0 00:00:02 MPPICFoam -parallel
hpcuser 54154 54148 81 09:54 pts/0 00:00:02 MPPICFoam -parallel
hpcuser 54166 47219 0 09:54 pts/0 00:00:00 ps -ef
hpcuser 54167 47219 0 09:54 pts/0 00:00:00 grep --color=auto hpcuser

[node040 B-3-22471]$ gdb -p 54148
GNU gdb (GDB) Red Hat Enterprise Linux 7.6.1-120.el7

(gdb) frame
#0 0x00002aaaac17fccd in poll () from /usr/lib64/libc.so.6
(gdb) where
#0 0x00002aaaac17fccd in poll () from /usr/lib64/libc.so.6
#1 0x00002aaaab096fc6 in poll_dispatch (base=0x659370, tv=0x0) at ../../../../../../../openmpi-4.0.3/opal/mca/event/libevent2022/libevent/poll.c:165
#2 0x00002aaaab08ec80 in opal_libevent2022_event_base_loop (base=0x659370, flags=1) at ../../../../../../../openmpi-4.0.3/opal/mca/event/libevent2022/libevent/event.c:1630
#3 0x0000000000401438 in orterun (argc=5, argv=0x7fffffffaae8) at ../../../../../openmpi-4.0.3/orte/tools/orterun/orterun.c:178
#4 0x0000000000400f6d in main (argc=5, argv=0x7fffffffaae8) at ../../../../../openmpi-4.0.3/orte/tools/orterun/main.c:13
(gdb) n
Single stepping until exit from function poll,
which has no line number information.
< ... freeze ... >

(gdb) disassemble
Dump of assembler code for function poll:
0x00002aaaac0ceca0 <+0>: cmpl $0x0,0x2d930d(%rip) # 0x2aaaac3a7fb4 <__libc_multiple_threads>
0x00002aaaac0ceca7 <+7>: jne 0x2aaaac0cecb9 <poll+25>
0x00002aaaac0ceca9 <+0>: mov $0x7,%eax
0x00002aaaac0cecae <+5>: syscall
0x00002aaaac0cecb0 <+7>: cmp $0xfffffffffffff001,%rax
0x00002aaaac0cecb6 <+13>: jae 0x2aaaac0cece9 <poll+73>
0x00002aaaac0cecb8 <+15>: retq
0x00002aaaac0cecb9 <+25>: sub $0x8,%rsp
0x00002aaaac0cecbd <+29>: callq 0x2aaaac0e7720 <__libc_enable_asynccancel>
0x00002aaaac0cecc2 <+34>: mov %rax,(%rsp)
0x00002aaaac0cecc6 <+38>: mov $0x7,%eax
0x00002aaaac0ceccb <+43>: syscall
=> 0x00002aaaac0ceccd <+45>: mov (%rsp),%rdi
0x00002aaaac0cecd1 <+49>: mov %rax,%rdx
0x00002aaaac0cecd4 <+52>: callq 0x2aaaac0e7780 <__libc_disable_asynccancel>
0x00002aaaac0cecd9 <+57>: mov %rdx,%rax
0x00002aaaac0cecdc <+60>: add $0x8,%rsp
0x00002aaaac0cece0 <+64>: cmp $0xfffffffffffff001,%rax
0x00002aaaac0cece6 <+70>: jae 0x2aaaac0cece9 <poll+73>
0x00002aaaac0cece8 <+72>: retq
0x00002aaaac0cece9 <+73>: mov 0x2d3160(%rip),%rcx # 0x2aaaac3a1e50
0x00002aaaac0cecf0 <+80>: neg %eax
0x00002aaaac0cecf2 <+82>: mov %eax,%fs:(%rcx)
0x00002aaaac0cecf5 <+85>: or $0xffffffffffffffff,%rax
0x00002aaaac0cecf9 <+89>: retq
End of assembler dump.

Rank 0 Process

[node040 B-3-22471]$ gdb -p 54152

GNU gdb (GDB) Red Hat Enterprise Linux 7.6.1-120.el7
Attaching to process 54152
Reading symbols from /mmfs1/tools/openfoam-8/debug/OpenFOAM-8/platforms/linux64GccDPInt32Debug/bin/MPPICFoam...done.
Reading symbols from /mmfs1/tools/openfoam-8/debug/OpenFOAM-8/platforms/linux64GccDPInt32Debug/lib/liblagrangian.so...done.
Loaded symbols for /mmfs1/tools/openfoam-8/debug/OpenFOAM-8/platforms/linux64GccDPInt32Debug/lib/liblagrangian.so
Reading symbols from /mmfs1/tools/openfoam-8/debug/OpenFOAM-8/platforms/linux64GccDPInt32Debug/lib/liblagrangianIntermediate.so...done.
Loaded symbols for /mmfs1/tools/openfoam-8/debug/OpenFOAM-8/platforms/linux64GccDPInt32Debug/lib/liblagrangianIntermediate.so
Reading symbols from /mmfs1/tools/openfoam-8/debug/OpenFOAM-8/platforms/linux64GccDPInt32Debug/lib/liblagrangianTurbulence.so...done.
Loaded symbols for /mmfs1/tools/openfoam-8/debug/OpenFOAM-8/platforms/linux64GccDPInt32Debug/lib/liblagrangianTurbulence.so
Reading symbols from /mmfs1/tools/openfoam-8/debug/OpenFOAM-8/platforms/linux64GccDPInt32Debug/lib/libincompressibleTransportModels.so...done.
Loaded symbols for /mmfs1/tools/openfoam-8/debug/OpenFOAM-8/platforms/linux64GccDPInt32Debug/lib/libincompressibleTransportModels.so
...
(gdb) disassemble
Dump of assembler code for function Foam::BarycentricTensor::BarycentricTensor(Foam::Vector const&, Foam::Vector const&, Foam::Vector const&, Foam::Vector const&):
0x00000000004aa9fc <+0>: push %rbp
0x00000000004aa9fd <+1>: mov %rsp,%rbp
0x00000000004aaa00 <+4>: sub $0x30,%rsp
0x00000000004aaa04 <+8>: mov %rdi,-0x8(%rbp)
=> 0x00000000004aaa08 <+12>: mov %rsi,-0x10(%rbp)
0x00000000004aaa0c <+16>: mov %rdx,-0x18(%rbp)
0x00000000004aaa10 <+20>: mov %rcx,-0x20(%rbp)
0x00000000004aaa14 <+24>: mov %r8,-0x28(%rbp)
0x00000000004aaa18 <+28>: mov -0x8(%rbp),%rax
0x00000000004aaa1c <+32>: mov %rax,%rdi
0x00000000004aaa1f <+35>: callq 0x4b9302 <Foam::MatrixSpace<Foam::BarycentricTensor, double, (unsigned char)4, (unsigned char)3>::MatrixSpace()>
0x00000000004aaa24 <+40>: mov -0x10(%rbp),%rax
0x00000000004aaa28 <+44>: mov %rax,%rdi
0x00000000004aaa2b <+47>: callq 0x4a7af8 <Foam::Vector::x() const>
0x00000000004aaa30 <+52>: mov (%rax),%rax
0x00000000004aaa33 <+55>: mov -0x8(%rbp),%rdx
0x00000000004aaa37 <+59>: mov %rax,(%rdx)
0x00000000004aaa3a <+62>: mov -0x18(%rbp),%rax
0x00000000004aaa3e <+66>: mov %rax,%rdi
0x00000000004aaa41 <+69>: callq 0x4a7af8 <Foam::Vector::x() const>
0x00000000004aaa46 <+74>: mov (%rax),%rax
0x00000000004aaa49 <+77>: mov -0x8(%rbp),%rdx
0x00000000004aaa4d <+81>: mov %rax,0x8(%rdx)
0x00000000004aaa51 <+85>: mov -0x20(%rbp),%rax
0x00000000004aaa55 <+89>: mov %rax,%rdi
0x00000000004aaa58 <+92>: callq 0x4a7af8 <Foam::Vector::x() const>
0x00000000004aaa5d <+97>: mov (%rax),%rax
0x00000000004aaa60 <+100>: mov -0x8(%rbp),%rdx
0x00000000004aaa64 <+104>: mov %rax,0x10(%rdx)
0x00000000004aaa68 <+108>: mov -0x28(%rbp),%rax
0x00000000004aaa6c <+112>: mov %rax,%rdi
0x00000000004aaa6f <+115>: callq 0x4a7af8 <Foam::Vector::x() const>
0x00000000004aaa74 <+120>: mov (%rax),%rax
0x00000000004aaa77 <+123>: mov -0x8(%rbp),%rdx
0x00000000004aaa7b <+127>: mov %rax,0x18(%rdx)
0x00000000004aaa7f <+131>: mov -0x10(%rbp),%rax
0x00000000004aaa83 <+135>: mov %rax,%rdi
0x00000000004aaa86 <+138>: callq 0x4a7b06 <Foam::Vector::y() const>
0x00000000004aaa8b <+143>: mov (%rax),%rax
0x00000000004aaa8e <+146>: mov -0x8(%rbp),%rdx

(gdb) frame
#0 0x00000000004a7c6d in Foam::operator^ (v1=..., v2=...) at /mmfs1/tools/openfoam-8/debug/OpenFOAM-8/src/OpenFOAM/lnInclude/VectorI.H:159

(gdb) frame
#0 0x00000000004a55a2 in Foam::tetIndices::faceTriIs (this=0x7fffffff4b10, mesh=...) at /mmfs1/tools/openfoam-8/debug/OpenFOAM-8/src/OpenFOAM/lnInclude/tetIndicesI.H:86
86 label facePtI = (tetPt() + faceBasePtI) % f.size();

#0 0x00002aaaaacfd654 in Foam::BarycentricTensor::d (this=0x7fffffff4620) at /mmfs1/tools/openfoam-8/debug/OpenFOAM-8/src/OpenFOAM/lnInclude/BarycentricTensorI.H:159
159 return Vector(this->v_[XD], this->v_[YD], this->v_[ZD]);

(gdb) where
#0 0x00000000004ceebd in Foam::Barycentric::Barycentric (this=0x7fffffff4be0, va=@0x7fffffff4cc0: -0.13335, vb=@0x7fffffff4cc8: -0.13716, vc=@0x7fffffff4cd0: -0.13716,
vd=@0x7fffffff4cd8: -0.12953999999999999) at /mmfs1/tools/openfoam-8/debug/OpenFOAM-8/src/OpenFOAM/lnInclude/BarycentricI.H:50
#1 0x00000000004b97f5 in Foam::BarycentricTensor::z (this=0x7fffffff4c80) at /mmfs1/tools/openfoam-8/debug/OpenFOAM-8/src/OpenFOAM/lnInclude/BarycentricTensorI.H:131
#2 0x00000000004aae13 in Foam::operator& (T=..., b=...) at /mmfs1/tools/openfoam-8/debug/OpenFOAM-8/src/OpenFOAM/lnInclude/BarycentricTensorI.H:177
#3 0x00000000004a6a1e in Foam::particle::position (this=0x2240a40) at /mmfs1/tools/openfoam-8/debug/OpenFOAM-8/src/lagrangian/basic/lnInclude/particleI.H:280
#4 0x00002aaaaacf837d in Foam::particle::deviationFromMeshCentre (this=0x2240a40) at particle/particle.C:1036
#5 0x000000000051ac3a in Foam::KinematicParcelFoam::particle::move<Foam::MPPICCloud<Foam::KinematicCloud<Foam::Cloud<Foam::MPPICParcel<Foam::KinematicParcelFoam::particle > > > > > (
this=0x2240a40, cloud=..., td=..., trackTime=1) at /mmfs1/tools/openfoam-8/debug/OpenFOAM-8/src/lagrangian/intermediate/lnInclude/KinematicParcel.C:309
#6 0x000000000050acc2 in Foam::MPPICParcel<Foam::KinematicParcelFoam::particle >::move<Foam::MPPICCloud<Foam::KinematicCloud<Foam::Cloud<Foam::MPPICParcel<Foam::KinematicParcelFoam::particle > > > > > (this=0x2240a40, cloud=..., td=..., trackTime=1) at /mmfs1/tools/openfoam-8/debug/OpenFOAM-8/src/lagrangian/intermediate/lnInclude/MPPICParcel.C:102
#7 0x00000000004f22f3 in Foam::Cloud<Foam::MPPICParcel<Foam::KinematicParcelFoam::particle > >::move<Foam::MPPICCloud<Foam::KinematicCloud<Foam::Cloud<Foam::MPPICParcel<Foam::KinematicParcelFoam::particle > > > > > (this=0x7fffffff7220, cloud=..., td=..., trackTime=1) at /mmfs1/tools/openfoam-8/debug/OpenFOAM-8/src/lagrangian/basic/lnInclude/Cloud.C:205
#8 0x00000000004f1e18 in Foam::MPPICCloud<Foam::KinematicCloud<Foam::Cloud<Foam::MPPICParcel<Foam::KinematicParcelFoam::particle > > > >::motion<Foam::MPPICCloud<Foam::KinematicCloud<Foam::Cloud<Foam::MPPICParcel<Foam::KinematicParcelFoam::particle > > > > > (this=0x7fffffff7220, cloud=..., td=...)
at /mmfs1/tools/openfoam-8/debug/OpenFOAM-8/src/lagrangian/intermediate/lnInclude/MPPICCloud.C:247
#9 0x00000000004da066 in Foam::KinematicCloud<Foam::Cloud<Foam::MPPICParcel<Foam::KinematicParcelFoam::particle > > >::evolveCloud<Foam::MPPICCloud<Foam::KinematicCloud<Foam::Cloud<Foam::MPPICParcel<Foam::KinematicParcelFoam::particle > > > > > (this=0x7fffffff7220, cloud=..., td=...)
at /mmfs1/tools/openfoam-8/debug/OpenFOAM-8/src/lagrangian/intermediate/lnInclude/KinematicCloud.C:210
#10 0x00000000004c3497 in Foam::KinematicCloud<Foam::Cloud<Foam::MPPICParcel<Foam::KinematicParcelFoam::particle > > >::solve<Foam::MPPICCloud<Foam::KinematicCloud<Foam::Cloud<Foam::MPPICParcel<Foam::KinematicParcelFoam::particle > > > > > (this=0x7fffffff7220, cloud=..., td=...) at /mmfs1/tools/openfoam-8/debug/OpenFOAM-8/src/lagrangian/intermediate/lnInclude/KinematicCloud.C:114
#11 0x00000000004afc73 in Foam::MPPICCloud<Foam::KinematicCloud<Foam::Cloud<Foam::MPPICParcel<Foam::KinematicParcelFoam::particle > > > >::evolve (this=0x7fffffff7220)
at /mmfs1/tools/openfoam-8/debug/OpenFOAM-8/src/lagrangian/intermediate/lnInclude/MPPICCloud.C:169
#12 0x000000000049e61e in main (argc=2, argv=0x7fffffffa258) at ../DPMFoam.C:109

Rank 1 Process

[node040 B-3-22471]$ gdb -p 54153

(gdb) disassemble
Dump of assembler code for function uct_rc_mlx5_iface_progress_cyclic:
0x00002aaaca088020 <+0>: push %r15
0x00002aaaca088022 <+2>: push %r14
0x00002aaaca088024 <+4>: push %r13
0x00002aaaca088026 <+6>: push %r12
0x00002aaaca088028 <+8>: push %rbp
0x00002aaaca088029 <+9>: push %rbx
0x00002aaaca08802a <+10>: mov %rdi,%rbx
0x00002aaaca08802d <+13>: sub $0x38,%rsp
0x00002aaaca088031 <+17>: movzwl 0x86c8(%rdi),%edx
0x00002aaaca088038 <+24>: movzwl 0x86c6(%rdi),%eax
0x00002aaaca08803f <+31>: imul %edx,%eax
0x00002aaaca088042 <+34>: mov 0x86b0(%rdi),%rdx
0x00002aaaca088049 <+41>: cltq
0x00002aaaca08804b <+43>: cmpw $0x0,0x2(%rdx,%rax,1)
=> 0x00002aaaca088051 <+49>: jne 0x2aaaca0887cd <uct_rc_mlx5_iface_progress_cyclic+1965>
0x00002aaaca088057 <+55>: mov 0x86f0(%rdi),%rax
0x00002aaaca08805e <+62>: mov 0x8708(%rdi),%edx
0x00002aaaca088064 <+68>: mov 0x870c(%rdi),%ecx
0x00002aaaca08806a <+74>: prefetcht0 (%rax)
0x00002aaaca08806d <+77>: mov 0x8700(%rdi),%eax
0x00002aaaca088073 <+83>: lea -0x1(%rdx),%ebp
0x00002aaaca088076 <+86>: and %eax,%ebp
0x00002aaaca088078 <+88>: shl %cl,%ebp
0x00002aaaca08807a <+90>: add 0x86f8(%rdi),%rbp
0x00002aaaca088081 <+97>: movzbl 0x3f(%rbp),%ecx
0x00002aaaca088085 <+101>: mov %ecx,%esi
0x00002aaaca088087 <+103>: and $0x1,%esi
0x00002aaaca08808a <+106>: test %eax,%edx
0x00002aaaca08808c <+108>: setne %dl
0x00002aaaca08808f <+111>: cmp %sil,%dl
0x00002aaaca088092 <+114>: jne 0x2aaaca088686 <uct_rc_mlx5_iface_progress_cyclic+1638>
0x00002aaaca088098 <+120>: test %cl,%cl
0x00002aaaca08809a <+122>: js 0x2aaaca088679 <uct_rc_mlx5_iface_progress_cyclic+1625>
0x00002aaaca0880a0 <+128>: add $0x1,%eax
0x00002aaaca0880a3 <+131>: mov %eax,0x8700(%rdi)
0x00002aaaca0880a9 <+137>: movzwl 0x3c(%rbp),%r13d
0x00002aaaca0880ae <+142>: movzwl 0x86c6(%rdi),%edi
0x00002aaaca0880b5 <+149>: movzwl 0x86c8(%rbx),%ecx
0x00002aaaca0880bc <+156>: mov 0x86b0(%rbx),%rdx
0x00002aaaca0880c3 <+163>: mov 0x2c(%rbp),%r12d
0x00002aaaca0880c7 <+167>: mov %r13d,%eax
0x00002aaaca0880ca <+170>: ror $0x8,%ax

(gdb) frame
#0 0x00002aaac936812d in uct_mm_iface_progress (tl_iface=) at ../../../src/uct/sm/mm/base/mm_iface.c:365

(gdb) frame
#0 0x00002aaaca08848a in uct_rc_mlx5_iface_progress_cyclic (arg=) at ../../../../src/uct/ib/rc/accel/rc_mlx5_iface.c:183
183 }

(gdb) where
#0 0x00002aaaca088484 in uct_rc_mlx5_iface_progress_cyclic (arg=) at ../../../../src/uct/ib/rc/accel/rc_mlx5_iface.c:183
#1 0x00002aaac90b608a in ucs_callbackq_dispatch (cbq=) at /home/hpcuser/build/ucx/build/../src/ucs/datastruct/callbackq.h:211
#2 uct_worker_progress (worker=) at /home/hpcuser/build/ucx/build/../src/uct/api/uct.h:2592
#3 ucp_worker_progress (worker=0xb9f390) at ../../../src/ucp/core/ucp_worker.c:2530
#4 0x00002aaac8c6c6d7 in mca_pml_ucx_progress () from /tools/openmpi-4.0.3/gcc/4.8.5/ucx/lib/openmpi/mca_pml_ucx.so
#5 0x00002aaab91c780c in opal_progress () from /tools/openmpi-4.0.3/gcc/4.8.5/ucx/lib/libopen-pal.so.40
#6 0x00002aaab85111bd in ompi_request_default_wait_all () from /tools/openmpi-4.0.3/gcc/4.8.5/ucx/lib/libmpi.so.40
#7 0x00002aaab8565398 in ompi_coll_base_alltoall_intra_basic_linear () from /tools/openmpi-4.0.3/gcc/4.8.5/ucx/lib/libmpi.so.40
#8 0x00002aaab85240d7 in PMPI_Alltoall () from /tools/openmpi-4.0.3/gcc/4.8.5/ucx/lib/libmpi.so.40
#9 0x00002aaab2569953 in Foam::UPstream::allToAll (sendData=..., recvData=..., communicator=0) at UPstream.C:367
#10 0x00002aaab0a162c1 in Foam::Pstream::exchangeSizes<Foam::List<Foam::DynamicList<char, 0u, 2u, 1u> > > (sendBufs=..., recvSizes=..., comm=0) at db/IOstreams/Pstreams/exchange.C:158
#11 0x00002aaab0a15d0d in Foam::PstreamBuffers::finishedSends (this=0x7fffffff4fe0, recvSizes=..., block=true) at db/IOstreams/Pstreams/PstreamBuffers.C:106
#12 0x00000000004f2670 in Foam::Cloud<Foam::MPPICParcel<Foam::KinematicParcelFoam::particle > >::move<Foam::MPPICCloud<Foam::KinematicCloud<Foam::Cloud<Foam::MPPICParcel<Foam::KinematicParcelFoam::particle > > > > > (this=0x7fffffff7220, cloud=..., td=..., trackTime=1) at /mmfs1/tools/openfoam-8/debug/OpenFOAM-8/src/lagrangian/basic/lnInclude/Cloud.C:283
#13 0x00000000004f1e18 in Foam::MPPICCloud<Foam::KinematicCloud<Foam::Cloud<Foam::MPPICParcel<Foam::KinematicParcelFoam::particle > > > >::motion<Foam::MPPICCloud<Foam::KinematicCloud<Foam::Cloud<Foam::MPPICParcel<Foam::KinematicParcelFoam::particle > > > > > (this=0x7fffffff7220, cloud=..., td=...)
at /mmfs1/tools/openfoam-8/debug/OpenFOAM-8/src/lagrangian/intermediate/lnInclude/MPPICCloud.C:247
#14 0x00000000004da066 in Foam::KinematicCloud<Foam::Cloud<Foam::MPPICParcel<Foam::KinematicParcelFoam::particle > > >::evolveCloud<Foam::MPPICCloud<Foam::KinematicCloud<Foam::Cloud<Foam::MPPICParcel<Foam::KinematicParcelFoam::particle > > > > > (this=0x7fffffff7220, cloud=..., td=...)
at /mmfs1/tools/openfoam-8/debug/OpenFOAM-8/src/lagrangian/intermediate/lnInclude/KinematicCloud.C:210
#15 0x00000000004c3497 in Foam::KinematicCloud<Foam::Cloud<Foam::MPPICParcel<Foam::KinematicParcelFoam::particle > > >::solve<Foam::MPPICCloud<Foam::KinematicCloud<Foam::Cloud<Foam::MPPICParcel<Foam::KinematicParcelFoam::particle > > > > > (this=0x7fffffff7220, cloud=..., td=...) at /mmfs1/tools/openfoam-8/debug/OpenFOAM-8/src/lagrangian/intermediate/lnInclude/KinematicCloud.C:114
#16 0x00000000004afc73 in Foam::MPPICCloud<Foam::KinematicCloud<Foam::Cloud<Foam::MPPICParcel<Foam::KinematicParcelFoam::particle > > > >::evolve (this=0x7fffffff7220)
at /mmfs1/tools/openfoam-8/debug/OpenFOAM-8/src/lagrangian/intermediate/lnInclude/MPPICCloud.C:169
#17 0x000000000049e61e in main (argc=2, argv=0x7fffffffa258) at ../DPMFoam.C:109

Rank 2 Process

[node040 B-3-22471]$ gdb -p 54154

(gdb) disassemble
Dump of assembler code for function uct_mm_iface_progress:
0x00002aaac9367f80 <+0>: push %r15
0x00002aaac9367f82 <+2>: push %r14
0x00002aaac9367f84 <+4>: push %r13
0x00002aaac9367f86 <+6>: push %r12
0x00002aaac9367f88 <+8>: push %rbp
0x00002aaac9367f89 <+9>: push %rbx
0x00002aaac9367f8a <+10>: mov %rdi,%rbx
0x00002aaac9367f8d <+13>: sub $0x168,%rsp
0x00002aaac9367f94 <+20>: mov 0x558(%rdi),%esi
0x00002aaac9367f9a <+26>: movl $0x0,0x50(%rsp)
0x00002aaac9367fa2 <+34>: test %esi,%esi
0x00002aaac9367fa4 <+36>: je 0x2aaac9368527 <uct_mm_iface_progress+1447>
0x00002aaac9367faa <+42>: lea 0x598(%rbx),%r14
0x00002aaac9367fb1 <+49>: lea 0x560(%rbx),%r13
0x00002aaac9367fb8 <+56>: lea 0x60(%rsp),%r12
0x00002aaac9367fbd <+61>: mov 0x538(%rdi),%rdx
0x00002aaac9367fc4 <+68>: movabs $0x7fffffffffffffff,%rbp
0x00002aaac9367fce <+78>: xor %edi,%edi
0x00002aaac9367fd0 <+80>: movzbl 0x548(%rbx),%ecx
0x00002aaac9367fd7 <+87>: mov 0x540(%rbx),%rax
0x00002aaac9367fde <+94>: movzbl (%rdx),%edx
0x00002aaac9367fe1 <+97>: shr %cl,%rax
=> 0x00002aaac9367fe4 <+100>: xor %rdx,%rax
0x00002aaac9367fe7 <+103>: and $0x1,%eax
0x00002aaac9367fea <+106>: jne 0x2aaac9368500 <uct_mm_iface_progress+1408>
0x00002aaac9367ff0 <+112>: mov 0x528(%rbx),%rdx
0x00002aaac9367ff7 <+119>: mov (%rdx),%rdx
0x00002aaac9367ffa <+122>: and %rbp,%rdx
0x00002aaac9367ffd <+125>: cmp %rdx,0x540(%rbx)
0x00002aaac9368004 <+132>: ja 0x2aaac93682c8 <uct_mm_iface_progress+840>
0x00002aaac936800a <+138>: mov 0x538(%rbx),%r10
0x00002aaac9368011 <+145>: testb $0x2,(%r10)
0x00002aaac9368015 <+149>: je 0x2aaac93681ef <uct_mm_iface_progress+623>
0x00002aaac936801b <+155>: mov 0x224f36(%rip),%r15 # 0x2aaac958cf58
0x00002aaac9368022 <+162>: cmpl $0x7,(%r15)
0x00002aaac9368026 <+166>: ja 0x2aaac9368198 <uct_mm_iface_progress+536>
0x00002aaac936802c <+172>: lea 0x1c(%r10),%rsi
0x00002aaac9368030 <+176>: movzbl 0x1(%r10),%r9d
0x00002aaac9368035 <+181>: movzwl 0x2(%r10),%edx
0x00002aaac936803a <+186>: cmp $0x1f,%r9b
0x00002aaac936803e <+190>: ja 0x2aaac9368164 <uct_mm_iface_progress+484>
0x00002aaac9368044 <+196>: movzbl %r9b,%r9d

(gdb) frame
#0 0x00002aaaca08823b in uct_ib_mlx5_get_cqe (cqe_index=51, cq=0xbf0778) at /home/hpcuser/build/ucx/build/../src/uct/ib/mlx5/ib_mlx5.inl:13
13 return UCS_PTR_BYTE_OFFSET(cq->cq_buf, ((cqe_index & (cq->cq_length - 1)) <<

(gdb) where
#0 0x00002aaaca08823b in uct_ib_mlx5_get_cqe (cqe_index=51, cq=0xbf0778) at /home/hpcuser/build/ucx/build/../src/uct/ib/mlx5/ib_mlx5.inl:13
#1 uct_ib_mlx5_poll_cq (cq=0xbf0778, iface=0xbe8050) at /home/hpcuser/build/ucx/build/../src/uct/ib/mlx5/ib_mlx5.inl:73
#2 uct_rc_mlx5_iface_poll_tx (iface=0xbe8050) at ../../../../src/uct/ib/rc/accel/rc_mlx5_iface.c:140
#3 uct_rc_mlx5_iface_progress (flags=2, arg=0xbe8050) at ../../../../src/uct/ib/rc/accel/rc_mlx5_iface.c:177
#4 uct_rc_mlx5_iface_progress_cyclic (arg=0xbe8050) at ../../../../src/uct/ib/rc/accel/rc_mlx5_iface.c:182
#5 0x00002aaac90b608a in ucs_callbackq_dispatch (cbq=) at /home/hpcuser/build/ucx/build/../src/ucs/datastruct/callbackq.h:211
#6 uct_worker_progress (worker=) at /home/hpcuser/build/ucx/build/../src/uct/api/uct.h:2592
#7 ucp_worker_progress (worker=0xb9f3d0) at ../../../src/ucp/core/ucp_worker.c:2530
#8 0x00002aaac8c6c6d7 in mca_pml_ucx_progress () from /tools/openmpi-4.0.3/gcc/4.8.5/ucx/lib/openmpi/mca_pml_ucx.so
#9 0x00002aaab91c780c in opal_progress () from /tools/openmpi-4.0.3/gcc/4.8.5/ucx/lib/libopen-pal.so.40
#10 0x00002aaab85111bd in ompi_request_default_wait_all () from /tools/openmpi-4.0.3/gcc/4.8.5/ucx/lib/libmpi.so.40
#11 0x00002aaab8565398 in ompi_coll_base_alltoall_intra_basic_linear () from /tools/openmpi-4.0.3/gcc/4.8.5/ucx/lib/libmpi.so.40
#12 0x00002aaab85240d7 in PMPI_Alltoall () from /tools/openmpi-4.0.3/gcc/4.8.5/ucx/lib/libmpi.so.40
#13 0x00002aaab2569953 in Foam::UPstream::allToAll (sendData=..., recvData=..., communicator=0) at UPstream.C:367
#14 0x00002aaab0a162c1 in Foam::Pstream::exchangeSizes<Foam::List<Foam::DynamicList<char, 0u, 2u, 1u> > > (sendBufs=..., recvSizes=..., comm=0) at db/IOstreams/Pstreams/exchange.C:158
#15 0x00002aaab0a15d0d in Foam::PstreamBuffers::finishedSends (this=0x7fffffff4fe0, recvSizes=..., block=true) at db/IOstreams/Pstreams/PstreamBuffers.C:106
#16 0x00000000004f2670 in Foam::Cloud<Foam::MPPICParcel<Foam::KinematicParcelFoam::particle > >::move<Foam::MPPICCloud<Foam::KinematicCloud<Foam::Cloud<Foam::MPPICParcel<Foam::KinematicParcelFoam::particle > > > > > (this=0x7fffffff7220, cloud=..., td=..., trackTime=1) at /mmfs1/tools/openfoam-8/debug/OpenFOAM-8/src/lagrangian/basic/lnInclude/Cloud.C:283
#17 0x00000000004f1e18 in Foam::MPPICCloud<Foam::KinematicCloud<Foam::Cloud<Foam::MPPICParcel<Foam::KinematicParcelFoam::particle > > > >::motion<Foam::MPPICCloud<Foam::KinematicCloud<Foam::Cloud<Foam::MPPICParcel<Foam::KinematicParcelFoam::particle > > > > > (this=0x7fffffff7220, cloud=..., td=...)
at /mmfs1/tools/openfoam-8/debug/OpenFOAM-8/src/lagrangian/intermediate/lnInclude/MPPICCloud.C:247
#18 0x00000000004da066 in Foam::KinematicCloud<Foam::Cloud<Foam::MPPICParcel<Foam::KinematicParcelFoam::particle > > >::evolveCloud<Foam::MPPICCloud<Foam::KinematicCloud<Foam::Cloud<Foam::MPPICParcel<Foam::KinematicParcelFoam::particle > > > > > (this=0x7fffffff7220, cloud=..., td=...)
at /mmfs1/tools/openfoam-8/debug/OpenFOAM-8/src/lagrangian/intermediate/lnInclude/KinematicCloud.C:210
#19 0x00000000004c3497 in Foam::KinematicCloud<Foam::Cloud<Foam::MPPICParcel<Foam::KinematicParcelFoam::particle > > >::solve<Foam::MPPICCloud<Foam::KinematicCloud<Foam::Cloud<Foam::MPPICParcel<Foam::KinematicParcelFoam::particle > > > > > (this=0x7fffffff7220, cloud=..., td=...) at /mmfs1/tools/openfoam-8/debug/OpenFOAM-8/src/lagrangian/intermediate/lnInclude/KinematicCloud.C:114
#20 0x00000000004afc73 in Foam::MPPICCloud<Foam::KinematicCloud<Foam::Cloud<Foam::MPPICParcel<Foam::KinematicParcelFoam::particle > > > >::evolve (this=0x7fffffff7220)
at /mmfs1/tools/openfoam-8/debug/OpenFOAM-8/src/lagrangian/intermediate/lnInclude/MPPICCloud.C:169
#21 0x000000000049e61e in main (argc=2, argv=0x7fffffffa258) at ../DPMFoam.C:109

strace

Root MPI

[node040 B-3-22471]$ strace -ff -p 54148
strace: Process 54148 attached with 4 threads
[pid 54151] select(19, [17 18], NULL, NULL, {tv_sec=3516, tv_usec=700411} <unfinished ...>
[pid 54150] select(16, [14 15], NULL, NULL, {tv_sec=0, tv_usec=963883} <unfinished ...>
[pid 54149] epoll_wait(10, <unfinished ...>
[pid 54148] restart_syscall(<... resuming interrupted restart_syscall ...> <unfinished ...>
[pid 54150] <... select resumed>) = 0 (Timeout)
[pid 54150] select(16, [14 15], NULL, NULL, {tv_sec=2, tv_usec=0}) = 0 (Timeout)
[pid 54150] select(16, [14 15], NULL, NULL, {tv_sec=2, tv_usec=0}) = 0 (Timeout)
[pid 54150] select(16, [14 15], NULL, NULL, {tv_sec=2, tv_usec=0}) = 0 (Timeout)
[pid 54150] select(16, [14 15], NULL, NULL, {tv_sec=2, tv_usec=0}) = 0 (Timeout)
[pid 54150] select(16, [14 15], NULL, NULL, {tv_sec=2, tv_usec=0}) = 0 (Timeout)

Rank 0

[node040 B-3-22471]$ strace -ff -p 54152
strace: Process 54152 attached with 4 threads
[pid 54163] epoll_wait(27, <unfinished ...>
[pid 54159] epoll_wait(11, <unfinished ...>
[pid 54156] restart_syscall(<... resuming interrupted poll ...>

Rank 1

[node040 B-3-22471]$ strace -ff -p 54153
strace: Process 54153 attached with 4 threads
[pid 54161] epoll_wait(27, <unfinished ...>
[pid 54160] epoll_wait(11, <unfinished ...>
[pid 54157] restart_syscall(<... resuming interrupted poll ...> <unfinished ...>
[pid 54153] poll([{fd=6, events=POLLIN}, {fd=16, events=POLLIN}], 2, 0) = 0 (Timeout)
[pid 54153] poll([{fd=6, events=POLLIN}, {fd=16, events=POLLIN}], 2, 0) = 0 (Timeout)
[pid 54153] poll([{fd=6, events=POLLIN}, {fd=16, events=POLLIN}], 2, 0) = 0 (Timeout)
[pid 54153] poll([{fd=6, events=POLLIN}, {fd=16, events=POLLIN}], 2, 0) = 0 (Timeout)
[pid 54153] poll([{fd=6, events=POLLIN}, {fd=16, events=POLLIN}], 2, 0) = 0 (Timeout)
....

Rank 2

[node040 B-3-22471]$ strace -ff -p 54154
strace: Process 54154 attached with 4 threads
[pid 54158] epoll_wait(11, <unfinished ...>
[pid 54155] restart_syscall(<... resuming interrupted poll ...> <unfinished ...>
[pid 54162] epoll_wait(27, <unfinished ...>
[pid 54154] poll([{fd=6, events=POLLIN}, {fd=16, events=POLLIN}], 2, 0) = 0 (Timeout)
[pid 54154] poll([{fd=6, events=POLLIN}, {fd=16, events=POLLIN}], 2, 0) = 0 (Timeout)
[pid 54154] poll([{fd=6, events=POLLIN}, {fd=16, events=POLLIN}], 2, 0) = 0 (Timeout)
[pid 54154] poll([{fd=6, events=POLLIN}, {fd=16, events=POLLIN}], 2, 0) = 0 (Timeout)
...

UCX Debug Log

[node040 B-3-22471]$ tail -f /tmp/ucx.log
[1621958056.506833] [node040:61982:0] mm_iface.c:250 UCX DATA RX [10315] oi am_id 2 len 12 EGR_O tag fffff30000000003
[1621958056.506835] [node040:61982:0] tag_match.inl:119 UCX DATA checking req 0xcf70c0 tag fffff30000000003/ffffffffffffffff with tag fffff30000000003
[1621958056.506838] [node040:61982:0] tag_match.inl:121 UCX REQ matched received tag fffff30000000003 to req 0xcf70c0
[1621958056.506840] [node040:61982:0] eager_rcv.c:25 UCX REQ found req 0xcf70c0
[1621958056.506842] [node040:61982:0] ucp_request.inl:603 UCX REQ req 0xcf70c0: unpack recv_data req_len 4 data_len 4 offset 0 last: yes
[1621958056.506845] [node040:61982:0] ucp_request.inl:205 UCX REQ completing receive request 0xcf70c0 (0xcf71d0) --e-cr- stag 0xfffff30000000003 len 4, Success
[1621958056.506847] [node040:61982:0] ucp_request.c:80 UCX REQ free request 0xcf70c0 (0xcf71d0) d-e-cr-
[1621958056.506849] [node040:61982:0] ucp_request.inl:181 UCX REQ put request 0xcf70c0
[1621958056.506867] [node040:61982:0] tag_send.c:246 UCX REQ send_nbx buffer 0x7fffffff50af count 1 tag 10000100003 to
[1621958056.506870] [node040:61982:0] mm_ep.c:289 UCX DATA TX [201] -i am_id 2 len

< ... freeze ... >
Uploading Compiled_MPPICFoam_Parallel_Troublshooting.pptx…

These messages appear in all my output logs now

--> FOAM Warning :
From function void* Foam::dlOpen(const Foam::fileName&, bool)
in file POSIX.C at line 1251
dlopen error : libfvMotionSolvers.dll: cannot open shared object file: No such file or directory

This is odd. It's expected that there should be no .dll files on a linux build - even under WSL.
Are these needed .so files mislabeled?
Deprecated libs?
blockMesh.log

Since the official bug reporting website seems to be currently unavailable, I report the issue here.

Both in codedFixedValueFvPatchFields.C and codedMixedFvPatchFields.C, the "codeOptions" key is missing from codeKeys_. Because of that codeOptions is not written at write time, which causes problems when restarting a case or trying to run in parallel after regular decomposition.

I believe it should be as follows:

template<>
const Foam::wordList
Foam::CodedBase<Foam::codedMixedFvPatchFieldBase>::codeKeys_ =
{
    "code",
    "codeInclude",
    "codeOptions",
    "localCode"
};

This issue also applies to the development version.

Version

OpenFoam-8/OpenFoam-2012
mpirun (Open MPI) 4.0.3

I have copied a cavity file here to test cluster parallel computing

OpenFOAM/OpenFOAM-8/tutorials/incompressible/icoFoam/cavity/cavity

After adding the machines and system/decomposepardict files, the test folder looks like this

$ ls
0  Allclean  constant  machines  system  test.sh

There is no need for passwords between nodes, and SSH login and hosts files have been configured correctly.

I can use any machine for parallel computing, and it can run normally. When I need more than one node to compute, I will report an error. The icofoam executable file of other nodes cannot be found, but this file is real and can run normally.

Maybe it's a variable setting error in openmpi, but I don't know where to modify it to use cluster for parallel computing.

Here is the output log, I hope someone can help me.

$ cat test.sh 
blockMesh && decomposePar && mpirun --hostfile machines -np 8 icoFoam -parallel > log

$ sh test.sh 
/*---------------------------------------------------------------------------*\
  =========                 |
  \\      /  F ield         | OpenFOAM: The Open Source CFD Toolbox
   \\    /   O peration     | Website:  https://openfoam.org
    \\  /    A nd           | Version:  8
     \\/     M anipulation  |
\*---------------------------------------------------------------------------*/
Build  : 8-3d62498be310
Exec   : blockMesh
Date   : May 17 2021
Time   : 11:50:01
Host   : "dyfluid"
PID    : 191394
I/O    : uncollated
Case   : /home/dyfluid/test/test2/cavity
nProcs : 1
sigFpe : Enabling floating point exception trapping (FOAM_SIGFPE).
fileModificationChecking : Monitoring run-time modified files using timeStampMaster (fileModificationSkew 10)
allowSystemOperations : Allowing user-supplied system call operations

// * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * //
Create time

Creating block mesh from
    "system/blockMeshDict"
Creating block edges
No non-planar block faces defined
Creating topology blocks
Creating topology patches

Creating block mesh topology

Check topology

	Basic statistics
		Number of internal faces : 0
		Number of boundary faces : 6
		Number of defined boundary faces : 6
		Number of undefined boundary faces : 0
	Checking patch -> block consistency

Creating block offsets
Creating merge list .

Creating polyMesh from blockMesh
Creating patches
Creating cells
Creating points with scale 0.1
    Block 0 cell size :
        i : 0.005 .. 0.005
        j : 0.005 .. 0.005
        k : 0.01

Writing polyMesh
----------------
Mesh Information
----------------
  boundingBox: (0 0 0) (0.1 0.1 0.01)
  nPoints: 882
  nCells: 400
  nFaces: 1640
  nInternalFaces: 760
----------------
Patches
----------------
  patch 0 (start: 760 size: 20) name: movingWall
  patch 1 (start: 780 size: 60) name: fixedWalls
  patch 2 (start: 840 size: 800) name: frontAndBack

End

/*---------------------------------------------------------------------------*\
  =========                 |
  \\      /  F ield         | OpenFOAM: The Open Source CFD Toolbox
   \\    /   O peration     | Website:  https://openfoam.org
    \\  /    A nd           | Version:  8
     \\/     M anipulation  |
\*---------------------------------------------------------------------------*/
Build  : 8-3d62498be310
Exec   : decomposePar
Date   : May 17 2021
Time   : 11:50:01
Host   : "dyfluid"
PID    : 191395
I/O    : uncollated
Case   : /home/dyfluid/test/test2/cavity
nProcs : 1
sigFpe : Enabling floating point exception trapping (FOAM_SIGFPE).
fileModificationChecking : Monitoring run-time modified files using timeStampMaster (fileModificationSkew 10)
allowSystemOperations : Allowing user-supplied system call operations

// * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * //
Create time



Decomposing mesh region0

Create mesh

Calculating distribution of cells
Selecting decompositionMethod scotch

Finished decomposition in 0 s

Calculating original mesh data

Distributing cells to processors

Distributing faces to processors

Distributing points to processors

Constructing processor meshes

Processor 0
    Number of cells = 50
    Number of faces shared with processor 1 = 10
    Number of faces shared with processor 3 = 5
    Number of faces shared with processor 6 = 5
    Number of faces shared with processor 7 = 5
    Number of processor patches = 4
    Number of processor faces = 25
    Number of boundary faces = 105

Processor 1
    Number of cells = 50
    Number of faces shared with processor 0 = 10
    Number of faces shared with processor 2 = 5
    Number of processor patches = 2
    Number of processor faces = 15
    Number of boundary faces = 115

Processor 2
    Number of cells = 50
    Number of faces shared with processor 1 = 5
    Number of faces shared with processor 3 = 10
    Number of processor patches = 2
    Number of processor faces = 15
    Number of boundary faces = 115

Processor 3
    Number of cells = 50
    Number of faces shared with processor 0 = 5
    Number of faces shared with processor 2 = 10
    Number of faces shared with processor 4 = 10
    Number of processor patches = 3
    Number of processor faces = 25
    Number of boundary faces = 105

Processor 4
    Number of cells = 50
    Number of faces shared with processor 3 = 10
    Number of faces shared with processor 5 = 10
    Number of faces shared with processor 7 = 5
    Number of processor patches = 3
    Number of processor faces = 25
    Number of boundary faces = 105

Processor 5
    Number of cells = 50
    Number of faces shared with processor 4 = 10
    Number of faces shared with processor 7 = 5
    Number of processor patches = 2
    Number of processor faces = 15
    Number of boundary faces = 115

Processor 6
    Number of cells = 50
    Number of faces shared with processor 0 = 5
    Number of faces shared with processor 7 = 10
    Number of processor patches = 2
    Number of processor faces = 15
    Number of boundary faces = 115

Processor 7
    Number of cells = 50
    Number of faces shared with processor 0 = 5
    Number of faces shared with processor 4 = 5
    Number of faces shared with processor 5 = 5
    Number of faces shared with processor 6 = 10
    Number of processor patches = 4
    Number of processor faces = 25
    Number of boundary faces = 105

Number of processor faces = 80
Max number of cells = 50 (0% above average 50)
Max number of processor patches = 4 (45.4545% above average 2.75)
Max number of faces between processors = 25 (25% above average 20)

Time = 0

Processor 0: field transfer
Processor 1: field transfer
Processor 2: field transfer
Processor 3: field transfer
Processor 4: field transfer
Processor 5: field transfer
Processor 6: field transfer
Processor 7: field transfer

End

--------------------------------------------------------------------------
mpirun was unable to find the specified executable file, and therefore
did not launch the job.  This error was first reported for process
rank 4; it may have occurred for other processes as well.

NOTE: A common cause for this error is misspelling a mpirun command
      line parameter option (remember that mpirun interprets the first
      unrecognized command line token as the executable).

Node:       node2
Executable: /home/dyfluid/OpenFOAM/OpenFOAM-8/platforms/linux64GccDPInt32Opt/bin/icoFoam
--------------------------------------------------------------------------

There was a helpful error message that the mesh quality controls have been moved to their own dict file. But I can find no documentation on this in the online docs.

1. Step:
   Matthek’s Triangles spline:
   This is a theme for snappy hex meshed and the volume fluid optimizer in openfoam.
   And also for the stress optimization of the solid solver.

See
http://bibliothek.fzk.de/mattheck/mattheck_postersammlung.pdf

and in Detail
https://publikationen.bibliothek.kit.edu/220061912/3814714

Look in the poster and you see the triangle method with 3 Triangles to construct spline points.
With this points a good edge curve is the result without FEM for beams, shells and solids or CFD for fluids.

there is a pdf Poster in german with stress Relation of matthek spline by triangles to circle arc .
https://publikationen.bibliothek.kit.edu/220076489

so an Optimization of edges of the calculation room at the beginning is here simple possible.

A result can be a model by splines by intelligent interpolation of the mesh surface.

2. Step:
   Spiro Splines by Levien is here an interesting solution and better than bezier splines.

Circular Arcs are simple but not the best curves in nature and in technics.
Bezier Splines are limited in precision and mathematics.
Spiro splines are Derivats of spirals.

See phd Levien at https://levien.com/phd/phd.html

There are here two things

1. Improvement of bezier splines:
   In this PhD are some pages for better bezier splines
   See page 135 and more with error picture in page 141.

2. The new spiro spline from Levien:
   The spiro spline is more than a better bezier spline.

See also https://www.levien.com/spiro/

with some code for his phd and his font program.
and for more clearness:
https://raphlinus.github.io/curves/2018/12/21/new-spline.html

The code in Inkscape with the spiro path tool was improved in last years.
So Inkscape source code is here perhaps a better source with spiro splines also for this project.

So Openfoam can calculate optimized meshes with splines surfaces for better smooth surface results back to CAD.
With both splines at beginning and end the evolution of best contours is much faster and elegant possible.

Hello :)

Thanks for providing the docker image as an easy way to run OpenFoam-8.
I'd have a slight improvement, allowing to call commands directly (docker run openfoam "cmd").
It really simplifies my setup and i'd like to provide my modification for the upstream docker image.

Where can i find the Docker build file for the OF-8 image?

Behavior is pretty clear from my perspective.

1. i provide a set of parameters for birdcarrau in my transportProperties
   

2. Results in the following failure
   

(line 45 is where a is defined)

Hi,

I am using OpenFoam-8 on a Suse Linux box.
I generated a simple 3D mesh with gmsh, and exported it in two flavors: airfoil.msh (native gmsh format) and airfoil.vtk.
Running gmshToFoam airfoil.msh, I get an error:

/*---------------------------------------------------------------------------*\
  =========                 |
  \\      /  F ield         | OpenFOAM: The Open Source CFD Toolbox
   \\    /   O peration     | Website:  https://openfoam.org
    \\  /    A nd           | Version:  8
     \\/     M anipulation  |
\*---------------------------------------------------------------------------*/
Build  : 8-563c11b49eaf
Exec   : gmshToFoam airfoil.msh
Date   : Jan 28 2021
Time   : 23:05:36
Host   : "localhost.localdomain"
PID    : 12854
I/O    : uncollated
Case   : /home/laurent/OpenFOAM/OpenFOAM-8/tutorials/compressible/rhoSimpleFoam/airfoil_nacaXX
nProcs : 1
sigFpe : Enabling floating point exception trapping (FOAM_SIGFPE).
fileModificationChecking : Monitoring run-time modified files using timeStampMaster (fileModificationSkew 10)
allowSystemOperations : Allowing user-supplied system call operations

// * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * //
Create time

Using #calcEntry at line 20 in file "/home/laurent/OpenFOAM/OpenFOAM-8/tutorials/compressible/rhoSimpleFoam/airfoil_nacaXX/system/controlDict/functions/forceCoeffsCompressible/forceCoeffsCompressible"
Using #codeStream with "/home/laurent/OpenFOAM/OpenFOAM-8/tutorials/compressible/rhoSimpleFoam/airfoil_nacaXX/dynamicCode/platforms/linux64GccDPInt32Opt/lib/libcodeStream_1cdcbfab46358bbcf2b09b5ffdb81074e9595ac2.so"
Using #codeStream at line 21 in file "/home/laurent/OpenFOAM/OpenFOAM-8/tutorials/compressible/rhoSimpleFoam/airfoil_nacaXX/system/controlDict/functions/forceCoeffsCompressible/forceCoeffsCompressible"
Using #codeStream with "/home/laurent/OpenFOAM/OpenFOAM-8/tutorials/compressible/rhoSimpleFoam/airfoil_nacaXX/dynamicCode/platforms/linux64GccDPInt32Opt/lib/libcodeStream_4e53ea6683014d43aaae7f1655fdc419be73fa84.so"
Using #codeStream at line 27 in file "/home/laurent/OpenFOAM/OpenFOAM-8/tutorials/compressible/rhoSimpleFoam/airfoil_nacaXX/system/controlDict/functions/forceCoeffsCompressible/forceCoeffsCompressible"
Using #codeStream with "/home/laurent/OpenFOAM/OpenFOAM-8/tutorials/compressible/rhoSimpleFoam/airfoil_nacaXX/dynamicCode/platforms/linux64GccDPInt32Opt/lib/libcodeStream_3a1ca79d5a971693a00301e3a44b841419f9a7ca.so"
Using #codeStream at line 33 in file "/home/laurent/OpenFOAM/OpenFOAM-8/tutorials/compressible/rhoSimpleFoam/airfoil_nacaXX/system/controlDict/functions/forceCoeffsCompressible/forceCoeffsCompressible"
Using #codeStream with "/home/laurent/OpenFOAM/OpenFOAM-8/tutorials/compressible/rhoSimpleFoam/airfoil_nacaXX/dynamicCode/platforms/linux64GccDPInt32Opt/lib/libcodeStream_2d58ca55a9c3d5f23a93b18c140764657d312776.so"
Starting to read mesh format at line 2
Read format version 4.1  ascii 0

Starting to read physical names at line 5
Physical names:4
    Line 1      airfoil
    Line 2      farfield
    Surface 1   mesh
    Volume 1    volume

Skipping tag $Entities at line 12
Starting to read points at line 1398
Vertices to be read:817


--> FOAM FATAL IO ERROR: 
Attempt to get back from bad stream

file: IStringStream.sourceFile at line 0.

    From function bool Foam::Istream::getBack(Foam::token&)
    in file db/IOstreams/IOstreams/Istream.C at line 56.

FOAM exiting

With vtkUnstructuredToFoam airfoil.vtk, I get

/*---------------------------------------------------------------------------*\
  =========                 |
  \\      /  F ield         | OpenFOAM: The Open Source CFD Toolbox
   \\    /   O peration     | Website:  https://openfoam.org
    \\  /    A nd           | Version:  8
     \\/     M anipulation  |
\*---------------------------------------------------------------------------*/
Build  : 8-563c11b49eaf
Exec   : vtkUnstructuredToFoam airfoil.vtk
Date   : Jan 28 2021
Time   : 23:08:44
Host   : "localhost.localdomain"
PID    : 13476
I/O    : uncollated
Case   : /home/laurent/OpenFOAM/OpenFOAM-8/tutorials/compressible/rhoSimpleFoam/airfoil_nacaXX
nProcs : 1
sigFpe : Enabling floating point exception trapping (FOAM_SIGFPE).
fileModificationChecking : Monitoring run-time modified files using timeStampMaster (fileModificationSkew 10)
allowSystemOperations : Allowing user-supplied system call operations

// * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * //
Create time

Using #calcEntry at line 20 in file "/home/laurent/OpenFOAM/OpenFOAM-8/tutorials/compressible/rhoSimpleFoam/airfoil_nacaXX/system/controlDict/functions/forceCoeffsCompressible/forceCoeffsCompressible"
Using #codeStream with "/home/laurent/OpenFOAM/OpenFOAM-8/tutorials/compressible/rhoSimpleFoam/airfoil_nacaXX/dynamicCode/platforms/linux64GccDPInt32Opt/lib/libcodeStream_1cdcbfab46358bbcf2b09b5ffdb81074e9595ac2.so"
Using #codeStream at line 21 in file "/home/laurent/OpenFOAM/OpenFOAM-8/tutorials/compressible/rhoSimpleFoam/airfoil_nacaXX/system/controlDict/functions/forceCoeffsCompressible/forceCoeffsCompressible"
Using #codeStream with "/home/laurent/OpenFOAM/OpenFOAM-8/tutorials/compressible/rhoSimpleFoam/airfoil_nacaXX/dynamicCode/platforms/linux64GccDPInt32Opt/lib/libcodeStream_4e53ea6683014d43aaae7f1655fdc419be73fa84.so"
Using #codeStream at line 27 in file "/home/laurent/OpenFOAM/OpenFOAM-8/tutorials/compressible/rhoSimpleFoam/airfoil_nacaXX/system/controlDict/functions/forceCoeffsCompressible/forceCoeffsCompressible"
Using #codeStream with "/home/laurent/OpenFOAM/OpenFOAM-8/tutorials/compressible/rhoSimpleFoam/airfoil_nacaXX/dynamicCode/platforms/linux64GccDPInt32Opt/lib/libcodeStream_3a1ca79d5a971693a00301e3a44b841419f9a7ca.so"
Using #codeStream at line 33 in file "/home/laurent/OpenFOAM/OpenFOAM-8/tutorials/compressible/rhoSimpleFoam/airfoil_nacaXX/system/controlDict/functions/forceCoeffsCompressible/forceCoeffsCompressible"
Using #codeStream with "/home/laurent/OpenFOAM/OpenFOAM-8/tutorials/compressible/rhoSimpleFoam/airfoil_nacaXX/dynamicCode/platforms/linux64GccDPInt32Opt/lib/libcodeStream_2d58ca55a9c3d5f23a93b18c140764657d312776.so"
Header   : "# vtk DataFile Version 2.0"
Title    : "1234, Created by Gmsh"
dataType : "ASCII"
line:4 tag:DATASET
geomType : UNSTRUCTURED_GRID
line:5 tag:POINTS
Reading 80754 numbers representing 80754 coordinates.
line:80761 tag:CELLS
Reading 160682 cells or faces.
line:241445 tag:CELL_TYPES
Read 80339 cells;80339 faces.
line:402129 tag:CELL_DATA
line:402130 tag:SCALARS
Reading scalar CellEntityIds of type int from lookup table
Read points:80754 cellShapes:80339 faces:80339 lines:4

Cell fields:
Read 1 labelField fields:
Size    Name
----    ----
160682  CellEntityIds



Point fields:


Other fields:
--> FOAM Warning : 
    From function Foam::polyMesh::polyMesh(const Foam::IOobject&, Foam::pointField&&, const cellShapeList&, const faceListList&, const wordList&, const wordList&, const Foam::word&, const Foam::word&, const wordList&, bool)
    in file meshes/polyMesh/polyMeshFromShapeMesh.C at line 595
    Found 161093 undefined faces in mesh; adding to default patch.
Writing mesh ...
End

I would prefer to work with the gmsh format though.
I can provide airfoil.msh for debugging.
Thanks for the help.

I tried installing OpenFOAM on ubuntu 21.04 but I got an error
E: The repository 'http://dl.openfoam.org/ubuntu hirsute Release' does not have a Release file.

Installed OpenFoam8 on Windows 10 following the guide. Whenever I run blockMesh I get an error. Error log posted below.

I was following this tutorial, using all those files she linked to. https://www.youtube.com/watch?v=oc4V79F78AQ

From the error message, it appears to be be an issue with From function int main(int, char**) in file blockMesh.C at line 408.

/*---------------------------------------------------------------------------*\
  =========                 |
  \\      /  F ield         | OpenFOAM: The Open Source CFD Toolbox
   \\    /   O peration     | Website:  https://openfoam.org
    \\  /    A nd           | Version:  8
     \\/     M anipulation  |
\*---------------------------------------------------------------------------*/
Build  : 8-340defec456f
Exec   : blockMesh
Date   : Dec 13 2020
Time   : 22:55:35
Host   : "DESKTOP-GN7NPG8"
PID    : 21448
I/O    : uncollated
Case   : /home/rfbomb/OpenFOAM/rfbomb-8/run/postExtruder
nProcs : 1
sigFpe : Enabling floating point exception trapping (FOAM_SIGFPE).
fileModificationChecking : Monitoring run-time modified files using timeStampMaster (fileModificationSkew 10)
allowSystemOperations : Allowing user-supplied system call operations

// * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * //
Create time

Not deleting polyMesh directory
    "/home/rfbomb/OpenFOAM/rfbomb-8/run/postExtruder/constant/polyMesh"
    because it contains blockMeshDict
Creating block mesh from
    "constant/polyMesh/blockMeshDict"
Creating block edges
No non-planar block faces defined
Creating topology blocks
Creating topology patches

Creating block mesh topology

Check topology

        Basic statistics
                Number of internal faces : 0
                Number of boundary faces : 6
                Number of defined boundary faces : 6
                Number of undefined boundary faces : 0
        Checking patch -> block consistency

Creating block offsets
Creating merge list .

Creating polyMesh from blockMesh
Creating patches
Creating cells
Creating points with scale 1
    Block 0 cell size :
        i : 0.04
        j : 0.04 .. 0.04
        k : 0.04

There are no merge patch pairs edges

Writing polyMesh


--> FOAM FATAL ERROR:
Failed writing polyMesh.

    From function int main(int, char**)
    in file blockMesh.C at line 408.

FOAM exiting

The standard procedure to install OpenFOAM-8 in Ubuntu fails with the following error:

ERROR: cannot verify dl.openfoam.org's certificate.
Issued certificate has expired.

Should be trivial to fix.

The tutorials folders are incomplete. They don´t have all archives.
For example in the folder:

• OpenFOAM-8/tutorials/incompressible/simpleFoam/pitzDailyExptInlet/system/

There is no the archive blockMeshDict.

The tutorial OpenFOAM-8/tutorials/incompressible/simpleFoam/motorBike doesn´t work neither.

I am having a tremendous amount of difficulty getting snappyHexMesh to snap on this and other models.
I have been trying resolve this over several weeks and have spent tens or hundreds of hours on it. Originally I thought this was a user error, but today this model twice snapped correctly and produced a smooth result - but the result was not reproducible.

That indicates the problem is a non-deterministic bug. It makes the package nearly useless by itself.

Fortunately I have been using SimFlow for mesh generation which has allowed me to work - but SimFlow doesn't expose enough features to support my project and I need to get this working ASAP.

The entire case is attached with an image. It's fast (< 2 minutes.)

The image shows that only two of the three feature edges were refined correctly. If desired I can open another ticket.

I have 30 years of experience writing various forms of meshing software - if you need someone to assist I'm looking for contract work. https://www.linkedin.com/in/bob-tipton-80444315/

CalHemisphere_Graded.zip

Environment：
OS: CentOS-7.6 x64
OpenFOAM : 8

paraFoam

WARN file does not exist:
./system/controlDict
./system/fvSchemes
./system/fvSolution
Cannot locate OpenFOAM-format case files
Would you like to open ParaView anyway <Y|n>:y
ERROR: In /root/OpenFOAM/ThirdParty-8/ParaView-5.6.3/VTK/Rendering/OpenGL2/vtkOpenGLRenderWindow.cxx, line 749
vtkXOpenGLRenderWindow (0x4fec6d0): Unable to find a valid OpenGL 3.2 or later implementation. Please update your video card driver to the latest version. If you are using Mesa please make sure you have version 11.2 or later and make sure your driver in Mesa supports OpenGL 3.2 such as llvmpipe or openswr. If you are on windows and using Microsoft remote desktop note that it only supports OpenGL 3.2 with nvidia quadro cards. You can use other remoting software such as nomachine to avoid this issue.

ERROR: In /root/OpenFOAM/ThirdParty-8/ParaView-5.6.3/VTK/Rendering/OpenGL2/vtkXOpenGLRenderWindow.cxx, line 796
vtkXOpenGLRenderWindow (0x4fec6d0): failed to create offscreen window

ERROR: In /root/OpenFOAM/ThirdParty-8/ParaView-5.6.3/VTK/Rendering/OpenGL2/vtkOpenGLRenderWindow.cxx, line 736
vtkXOpenGLRenderWindow (0x4fec6d0): GLEW could not be initialized.

I just spent a great deal of time discovering that surfaceFeaturesExtract doesn't exist at all in V8.
I did find that surfaceFeatures works instead.
I have found no documentation or notice about this change. If it exists it is not very discoverable.
When running surfaceFeaturesExtract a message appears - 'sudo apt-get install OpenFoam'. I did that and then got multiple versions which made things much worse. I since complete wiped my WSL/Ubuntu install and reinstalled everything.

Hello everyone,
Does anyone know that Is it possible to simulate a spray of cryogenic fluid?
I heard someone has created a flashFoam through OpenFoam, but I am not able to find its repository.

Could anyone please help me out?

Thank you.
Neel Shah