PeleLMeX is a solver for high fidelity reactive flow simulations, namely direct numerical simulation (DNS) and large eddy simulation (LES). The solver combines a low Mach number approach, adaptive mesh refinement (AMR), embedded boundary (EB) geometry treatment and high performance computing (HPC) to provide a flexible tool to address research questions on platforms ranging from small workstations to the world's largest GPU-accelerated supercomputers. PeleLMeX has been used to study complex flame/turbulence interactions in RCCI engines and hydrogen combustion or the effect of sustainable aviation fuel on gas turbine combustion.

PeleLMeX is part of the Pele combustion Suite.

PeleLMeX is a non-subcycling version of PeleLM based on AMReX's AmrCore and borrowing from the incompressible solver incflo. It solves of the multispecies reactive Navier-Stokes equations in the low Mach number limit as described in the documentation. It inherits most of PeleLM algorithmic features, but differs significantly in its implementation stemming from the non-subcycling approach. PeleLM is no longer under active development; PeleLMeX should be used for simulations of low Mach number reacting flows and PeleC for simulations of flows with higher Mach numbers where compressibility effects are significant.

A overview of PeleLMeX controls is provided in the documentation.

The PeleLMeX governing equations and core algorithms are described in:

https://amrex-combustion.github.io/PeleLMeX/manual/html/Model.html#mathematical-background

https://amrex-combustion.github.io/PeleLMeX/manual/html/Model.html#pelelmex-algorithm

A set of self-contained tutorials describing more complex problems is also provided:

https://amrex-combustion.github.io/PeleLMeX/manual/html/Tutorials.html

The compilations of PeleLMeX requires a C++17 compatible compiler (GCC >= 8 or Clang >= 3.6) as well as CMake >= 3.23 for compiling the SUNDIALS third party library.

Most of the examples provided hereafter and in the tutorials will use MPI to run in parallel. Although not mandatory, it is advised to build PeleLMeX with MPI support from the get go if more than a single core is available to you. Any of mpich or open-mpi is a suitable option if MPI is not already available on your platform.

Finally, when building with GPU support, CUDA >= 11 is required with NVIDIA GPUs and ROCm >= 5.2 is required with AMD GPUs.

The preferred method consists of cloning PeleLMeX and its submodules (amrex, AMReX-Hydro, PelePhysics, PeleMP), and SUNDIALS using a recursive git clone:

git clone --recursive https://github.com/AMReX-Combustion/PeleLMeX.git

Alternatively, you can use a separate git clone of each of the submodules. The default location for PeleLMeX dependencies is the Submodules folder but you optionally setup the following environment variables (e.g. using bash) to any other location:

export PELELMEX_HOME=<path_to_PeleLMeX>
export AMREX_HOME=${PELELMEX_HOME}/Submodules/amrex
export AMREX_HYDRO_HOME=${PELELMEX_HOME}/Submodules/AMReX-Hydro
export PELE_PHYSICS_HOME=${PELELMEX_HOME}/Submodules/PelePhysics
export PELEMP_HOME=${PELELMEX_HOME}/Submodules/PeleMP
export SUNDIALS_HOME=${PELELMEX_HOME}/Submodules/sundials

Both GNUmake and CMake can be used to build a PeleLMeX executable, but GNUmake is the preferred choice. The code handling the initial condition and boundary conditions is unique to each case, and subfolders in the Exec directory provide a number of examples.

For instance, to compile the executable for the case of a rising hot bubble, move into the HotBubble folder:

cd PeleLMeX/Exec/RegTests/HotBubble

If this is a clean install, you will need to make the third party libraries with: make TPL (note: if on macOS, you might need to specify COMP=llvm in the make statements).

Finally, make with: make -j, or if on macOS: make -j COMP=llvm. To clean the installation, use either make clean or make realclean. If running into compile errors after changing compile time options in PeleLMeX (e.g., the chemical mechanism), the first thing to try is to clean your build by running make TPLrealclean && make realclean, then try to rebuild the third party libraries and PeleLMeX with make TPL && make -j. See the Tutorial for this case for instructions on how to compile with different options (for example, to compile without MPI support or to compile for GPUs) and how to run the code once compiled.

When using CMake (cmake version >= 3.23 is required), first configure CMake directly in PeleLMeX root folder:

cmake -S . -B buildHotBubble -DPELELMEX_MPI=ON -DPELELMEX_CASE=HotBubble

Then build the executable:

cmake --build buildHotBubble --parallel 4

Do you have a question ? Found an issue ? Please use the GitHub Discussions to engage with the development team or open a new GitHub issue to report a bug. The development team also encourages users to take an active role in respectfully answering each other's questions in these spaces. When reporting a bug, it is helpful to provide as much detail as possible, including a case description and the major compile and runtime options being used. Though not required, it is most effective to create a fork of this repository and share a branch of that fork with a case that minimally reproduces the error.

New contributions to PeleLMeX are welcome ! Contributing Guidelines are provided in CONTRIBUTING.md.

This research was supported by the Exascale Computing Project (ECP), Project Number: 17-SC-20-SC, a collaborative effort of two DOE organizations -- the Office of Science and the National Nuclear Security Administration -- responsible for the planning and preparation of a capable exascale ecosystem -- including software, applications, hardware, advanced system engineering, and early testbed platforms -- to support the nation's exascale computing imperative.