(Please visit our project Homepage and Documentation)
FELTOR (Full-F ELectromagnetic code in TORoidal geometry) is both a numerical library and a scientific software package built on top of it.
Its main physical target are plasma edge and scrape-off layer (gyro-)fluid simulations. The numerical methods centre around discontinuous Galerkin methods on structured grids. Our core level functions are parallelized for a variety of hardware from multi-core cpu to hybrid MPI+GPU, which makes the library incredibly fast. Note that the library ships with a multitude of test and benchmark programs.
This guide discusses how to setup, test and benchmark the FELTOR library installation. Please read it before you proceed to the online tutorial to learn how to use the library in your own programs. We also present how to use the FELTOR software package, which requires additional external libraries to be installed on the system.
The first part assumes a Linux operating system. If you want to work on Windows, jump to Using Feltor on Windows.
Open a terminal and clone the repository into any folder you like
git clone https://www.github.com/feltor-dev/feltor
You also need to clone thrust and cusp distributed under the Apache-2.0 license. Also, we need Agner Fog’s vcl library (GPL-3.0). So again in a folder of your choice
git clone https://www.github.com/thrust/thrust
git clone https://www.github.com/cusplibrary/cusplibrary
git clone https://www.github.com/feltor-dev/vcl
Our code only depends on external libraries that are themselves openly available.
Minimum system requirements | Recommended system requirements | |
---|---|---|
CPU |
Any |
support for AVX and FMA instruction set |
Compiler |
gcc-4.9 or msvc-15 or icc-15.0 (C++-11 standard) |
OpenMP-4 support, avx, fma instruction set flags |
GPU |
- |
NVidia GPU with compute-capability > 6 and nvcc-8.0 |
MPI |
- |
mpi installation compatible with compiler (must be cuda-aware in case hybrid MPI+GPU is the target system) |
Our GPU backend uses the Nvidia-CUDA programming environment and in order to compile and run a program for a GPU a user needs at least the nvcc-7.5 compiler (available free of charge) and a NVidia GPU. However, we explicitly note here that due to the modular design of our software a user does not have to possess a GPU nor the nvcc compiler. The CPU version of the backend is equally valid and provides the same functionality. Analogously, an MPI installation is only required if the user targets a distributed memory system.
In order to compile one of the many test and benchmark codes
inside the FELTOR library you need to tell
the FELTOR configuration where the external libraries are located on
your computer. The default way to do this is to go into your HOME
directory, make an include directory and link the paths in this
directory
cd ~
mkdir include
cd include
ln -s path/to/thrust/thrust
ln -s path/to/cusplibrary/cusp
ln -s path/to/vcl
If you do not like this, you can also set the include paths in your own config file as described here.
Now let us compile the first benchmark program.
cd path/to/feltor/inc/dg
make blas_b device=omp #(for an OpenMP version)
#or
make blas_b device=gpu #(if you have a gpu and nvcc )
Run the code with
./blas_b
and when prompted for input vector sizes type for example 3 100 100 10
which makes a grid with 3 polynomial coefficients, 100 cells in x, 100
cells in y and 10 in z. If you compiled for OpenMP, you can set the
number of threads with e.g. export OMP_NUM_THREADS=4
.
This is a benchmark program to benchmark various elemental functions the library is built on. Go ahead and vary the input parameters and see how your hardware performs. You can compile and run any other program that ends in
_t.cu
(test programs) or_b.cu
(benchmark programs) infeltor/inc/dg
in this way.
Now, let us test the mpi setup
You can of course skip this if you don’t have mpi installed on your computer. If you intend to use the MPI backend, an implementation library of the mpi standard is required. Per default
mpic++
is used for compilation.
cd path/to/feltor/inc/dg
make blas_mpib device=omp # (for MPI+OpenMP)
# or
make blas_mpib device=gpu # (for MPI+GPU, requires CUDA-aware MPI installation)
Run the code with $ mpirun -n '# of procs' ./blas_mpib
then tell how
many process you want to use in the x-, y- and z- direction, for
example: 2 2 1
(i.e. 2 procs in x, 2 procs in y and 1 in z; total
number of procs is 4) when prompted for input vector sizes type for
example 3 100 100 10
(number of cells divided by number of procs must
be an integer number). If you compiled for MPI+OpenMP, you can set the
number of OpenMP threads with e.g. export OMP_NUM_THREADS=2
.
FELTOR’s library is the dg-library (from discontinuous Galerkin). Note that the library is header-only, which means that you just have to include the relevant header(s) and you’re good to go. For example in the following program we compute the square L2 norm of a function:
#include <iostream>
//include the basic dg-library
#include "dg/algorithm.h"
//optional: include the geometries expansion
#include "dg/geometries/geometries.h"
double function(double x, double y){return exp(x)*exp(y);}
int main()
{
//create a 2d discretization of [0,2]x[0,2] with 3 polynomial coefficients
dg::CartesianGrid2d g2d( 0, 2, 0, 2, 3, 20, 20);
//discretize a function on this grid
const dg::DVec x = dg::evaluate( function, g2d);
//create the volume element
const dg::DVec vol2d = dg::create::volume( g2d);
//compute the square L2 norm on the device
double norm = dg::blas2::dot( x, vol2d, x);
// norm is now: (exp(4)-exp(0))^2/4
std::cout << norm <<std::endl;
return 0;
}
To compile and run this code for a GPU use
nvcc -x cu -std=c++11 -Ipath/to/feltor/inc -Ipath/to/thrust/thrust -Ipath/to/cusplibrary/cusp test.cpp -o test
./test
Or if you want to use OpenMP and gcc instead of CUDA for the device functions you can also use
g++ -std=c++11 -fopenmp -mavx -mfma -DTHRUST_DEVICE_SYSTEM=THRUST_DEVICE_SYSTEM_OMP -Ipath/to/feltor/inc -Ipath/to/thrust/thrust -Ipath/to/cusplibrary/cusp test.cpp -o test
export OMP_NUM_THREADS=4
./test
If you want to use mpi, just include the MPI header before any other FELTOR header and use our convenient typedefs like so:
#include <iostream>
//activate MPI in FELTOR
#include "mpi.h"
#include "dg/algorithm.h"
double function(double x, double y){return exp(x)*exp(y);}
int main(int argc, char* argv[])
{
//init MPI and create a 2d Cartesian Communicator assuming 4 MPI threads
MPI_Init( &argc, &argv);
int periods[2] = {true, true}, np[2] = {2,2};
MPI_Comm comm;
MPI_Cart_create( MPI_COMM_WORLD, 2, np, periods, true, &comm);
//create a 2d discretization of [0,2]x[0,2] with 3 polynomial coefficients
dg::CartesianMPIGrid2d g2d( 0, 2, 0, 2, 3, 20, 20, comm);
//discretize a function on this grid
const dg::MDVec x = dg::evaluate( function, g2d);
//create the volume element
const dg::MDVec vol2d = dg::create::volume( g2d);
//compute the square L2 norm
double norm = dg::blas2::dot( x, vol2d, x);
//on every thread norm is now: (exp(4)-exp(0))^2/4
//be a good MPI citizen and clean up
MPI_Finalize();
return 0;
}
Compile e.g. for a hybrid MPI + OpenMP hardware platform with
mpic++ -std=c++11 -mavx -mfma -fopenmp -DTHRUST_DEVICE_SYSTEM=THRUST_DEVICE_SYSTEM_OMP -Ipath/to/feltor/inc -Ipath/to/thrust/thrust -Ipath/to/cusplibrary/cusp test_mpi.cpp -o test_mpi
export OMP_NUM_THREADS=2
mpirun -n 4 ./test_mpi
Note the striking similarity to the previous program. Especially the line calling the dot function did not change at all. The compiler chooses the correct implementation for you! This is a first example of a container free numerical algorithm.
Now, we want to compile and run a simulation program. To this end, we have to download and install some additional libraries for I/O-operations.
First, we need to install jsoncpp (distributed under the MIT License).
The easiest way to do this on Linux is to install libjsoncpp-dev
through the package managment system. However, if you have
a GPU or if you do not have sudo privileges you have to install the library manually. That means you have to clone JsonCpp and follow the build instructions in the README. After this, link the
include path
cd ~/include
ln -s /usr/include/jsoncpp/json # if installed as a system library
# or
ln -s path/to/jsoncpp/include/json # if installed manually
or append the respective path as well as the path to the object library to the INCLUDE
and JSONLIB
variables as
described under config.
For data output we use the
NetCDF-C library under an
MIT - like license (we use the netcdf-4 file format).
The underlying HDF5
library also uses a very permissive license.
Both can be installed easily on Linux through the libnetcdf-dev
package.
For a manual build follow the build instructions in the netcdf-documentation.
Note that for the mpi
versions of applications you need to build hdf5 and netcdf with the
--enable-parallel
flag. Do NOT use the pnetcdf library, which uses the
classic netcdf file format.
Some desktop applications in FELTOR use the
draw library (developed by us
also under MIT), which depends on
glfw3, an OpenGL development library under a
BSD-like license. There is a libglfw3-dev
package for convenient installation. Again, link path/to/draw
in the include
folder.
We are now ready to compile and run a simulation program
cd path/to/feltor/src/toefl # or any other project in the src folder
make toeflR device=gpu # (compile on gpu or omp)
./toeflR <inputfile.json> # (behold a live simulation with glfw output on screen)
# or
make toefl_hpc device=gpu # (compile on gpu or omp)
./toefl_hpc <inputfile.json> <outputfile.nc> # (a single node simulation with output stored in a file)
# or
make toefl_mpi device=omp # (compile on gpu or omp)
export OMP_NUM_THREADS=2 # (set OpenMP thread number to 1 for pure MPI)
echo 2 2 | mpirun -n 4 ./toefl_mpi <inputfile.json> <outputfile.nc>
# (a multi node simulation with now in total 8 threads with output stored in a file)
# The mpi program will wait for you to type the number of processes in x and y direction before
# running. That is why the echo is there.
A default input file is located in path/to/feltor/src/toefl/input
. All
three programs solve the same equations. The technical documentation on
what equations are discretized, input/output parameters, etc. can be
generated as a pdf with make doc
in the path/to/feltor/src/toefl
directory.
FELTOR has been developed mostly on Linux machines. Recently, it has become possible to develop also on Windows using Microsoft Visual Studio. We here describe how to work with FELTOR’s OpenMP shared memory backend on Windows.
Unfortunately, the msvc compiler only supports an outdated OpenMP version so consider a performance penalty of approximately a factor 2, when running the OpenMP backend on Windows.
We suggest to install the Github desktop https://desktop.github.com.
Please clone all four of the following URLs using File → Clone repository…
https://www.github.com/feltor-dev/feltor
https://www.github.com/thrust/thrust
https://www.github.com/cusplibrary/cusplibrary
https://www.github.com/feltor-dev/vcl
Please also have a look at the relevant system requirements Table.
In Visual Studio we suggest to create a Property Sheet for FELTOR.
The Property Sheet can then be conveniently added to any project that includes
the FELTOR library headers dg/algorithm.h
and/or dg/geometries/geometries.h
-
Open an existing solution in Visual Studio or create a new project with
File → New → Projet …
selectingEmpty Project
in Visual C++. -
In the Solution Explorer change to the
Property Manager
tab, then click onAdd New Project Property Sheet
, name itFeltorPropertySheet.props
and save it to a convenient location. -
Double click on
FeltorPropertySheet
(expand your solution and any of the Debug or Release tabs to find it)-
In
VC++ Directories → Include Directories
click onEdit
Then add the four linespath\to\feltor\inc
,path\to\thrust
,path\to\cusplibrary
andpath\to\folder_containing_vcl
-
In
C/C++ → Optimization → Enable Intrinsic Functions
selectYes (/Oi)
-
In
C/C++ → Preprocessor → Preprocessor Definitions
selectEdit
and add the lineTHRUST_DEVICE_SYSTEM=THRUST_DEVICE_SYSTEM_OMP
(Selects the OpenMP backend in FELTOR) -
In
C/C++ → Code Generation → Enable Enhanced Instruction Set
selectAdvanced Vector Extensions 2 (/arch:AVX2)
(If your CPU supports it, of course) -
In
C/C++ → Language → Open MP Support
selectYes (/openmp)
-
-
Don’t forget to click
Apply
in the end.
That’s it.
You can add your Feltor Property Sheet to any new project
by switching to the Property Manager
again:
click Add Existing Property Sheet
and select FeltorPropertySheet
.
We suggest that you generate a new project for each executable program.
In order to test the Feltor Property Sheet let us add a source file to the project and compile
-
In the Solution Explorer right click on
Source Files → Add → New Item … → C++ File (.cpp)
. As an example we name ittest.cpp
and copy the contents of test.cpp -
Change the Platform from x86 to x64.
-
Compile with
Ctrl + F5
then run the code
If you want to prevent the console from closing on program exit, set
Properties → Linker → System → SubSystem → Console (/SUBSYSTEM:CONSOLE)
in your Property Sheet.
Our simulation codes typically depend on jsoncpp for parameter input, glfw3 for plotting or netcdf-4 for file output and come with a LaTeX file containing documentation. You will need to download these additional libraries and adapt the project properties accordingly.
- jsoncpp
-
-
Download and Install Anaconda. (Once Anaconda is installed don’t do anything yet, it’s just to get a working python3 installation)
-
In Github desktop:
File → Clone repository…
clonehttps://github.com/open-source-parsers/jsoncpp
-
Execute the file
path\to\jsconcpp\amalgamate.py
(double click). The only way to confirm its success is to look for adist
folder containingjsconcpp.cpp
and a folder containing two header files. -
Add
path\to\jsoncpp\dist
toProperties → VC++ Directories → Include Directories
-
In the Solution Explorer Right click
Source Files → Add → Existing Item
and selectpath\to\jsconcpp\dist\jsoncpp.cpp
-
- Glfw3
-
-
In Github desktop:
File → Clone repository…
clonehttps://github.com/feltor-dev/draw
-
Download and extract the Windows binaries from https://www.glfw.org/download.html
-
Add
path\to\glfw-3\include
andpath\to\folder_containing_draw
toProperties → VC++ Directories → Include Directories
-
In
Properties → Linker → General → Additional Library Directories
addpath\to\glfw-3\lib-vc2015
-
Finally, in
Properties → Linker → Input → Additional Dependencies
add the linesglfw3.lib
andopengl32.lib
(there needs to be a newline in between!)
-
- NetCDF
-
-
Download and install the
NetCDF-4.x.x-NC4-64.exe
package from https://www.unidata.ucar.edu/downloads/netcdf/index.jsp (make sure to Check "Add netCDF to system PATH" during the installation process) -
Add
path\to\netCDF\include
toProperties → VC++ Directories → Include Directories
-
In
Properties → Linker → General → Additional Library Directories
addpath\to\netCDF\lib
-
Finally, in
Properties → Linker → Input → Additional Dependencies
add the linenetcdf.lib
-
- LaTeX
-
Install MikTex and TeXstudio (in that order) in order to be able to compile the tex file(s) of the documentation.
The
documentation
of the dG library was generated with
Doxygen and LateX. You can generate a local
version including informative pdf writeups on implemented numerical
methods directly from source code. This depends on the doxygen
,
libjs-mathjax
and graphviz
packages and LateX. Type make doc
in
the folder path/to/feltor/doc
and open index.html
(a symbolic link
to dg/html/modules.html
) with your favorite browser.
Finally, also note the documentations of thrust
and cusp.
We maintain tex files in every src folder for
technical documentation, which can be compiled using pdflatex with
make doc
in the respective src folder.
For details on how FELTOR’s internal Makefiles are configured please see the config folder.
FELTOR has been developed by Matthias Wiesenberger and Markus Held. Please see the list of contributors and funding. Also check out our homepage for general information, wiki pages, troubleshooting and guides on how to contribute.
This project is licensed under the MIT license - see LICENSE for details.