darma-tasking / cfd-mini-app Goto Github PK

View Code? Open in Web Editor NEW

0.0 0.0 0.0 391 KB

License: Other

Python 21.41% Shell 2.56% CMake 2.32% C++ 71.59% Dockerfile 1.18% Perl 0.95%

cfd-mini-app's People

Contributors

Watchers

cfd-mini-app's Issues

Integrate new parallel mesh handling in solver

This includes removing current mesh chunk object passed as argument to solver.

Move predictor step to mesh objects

One part must be implemented at the parallel mesh level; another one at the mesh chunk level

Checkpoint on issues (08/31)

Major issues
- error in vertical velocity vector component in correction step (in multi color, chunk 0 appears to be most impacted ?)
- single rank single color computation error (originates from correction step)
Minor issues
- VTK file export:
  - fix ghost/shared_owned points erroneous value exported
  - improve usability when dealing with multiple parallel meshes (#58 )
- Docker build needs MPI library
- whitespace and tabs
Next big steps
- scatter and gather pressure in multi node (MPI)
- continue chunk border sending and receiving (MPI)
Miscellaneous
- update unit tests
- document and make diagrams for recent changes

Make Kokkos::Views 2-dimensional to follow cartesian grid

The velocity could look like Kokkos::View<double**[2]> velocity;, as there is a u and v component to it.
The pressure would just be Kokkos::View<double**> pressure;.

Fix usage of wrong factor in predictor step

inv_size_db was removed and replaced by inv_sz2 which isn't the correct factor

Make Laplacian a Kokkos Kernels BlockCRS matrix and use Gauss Seidel linear solve

This is in order to be less dependent on a single part of the application currently implemented by means of a naïve conjugate gradient solve.

Make build of VTK optional

This is to remove the requirement to have a VTK install if visualization is not needed

Fix errors related to Kokkos updates

There have been changes in Kokkos which cause problems in the mini-app; this has to be fixed (example: https://github.com/kokkos/kokkos-kernels/commit/f0c98c4534cc403a170513b63f6dc7fc81fdb2ad "Check that ordinal types are signed" in kokkos-kernels causes a compiler error for different unsigned ints.

Create the MeshChunk parallel object

Add visualization harness

The goal of this issue is to add a visualization backend to this mini-app, for instance by outputting time-varying solutions on a constant mesh via Exodus files (so these might be visualized with. e.g., ParaView).

Add MPI basic integration

No parallel computation functionality, only a ready to use MPI setup.

Docker container does not have VTK which results in a build error during CI

Is this something that you would like to see fixed in CI @lifflander ?

Create a parallel mesh object with indexing scheme

GlobalToLocalNodeIndices -> (i,j) -> ((u,v),(k,l))
GlobalToLocalCellIndices -> (i,j) -> ((u,v),(k,l))
LocalToGlobalNodeIndices -> ((u,v),(k,l)) -> (i,j)
LocalToGlobalCellIndices -> ((u,v),(k,l)) -> (i,j)

Fix index bounds in predictor step

m and n are initialised 1 int to low

memory error in allTests (Clang 11.0.0 build)

Error log with AddressSanitizer on:

./allTests
[==========] Running 1 test from 1 test suite.
[----------] Global test environment set-up.
[----------] 1 test from solver_test
[ RUN      ] solver_test.Laplacian_values_test
=================================================================
==56370==ERROR: AddressSanitizer: stack-use-after-scope on address 0x7ffeea6a99c0 at pc 0x00010556974e bp 0x7ffeea6a91f0 sp 0x7ffeea6a91e8
READ of size 8 at 0x7ffeea6a99c0 thread T0
    #0 0x10556974d in MeshChunk::get_n_cells_x() const mesh_chunk.h:41
    #1 0x105566edd in Solver::solve(Solver::stopping_point, Solver::linear_solver, Solver::adaptative_time_step) solver.cpp:16
    #2 0x10557fa22 in solver_test_Laplacian_values_test_Test::TestBody() allTests.cpp:57
    #3 0x105ed2e3d in void testing::internal::HandleSehExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) gtest.cc:2433
    #4 0x105e9d7da in void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) gtest.cc:2469
    #5 0x105e9d712 in testing::Test::Run() gtest.cc:2508
    #6 0x105e9e951 in testing::TestInfo::Run() gtest.cc:2684
    #7 0x105e9f8e6 in testing::TestSuite::Run() gtest.cc:2816
    #8 0x105eac6f5 in testing::internal::UnitTestImpl::RunAllTests() gtest.cc:5338
    #9 0x105ed72ed in bool testing::internal::HandleSehExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) gtest.cc:2433
    #10 0x105eac06a in bool testing::internal::HandleExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) gtest.cc:2469
    #11 0x105eabf3f in testing::UnitTest::Run() gtest.cc:4925
    #12 0x1055816e0 in RUN_ALL_TESTS() gtest.h:2473
    #13 0x10558157c in main allTests.cpp:107
    #14 0x7fff70a373d4 in start (libdyld.dylib:x86_64+0x163d4)

Address 0x7ffeea6a99c0 is located in stack of thread T0 at offset 64 in frame
    #0 0x10557f5af in solver_test_Laplacian_values_test_Test::TestBody() allTests.cpp:55

  This frame has 28 object(s):
    [32, 176) 'sparse_Laplacian' (line 61) <== Memory access at offset 64 is inside this variable
    [240, 272) 'row' (line 65)
    [304, 312) 'ref.tmp' (line 68)
    [336, 344) 'ref.tmp16' (line 69)
    [368, 384) 'gtest_ar' (line 84)
    [400, 404) 'ref.tmp66' (line 84)
    [416, 424) 'ref.tmp74' (line 84)
    [448, 456) 'ref.tmp77' (line 84)
    [480, 496) 'gtest_ar90' (line 85)
    [512, 516) 'ref.tmp100' (line 85)
    [528, 536) 'ref.tmp109' (line 85)
    [560, 568) 'ref.tmp112' (line 85)
    [592, 608) 'gtest_ar129' (line 88)
    [624, 628) 'ref.tmp137' (line 88)
    [640, 648) 'ref.tmp146' (line 88)
    [672, 680) 'ref.tmp149' (line 88)
    [704, 720) 'gtest_ar192' (line 92)
    [736, 740) 'ref.tmp197' (line 92)
    [752, 760) 'ref.tmp206' (line 92)
    [784, 792) 'ref.tmp209' (line 92)
    [816, 832) 'gtest_ar224' (line 95)
    [848, 852) 'ref.tmp229' (line 95)
    [864, 872) 'ref.tmp238' (line 95)
    [896, 904) 'ref.tmp241' (line 95)
    [928, 944) 'gtest_ar265' (line 99)
    [960, 964) 'ref.tmp268' (line 99)
    [976, 984) 'ref.tmp277' (line 99)
    [1008, 1016) 'ref.tmp280' (line 99)
HINT: this may be a false positive if your program uses some custom stack unwind mechanism, swapcontext or vfork
      (longjmp and C++ exceptions *are* supported)
SUMMARY: AddressSanitizer: stack-use-after-scope mesh_chunk.h:41 in MeshChunk::get_n_cells_x() const
Shadow bytes around the buggy address:
  0x1fffdd4d52e0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x1fffdd4d52f0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x1fffdd4d5300: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x1fffdd4d5310: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x1fffdd4d5320: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
=>0x1fffdd4d5330: f1 f1 f1 f1 f8 f8 f8 f8[f8]f8 f8 f8 f8 f8 f8 f8
  0x1fffdd4d5340: f8 f8 f8 f8 f8 f8 f2 f2 f2 f2 f2 f2 f2 f2 f8 f8
  0x1fffdd4d5350: f8 f8 f2 f2 f2 f2 f8 f2 f2 f2 f8 f2 f2 f2 f8 f8
  0x1fffdd4d5360: f2 f2 f8 f2 f8 f2 f2 f2 f8 f2 f2 f2 f8 f8 f2 f2
  0x1fffdd4d5370: f8 f2 f8 f2 f2 f2 f8 f2 f2 f2 f8 f8 f2 f2 f8 f2
  0x1fffdd4d5380: f8 f2 f2 f2 f8 f2 f2 f2 f8 f8 f2 f2 f8 f2 f8 f2
Shadow byte legend (one shadow byte represents 8 application bytes):
  Addressable:           00
  Partially addressable: 01 02 03 04 05 06 07 
  Heap left redzone:       fa
  Freed heap region:       fd
  Stack left redzone:      f1
  Stack mid redzone:       f2
  Stack right redzone:     f3
  Stack after return:      f5
  Stack use after scope:   f8
  Global redzone:          f9
  Global init order:       f6
  Poisoned by user:        f7
  Container overflow:      fc
  Array cookie:            ac
  Intra object redzone:    bb
  ASan internal:           fe
  Left alloca redzone:     ca
  Right alloca redzone:    cb
  Shadow gap:              cc
==56370==ABORTING
in solve: zsh: abort      ./allTests

Implement parallel meshes as smaller instances of a global mesh

This is in order to be able to consider p parallel meshes that each contain mesh chunks, which are all part of one main computational mesh.

Improve CG performance

Separate testing of serial vs parallel mesh cases

Currently all are glommed in test.cpp

Create branch for parallel version of the code

Docker build failing

Docker build is failing on latest commits as in https://github.com/DARMA-tasking/cfd-mini-app/runs/7769486077?check_suite_focus=true

Add MPI parallel computation functionality in velocity prediction

Add capability to graph parallel speedups with OpenMP

Create solver class and test it

The goal of this issue is to implement the solver class using Kokkos kernels for its linear algebra needs.

Test intermediate steps within the solver flow

Main test points:

after applying BCs
after computing Laplacian
after computing RHS
after first complete iteration (predictor / PPE solve / corrector)

Clean up vtk file writing

When writing vtk files on different parallel meshes, the exported vtk files become quickly messy (unorganized directories etc.) and hard to track. A better way of exporting vtk files is needed.

test code output results and compare vs Python implementation

This is to ensure consistency between the 2 implementations.

create comparative plots along lines across mesh
create comparative 2D visualizations
for both OoIs (velocity and pressure).

create unit test for Laplacian operator matrix computation

There is an algebraic solution for the 3x3 mesh (i.e. 9x9 Laplacian matrix) that can be easily derived manually and verified programmatically.
This should be a good first unit test.

Implement linear solver stub and baseline implementation

The baseline should be an iterative CG solver using Kokkos, as the Laplacian with zero-flow von Neumann boundary conditions result in a positive, symmetric sparse matrix.
N.B.: without a specific treatment said matrix is singular (positive semi definite). It should be verified that the RHS is indeed orthogonal to the (non-trivial) null space.

Turn on adaptive time stepping and add timers between all stages of the main solve routine

This should finalize the alpha version of this miniapp.

Fix velocity update during correction step

See lines:

cfd-mini-app/cppfd/solver.cpp

Lines 349 to 356 in 6760f95

 void Solver::correct_velocity(){ 

 for(uint64_t j = 1; j < this->mesh.get_n_points_y() - 1; j++){ 

 for(uint64_t i = 1; i < this->mesh.get_n_points_x() - 1; i++){ 

 this->mesh.set_velocity_u(i, j, (this->mesh.get_velocity_u(i, j) - (this->delta_t/this->rho) * (this->mesh.get_pressure(i, j) - this->mesh.get_pressure(i-1, j)) * 1/this->mesh.get_cell_size())); 

 this->mesh.set_velocity_v(i, j, (this->mesh.get_velocity_v(i, j) - (this->delta_t/this->rho) * (this->mesh.get_pressure(i, j) - this->mesh.get_pressure(i, j-1)) * 1/this->mesh.get_cell_size())); 

 } 

 } 

 }

Currently the correction is asymmetrical because it is only using pressure contributions from the top and right cells (not from the lower left).

Create unit test for GlobalToLocalCellIndices() and LocalToGlobalCellIndices()

Use e.g. the hard-coded example now in the manual test.cpp:

  uint64_t n_p = 3;
  uint64_t n_q = 2;
  ParallelMesh p_mesh(n_cells, n_cells, 1. / n_cells, n_p, n_q);
  std::map<std::array<uint64_t,2>, uint64_t> block_counts = {};
  for (uint64_t n = 0; n < n_cells; n++)
    for (uint64_t m = 0; m < n_cells; m++){
      LocalCoordinates loc = p_mesh.GlobalToLocalCellIndices(m, n);
      block_counts[{loc.block[0], loc.block[1]}] ++;
    }
  std::cout << "Mesh block counts in "
	    << n_p << " x " << n_q
	    << " partition of "
	    << n_cells << "x" << n_cells
	    << " parallel mesh:\n";
  uint64_t c_total = 0;
  for (const auto& it_block_counts : block_counts){
    auto n_in_block = it_block_counts.second;
    c_total += n_in_block;
    std:: cout << "  block ( "
	       << it_block_counts.first[0] << " ; "
	       << it_block_counts.first[1] << " ): "
	       << n_in_block << "\n";
  }
  std::cout << "  grand total: " << c_total << "\n\n";

and verify that for a 35x35 parallel mesh the following results are obtained:

  block ( 0 ; 0): 216
  block ( 0 ; 1): 204
  block ( 1 ; 0): 216
  block ( 1 ; 1): 204
  block ( 2 ; 0): 198
  block ( 2 ; 1): 187
  grand total: 1225

Regarding the reciprocal function LocalToGlobalCellIndices(), one may verify that the composition of both functions amounts the identity, at least in one direction, as such:

      auto g = p_mesh.LocalToGlobalCellIndices(loc);
      if (m != g[0])
	std::cout << "** ERROR: "
		  << m << " != " << g[0] << "\n";
      if (n != g[1])
	std::cout << "** ERROR: "
		  << n << " != " << g[1] << "\n";

	void Solver::correct_velocity(){
	for(uint64_t j = 1; j < this->mesh.get_n_points_y() - 1; j++){
	for(uint64_t i = 1; i < this->mesh.get_n_points_x() - 1; i++){
	this->mesh.set_velocity_u(i, j, (this->mesh.get_velocity_u(i, j) - (this->delta_t/this->rho) * (this->mesh.get_pressure(i, j) - this->mesh.get_pressure(i-1, j)) * 1/this->mesh.get_cell_size()));
	this->mesh.set_velocity_v(i, j, (this->mesh.get_velocity_v(i, j) - (this->delta_t/this->rho) * (this->mesh.get_pressure(i, j) - this->mesh.get_pressure(i, j-1)) * 1/this->mesh.get_cell_size()));
	}
	}
	}

darma-tasking / cfd-mini-app Goto Github PK

cfd-mini-app's People

Contributors

Watchers

cfd-mini-app's Issues

Recommend Projects

Recommend Topics

Recommend Org