idharmateja / moose-core Goto Github PK

Not all compartments are stimulated in a given simulation. Right now inject values for all compartments are transferred irrespective of whether they are stimulated or not. Decrease the amount of transfer by sending only required information.

Right now, cuda calls are bare and don't give any notification if it is not successful. They have to wrapped appropriately.

In this method also, there is a choice to use WPT(work per thread) or CSRMV(matrix vector multiplication). It is observed that load per thread is less and load balance is minimum. Time can be saved by unifying kernels and using WPT approach.

Each module in hsolve step function has memory transfers before and after the computation. Remove some of them which are not necessary/redundant etc..

It is not necessary to wait for solver to complete to send Vmid values to GPU. One can compute (i+1)th part of V array while transferring ith part. This way memory transfer can be hidden.

• Store gates in a rowPtr fashion instead of 3*#channels. This will decrease memory storage.
• Get rid of compressing step.
• Decrease work in calculate channel currents.

How to generate hines matrix from a swc file ?
How to fill hines matrix using conductance values of compartments ?

• 3-way overlap experiment on updateMatrix method.

• Concurrent kernel execution in advanceChannels.
• Hiding memory transfers using streams and async calls.
• Shared memory optimizations ?
• Gauss elimination on GPU.
• Heterogeneous computing.
• Coalesce data access pattern in update matrix.
• Using structureOfArrays instead of arrayOfStructue.

The very first call cusparseCreate() is returning not initialized status.

/usr/bin/ld: hsolve/libhsolve_cuda.a(cuda_compile_generated_AdvanceChannel.cu.o): relocation R_X86_64_32S against .bss' can not be used when making a shared object; recompile with -fPIC hsolve/libhsolve_cuda.a: error adding symbols: Bad value collect2: error: ld returned 1 exit status make[2]: *** [libmoose.so] Error 1 make[1]: *** [CMakeFiles/libmoose.dir/all] Error 2 make[1]: *** Waiting for unfinished jobs.... /usr/bin/ld: hsolve/libhsolve_cuda.a(cuda_compile_generated_AdvanceChannel.cu.o): relocation R_X86_64_32S against.bss' can not be used when making a shared object; recompile with -fPIC
hsolve/libhsolve_cuda.a: error adding symbols: Bad value
collect2: error: ld returned 1 exit status
make[2]: *** [python/moose/_moose.so] Error 1
make[1]: *** [CMakeFiles/_moose.dir/all] Error 2
make: *** [all] Error 2

• Right now matrix information is stored in a vector, which cannot be pinned. Store it in an array form and assign pinned memory.
• Vector is of size 4_nCompt. Only 2_nCompt of that array is required to be transferred from GPU to CPU. Let code view 2*N portion as a row major format.

Write a version of pervasive flow which takes care of above.

Previous approach
It was a two step process. First step is to accumulate currents of channels in a compartment. Second step is to update the matrix data.

It is observed that having multiple kernels results in more simulation time. So unify these two steps, when using work per thread approach.

Is there a specific memory where lookup tables can be stored for better performance.

[ 71%] Building NVCC (Device) object hsolve/CMakeFiles/cuda_compile.dir//./cuda_compile_generated_AdvanceChannel.cu.o
In file included from /usr/include/c++/4.8/random:35:0,
from /home1/teja/Work/moose-core/randnum/RNG.h:33,
from /home1/teja/Work/moose-core/hsolve/../basecode/global.h:27,
from /home1/teja/Work/moose-core/hsolve/HSolveUtils.h:14,
from /home1/teja/Work/moose-core/hsolve/HSolveActive.h:23,
from /home1/teja/Work/moose-core/hsolve/AdvanceChannel.cu:9:
/usr/include/c++/4.8/bits/c++0x_warning.h:32:2: error: #error This file requires compiler and library support for the ISO C++ 2011 standard. This support is currently experimental, and must be enabled with the -std=c++11 or -std=gnu++11 compiler options.
#error This file requires compiler and library support for the
^
CMake Error at cuda_compile_generated_AdvanceChannel.cu.o.cmake:206 (message):
Error generating
/home1/teja/Work/moose-core/_build/hsolve/CMakeFiles/cuda_compile.dir//./cuda_compile_generated_AdvanceChannel.cu.o

make[2]: *** [hsolve/CMakeFiles/cuda_compile.dir/./cuda_compile_generated_AdvanceChannel.cu.o] Error 1
make[1]: *** [hsolve/CMakeFiles/hsolve_cuda.dir/all] Error 2
make: *** [all] Error 2

Error limit reached.
100 errors detected in the compilation of "/tmp/tmpxft_00001351_00000000-7_AdvanceChannel.cpp1.ii".
Compilation terminated.
CMake Error at cuda_compile_generated_AdvanceChannel.cu.o.cmake:264 (message):
Error generating file
~/Work/ncbs/moose-core/build/hsolve/CMakeFiles/cuda_compile.dir//./cuda_compile_generated_AdvanceChannel.cu.o

Commit BhallaLab@d734228 had made changes to hsolve CPU code which have to be taken care in GPU.

Problem : Input is a set of sets and output is a set where each element is the cumulative sum of a set in input. One can assign each thread to take care or map it to SPMV.
In hsolve, the pattern of communication remains same for each time-step. One can do some dry runs and determine which is faster and use it in the simulation.

One can either use WPT(work per thread) approach or SPMV(sparse matrix vector multiplication) approach to compute advanceCalcium method. No one approach is better than other in all use cases. Do some pre-runs and decide before starting simulation.

[ 71%] Building NVCC (Device) object hsolve/CMakeFiles/cuda_compile.dir//./cuda_compile_generated_AdvanceChannel.cu.o
In file included from /usr/include/c++/4.8/random:35:0,
from /home1/teja/Work/moose-core/randnum/RNG.h:33,
from /home1/teja/Work/moose-core/hsolve/../basecode/global.h:27,
from /home1/teja/Work/moose-core/hsolve/HSolveUtils.h:14,
from /home1/teja/Work/moose-core/hsolve/HSolveActive.h:23,
from /home1/teja/Work/moose-core/hsolve/AdvanceChannel.cu:9:
/usr/include/c++/4.8/bits/c++0x_warning.h:32:2: error: #error This file requires compiler and library support for the ISO C++ 2011 standard. This support is currently experimental, and must be enabled with the -std=c++11 or -std=gnu++11 compiler options.
#error This file requires compiler and library support for the
^
CMake Error at cuda_compile_generated_AdvanceChannel.cu.o.cmake:206 (message):
Error generating
/home1/teja/Work/moose-core/_build/hsolve/CMakeFiles/cuda_compile.dir//./cuda_compile_generated_AdvanceChannel.cu.o

make[2]: *** [hsolve/CMakeFiles/cuda_compile.dir/./cuda_compile_generated_AdvanceChannel.cu.o] Error 1
make[1]: *** [hsolve/CMakeFiles/hsolve_cuda.dir/all] Error 2
make: *** [all] Error 2

In order to reduce memory transfer between host and device, we can send Vmid to device and calculate V from Vmid.

GPU alternative for advanceCalcium does not give same results as CPU.

The following script is giving different results when compared to CPU results.
https://github.com/dharmasam9/moose-examples/blob/master/cuda/testScript.py

The CPU build is taking less time, where as CUDA build is taking is much more time.