bsc-performance-tools / extrae Goto Github PK

Instrumentation framework to generate execution traces of the most used parallel runtimes.

License: GNU Lesser General Public License v2.1

Makefile 1.81% Shell 1.58% M4 2.82% Fortran 0.44% LLVM 0.01% Java 0.24% C 86.83% Cuda 0.08% C++ 5.67% Python 0.50% Assembly 0.03%

extrae's Introduction

Extrae

A dynamic instrumentation package to trace programs compiled and run with the shared memory model (like OpenMP and pthreads), the message passing (MPI) programming model or both programming models (different MPI processes using OpenMP or pthreads within each MPI process). Extrae generates trace files that can be later visualized with Paraver.

Optional dependencies

Please, make sure to install the development versions of all packages (-dev, -devel) when available.

libunwind (>=1.0)
Used to access the callstack within Extrae. This lets the analyst gather MPI call-sites and emit information on manually added events. Required for Intel x86_64 and ia64 architectures.
PAPI
Used to access the HW counters of the microprocessor. This increases the richness of the gathered traces. It is highly recommended to install and use PAPI with extrae, altough on some old kernels this may require patching.
libiberty and libbfd (from the binutils package)
These two libraries are required to translate gathered application addresses into source code information (file name, address line and function name). Highly recommended.
libxml2 (>=2.5.0)
Used to parse the configuration of the instrumentation package instead of using environment variables. Highly recommended.
Dyninst Support is in experimental status. It is known to work on Linux PPC32/64 systems and on Linux x86/x86_64 systems.
libz
libiberty and libbfd may require libz. Also libz may be used to generate compressed traces directly.
MPI
Execute MPI jobs. Tested with MPICH, MPICH2 and OpenMPI. Others may work.
OpenMP runtime (gcc, ibm or icc)
Execute OpenMP jobs.
CUDA / CUPTI
Execute CUDA-based applications.
OpenCL
Execute OpenCL-based applications.

Installation instructions

If installing from Git, you will need to generate a configure file by running:

$ ./boostrap

then continue with installation as if from a release.

Refer to the INSTALL file for general installation instructions.
Refer to the INSTALL-examples file for examples of specific installation instructions.

extrae's People

Contributors

Stargazers

Watchers

extrae's Issues

Identifying subroutines with too many arguments for Extrae

Hi there. I am attempting to put the IFS model through Extrae, which is an enormous Fortran code base of a few million lines. I am seeing this error

Extrae: Error! Can't retrieve handler to stub '__kmpc_parallel_sched_278_args' (278 arguments)! Quitting!
Extrae:        Recompile Extrae to support this number of arguments!
Extrae:        Use src/tracer/wrappers/OMP/genstubs-kmpc-11.sh to do so.

and wanted to ask about how to interpret it. Does it literally mean that Extrae has encountered a Fortran subroutine with 278 arguments, and that this is above the maximum allowed number which is set during compilation?

There are some subroutines with this many arguments and many more in the IFS (I know...) but it is actually pretty difficult to track them down given the size of the code. If there is a way to get information on the subroutine name from Extrae, that would be very useful.

other bashisms in configure

extrae version 4.0.1

I've run checkbashisms on configure

$ checkbashisms extrae-4.0.1/configure
possible bashism in extrae-4.0.1/configure line 355 ([^] should be [!]):
    sed '/^X\(.*[^/]\)\/\/*[^/][^/]*\/*$/{
possible bashism in extrae-4.0.1/configure line 359 ([^] should be [!]):
	  /^X\(\/\/\)[^/].*/{
possible bashism in extrae-4.0.1/configure line 473 ([^] should be [!]):
    sed '/^.*\/\([^/][^/]*\)\/*$/{
possible bashism in extrae-4.0.1/configure line 537 (unsafe echo with backslash):
  case `echo 'xy\c'` in
possible bashism in extrae-4.0.1/configure line 5000 (should be 'b = a'):
                 if test "${target_cpu}" == "aarch64" ; then
possible bashism in extrae-4.0.1/configure line 5039 (should be 'b = a'):
				 if test "${target_cpu}" == "riscv64" ; then
possible bashism in extrae-4.0.1/configure line 5212 (should be 'b = a'):
if test ${cross_compiling} == "no"; then
possible bashism in extrae-4.0.1/configure line 8700 ([^] should be [!]):
    ac_fc_v_output=`echo $ac_fc_v_output | sed "\
	s/-cmdline  *'[^']*'/ /g; s/-cmdline  *\"[^\"]*\"/ /g
possible bashism in extrae-4.0.1/configure line 8809 ([^] should be [!]):
    ac_fc_v_output=`echo $ac_fc_v_output | sed "\
	s/-cmdline  *'[^']*'/ /g; s/-cmdline  *\"[^\"]*\"/ /g
possible bashism in extrae-4.0.1/configure line 11011 (should be >word 2>&1):
		which dpkg-architecture &> /dev/null
possible bashism in extrae-4.0.1/configure line 11028 (should be 'b = a'):
			if test "C" == "C" ; then
possible bashism in extrae-4.0.1/configure line 11193 (should be >word 2>&1):
		which dpkg-architecture &> /dev/null
possible bashism in extrae-4.0.1/configure line 11210 (should be 'b = a'):
			if test "C++" == "C" ; then
possible bashism in extrae-4.0.1/configure line 12928 (should be 'b = a'):
   if test "x$with_pmpi_hook" == "xdlsym"
possible bashism in extrae-4.0.1/configure line 13242 (should be VAR="${VAR}foo"):
            GASPI_LIBS+="-lGPI2 "
possible bashism in extrae-4.0.1/configure line 13253 ('((' should be '$(('):
      if (( $found == 0 )); then
possible bashism in extrae-4.0.1/configure line 14833 (shopt):
    if ( shopt | grep nocaseglob ) >/dev/null 2>&1; then
possible bashism in extrae-4.0.1/configure line 43705 ([^] should be [!]):
s/^ \('"$ac_word_re"'\)\(([^()]*)\)[	 ]*\(.*\)/P["\1"]="\2"\
D["\1"]=" \3"/p
possible bashism in extrae-4.0.1/configure line 45028 (should be VAR="${VAR}foo"):
			MPI_EXTRA_CAPABILITIES+=" peruse"
possible bashism in extrae-4.0.1/configure line 45031 (should be VAR="${VAR}foo"):
			MPI_EXTRA_CAPABILITIES+=" 1-sided"
possible bashism in extrae-4.0.1/configure line 45034 (should be VAR="${VAR}foo"):
			MPI_EXTRA_CAPABILITIES+=" I/O"
possible bashism in extrae-4.0.1/configure line 45037 (should be VAR="${VAR}foo"):
			MPI_EXTRA_CAPABILITIES+=" MPI_Comm_spawn"
possible bashism in extrae-4.0.1/configure line 45040 (should be VAR="${VAR}foo"):
			MPI_EXTRA_CAPABILITIES+=" MPI3"
possible bashism in extrae-4.0.1/configure line 45042 (echo -e):
		echo -e \\\tMPI home:             ${MPI_HOME}
possible bashism in extrae-4.0.1/configure line 45044 (echo -e):
		echo -e \\\tMPI launcher:         ${MPIRUN}
possible bashism in extrae-4.0.1/configure line 45046 (echo -e):
		echo -e \\\tMPI launcher was not found. It is NOT necessary to compile Extrae but it is necessary to execute the regression tests.
possible bashism in extrae-4.0.1/configure line 45048 (echo -e):
		echo -e \\\tFortran decoration:   ${FORTRAN_DECORATION}
possible bashism in extrae-4.0.1/configure line 45049 (echo -e):
		echo -e \\\tmixed C/Fortran libraries? ${mpi_lib_contains_c_and_fortran}
possible bashism in extrae-4.0.1/configure line 45050 (echo -e):
		echo -e \\\tshared libraries?     ${MPI_SHARED_LIB_FOUND}
possible bashism in extrae-4.0.1/configure line 45051 (echo -e):
		echo -e \\\tMPI capabilities:     ${MPI_EXTRA_CAPABILITIES}
possible bashism in extrae-4.0.1/configure line 45057 (echo -e):
		echo -e \\\tGASPI home:             ${GASPI_HOME}
possible bashism in extrae-4.0.1/configure line 45059 (echo -e):
		echo -e \\\tGASPI launcher:         ${GASPI_BIN_gaspi_run}
possible bashism in extrae-4.0.1/configure line 45061 (echo -e):
		echo -e \\\tGASPI launcher was not found. It is NOT necessary to compile Extrae but it is necessary to execute the regression tests.
possible bashism in extrae-4.0.1/configure line 45069 (echo -e):
			echo -e \\\tGNU OpenMP: yes
possible bashism in extrae-4.0.1/configure line 45071 (echo -e):
			echo -e \\\tGNU OpenMP: no
possible bashism in extrae-4.0.1/configure line 45073 (echo -e):
		echo -e \\\tIBM OpenMP: ${enable_openmp_ibm}
possible bashism in extrae-4.0.1/configure line 45074 (echo -e):
		echo -e \\\tIntel OpenMP: ${enable_openmp_intel}
possible bashism in extrae-4.0.1/configure line 45075 (echo -e):
		echo -e \\\tOMPT: ${enable_openmp_ompt}
possible bashism in extrae-4.0.1/configure line 45083 (echo -e):
    echo -e \\\tOpenSHMEM home:            ${OPENSHMEM_HOME}
possible bashism in extrae-4.0.1/configure line 45091 (echo -e):
		echo -e \\\tSupport for pthread_barrier_wait: ${pthread_barrier_wait_exists}
possible bashism in extrae-4.0.1/configure line 45092 (echo -e):
		echo -e \\\tSupport for pthread_cond_* calls: ${enable_pthread_cond_calls}
possible bashism in extrae-4.0.1/configure line 45100 (echo -e):
		echo -e \\\tCUDA : ${CUDA_HOME}
possible bashism in extrae-4.0.1/configure line 45101 (echo -e):
		echo -e \\\tCUPTI: ${CUPTI_HOME}
possible bashism in extrae-4.0.1/configure line 45105 (echo -e):
			echo -e \\\tCUDA: ${CUDA_HOME}
possible bashism in extrae-4.0.1/configure line 45115 (echo -e):
		echo -e \\\tOpenCL home: ${opencl_path}
possible bashism in extrae-4.0.1/configure line 45123 (echo -e):
		echo -e \\\tOPENACC home:             ${OPENACC_HOME}
possible bashism in extrae-4.0.1/configure line 45129 (echo -e):
		echo -e \\\tJVMTI support: ${JVMTI_found}
possible bashism in extrae-4.0.1/configure line 45131 (echo -e):
			echo -e \\\tAspectJ support: yes
possible bashism in extrae-4.0.1/configure line 45132 (echo -e):
			echo -e \\\tAspectJ compiler: ${AJC}
possible bashism in extrae-4.0.1/configure line 45133 (echo -e):
			echo -e \\\tAspectJ weaver: ${ASPECT_WEAVER_JAR}
possible bashism in extrae-4.0.1/configure line 45135 (echo -e):
			echo -e \\\tAspectJ support: no
possible bashism in extrae-4.0.1/configure line 45146 (echo -e):
			echo -e \\\tPerformance API:  PMAPI
possible bashism in extrae-4.0.1/configure line 45148 (echo -e):
			echo -e \\\tPerformance API:  PAPI
possible bashism in extrae-4.0.1/configure line 45149 (echo -e):
			echo -e \\\tPAPI home:        ${PAPI_HOME}
possible bashism in extrae-4.0.1/configure line 45150 (echo -e):
			echo -e \\\tSampling support: ${PAPI_SAMPLING_ENABLED}
possible bashism in extrae-4.0.1/configure line 45203 (echo -e):
		echo -e callstack access: through libunwind \(${UNWIND_HOME}\)
possible bashism in extrae-4.0.1/configure line 45226 (echo -e):
		echo -e \\\tClustering suite available: ${CLUSTERING_INSTALLED}
possible bashism in extrae-4.0.1/configure line 45227 (echo -e):
		echo -e \\\tSpectral analysis available: ${SPECTRAL_INSTALLED}

Compile OpenMP code in fortran for extrae

Hello,

I need some help to compile a fortran file that's using OpenMP.
I don't know what I missed from the documentation.

Here is how I compile my application:

include Makefile.inc
CFLAGS = -O -fopenmp -g -I$(EXTRAE_HOME)/include
CLIBS = -L$(EXTRAE_HOME)/lib -lomptrace
app_omp:
	gfortran $(CFLAGS) -O3 app.f90 -o app_omp $(CLIBS)

Here is how I execute it:

        source /TOOLS/extrae/current/etc/extrae.sh
        export LD_PRELOAD=${EXTRAE_HOME}/lib/libomptrace.so
        export OMP_NUM_THREADS=20
        export EXTRAE_CONFIG_FILE=extrae.xml
       ./app_omp

And I get the following error:

Extrae: WARNING! omp_get_thread_num_real is a NULL pointer. Did the initialization of this module trigger? Retrying initialization...
Extrae: WARNING! omp_get_thread_num_real is a NULL pointer. Did the initialization of this module trigger? Retrying initialization...
Extrae: WARNING! You have ended executing Extrae's omp_get_thread_num weak symbol! That's likely to happen when you load the tracing library for OpenMP, but your application is not compiled/linked against OpenMP.
Welcome to Extrae 3.5.3
tee: symbol lookup error: /TOOLS/extrae/current/lib/libomptrace.so: undefined symbol: omp_get_max_threads

Any help will be appreciated :)

Jean

undefined reference to symbol 'pfm_get_pmu_info'

configured with
./configure --prefix=/usr --build=x86_64-pc-linux-gnu --host=x86_64-pc-linux-gnu --mandir=/usr/share/man --infodir=/usr/share/info --datadir=/usr/share --sysconfdir=/etc --localstatedir=/var/lib --datarootdir=/usr/share --disable-silent-rules --disable-static --docdir=/usr/share/doc/extrae-4.0.1 --htmldir=/usr/share/doc/extrae-4.0.1/html --with-sysroot=/ --libdir=/usr/lib64 --disable-mic --disable-pmapi --disable-static --enable-shared --without-cupti --without-gm --without-mx --datadir=/var/tmp/portage/sys-cluster/extrae-4.0.1/temp --datarootdir=/var/tmp/portage/sys-cluster/extrae-4.0.1/temp --with-librt=/usr --with-mpi=/usr --with-papi=/usr --with-unwind=/usr --disable-doc --disable-heterogeneous --disable-inotify --enable-instrument_dynamic-memory --enable-instrument_io --enable-instrument_syscall --disable-merge-in-trace --disable-nanos --disable-online --enable-openmp --enable-parallel-merge --disable-pebs-sampling --disable-peruse --enable-posix-clock --disable-pthread --disable-sampling --enable-single-mpi-lib --disable-smpss --enable-xml --without-java-aspectj-weaver --without-java-aspectj --without-clustering --without-boost --without-dyninst --without-dwarf --without-elf --without-java-jdk --without-memkind --without-synapse --without-opencl --without-openshmem --without-fft --without-spectral

the build fails with

/bin/sh ../../libtool  --tag=CC   --mode=link x86_64-pc-linux-gnu-gcc -I../../include -I../../src/common -I../../src/tracer -I../../src/tracer/wrappers/API -I../../src/tracer/clocks -I../../src/tracer/hwc -O -g -O2 -pipe -march=x86-64 -frecord-gcc-switches -fno-diagnostics-color -fmessage-length=0 -fno-optimize-sibling-calls -Wall -W   -L/usr/lib64  -R/usr/lib64 -lpapi -lpfm  -Wl,-O1 -Wl,--as-needed -Wl,--defsym=__gentoo_check_ldflags__=0 -o extrae-cmd extrae_cmd-extrae-cmd.o extrae_cmd-extrae-cmd-init.o extrae_cmd-extrae-cmd-emit.o extrae_cmd-extrae-cmd-fini.o ../../src/tracer/libseqtrace.la ../../src/merger/libmpi2prv.la -lm -L/usr/lib64 -R/usr/lib64 -lbfd -L/usr/lib64 -R/usr/lib64 -liberty   
libtool: link: x86_64-pc-linux-gnu-gcc -I../../include -I../../src/common -I../../src/tracer -I../../src/tracer/wrappers/API -I../../src/tracer/clocks -I../../src/tracer/hwc -O -g -O2 -pipe -march=x86-64 -frecord-gcc-switches -fno-diagnostics-color -fmessage-length=0 -fno-optimize-sibling-calls -Wall -W -Wl,-O1 -Wl,--defsym=__gentoo_check_ldflags__=0 -o .libs/extrae-cmd extrae_cmd-extrae-cmd.o extrae_cmd-extrae-cmd-init.o extrae_cmd-extrae-cmd-emit.o extrae_cmd-extrae-cmd-fini.o  -L/usr/lib64 -Wl,--as-needed ../../src/tracer/.libs/libseqtrace.so -lpapi -lpfm -lunwind -lxml2 -ldl -lrt ../../src/merger/.libs/libmpi2prv.a -lm -lbfd -liberty -Wl,-rpath -Wl,/usr/lib64
/usr/lib/gcc/x86_64-pc-linux-gnu/11.3.0/../../../../x86_64-pc-linux-gnu/bin/ld: ../../src/merger/.libs/libmpi2prv.a(libmpi2prv_la-HardwareCounters.o): undefined reference to symbol 'pfm_get_pmu_info'
/usr/lib/gcc/x86_64-pc-linux-gnu/11.3.0/../../../../x86_64-pc-linux-gnu/bin/ld: /usr/lib64/libpfm.so: error adding symbols: DSO missing from command line
collect2: error: ld returned 1 exit status

See https://bugs.gentoo.org/849617
log https://bugs.gentoo.org/attachment.cgi?id=782648

Extrae build fails with binutils-2.34

binutils-2.34 changes some functions signatures causing undefined reference error when linking Extrae. The binutils porting problem has been reported here

The problem is easily circumvented by replacing the undefined references with the actual names:

vma = bfd_get_section_vma (abfd, section);; -> vma = bfd_section_vma (section);; 
size = bfd_get_section_size (section); -> size = bfd_section_size (section);
if ((bfd_get_section_flags (abfd, section) & SEC_ALLOC) == 0) -> if ((bfd_section_flags (section) & SEC_ALLOC) == 0)

1a215e849 breaks instrumentation for applications with signals

dladdr is not reentrant, in fact it calls a mutex. So if an application has signals (say SIGALRM to execute some code periodically), the signal may happen while Extrae's LookForUFaddress is inside dladdr, and it will call it again to instrument the function in the signal handler.

This causes the application to hang forever with a backtrace looking like this (where trigger is the signal handler):

#0  0x00007ffff7402c94 in __lll_lock_wait () from /usr/lib64/libpthread.so.0
#1  0x00007ffff73fb6a0 in pthread_mutex_lock () from /usr/lib64/libpthread.so.0
#2  0x00007ffff7981a38 in _dl_addr () from /lib64/libc.so.6
#3  0x00007ffff74a1d50 in dladdr () from /usr/lib64/libdl.so.2
#4  0x00007ffff7c18744 in LookForUFaddress (address=<optimized out>) at ../../../src/tracer/UF_gcc_instrument.c:157
#5  0x00007ffff7c1885c in __cyg_profile_func_enter (this_fn=0x7ffff7b78e90 <trigger>, call_site=<optimized out>)
    at ../../../src/tracer/UF_gcc_instrument.c:84
#6  0x00007ffff7b78edc in trigger (signo=<optimized out>, siginfo=0x7fffffffaa58, ctx=<optimized out>) at vulnerability.c:418
#7  <signal handler called>
#8  0x00007ffff73fb680 in pthread_mutex_lock () from /usr/lib64/libpthread.so.0
#9  0x00007ffff7981a38 in _dl_addr () from /lib64/libc.so.6
#10 0x00007ffff74a1d50 in dladdr () from /usr/lib64/libdl.so.2
#11 0x00007ffff7c18744 in LookForUFaddress (address=<optimized out>) at ../../../src/tracer/UF_gcc_instrument.c:157
#12 0x00007ffff7c189f4 in __cyg_profile_func_exit (this_fn=<optimized out>, call_site=<optimized out>) at ../../../src/tracer/UF_gcc_instrument.c:110
[...]

This has happened to me with a preloaded extrae and user functions enabled, using gcc's -finstrument-functions.

A solution could be to user the addresses of the symbols directly rather than their relative address in the library, or to mask signals for the call and restore them afterwards, or even to use a reentrant alternative to dladdr.

Attempting to use an MPI routine after finalizing MPICH (ParaStationMPI)

I think this is unrelated to the above, but when I tried using my self-built Extrae I got this error on exiting the program (otherwise normally)

Attempting to use an MPI routine after finalizing MPICH

As far as I can tell, this is indeed emitted after I've called MPI_FINALIZE. In fact I don't think there are any subroutine or function calls at all after MPI_FINALIZE.

I'm actually using ParastationMPI but this is a fork of MPICH. Have you any ideas what is happening here?

Originally posted by @samhatfield in #76 (comment)

Getting unknown type name error while running make during installation

common/bfd_manager.c:228:78: error: unknown type name ‘PTR’
  228 | static void BFDmanager_findAddressInSection (bfd * abfd, asection * section, PTR data)
      |                                                                              ^~~
common/bfd_manager.c: In function ‘BFDmanager_translateAddress’:
common/bfd_manager.c:300:36: error: ‘BFDmanager_findAddressInSection’ undeclared (first use in this function)

Added below line in common/bfd_manager.c for it to work:

#define PTR char*

undefined reference to pthread_mutex_trylock

Hi,

I am trying to build extrae but I get this error:

/usr/bin/ld: ../../src/tracer/.libs/libseqtrace.so: undefined reference to `pthread_mutex_trylock'
collect2: error: ld returned 1 exit status

I found some information about this problem here but I am not sure how to resolve it for extrae.

Could you provide any help?

Failure on Fedora30 (Dyninst 10?)

Building fails on Fedora 30. It's probably due to Dyninst 10, but I haven't investigated as F30 isn't of interest for using extrae. There's output from the failed build that won't last too long at https://copr-be.cloud.fedoraproject.org/results/loveshack/livhpc/fedora-30-x86_64/00920021-extrae/build.log.gz

Extrae for an MPI Python script, in a Singularity container

Hi,

I have set up a Singularity container on a cluster, and I am trying to profile my MPI Python application. Before I ask other detailed questions, is it possible to profile an MPI Python script with Extrae? It is not in the documentation, and the examples and I suspect it is actually not possible given that I have run into error both with the static and dynamic approach.

Thanks,
Best regards,

errors in test scripts

When trying to get make check working, I noticed that some test scripts use $(...) rather than ${...}. There's a patch at
https://copr-dist-git.fedorainfracloud.org/cgit/loveshack/livhpc/extrae.git/plain/extrae-tests.patch?h=epel7 reproduced below. (I haven't yet tested the current version of the RPM packaging it's from, in case you're interested in that.)

diff -rN -u old/tests/functional/launcher/test-binary-rewrite.sh new/tests/functional/launcher/test-binary-rewrite.sh
--- old/tests/functional/launcher/test-binary-rewrite.sh	2018-09-24 21:07:18.350597378 +0100
+++ new/tests/functional/launcher/test-binary-rewrite.sh	2018-09-24 21:07:18.450597378 +0100
@@ -9,7 +9,7 @@
 rm -fr TRACE.sym TRACE.mpits set-0
 
 #export EXTRAE_HOME=`cat ../../../PREFIX`
-export EXTRAE_HOME=$(top_builddir)
+export EXTRAE_HOME=${top_builddir}
 
 if test ! -x ${EXTRAE_HOME}/bin/extrae ; then
 	echo "Could not execute binary rewriter test because ${EXTRAE_HOME}/bin/extrae has not been installed. Run make install first."
diff -rN -u old/tests/functional/launcher/test-dyninst-2.sh new/tests/functional/launcher/test-dyninst-2.sh
--- old/tests/functional/launcher/test-dyninst-2.sh	2018-09-24 21:07:18.350597378 +0100
+++ new/tests/functional/launcher/test-dyninst-2.sh	2018-09-24 21:07:18.450597378 +0100
@@ -9,7 +9,7 @@
 rm -fr TRACE.sym TRACE.mpits set-0
 
 #export EXTRAE_HOME=`cat ../../../PREFIX`
-export EXTRAE_HOME=$(top_builddir)
+export EXTRAE_HOME=${top_builddir}
 
 if test ! -x ${EXTRAE_HOME}/bin/extrae ; then
 	echo "Could not execute binary rewriter test because ${EXTRAE_HOME}/bin/extrae has not been installed. Run make install first."
diff -rN -u old/tests/functional/launcher/test-dyninst.sh new/tests/functional/launcher/test-dyninst.sh
--- old/tests/functional/launcher/test-dyninst.sh	2018-09-24 21:07:18.350597378 +0100
+++ new/tests/functional/launcher/test-dyninst.sh	2018-09-24 21:07:18.450597378 +0100
@@ -9,7 +9,7 @@
 rm -fr TRACE.sym TRACE.mpits set-0
 
 #export EXTRAE_HOME=`cat ../../../PREFIX`
-export EXTRAE_HOME=$(top_builddir)
+export EXTRAE_HOME=${top_builddir}
 
 if test ! -x ${EXTRAE_HOME}/bin/extrae ; then
 	echo "Could not execute binary rewriter test because ${EXTRAE_HOME}/bin/extrae has not been installed. Run make install first."

Feature Request: Allow specification of binary when calling mpi2prv

I am using Extrae (v3.5.4) on our Cray system. The aprun command copies the binary to a temporary directory on the respective node(s) and launches it from there, even if the working directory is shared. Because I am currently running my MPI application with only one process I have to run mpi2prv manually, which cannot find the binary in the temporary directory:

mpi2prv: Warning: Couldn't open /var/opt/cray/alps/spool/1666386/a.out for reading, addresses may not be translated.

It would be helpful to be able to specify the path to the original binary when invoking mpi2prv so that extrae can extract symbols from there. I did not see any command line option for this.

MPI+OmpSs: Found X unmatched/pending communications

Using below example with Extrae 3.5.3, I get the following error (although it more sounds like a warning):

mpi2prv: Error! Found 2 unmatched communications. Resulting tracefile may be inconsistent.
mpi2prv: Error! Found 5846 pending communications. Resulting tracefile may be inconsistent.

The code:

#include <mpi.h>
#include <assert.h>
#include <stdio.h>

static void wait(MPI_Request *req)
{
  do {
    int flag;
    MPI_Testall(1, req, &flag, MPI_STATUSES_IGNORE);
    if (flag) break;
  } while (1);
}

int main(int argc, char **argv)
{
  int provided, rank, size;
  MPI_Init_thread(&argc, &argv, MPI_THREAD_MULTIPLE, &provided);
  assert(provided == MPI_THREAD_MULTIPLE);
  MPI_Comm_rank(MPI_COMM_WORLD, &rank);
  MPI_Comm_size(MPI_COMM_WORLD, &size);
  int val = 0;

#pragma omp parallel
{
#pragma omp single
{
  for (int i = 0; i < 10000; ++i) {
#pragma omp task 
{
    MPI_Request req = MPI_REQUEST_NULL;
    if (rank == 0) {
      MPI_Isend(&val, 1, MPI_INT, 1, i, MPI_COMM_WORLD, &req);
      wait(&req);
    } else if (rank == 1) {
      MPI_Irecv(&val, 1, MPI_INT, 0, i, MPI_COMM_WORLD, &req);
      wait(&req);
    }
}

  }
} // pragma omp single
#pragma omp taskwait
} // pragma omp parallel

  MPI_Finalize();
  return 0;
}

I have seen this error both on a Cray XC40 and a KNL-based system. I wonder whether this is a bug in Extrae?

I also see the following fatal error at times with the same code above but cannot reproduce it reliably:

mpi2prv: Error! Invalid group for foreign receive. Dying...

Not sure what could cause this...

Segmentation fault when EXTRAE_CONFIG_FILE is wrong

I noticed that when the EXTRAE_CONFIG_FILE environment variable points to an unexisting file the library crashes (at least when is used with Nanos++).
For example:

[user@machine ompss_testbench]$ export EXTRAE_CONFIG_FILE=""
[user@machine ompss_testbench]$ NX_ARGS="--instrumentation=extrae --schedule=dbf" ./matmul/mm-i 2048 128 0
Welcome to Extrae 3.4.3
Extrae: Parsing the configuration file () begins
I/O warning : failed to load external entity ""
Extrae: Error! xmlParseFile routine failed!
Extrae: Dynamic memory instrumentation is disabled.
Extrae: Basic I/O memory instrumentation is disabled.
Extrae: System calls instrumentation is disabled.
Extrae: Parsing the configuration file () has ended
Segmentation fault (core dumped)
[user@machine ompss_testbench]$ unset EXTRAE_CONFIG_FILE
[user@machine ompss_testbench]$ NX_ARGS="--instrumentation=extrae --schedule=dbf" ./matmul/mm-i 2048 128 0
Welcome to Extrae 3.4.3
Extrae: Generating intermediate files for Paraver traces.
Extrae: Intermediate files will be stored in "awesome folder"
Extrae: Tracing buffer can hold 500000 events
Extrae: Tracing mode is set to: Detail.
Extrae: Successfully initiated with 1 tasks and 64 threads


_EXEC_TIME_CYCLE_H_: 721433916

_USER_TIME_CYCLE_H_: 45000842483

_SYSTEM_TIME_CYCLE_H_: 380714
Test successfully executed!
(.......)
Extrae: Deallocating memory.
Extrae: Application has ended. Tracing has been terminated.

Configure does not detect libbfd

I'm natively building on an arm64 machine with libbfd installed (see below find) and extrae configure is not able to find the library. It is not found either with the --with-binutils option not without it.

Output of find command:

user@Node-arm64:~/$ find /usr/ -name libbfd\*.so
/usr/lib/aarch64-linux-gnu/libbfd-2.26-system.so
/usr/lib/aarch64-linux-gnu/libbfd.so

Output of configure:

user@Node-arm64:~/gitrepos/extrae/build-arm64$ ../configure --without-mpi --without-unwind --enable-posix-clock --without-dyninst --enable-openmp-ompt --without-papi --enable-arm64 --with-binutils=/usr
...
checking for binutils... notfound
configure: Warning! Cannot find the libbfd library in the given binutils home. Please, make sure that the binutils packages is correctly installed. If you have installed the binutils package by hand from their source code, make sure to add --enable-shared in its configure execution.
configure: libiberty library directory: /usr/lib/aarch64-linux-gnu
configure: error: Error! Cannot find binutils home in the given path! Check for the previous warning messages. Check for the given path or whether the binutils development packages -binutils-dev or binutils-devel- are installed.

user@Node-arm64:~/gitrepos/extrae/build-arm64$ ../configure --without-mpi --without-unwind --enable-posix-clock --without-dyninst --enable-openmp-ompt --without-papi --enable-arm64
...
checking for binutils... notfound
configure: Warning! Cannot find the libbfd library in the given binutils home. Please, make sure that the binutils packages is correctly installed. If you have installed the binutils package by hand from their source code, make sure to add --enable-shared in its configure execution.
configure: libiberty library directory: /usr/lib/aarch64-linux-gnu
checking bfd.h usability... yes
checking bfd.h presence... yes
checking for bfd.h... yes
checking whether libbfd and libiberty work... no, see config.log for further details
configure: error: You can gather call-site information which must be translated using binutils, but either libbfd or libiberty are not found. Please make sure that the binutils-dev package is installed and specify where to find these libraries through --with-binutils. The latest source can be downloaded from http://www.gnu.org/software/binutils

The problem is in config/macros.m4 where the find command is limited to 1 depth level. This restriction is valid if we are cross-compiling and multiarch_triplet is set.

Intercepting MPI calls made in external library

Note: This problem is related to the usage of Extrae with our solver.

Question

Is it possible to intercept the MPI calls made via an external library? If so, could you please explain how?

MPI usage

Normal case: I have an external MPI library plugin (say e.g. mpi-wrapper.so) that I link to the solver (say e.g. EXE). At the runtime, the MPI call hierarchy is: EXE -> mpi-wrapper.so -> libmpi.so. In this case, there is no MPI information recorded in the trace when I use Extrae.

Test case: If I change the MPI call hierarchy as EXE -> libmpi.so, MPI calls are intercepted by Extrae.

Attempts to trace MPI calls

LD_PRELOAD => NOK
Linking libmpitrace.so to EXE => NOK
Linking libmpitrace.so to EXE and to mpi-wrapper.so => NOK

Is there any way to generate MPI information in the trace using Extrae for the normal case mentioned above? Do you have any suggestions for this problem?

Extrae build failure on IBM power 8

Dear BSC Developers

I am trying to build extrae on IBM power 8 machine but it failed compiling commonSnippets.C. The following are the build environments and error messages.

Build environment:
    Machine: IBM Power 8
    OS: Red Hat Enterprise Linux Server 7.4 (Maipo)
    Compiler: GCC/8.2.0
    CUDA: cuda/10.1
    OpenMPI: 4.0.1
    Boost:  1.72.0
    dyninst: 10.1.0
    extrae commit: 5ccdf0c, 2020-08-21 19:43:13, tag 3.8.3


Error messages:
    commonSnippets.C: In function std::__cxx11::string decodeBasicBlocks(BPatch_function*, std::__cxx11::string):
    commonSnippets.C:486:34: error: no match for operator= (operand types are Dyninst::InstructionAPI::Instruction::Ptr 
    {aka boost::shared_ptr<Dyninst::InstructionAPI::Instruction>} and Dyninst::InstructionAPI::Instruction)
     while((insn = dec.decode())) {
                              ^
     In file included from ....../boost/1.72.0/ppc64le/include/boost/shared_ptr.hpp:17,
             from ... .../dyninst-10.1.0/build/include/dyn_regs.h:36,
             from ... .../dyninst-10.1.0/build/include/dyntypes.h:183,
             from ... .../dyninst-10.1.0/build/include/util.h:223,
             from ... .../dyninst-10.1.0/build/include/InstructionAST.h:41,
             from ... .../dyninst-10.1.0/build/include/Expression.h:35,
             from ... .../dyninst-10.1.0/build/include/Instruction.h:38,
             from ... .../dyninst-10.1.0/build/include/BPatch_point.h:57,
             from commonSnippets.C:48:
... .../include/boost/smart_ptr/shared_ptr.hpp:547:18: note: candidate: boost::shared_ptr<T>& 
 boost::shared_ptr<T>::operator=(const boost::shared_ptr<T>&) [with T = Dyninst::InstructionAPI::Instruction]
 shared_ptr & operator=( shared_ptr const & r ) BOOST_SP_NOEXCEPT

Any advice to fix this? Thank you very much!

Shelton Ma

papi libs are not found in their default location

Using a standard configure && make && make install process for papi on ubuntu 18.04 it installs itself in "lib" but extrae configure looks at "lib64". This may be considered a papi bug as well.

error in compiling commonSnippets.C

When compiling Extrae-3.7.1 with Dyninst-1.10.0, I get the following error:

commonSnippets.C:486:34: error: no match for ‘operator=’ (operand types are ‘Dyninst::InstructionAPI::Instruction::Ptr’ {aka 
‘boost::shared_ptr<Dyninst::InstructionAPI::Instruction>’} and ‘Dyninst::InstructionAPI::Instruction’)
  486 |         while((insn = dec.decode())) {
      |                                  ^
In file included from /u/aragagnin/opt/tools/dyninst-10.1.0//include/boost/shared_ptr.hpp:17,
             from /opt/tools/dyninst-10.1.0//include/dyn_regs.h:36,
             from /opt/tools/dyninst-10.1.0//include/dyntypes.h:183,
             from /opt/tools/dyninst-10.1.0//include/util.h:223,
             from /opt/tools/dyninst-10.1.0//include/InstructionAST.h:41,
             from /opt/tools/dyninst-10.1.0//include/Expression.h:35,
             from /opt/tools/dyninst-10.1.0//include/Instruction.h:38,
             from /opt/tools/dyninst-10.1.0//include/BPatch_point.h:57,

Looks like decode() returns a object Instruction while Extrae expects a Instruction::Ptr.
In fact, in the source code of Dyninst, we can see that the function signature returns a object:

$grep decode -A10 -B10  /opt/tools/dyninst-10.1.0//include/InstructionDecoder.h
[...]
class InstructionDecoderImpl;

class INSTRUCTION_EXPORT InstructionDecoder
{
  friend class Instruction;
    public:
    static const unsigned int maxInstructionLength = 16;
[...]
 /// Decode the current instruction in this %InstructionDecoder object's buffer, interpreting it as
  /// machine language of the type understood by this %InstructionDecoder.
  /// If the buffer does not contain a valid instruction stream, a null %Instruction pointer
  /// will be returned.  The %Instruction's \c size field will contain the size of the instruction decoded.
  Instruction decode();
[...]

libunwind should be found in default location

On a standard ubuntu 18.04 you still need to give --with-unwind=/usr

OMPT initialization avoided

When using OMPT tracing from Nanos++ runtime, the call to ompt_initialize does not do anything.
I realized that ompt_enabled in this line is initialized during the XML parsing. However, the Extrae_init is called two lines below so the ompt is never initialized.

configure fail with POSIX shell

When configuring with the dash shell it fail with:
./configure: 10549: Syntax error: redirection unexpected
Please make IFS=' ' read -ra MPICC_parts <<< "${MPICC}" in config/mpi-macros.m4 POSIX compatible

Deadlock in random() function

The PrepareNextAlarm function is called in a signal handler and uses random(), causing a deadlock if the signal has interrupted a call to random() done by the application. Using random_r() instead of random() should solve the issue.

configure fails to detect missing libiberty

Running configure on Extrae 3.5.2 with the following result on my local laptop:

libbfd available: yes (/usr/lib/x86_64-linux-gnu)
libiberty available: yes ()
zlib available: yes (/usr)
libxml2 available: yes (/usr)
BOOST available: no
callstack access: through backtrace (from linux)

I get the following error during linking:

make[4]: Entering directory '/home/joseph/src/extrae-3.5.2/build/src/merger/parallel'
  CCLD     mpimpi2prv
libtool:   error: require no space between '-L' and '-liberty'

It appears that configure does not correctly detect if libiberty is not installed correctly. Installing the package libiberty-dev fixes the problem but imo configure should detect the missing library.

missing DESTDIR in makefile

I had to add DESTDIR in some places to build an rpm. I used this patch with 3.6.1: https://copr-dist-git.fedorainfracloud.org/cgit/loveshack/livhpc/extrae.git/plain/extrae-inst.patch?h=epel7

fix for installation

I needed this change to build a package from 3.7.1 with $MPI_HOME/etc as sysconfig and non-null DESTDIR.

--- extrae-3.7.1/src/tracer/online/Makefile.am~	2019-07-05 11:24:32.000000000 +0000
+++ extrae-3.7.1/src/tracer/online/Makefile.am	2019-07-21 13:52:50.714653738 +0000
@@ -151,7 +151,7 @@
 .PHONY : $(SCRIPT_ONLINE_ENV) $(SCRIPT_ONLINE_ROOT)
 
 install-data-hook:
-	$(MKDIR_P) $(prefix)/etc
+	$(MKDIR_P) $(DESTDIR)$(sysconfdir)
 	cp $(SCRIPT_ONLINE_ENV) $(DESTDIR)$(sysconfdir)
 	cp $(SCRIPT_ONLINE_ROOT) $(DESTDIR)$(bindir)
 	chmod a+x $(DESTDIR)$(bindir)/$(SCRIPT_ONLINE_ROOT)

fix for using shared synapse lib

I needed this patch to build 3.6.1 with a shared synapse library: https://copr-dist-git.fedorainfracloud.org/cgit/loveshack/livhpc/extrae.git/plain/extrae-m4.patch?h=epel7

avoid `AC_TRY_RUN` in autoconf scriptage to allow cross-compiling

Hello,

while trying to package extrae for Julia binarybuilder, we hit a problem where it cannot be easily cross-compiled to many platforms. The configure script fails in almost all cases with a message like

configure: error: cannot run test program while cross compiling

because AC_TRY_RUN macros cannot be used in cross builds.

The problematic uses include:

checking for various sizes of MPI things at 3 places here https://github.com/bsc-performance-tools/extrae/blob/master/config/mpi-macros.m4#L414 , which can be easily done without running the program -- you can force the compiler to produce a message that contains the statically obtained result (say using #warn) and grab it from there. Btw. the latter 2 checks should use offsetof.)
checking for pointer sizes and endianiness around https://github.com/bsc-performance-tools/extrae/blob/master/config/macros.m4#L458 , which should be always taken from the build environment (and if not, it can still be determined statically)
checking for libxml2 version here https://github.com/bsc-performance-tools/extrae/blob/master/config/libxml2.m4#L61 which should instead be done with proper featurechecks

The tracking PR for Binarybuilder is at JuliaPackaging/Yggdrasil#5710 -- I disabled most of the architectures so it looks pretty much okay now (I also squashed git history so the failed tests cannot be seen, but you can reproduce the failures simply with julia build_tarballs.sh). Having a better platform coverage would be great.

As an alternative, is there a way to somehow pass the test values to configure script manually, so that it doesn't need to run the tests?

Thanks!
-mk

dyninst is considered experimental but still needs to be disabled explicitly

experimental packages should probably be disabled by default

Race condition in handling MPI requests

The Symptom

I am seeing my application crash randomly with (Intel) MPI complaining about negative ranks being passed to MPI_Group_translate_ranks when running under Extrae 3.5.2 on a KNL system with 64 pthreads managed by my own runtime. All threads perform various asynchronous MPI operations, e.g., MPI_Isend, MPI_Irecv, MPI_Rget, and test them using MPI_Testsome and MPI_Test. Everything works fine if Extrae is not used.

Behavior of Extrae

It appears that the wrapper for MPI_Testsome calls the test function on the array of requests and iterates over the the returned status objects to perform some book-keeping for completed requests that were previously added to the hash table, i.e., requests created using MPI_Irecv. Eventually, the requests are removed from the hash table one-by-one.

Problem Analysis

There is a race condition between the time the requests have completed (MPI_Testsome returns) and the time the requests are removed from the hash table. That time can be significant for large numbers of requests (1k in my case). Unfortunately, MPI implementations tend to re-use requests that are finished. That means that during that time another thread might receive a request from a different MPI function that is still stored in the hash table and test for its completion. In that case, get_Irank_obj_C is called for that request and if the request was not re-used in a call to MPI_Irecv (but for example MPI_Rget) the returned status contains undefined values as the MPI standard does not mandate the status to contain anything meaningful in that case. Hence, the call to MPI_Group_translate_ranks fails. If the the request was re-used in MPI_Irecv the request would be removed from the hash table even though it's still active (again), leading to an error in Extrae.

** Possible Solution **

The only clean solution I see is to remove all requests from the hash table before testing them and later adding those that have not completed. There might be other solutions, though.

Infinite loop in signal handler

I am having trouble terminating Extrae (v3.5.4) in case of an error, e.g., MPI error or segmentation fault. I am seeing an endless stream of messages like the following:

Extrae: Attention! Signal 11 (Segmentation fault) caught. Notifying to flush buffers whenever possible.
Extrae: Attention! Signal 11 (Segmentation fault) caught. Notifying to flush buffers whenever possible.
Extrae: Attention! Signal 11 (Segmentation fault) caught. Notifying to flush buffers whenever possible.

Repeated attempts to quick the application using Ctrl+C don't help. My only option was to kill the interactive session...

Looking at the signal handler SigHandler_FlushAndTerminate the above line is printed every time a signal is caught after the first signal has been caught. While signal handlers are quite limited in what they should do (I/O is not part of it) it is good to attempt to flush the buffers before exiting. However, if the application is in an invalid state more signals might occur, leading to this infinite loop. The signal handler should instead give up (potentially after N signals caught) and exit the application without flushing the buffer to avoid waiting for the application to run into either the walltime limit or the filesystem quota (the log file grows fairly quickly...).

Even for SIGINT, the signal handler should not wait for the flush to finish. The user may want to interrupt flushing buffers, e.g., because he deems the measurements unusable based on the output of the application. In all cases, an infinite loop should be avoided.

[mpi2prv] Addresses not resolved or resolved incorrectly with PIE executables

While generating traces with Extrae and analyzing them with wxParaver on my local computer (at updated Arch Linux install), I observed that the generated traces in Paraver format did either not resolve the names of source functions (they were marked as 'Unresolved'), or they were mapped incorrectly, i.e. the function that was associated was not the one that was actually running at that point in time. However, while extracting traces in the MareNostrum 4 environment, I could not replicate this problem.

This happens even with the simplest of applications, compiled with debug symbols and no optimizations. I attach one small OpenMP application where I can reproduce the problem on my computer:
dsym.tar.gz

After some testing, I think I have mostly diagnosed the source of the problem. The first clue is that the main executable on my computer is mapped to some high address (which is different every time I run the application):

TRACE@[...].sym generated on my local environment:

B 0 "55ad32e3f000-55ad32e40000 1000 /home/zealcharm/dsym/trace_arch_pie/exes/main"
B 0 "7f739b9d4000-7f739ba8c000 8a000 /home/zealcharm/dsym/trace_arch_pie/exes/libstdc++.so.6.0.25"
[...]

TRACE@[...].sym generated on the MareNostrum 4 environment:

B 0 "400000-401000 0 /home/zealcharm/dsym/trace_mn4/exes/main"
B 0 "7fe34c1d2000-7fe34c33b000 0 /home/zealcharm/dsym/trace_mn4/exes/libpfm.so.4.10.1"
[...]

Indeed, this appears to be due to Arch Linux enabling PIE executables on the default GCC configuration. Note the --enable-default-pie:

$ gcc -v
Using built-in specs.
COLLECT_GCC=gcc
COLLECT_LTO_WRAPPER=/usr/lib/gcc/x86_64-pc-linux-gnu/8.3.0/lto-wrapper
Target: x86_64-pc-linux-gnu
Configured with: /build/gcc/src/gcc/configure --prefix=/usr --libdir=/usr/lib --libexecdir=/usr/lib --mandir=/usr/share/man --infodir=/usr/share/info --with-bugurl=https://bugs.archlinux.org/ --enable-languages=c,c++,ada,fortran,go,lto,objc,obj-c++ --enable-shared --enable-threads=posix --enable-libmpx --with-system-zlib --with-isl --enable-__cxa_atexit --disable-libunwind-exceptions --enable-clocale=gnu --disable-libstdcxx-pch --disable-libssp --enable-gnu-unique-object --enable-linker-build-id --enable-lto --enable-plugin --enable-install-libiberty --with-linker-hash-style=gnu --enable-gnu-indirect-function --enable-multilib --disable-werror --enable-checking=release --enable-default-pie --enable-default-ssp --enable-cet=auto
Thread model: posix
gcc version 8.3.0 (GCC)

Running file on the executables also reveals that the one I built on my local computer is compiled with PIE, while the one on MN4 isn't.

$ file dsym/trace_arch_pie/exes/main
dsym/trace_arch_pie/exes/main: ELF 64-bit LSB pie executable, x86-64 [...]
$ file dsym/trace_mn4/exes/main 
dsym/trace_mn4/exes/main: ELF 64-bit LSB executable, x86-64  [...]

To further confirm this, compiling the executable with gcc's -no-pie on my local computer fixes the problem.

Here are some traces I generated of the three scenarios (I have included all binaries references in the .sym file, but you may need to fix the paths to wherever you extract them):
MareNostrum 4 (OK): trace_mn4.tar.gz
Local Arch Linux, with default PIE (KO): trace_arch_pie.tar.gz
Local Arch Linux, with -no-pie (OK): trace_arch_no_pie.tar.gz

As for the actual point where the problem happens in Extrae, from some tests I have done, I believe the function addresses are correctly recorded in the .mpit files, but the trace merger (mpi2prv) is failing to map them correctly.

The data that arrives at Translate_Address@src/merger/paraver/addr2info.c appears to be correct, but neither of the two calls at BFDmanager_translateAddress@src/merger/common/bfd_manager.c is able to resolve the address. The first call is done with address (the mapped/relocated PC of the function, i.e. the big address like 0x55ad32[...]), which is not found in the binary since the mapping is done at runtime. The second call is done with address - start_address, which is better but still wrong by an offset of 0x1000. So I guess there's some missing address/section mapping to account for PIE around that place in the source, though I'm not knowledgeable enough to find out which.

Regards.

Parallel make install fails

I frequently use make install -j4 to combine building and installation of configured projects into one command. For Extrae, this breaks:

$ make install -j4
[...]
cp extrae-vars.sh $HOME/opt/extrae-3.5.4-gnu7.3.0-shared/etc
rm -fr $HOME/opt/extrae-3.5.4-gnu7.3.0-shared/etc/extrae-vars.sh $HOME/opt/extrae-3.5.4-gnu7.3.0-shared/etc/extrae-vars.csh
Preparing configured script for SH
echo "export PACKAGE_NAME=Extrae" >> extrae-vars.sh
echo "export [email protected]" >> extrae-vars.sh
chmod 0644 $HOME/opt/extrae-3.5.4-gnu7.3.0-shared/etc/extrae-vars.sh
chmod: cannot access '$HOME/opt/extrae-3.5.4-gnu7.3.0-shared/etc/extrae-vars.sh': No such file or directory
Makefile:757: recipe for target 'SHscripts' failed
make[3]: *** [SHscripts] Error 1
make[3]: *** Waiting for unfinished jobs....
echo "export CONFIGURE_LINE=\"`head -7 ../config.log | tail -1`\"" >> extrae-vars.sh
cp ../../etc/configured.sh $HOME/opt/extrae-3.5.4-gnu7.3.0-shared/etc
chmod 0755 $HOME/opt/extrae-3.5.4-gnu7.3.0-shared/etc/configured.sh
make[3]: Leaving directory '$HOME/src/extrae/extrae-3.5.4/build/etc'
Makefile:659: recipe for target 'install-data-am' failed
make[2]: *** [install-data-am] Error 2
make[2]: Leaving directory '$HOME/src/extrae/extrae-3.5.4/build/etc'
Makefile:614: recipe for target 'install-am' failed
make[1]: *** [install-am] Error 2
make[1]: Leaving directory '$HOME/src/extrae/extrae-3.5.4/build/etc'
Makefile:1097: recipe for target 'install-recursive' failed
make: *** [install-recursive] Error 1

It seems that there is a race in the creation and installation of the extrae-vars.sh file.

TLS free() interception causes a segfault with extrae 4.0.0

The new version seems to have introduced a problem with the free interception. I'm using a OmpSs-2 program with the libnanosmpitrace.so library, following this example.

It seems that the function xtr_mem_tracked_allocs_remove is called in the free wrapper, which in turn executes the TLS implementation of the glibc, which calls the free function, causing an infinite loop, and later a segfault. I believe this problem is caused by the access to the mallocentries variable:

extrae/src/tracer/wrappers/MALLOC/malloc_wrapper.c

Line 143 in f5f80f6

if (mallocentries == NULL) xtr_mem_tracked_allocs_initlist();

extrae/src/tracer/wrappers/MALLOC/malloc_wrapper.c

Line 87 in f5f80f6

#define NMALLOCENTRIES_MALLOC 16*1024

Here is the backtrace:

(gdb) bt 20
#0  0x00007f2b49132e63 in __tls_get_addr () from /nix/store/33idnvrkvfgd5lsx2pwgwwi955adl6sk-glibc-2.31/lib/ld-linux-x86-64.so.2
#1  0x00007f2b486cfec5 in xtr_mem_tracked_allocs_remove () from /nix/store/pl4zhp9sbklcgj0031f17n12gbwszhqv-extrae-4.0.0/lib/libnanosmpitrace.so
#2  0x00007f2b486d0019 in free () from /nix/store/pl4zhp9sbklcgj0031f17n12gbwszhqv-extrae-4.0.0/lib/libnanosmpitrace.so
#3  0x00007f2b4912d7d6 in _dl_update_slotinfo () from /nix/store/33idnvrkvfgd5lsx2pwgwwi955adl6sk-glibc-2.31/lib/ld-linux-x86-64.so.2
#4  0x00007f2b4912d8fc in update_get_addr () from /nix/store/33idnvrkvfgd5lsx2pwgwwi955adl6sk-glibc-2.31/lib/ld-linux-x86-64.so.2
#5  0x00007f2b49132e68 in __tls_get_addr () from /nix/store/33idnvrkvfgd5lsx2pwgwwi955adl6sk-glibc-2.31/lib/ld-linux-x86-64.so.2
#6  0x00007f2b486cfec5 in xtr_mem_tracked_allocs_remove () from /nix/store/pl4zhp9sbklcgj0031f17n12gbwszhqv-extrae-4.0.0/lib/libnanosmpitrace.so
#7  0x00007f2b486d0019 in free () from /nix/store/pl4zhp9sbklcgj0031f17n12gbwszhqv-extrae-4.0.0/lib/libnanosmpitrace.so
#8  0x00007f2b4912d7d6 in _dl_update_slotinfo () from /nix/store/33idnvrkvfgd5lsx2pwgwwi955adl6sk-glibc-2.31/lib/ld-linux-x86-64.so.2
#9  0x00007f2b4912d8fc in update_get_addr () from /nix/store/33idnvrkvfgd5lsx2pwgwwi955adl6sk-glibc-2.31/lib/ld-linux-x86-64.so.2
#10 0x00007f2b49132e68 in __tls_get_addr () from /nix/store/33idnvrkvfgd5lsx2pwgwwi955adl6sk-glibc-2.31/lib/ld-linux-x86-64.so.2
#11 0x00007f2b486cfec5 in xtr_mem_tracked_allocs_remove () from /nix/store/pl4zhp9sbklcgj0031f17n12gbwszhqv-extrae-4.0.0/lib/libnanosmpitrace.so
#12 0x00007f2b486d0019 in free () from /nix/store/pl4zhp9sbklcgj0031f17n12gbwszhqv-extrae-4.0.0/lib/libnanosmpitrace.so
#13 0x00007f2b4912d7d6 in _dl_update_slotinfo () from /nix/store/33idnvrkvfgd5lsx2pwgwwi955adl6sk-glibc-2.31/lib/ld-linux-x86-64.so.2
#14 0x00007f2b4912d8fc in update_get_addr () from /nix/store/33idnvrkvfgd5lsx2pwgwwi955adl6sk-glibc-2.31/lib/ld-linux-x86-64.so.2
#15 0x00007f2b49132e68 in __tls_get_addr () from /nix/store/33idnvrkvfgd5lsx2pwgwwi955adl6sk-glibc-2.31/lib/ld-linux-x86-64.so.2
#16 0x00007f2b486cfec5 in xtr_mem_tracked_allocs_remove () from /nix/store/pl4zhp9sbklcgj0031f17n12gbwszhqv-extrae-4.0.0/lib/libnanosmpitrace.so
#17 0x00007f2b486d0019 in free () from /nix/store/pl4zhp9sbklcgj0031f17n12gbwszhqv-extrae-4.0.0/lib/libnanosmpitrace.so
#18 0x00007f2b4912d7d6 in _dl_update_slotinfo () from /nix/store/33idnvrkvfgd5lsx2pwgwwi955adl6sk-glibc-2.31/lib/ld-linux-x86-64.so.2
#19 0x00007f2b4912d8fc in update_get_addr () from /nix/store/33idnvrkvfgd5lsx2pwgwwi955adl6sk-glibc-2.31/lib/ld-linux-x86-64.so.2
(More stack frames follow...)
(gdb) bt -20
#29489 0x00007f2b4912d8fc in update_get_addr () from /nix/store/33idnvrkvfgd5lsx2pwgwwi955adl6sk-glibc-2.31/lib/ld-linux-x86-64.so.2
#29490 0x00007f2b49132e68 in __tls_get_addr () from /nix/store/33idnvrkvfgd5lsx2pwgwwi955adl6sk-glibc-2.31/lib/ld-linux-x86-64.so.2
#29491 0x00007f2b486cfec5 in xtr_mem_tracked_allocs_remove () from /nix/store/pl4zhp9sbklcgj0031f17n12gbwszhqv-extrae-4.0.0/lib/libnanosmpitrace.so
#29492 0x00007f2b486d0019 in free () from /nix/store/pl4zhp9sbklcgj0031f17n12gbwszhqv-extrae-4.0.0/lib/libnanosmpitrace.so
#29493 0x00007f2b4912d7d6 in _dl_update_slotinfo () from /nix/store/33idnvrkvfgd5lsx2pwgwwi955adl6sk-glibc-2.31/lib/ld-linux-x86-64.so.2
#29494 0x00007f2b4912d8fc in update_get_addr () from /nix/store/33idnvrkvfgd5lsx2pwgwwi955adl6sk-glibc-2.31/lib/ld-linux-x86-64.so.2
#29495 0x00007f2b49132e68 in __tls_get_addr () from /nix/store/33idnvrkvfgd5lsx2pwgwwi955adl6sk-glibc-2.31/lib/ld-linux-x86-64.so.2
#29496 0x00007f2b486cfec5 in xtr_mem_tracked_allocs_remove () from /nix/store/pl4zhp9sbklcgj0031f17n12gbwszhqv-extrae-4.0.0/lib/libnanosmpitrace.so
#29497 0x00007f2b486d0019 in free () from /nix/store/pl4zhp9sbklcgj0031f17n12gbwszhqv-extrae-4.0.0/lib/libnanosmpitrace.so
#29498 0x00007f2b4912d7d6 in _dl_update_slotinfo () from /nix/store/33idnvrkvfgd5lsx2pwgwwi955adl6sk-glibc-2.31/lib/ld-linux-x86-64.so.2
#29499 0x00007f2b4912d8fc in update_get_addr () from /nix/store/33idnvrkvfgd5lsx2pwgwwi955adl6sk-glibc-2.31/lib/ld-linux-x86-64.so.2
#29500 0x00007f2b49132e68 in __tls_get_addr () from /nix/store/33idnvrkvfgd5lsx2pwgwwi955adl6sk-glibc-2.31/lib/ld-linux-x86-64.so.2
#29501 0x00007f2b486cfec5 in xtr_mem_tracked_allocs_remove () from /nix/store/pl4zhp9sbklcgj0031f17n12gbwszhqv-extrae-4.0.0/lib/libnanosmpitrace.so
#29502 0x00007f2b486d0019 in free () from /nix/store/pl4zhp9sbklcgj0031f17n12gbwszhqv-extrae-4.0.0/lib/libnanosmpitrace.so
#29503 0x00007f2b4912d7d6 in _dl_update_slotinfo () from /nix/store/33idnvrkvfgd5lsx2pwgwwi955adl6sk-glibc-2.31/lib/ld-linux-x86-64.so.2
#29504 0x00007f2b4912d8fc in update_get_addr () from /nix/store/33idnvrkvfgd5lsx2pwgwwi955adl6sk-glibc-2.31/lib/ld-linux-x86-64.so.2
#29505 0x00007f2b49132e68 in __tls_get_addr () from /nix/store/33idnvrkvfgd5lsx2pwgwwi955adl6sk-glibc-2.31/lib/ld-linux-x86-64.so.2
#29506 0x00007f2b486cfec5 in xtr_mem_tracked_allocs_remove () from /nix/store/pl4zhp9sbklcgj0031f17n12gbwszhqv-extrae-4.0.0/lib/libnanosmpitrace.so
#29507 0x00007f2b486d0019 in free () from /nix/store/pl4zhp9sbklcgj0031f17n12gbwszhqv-extrae-4.0.0/lib/libnanosmpitrace.so
#29508 0x00007f2b4912d7d6 in _dl_update_slotinfo () from /nix/store/33idnvrkvfgd5lsx2pwgwwi955adl6sk-glibc-2.31/lib/ld-linux-x86-64.so.2

CUDA build fails

Building --with-cuda fails because Extrae_cudaLaunch_Enter is called with only one argument at line 144 in src/tracer/wrappers/CUDA/cuda_wrapper.c. Should the second argument there be NULL?

Minor corrections

Using this issue for any minor issues I find as I move through Extrae.

diff --git a/src/merger/common/mpi2out.c b/src/merger/common/mpi2out.c
index b04e435e..e83183e3 100644
--- a/src/merger/common/mpi2out.c
+++ b/src/merger/common/mpi2out.c
@@ -1396,0 +1396,0 @@ int merger_post (int numtasks, int taskid)

 	if (taskid == 0)
 	{
-		fprintf (stdout, "mpi2prv: Checking for target directory existance...");
+		fprintf (stdout, "mpi2prv: Checking for target directory existence...");
 		char *dirn = dirname(strdup(__Extrae_Utils_trim(get_merge_OutputTraceName())));
 		if (!__Extrae_Utils_directory_exists(dirn))
 		{

tests are failing

gentoo CI tried to run the tests and failed
https://bugs.gentoo.org/804831

>>> Test phase: sys-cluster/extrae-3.8.3-r8
make -j4   check
Making check in src
make[1]: Entering directory '/var/tmp/portage/sys-cluster/extrae-3.8.3-r8/work/extrae-3.8.3/src'
Making check in loader
make[2]: Entering directory '/var/tmp/portage/sys-cluster/extrae-3.8.3-r8/work/extrae-3.8.3/src/loader'
make[2]: Nothing to be done for 'check'.
make[2]: Leaving directory '/var/tmp/portage/sys-cluster/extrae-3.8.3-r8/work/extrae-3.8.3/src/loader'
Making check in launcher
make[2]: Entering directory '/var/tmp/portage/sys-cluster/extrae-3.8.3-r8/work/extrae-3.8.3/src/launcher'
Making check in static
make[3]: Entering directory '/var/tmp/portage/sys-cluster/extrae-3.8.3-r8/work/extrae-3.8.3/src/launcher/static'
make[3]: Nothing to be done for 'check'.
make[3]: Leaving directory '/var/tmp/portage/sys-cluster/extrae-3.8.3-r8/work/extrae-3.8.3/src/launcher/static'
make[3]: Entering directory '/var/tmp/portage/sys-cluster/extrae-3.8.3-r8/work/extrae-3.8.3/src/launcher'
make[3]: Nothing to be done for 'check-am'.
make[3]: Leaving directory '/var/tmp/portage/sys-cluster/extrae-3.8.3-r8/work/extrae-3.8.3/src/launcher'
make[2]: Leaving directory '/var/tmp/portage/sys-cluster/extrae-3.8.3-r8/work/extrae-3.8.3/src/launcher'
Making check in common
make[2]: Entering directory '/var/tmp/portage/sys-cluster/extrae-3.8.3-r8/work/extrae-3.8.3/src/common'
Making check in MPI
make[3]: Entering directory '/var/tmp/portage/sys-cluster/extrae-3.8.3-r8/work/extrae-3.8.3/src/common/MPI'
make[3]: Nothing to be done for 'check'.
make[3]: Leaving directory '/var/tmp/portage/sys-cluster/extrae-3.8.3-r8/work/extrae-3.8.3/src/common/MPI'
make[3]: Entering directory '/var/tmp/portage/sys-cluster/extrae-3.8.3-r8/work/extrae-3.8.3/src/common'
make[3]: Nothing to be done for 'check-am'.
make[3]: Leaving directory '/var/tmp/portage/sys-cluster/extrae-3.8.3-r8/work/extrae-3.8.3/src/common'
make[2]: Leaving directory '/var/tmp/portage/sys-cluster/extrae-3.8.3-r8/work/extrae-3.8.3/src/common'
Making check in merger
make[2]: Entering directory '/var/tmp/portage/sys-cluster/extrae-3.8.3-r8/work/extrae-3.8.3/src/merger'
Making check in parallel
make[3]: Entering directory '/var/tmp/portage/sys-cluster/extrae-3.8.3-r8/work/extrae-3.8.3/src/merger/parallel'
make[3]: Nothing to be done for 'check'.
make[3]: Leaving directory '/var/tmp/portage/sys-cluster/extrae-3.8.3-r8/work/extrae-3.8.3/src/merger/parallel'
make[3]: Entering directory '/var/tmp/portage/sys-cluster/extrae-3.8.3-r8/work/extrae-3.8.3/src/merger'
make[3]: Nothing to be done for 'check-am'.
make[3]: Leaving directory '/var/tmp/portage/sys-cluster/extrae-3.8.3-r8/work/extrae-3.8.3/src/merger'
make[2]: Leaving directory '/var/tmp/portage/sys-cluster/extrae-3.8.3-r8/work/extrae-3.8.3/src/merger'
Making check in tracer
make[2]: Entering directory '/var/tmp/portage/sys-cluster/extrae-3.8.3-r8/work/extrae-3.8.3/src/tracer'
Making check in clocks
make[3]: Entering directory '/var/tmp/portage/sys-cluster/extrae-3.8.3-r8/work/extrae-3.8.3/src/tracer/clocks'
make[3]: Nothing to be done for 'check'.
make[3]: Leaving directory '/var/tmp/portage/sys-cluster/extrae-3.8.3-r8/work/extrae-3.8.3/src/tracer/clocks'
Making check in hwc
make[3]: Entering directory '/var/tmp/portage/sys-cluster/extrae-3.8.3-r8/work/extrae-3.8.3/src/tracer/hwc'
make[3]: Nothing to be done for 'check'.
make[3]: Leaving directory '/var/tmp/portage/sys-cluster/extrae-3.8.3-r8/work/extrae-3.8.3/src/tracer/hwc'
Making check in interfaces
make[3]: Entering directory '/var/tmp/portage/sys-cluster/extrae-3.8.3-r8/work/extrae-3.8.3/src/tracer/interfaces'
Making check in API
make[4]: Entering directory '/var/tmp/portage/sys-cluster/extrae-3.8.3-r8/work/extrae-3.8.3/src/tracer/interfaces/API'
make[4]: Nothing to be done for 'check'.
make[4]: Leaving directory '/var/tmp/portage/sys-cluster/extrae-3.8.3-r8/work/extrae-3.8.3/src/tracer/interfaces/API'
Making check in MPI
make[4]: Entering directory '/var/tmp/portage/sys-cluster/extrae-3.8.3-r8/work/extrae-3.8.3/src/tracer/interfaces/MPI'
make[4]: Nothing to be done for 'check'.
make[4]: Leaving directory '/var/tmp/portage/sys-cluster/extrae-3.8.3-r8/work/extrae-3.8.3/src/tracer/interfaces/MPI'
make[4]: Entering directory '/var/tmp/portage/sys-cluster/extrae-3.8.3-r8/work/extrae-3.8.3/src/tracer/interfaces'
make[4]: Nothing to be done for 'check-am'.
make[4]: Leaving directory '/var/tmp/portage/sys-cluster/extrae-3.8.3-r8/work/extrae-3.8.3/src/tracer/interfaces'
make[3]: Leaving directory '/var/tmp/portage/sys-cluster/extrae-3.8.3-r8/work/extrae-3.8.3/src/tracer/interfaces'
Making check in wrappers
make[3]: Entering directory '/var/tmp/portage/sys-cluster/extrae-3.8.3-r8/work/extrae-3.8.3/src/tracer/wrappers'
Making check in MALLOC
make[4]: Entering directory '/var/tmp/portage/sys-cluster/extrae-3.8.3-r8/work/extrae-3.8.3/src/tracer/wrappers/MALLOC'
make[4]: Nothing to be done for 'check'.
make[4]: Leaving directory '/var/tmp/portage/sys-cluster/extrae-3.8.3-r8/work/extrae-3.8.3/src/tracer/wrappers/MALLOC'
Making check in IO
make[4]: Entering directory '/var/tmp/portage/sys-cluster/extrae-3.8.3-r8/work/extrae-3.8.3/src/tracer/wrappers/IO'
make[4]: Nothing to be done for 'check'.
make[4]: Leaving directory '/var/tmp/portage/sys-cluster/extrae-3.8.3-r8/work/extrae-3.8.3/src/tracer/wrappers/IO'
Making check in SYSCALL
make[4]: Entering directory '/var/tmp/portage/sys-cluster/extrae-3.8.3-r8/work/extrae-3.8.3/src/tracer/wrappers/SYSCALL'
make[4]: Nothing to be done for 'check'.
make[4]: Leaving directory '/var/tmp/portage/sys-cluster/extrae-3.8.3-r8/work/extrae-3.8.3/src/tracer/wrappers/SYSCALL'
Making check in fork
make[4]: Entering directory '/var/tmp/portage/sys-cluster/extrae-3.8.3-r8/work/extrae-3.8.3/src/tracer/wrappers/fork'
make[4]: Nothing to be done for 'check'.
make[4]: Leaving directory '/var/tmp/portage/sys-cluster/extrae-3.8.3-r8/work/extrae-3.8.3/src/tracer/wrappers/fork'
Making check in OMP
make[4]: Entering directory '/var/tmp/portage/sys-cluster/extrae-3.8.3-r8/work/extrae-3.8.3/src/tracer/wrappers/OMP'
Making check in intel-kmpc-11-intermediate
make[5]: Entering directory '/var/tmp/portage/sys-cluster/extrae-3.8.3-r8/work/extrae-3.8.3/src/tracer/wrappers/OMP/intel-kmpc-11-intermediate'
make[5]: Nothing to be done for 'check'.
make[5]: Leaving directory '/var/tmp/portage/sys-cluster/extrae-3.8.3-r8/work/extrae-3.8.3/src/tracer/wrappers/OMP/intel-kmpc-11-intermediate'
make[5]: Entering directory '/var/tmp/portage/sys-cluster/extrae-3.8.3-r8/work/extrae-3.8.3/src/tracer/wrappers/OMP'
make[5]: Nothing to be done for 'check-am'.
make[5]: Leaving directory '/var/tmp/portage/sys-cluster/extrae-3.8.3-r8/work/extrae-3.8.3/src/tracer/wrappers/OMP'
make[4]: Leaving directory '/var/tmp/portage/sys-cluster/extrae-3.8.3-r8/work/extrae-3.8.3/src/tracer/wrappers/OMP'
Making check in MPI
make[4]: Entering directory '/var/tmp/portage/sys-cluster/extrae-3.8.3-r8/work/extrae-3.8.3/src/tracer/wrappers/MPI'
make[4]: Nothing to be done for 'check'.
make[4]: Leaving directory '/var/tmp/portage/sys-cluster/extrae-3.8.3-r8/work/extrae-3.8.3/src/tracer/wrappers/MPI'
make[4]: Entering directory '/var/tmp/portage/sys-cluster/extrae-3.8.3-r8/work/extrae-3.8.3/src/tracer/wrappers'
make[4]: Nothing to be done for 'check-am'.
make[4]: Leaving directory '/var/tmp/portage/sys-cluster/extrae-3.8.3-r8/work/extrae-3.8.3/src/tracer/wrappers'
make[3]: Leaving directory '/var/tmp/portage/sys-cluster/extrae-3.8.3-r8/work/extrae-3.8.3/src/tracer/wrappers'
Making check in stats
make[3]: Entering directory '/var/tmp/portage/sys-cluster/extrae-3.8.3-r8/work/extrae-3.8.3/src/tracer/stats'
make[3]: Nothing to be done for 'check'.
make[3]: Leaving directory '/var/tmp/portage/sys-cluster/extrae-3.8.3-r8/work/extrae-3.8.3/src/tracer/stats'
Making check in sampling
make[3]: Entering directory '/var/tmp/portage/sys-cluster/extrae-3.8.3-r8/work/extrae-3.8.3/src/tracer/sampling'
Making check in common
make[4]: Entering directory '/var/tmp/portage/sys-cluster/extrae-3.8.3-r8/work/extrae-3.8.3/src/tracer/sampling/common'
make[4]: Nothing to be done for 'check'.
make[4]: Leaving directory '/var/tmp/portage/sys-cluster/extrae-3.8.3-r8/work/extrae-3.8.3/src/tracer/sampling/common'
Making check in timer
make[4]: Entering directory '/var/tmp/portage/sys-cluster/extrae-3.8.3-r8/work/extrae-3.8.3/src/tracer/sampling/timer'
make[4]: Nothing to be done for 'check'.
make[4]: Leaving directory '/var/tmp/portage/sys-cluster/extrae-3.8.3-r8/work/extrae-3.8.3/src/tracer/sampling/timer'
make[4]: Entering directory '/var/tmp/portage/sys-cluster/extrae-3.8.3-r8/work/extrae-3.8.3/src/tracer/sampling'
make[4]: Nothing to be done for 'check-am'.
make[4]: Leaving directory '/var/tmp/portage/sys-cluster/extrae-3.8.3-r8/work/extrae-3.8.3/src/tracer/sampling'
make[3]: Leaving directory '/var/tmp/portage/sys-cluster/extrae-3.8.3-r8/work/extrae-3.8.3/src/tracer/sampling'
make[3]: Entering directory '/var/tmp/portage/sys-cluster/extrae-3.8.3-r8/work/extrae-3.8.3/src/tracer'
make[3]: Nothing to be done for 'check-am'.
make[3]: Leaving directory '/var/tmp/portage/sys-cluster/extrae-3.8.3-r8/work/extrae-3.8.3/src/tracer'
make[2]: Leaving directory '/var/tmp/portage/sys-cluster/extrae-3.8.3-r8/work/extrae-3.8.3/src/tracer'
Making check in others
make[2]: Entering directory '/var/tmp/portage/sys-cluster/extrae-3.8.3-r8/work/extrae-3.8.3/src/others'
make[2]: Nothing to be done for 'check'.
make[2]: Leaving directory '/var/tmp/portage/sys-cluster/extrae-3.8.3-r8/work/extrae-3.8.3/src/others'
Making check in cmd-line
make[2]: Entering directory '/var/tmp/portage/sys-cluster/extrae-3.8.3-r8/work/extrae-3.8.3/src/cmd-line'
make[2]: Nothing to be done for 'check'.
make[2]: Leaving directory '/var/tmp/portage/sys-cluster/extrae-3.8.3-r8/work/extrae-3.8.3/src/cmd-line'
make[2]: Entering directory '/var/tmp/portage/sys-cluster/extrae-3.8.3-r8/work/extrae-3.8.3/src'
make[2]: Nothing to be done for 'check-am'.
make[2]: Leaving directory '/var/tmp/portage/sys-cluster/extrae-3.8.3-r8/work/extrae-3.8.3/src'
make[1]: Leaving directory '/var/tmp/portage/sys-cluster/extrae-3.8.3-r8/work/extrae-3.8.3/src'
Making check in include
make[1]: Entering directory '/var/tmp/portage/sys-cluster/extrae-3.8.3-r8/work/extrae-3.8.3/include'
make[1]: Nothing to be done for 'check'.
make[1]: Leaving directory '/var/tmp/portage/sys-cluster/extrae-3.8.3-r8/work/extrae-3.8.3/include'
Making check in scripts
make[1]: Entering directory '/var/tmp/portage/sys-cluster/extrae-3.8.3-r8/work/extrae-3.8.3/scripts'
make[1]: Nothing to be done for 'check'.
make[1]: Leaving directory '/var/tmp/portage/sys-cluster/extrae-3.8.3-r8/work/extrae-3.8.3/scripts'
Making check in etc
make[1]: Entering directory '/var/tmp/portage/sys-cluster/extrae-3.8.3-r8/work/extrae-3.8.3/etc'
make[1]: Nothing to be done for 'check'.
make[1]: Leaving directory '/var/tmp/portage/sys-cluster/extrae-3.8.3-r8/work/extrae-3.8.3/etc'
Making check in tests
make[1]: Entering directory '/var/tmp/portage/sys-cluster/extrae-3.8.3-r8/work/extrae-3.8.3/tests'
Making check in src
make[2]: Entering directory '/var/tmp/portage/sys-cluster/extrae-3.8.3-r8/work/extrae-3.8.3/tests/src'
Making check in common
make[3]: Entering directory '/var/tmp/portage/sys-cluster/extrae-3.8.3-r8/work/extrae-3.8.3/tests/src/common'
make  extrae_vector
make[4]: Entering directory '/var/tmp/portage/sys-cluster/extrae-3.8.3-r8/work/extrae-3.8.3/tests/src/common'
x86_64-pc-linux-gnu-gcc -DHAVE_CONFIG_H -I. -I../../..    -I../../../src/common -O2 -pipe -march=x86-64 -frecord-gcc-switches -fno-optimize-sibling-calls -Wall -W -c -o extrae_vector-check_extrae_vector.o `test -f 'check_extrae_vector.c' || echo './'`check_extrae_vector.c
/bin/sh ../../../libtool  --tag=CC   --mode=link x86_64-pc-linux-gnu-gcc -I../../../src/common -O2 -pipe -march=x86-64 -frecord-gcc-switches -fno-optimize-sibling-calls -Wall -W  -Wl,-O1 -Wl,--as-needed -Wl,--defsym=__gentoo_check_ldflags__=0 -o extrae_vector extrae_vector-check_extrae_vector.o -L../../../src/common -lcommon 
libtool: link: x86_64-pc-linux-gnu-gcc -I../../../src/common -O2 -pipe -march=x86-64 -frecord-gcc-switches -fno-optimize-sibling-calls -Wall -W -Wl,-O1 -Wl,--defsym=__gentoo_check_ldflags__=0 -o extrae_vector extrae_vector-check_extrae_vector.o  -Wl,--as-needed -L../../../src/common /var/tmp/portage/sys-cluster/extrae-3.8.3-r8/work/extrae-3.8.3/src/common/.libs/libcommon.a
make[4]: Leaving directory '/var/tmp/portage/sys-cluster/extrae-3.8.3-r8/work/extrae-3.8.3/tests/src/common'
make  check-TESTS
make[4]: Entering directory '/var/tmp/portage/sys-cluster/extrae-3.8.3-r8/work/extrae-3.8.3/tests/src/common'
make[5]: Entering directory '/var/tmp/portage/sys-cluster/extrae-3.8.3-r8/work/extrae-3.8.3/tests/src/common'
PASS: extrae_vector
============================================================================
Testsuite summary for Extrae 3.8.3
============================================================================
# TOTAL: 1
# PASS:  1
# SKIP:  0
# XFAIL: 0
# FAIL:  0
# XPASS: 0
# ERROR: 0
============================================================================
make[5]: Leaving directory '/var/tmp/portage/sys-cluster/extrae-3.8.3-r8/work/extrae-3.8.3/tests/src/common'
make[4]: Leaving directory '/var/tmp/portage/sys-cluster/extrae-3.8.3-r8/work/extrae-3.8.3/tests/src/common'
make[3]: Leaving directory '/var/tmp/portage/sys-cluster/extrae-3.8.3-r8/work/extrae-3.8.3/tests/src/common'
Making check in tracer
make[3]: Entering directory '/var/tmp/portage/sys-cluster/extrae-3.8.3-r8/work/extrae-3.8.3/tests/src/tracer'
Making check in clocks
make[4]: Entering directory '/var/tmp/portage/sys-cluster/extrae-3.8.3-r8/work/extrae-3.8.3/tests/src/tracer/clocks'
make  check_clock_sleep check_clock_usleep check_clock_nanosleep
make[5]: Entering directory '/var/tmp/portage/sys-cluster/extrae-3.8.3-r8/work/extrae-3.8.3/tests/src/tracer/clocks'
x86_64-pc-linux-gnu-gcc -DHAVE_CONFIG_H -I. -I../../../..    -I../../../../src/tracer/clocks -I../../../../src/common -I../../../../src/tracer -O2 -pipe -march=x86-64 -frecord-gcc-switches -fno-optimize-sibling-calls -Wall -W -c -o check_clock_sleep-check_clock_sleep.o `test -f 'check_clock_sleep.c' || echo './'`check_clock_sleep.c
x86_64-pc-linux-gnu-gcc -DHAVE_CONFIG_H -I. -I../../../..    -I../../../../src/tracer/clocks -I../../../../src/common -I../../../../src/tracer -O2 -pipe -march=x86-64 -frecord-gcc-switches -fno-optimize-sibling-calls -Wall -W -c -o check_clock_usleep-check_clock_usleep.o `test -f 'check_clock_usleep.c' || echo './'`check_clock_usleep.c
x86_64-pc-linux-gnu-gcc -DHAVE_CONFIG_H -I. -I../../../..    -I../../../../src/tracer/clocks -I../../../../src/common -I../../../../src/tracer -O2 -pipe -march=x86-64 -frecord-gcc-switches -fno-optimize-sibling-calls -Wall -W -c -o check_clock_nanosleep-check_clock_nanosleep.o `test -f 'check_clock_nanosleep.c' || echo './'`check_clock_nanosleep.c
/bin/sh ../../../../libtool  --tag=CC   --mode=link x86_64-pc-linux-gnu-gcc -I../../../../src/tracer/clocks -I../../../../src/common -I../../../../src/tracer -O2 -pipe -march=x86-64 -frecord-gcc-switches -fno-optimize-sibling-calls -Wall -W ../../../../src/tracer/clocks/libclock.la  -Wl,-O1 -Wl,--as-needed -Wl,--defsym=__gentoo_check_ldflags__=0 -o check_clock_nanosleep check_clock_nanosleep-check_clock_nanosleep.o  
/bin/sh ../../../../libtool  --tag=CC   --mode=link x86_64-pc-linux-gnu-gcc -I../../../../src/tracer/clocks -I../../../../src/common -I../../../../src/tracer -O2 -pipe -march=x86-64 -frecord-gcc-switches -fno-optimize-sibling-calls -Wall -W ../../../../src/tracer/clocks/libclock.la  -Wl,-O1 -Wl,--as-needed -Wl,--defsym=__gentoo_check_ldflags__=0 -o check_clock_usleep check_clock_usleep-check_clock_usleep.o  
/bin/sh ../../../../libtool  --tag=CC   --mode=link x86_64-pc-linux-gnu-gcc -I../../../../src/tracer/clocks -I../../../../src/common -I../../../../src/tracer -O2 -pipe -march=x86-64 -frecord-gcc-switches -fno-optimize-sibling-calls -Wall -W ../../../../src/tracer/clocks/libclock.la  -Wl,-O1 -Wl,--as-needed -Wl,--defsym=__gentoo_check_ldflags__=0 -o check_clock_sleep check_clock_sleep-check_clock_sleep.o  
libtool: link: x86_64-pc-linux-gnu-gcc -I../../../../src/tracer/clocks -I../../../../src/common -I../../../../src/tracer -O2 -pipe -march=x86-64 -frecord-gcc-switches -fno-optimize-sibling-calls -Wall -W -Wl,-O1 -Wl,--defsym=__gentoo_check_ldflags__=0 -o check_clock_usleep check_clock_usleep-check_clock_usleep.o  ../../../../src/tracer/clocks/.libs/libclock.a -lrt -Wl,--as-needed
libtool: link: x86_64-pc-linux-gnu-gcc -I../../../../src/tracer/clocks -I../../../../src/common -I../../../../src/tracer -O2 -pipe -march=x86-64 -frecord-gcc-switches -fno-optimize-sibling-calls -Wall -W -Wl,-O1 -Wl,--defsym=__gentoo_check_ldflags__=0 -o check_clock_nanosleep check_clock_nanosleep-check_clock_nanosleep.o  ../../../../src/tracer/clocks/.libs/libclock.a -lrt -Wl,--as-needed
libtool: link: x86_64-pc-linux-gnu-gcc -I../../../../src/tracer/clocks -I../../../../src/common -I../../../../src/tracer -O2 -pipe -march=x86-64 -frecord-gcc-switches -fno-optimize-sibling-calls -Wall -W -Wl,-O1 -Wl,--defsym=__gentoo_check_ldflags__=0 -o check_clock_sleep check_clock_sleep-check_clock_sleep.o  ../../../../src/tracer/clocks/.libs/libclock.a -lrt -Wl,--as-needed
make[5]: Leaving directory '/var/tmp/portage/sys-cluster/extrae-3.8.3-r8/work/extrae-3.8.3/tests/src/tracer/clocks'
make  check-TESTS
make[5]: Entering directory '/var/tmp/portage/sys-cluster/extrae-3.8.3-r8/work/extrae-3.8.3/tests/src/tracer/clocks'
make[6]: Entering directory '/var/tmp/portage/sys-cluster/extrae-3.8.3-r8/work/extrae-3.8.3/tests/src/tracer/clocks'
PASS: check_clock_sleep
PASS: check_clock_nanosleep
PASS: check_clock_usleep
============================================================================
Testsuite summary for Extrae 3.8.3
============================================================================
# TOTAL: 3
# PASS:  3
# SKIP:  0
# XFAIL: 0
# FAIL:  0
# XPASS: 0
# ERROR: 0
============================================================================
make[6]: Leaving directory '/var/tmp/portage/sys-cluster/extrae-3.8.3-r8/work/extrae-3.8.3/tests/src/tracer/clocks'
make[5]: Leaving directory '/var/tmp/portage/sys-cluster/extrae-3.8.3-r8/work/extrae-3.8.3/tests/src/tracer/clocks'
make[4]: Leaving directory '/var/tmp/portage/sys-cluster/extrae-3.8.3-r8/work/extrae-3.8.3/tests/src/tracer/clocks'
make[4]: Entering directory '/var/tmp/portage/sys-cluster/extrae-3.8.3-r8/work/extrae-3.8.3/tests/src/tracer'
make[4]: Nothing to be done for 'check-am'.
make[4]: Leaving directory '/var/tmp/portage/sys-cluster/extrae-3.8.3-r8/work/extrae-3.8.3/tests/src/tracer'
make[3]: Leaving directory '/var/tmp/portage/sys-cluster/extrae-3.8.3-r8/work/extrae-3.8.3/tests/src/tracer'
make[3]: Entering directory '/var/tmp/portage/sys-cluster/extrae-3.8.3-r8/work/extrae-3.8.3/tests/src'
make[3]: Nothing to be done for 'check-am'.
make[3]: Leaving directory '/var/tmp/portage/sys-cluster/extrae-3.8.3-r8/work/extrae-3.8.3/tests/src'
make[2]: Leaving directory '/var/tmp/portage/sys-cluster/extrae-3.8.3-r8/work/extrae-3.8.3/tests/src'
Making check in functional
make[2]: Entering directory '/var/tmp/portage/sys-cluster/extrae-3.8.3-r8/work/extrae-3.8.3/tests/functional'
Making check in launcher
make[3]: Entering directory '/var/tmp/portage/sys-cluster/extrae-3.8.3-r8/work/extrae-3.8.3/tests/functional/launcher'
make  
make[4]: Entering directory '/var/tmp/portage/sys-cluster/extrae-3.8.3-r8/work/extrae-3.8.3/tests/functional/launcher'
make[4]: Nothing to be done for 'all'.
make[4]: Leaving directory '/var/tmp/portage/sys-cluster/extrae-3.8.3-r8/work/extrae-3.8.3/tests/functional/launcher'
make  check-TESTS
make[4]: Entering directory '/var/tmp/portage/sys-cluster/extrae-3.8.3-r8/work/extrae-3.8.3/tests/functional/launcher'
make[5]: Entering directory '/var/tmp/portage/sys-cluster/extrae-3.8.3-r8/work/extrae-3.8.3/tests/functional/launcher'
============================================================================
Testsuite summary for Extrae 3.8.3
============================================================================
# TOTAL: 0
# PASS:  0
# SKIP:  0
# XFAIL: 0
# FAIL:  0
# XPASS: 0
# ERROR: 0
============================================================================
make[5]: Leaving directory '/var/tmp/portage/sys-cluster/extrae-3.8.3-r8/work/extrae-3.8.3/tests/functional/launcher'
make[4]: Leaving directory '/var/tmp/portage/sys-cluster/extrae-3.8.3-r8/work/extrae-3.8.3/tests/functional/launcher'
make[3]: Leaving directory '/var/tmp/portage/sys-cluster/extrae-3.8.3-r8/work/extrae-3.8.3/tests/functional/launcher'
Making check in tracer
make[3]: Entering directory '/var/tmp/portage/sys-cluster/extrae-3.8.3-r8/work/extrae-3.8.3/tests/functional/tracer'
Making check in OTHER
make[4]: Entering directory '/var/tmp/portage/sys-cluster/extrae-3.8.3-r8/work/extrae-3.8.3/tests/functional/tracer/OTHER'
make  auto-init-fini define_event_type_gen_pcf define_event_type_gen_pcf_f
make[5]: Entering directory '/var/tmp/portage/sys-cluster/extrae-3.8.3-r8/work/extrae-3.8.3/tests/functional/tracer/OTHER'
x86_64-pc-linux-gnu-gcc -DHAVE_CONFIG_H -I. -I../../../..    -I../../../../include -I../../../../src/common -O2 -pipe -march=x86-64 -frecord-gcc-switches -fno-optimize-sibling-calls -Wall -W -c -o auto_init_fini-auto-init-fini.o `test -f 'auto-init-fini.c' || echo './'`auto-init-fini.c
x86_64-pc-linux-gnu-gcc -DHAVE_CONFIG_H -I. -I../../../..    -I../../../../include -I../../../../src/common -O2 -pipe -march=x86-64 -frecord-gcc-switches -fno-optimize-sibling-calls -Wall -W -c -o define_event_type_gen_pcf-define_event_type_gen_pcf.o `test -f 'define_event_type_gen_pcf.c' || echo './'`define_event_type_gen_pcf.c
/usr/bin/mpif90 -I../../../../include -I../../../../src/common -O2 -pipe -march=x86-64 -frecord-gcc-switches -c -o define_event_type_gen_pcf_f-define_event_type_gen_pcf.o `test -f 'define_event_type_gen_pcf.f90' || echo './'`define_event_type_gen_pcf.f90
/usr/bin/mpif90 -I../../../../include -I../../../../src/common -O2 -pipe -march=x86-64 -frecord-gcc-switches -c -o ../../../../include/define_event_type_gen_pcf_f-extrae_module.o `test -f '../../../../include/extrae_module.f90' || echo './'`../../../../include/extrae_module.f90
define_event_type_gen_pcf.f90:4:14:

    4 |          USE EXTRAE_MODULE
      |              1
Fatal Error: File â��../../../../include/extrae_module.modâ�� opened at (1) is not a GNU Fortran module file
compilation terminated.
make[5]: *** [Makefile:980: define_event_type_gen_pcf_f-define_event_type_gen_pcf.o] Error 1
/bin/sh ../../../../libtool  --tag=CC   --mode=link x86_64-pc-linux-gnu-gcc -I../../../../include -I../../../../src/common -O2 -pipe -march=x86-64 -frecord-gcc-switches -fno-optimize-sibling-calls -Wall -W -L../../../../src/tracer -lseqtrace -L/usr/lib64 -R /usr/lib64 -lpapi  -Wl,-O1 -Wl,--as-needed -Wl,--defsym=__gentoo_check_ldflags__=0 -o auto-init-fini auto_init_fini-auto-init-fini.o  
make[5]: Target 'define_event_type_gen_pcf_f' not remade because of errors.
/bin/sh ../../../../libtool  --tag=CC   --mode=link x86_64-pc-linux-gnu-gcc -I../../../../include -I../../../../src/common -O2 -pipe -march=x86-64 -frecord-gcc-switches -fno-optimize-sibling-calls -Wall -W -L../../../../src/tracer -lseqtrace -L/usr/lib64 -R /usr/lib64 -lpapi  -Wl,-O1 -Wl,--as-needed -Wl,--defsym=__gentoo_check_ldflags__=0 -o define_event_type_gen_pcf define_event_type_gen_pcf-define_event_type_gen_pcf.o  
libtool: link: x86_64-pc-linux-gnu-gcc -I../../../../include -I../../../../src/common -O2 -pipe -march=x86-64 -frecord-gcc-switches -fno-optimize-sibling-calls -Wall -W -Wl,-O1 -Wl,--defsym=__gentoo_check_ldflags__=0 -o .libs/auto-init-fini auto_init_fini-auto-init-fini.o  -L../../../../src/tracer /var/tmp/portage/sys-cluster/extrae-3.8.3-r8/work/extrae-3.8.3/src/tracer/.libs/libseqtrace.so -L/usr/lib64 -lunwind -lxml2 -ldl -lz -lrt -lpapi -Wl,--as-needed -Wl,-rpath -Wl,/usr/lib64
libtool: link: x86_64-pc-linux-gnu-gcc -I../../../../include -I../../../../src/common -O2 -pipe -march=x86-64 -frecord-gcc-switches -fno-optimize-sibling-calls -Wall -W -Wl,-O1 -Wl,--defsym=__gentoo_check_ldflags__=0 -o .libs/define_event_type_gen_pcf define_event_type_gen_pcf-define_event_type_gen_pcf.o  -L../../../../src/tracer /var/tmp/portage/sys-cluster/extrae-3.8.3-r8/work/extrae-3.8.3/src/tracer/.libs/libseqtrace.so -L/usr/lib64 -lunwind -lxml2 -ldl -lz -lrt -lpapi -Wl,--as-needed -Wl,-rpath -Wl,/usr/lib64
make[5]: Leaving directory '/var/tmp/portage/sys-cluster/extrae-3.8.3-r8/work/extrae-3.8.3/tests/functional/tracer/OTHER'
make[4]: *** [Makefile:1253: check-am] Error 2
make[4]: Target 'check' not remade because of errors.
make[4]: Leaving directory '/var/tmp/portage/sys-cluster/extrae-3.8.3-r8/work/extrae-3.8.3/tests/functional/tracer/OTHER'
Making check in MPI
make[4]: Entering directory '/var/tmp/portage/sys-cluster/extrae-3.8.3-r8/work/extrae-3.8.3/tests/functional/tracer/MPI'
make  pass_argument_MPIRUN mpi_initfini_c_nolinked mpi_initfini_c_linked mpi_initfini_f mpi_commranksize_c mpi_commranksize_f mpi_barrier_c mpi_bcast_c mpi_reduce_c mpi_gather_c mpi_scatter_c mpi_sendirecv_c mpi_isendirecv_c mpi_isendirecvwaitall_c mpi_sendirecvprobewait_c mpi_sendirecviprobewait_c mpi_sendirecvtestwait_c mpi_ibarrier_c mpi_ibcast_c mpi_ireduce_c
make[5]: Entering directory '/var/tmp/portage/sys-cluster/extrae-3.8.3-r8/work/extrae-3.8.3/tests/functional/tracer/MPI'
/usr/bin/mpicc -DHAVE_CONFIG_H -I. -I../../../..    -DMPIRUN="\"/usr/bin/mpirun\"" -O2 -pipe -march=x86-64 -frecord-gcc-switches -fno-optimize-sibling-calls -Wall -W -c -o pass_argument_MPIRUN-pass_argument_MPIRUN.o `test -f 'pass_argument_MPIRUN.c' || echo './'`pass_argument_MPIRUN.c
/usr/bin/mpicc -DHAVE_CONFIG_H -I. -I../../../..     -O2 -pipe -march=x86-64 -frecord-gcc-switches -fno-optimize-sibling-calls -Wall -W -c -o mpi_initfini_c.o mpi_initfini_c.c
/usr/bin/mpif90  -O2 -pipe -march=x86-64 -frecord-gcc-switches -c -o mpi_initfini_f.o mpi_initfini_f.f90
/usr/bin/mpicc -DHAVE_CONFIG_H -I. -I../../../..     -O2 -pipe -march=x86-64 -frecord-gcc-switches -fno-optimize-sibling-calls -Wall -W -c -o mpi_commranksize_c.o mpi_commranksize_c.c
/usr/bin/mpif90  -O2 -pipe -march=x86-64 -frecord-gcc-switches -c -o mpi_commranksize_f.o mpi_commranksize_f.f90
/usr/bin/mpicc -DHAVE_CONFIG_H -I. -I../../../..     -O2 -pipe -march=x86-64 -frecord-gcc-switches -fno-optimize-sibling-calls -Wall -W -c -o mpi_barrier_c.o mpi_barrier_c.c
/usr/bin/mpicc -DHAVE_CONFIG_H -I. -I../../../..     -O2 -pipe -march=x86-64 -frecord-gcc-switches -fno-optimize-sibling-calls -Wall -W -c -o mpi_bcast_c.o mpi_bcast_c.c
/usr/bin/mpicc -DHAVE_CONFIG_H -I. -I../../../..     -O2 -pipe -march=x86-64 -frecord-gcc-switches -fno-optimize-sibling-calls -Wall -W -c -o mpi_reduce_c.o mpi_reduce_c.c
/usr/bin/mpicc -DHAVE_CONFIG_H -I. -I../../../..     -O2 -pipe -march=x86-64 -frecord-gcc-switches -fno-optimize-sibling-calls -Wall -W -c -o mpi_gather_c.o mpi_gather_c.c
/usr/bin/mpicc -DHAVE_CONFIG_H -I. -I../../../..     -O2 -pipe -march=x86-64 -frecord-gcc-switches -fno-optimize-sibling-calls -Wall -W -c -o mpi_scatter_c.o mpi_scatter_c.c
/usr/bin/mpicc -DHAVE_CONFIG_H -I. -I../../../..     -O2 -pipe -march=x86-64 -frecord-gcc-switches -fno-optimize-sibling-calls -Wall -W -c -o mpi_sendirecv_c.o mpi_sendirecv_c.c
/usr/bin/mpicc -DHAVE_CONFIG_H -I. -I../../../..     -O2 -pipe -march=x86-64 -frecord-gcc-switches -fno-optimize-sibling-calls -Wall -W -c -o mpi_isendirecv_c.o mpi_isendirecv_c.c
/usr/bin/mpicc -DHAVE_CONFIG_H -I. -I../../../..     -O2 -pipe -march=x86-64 -frecord-gcc-switches -fno-optimize-sibling-calls -Wall -W -c -o mpi_isendirecvwaitall_c.o mpi_isendirecvwaitall_c.c
/usr/bin/mpicc -DHAVE_CONFIG_H -I. -I../../../..     -O2 -pipe -march=x86-64 -frecord-gcc-switches -fno-optimize-sibling-calls -Wall -W -c -o mpi_sendirecvprobewait_c.o mpi_sendirecvprobewait_c.c
/usr/bin/mpicc -DHAVE_CONFIG_H -I. -I../../../..     -O2 -pipe -march=x86-64 -frecord-gcc-switches -fno-optimize-sibling-calls -Wall -W -c -o mpi_sendirecviprobewait_c.o mpi_sendirecviprobewait_c.c
/usr/bin/mpicc -DHAVE_CONFIG_H -I. -I../../../..     -O2 -pipe -march=x86-64 -frecord-gcc-switches -fno-optimize-sibling-calls -Wall -W -c -o mpi_sendirecvtestwait_c.o mpi_sendirecvtestwait_c.c
/usr/bin/mpicc -DHAVE_CONFIG_H -I. -I../../../..     -O2 -pipe -march=x86-64 -frecord-gcc-switches -fno-optimize-sibling-calls -Wall -W -c -o mpi_ibarrier_c.o mpi_ibarrier_c.c
mpi_sendirecviprobewait_c.c: In function â��mainâ��:
mpi_sendirecviprobewait_c.c:35:17: warning: implicit declaration of function â�-Wimplicit-function-declaration][]]
   35 |                 sleep(5);
      |                 ^~~~~
/usr/bin/mpicc -DHAVE_CONFIG_H -I. -I../../../..     -O2 -pipe -march=x86-64 -frecord-gcc-switches -fno-optimize-sibling-calls -Wall -W -c -o mpi_ibcast_c.o mpi_ibcast_c.c
/usr/bin/mpicc -DHAVE_CONFIG_H -I. -I../../../..     -O2 -pipe -march=x86-64 -frecord-gcc-switches -fno-optimize-sibling-calls -Wall -W -c -o mpi_ireduce_c.o mpi_ireduce_c.c
/bin/sh ../../../../libtool  --tag=CC   --mode=link /usr/bin/mpicc -DMPIRUN="\"/usr/bin/mpirun\"" -O2 -pipe -march=x86-64 -frecord-gcc-switches -fno-optimize-sibling-calls -Wall -W  -Wl,-O1 -Wl,--as-needed -Wl,--defsym=__gentoo_check_ldflags__=0 -o pass_argument_MPIRUN pass_argument_MPIRUN-pass_argument_MPIRUN.o  
/bin/sh ../../../../libtool  --tag=CC   --mode=link /usr/bin/mpicc  -O2 -pipe -march=x86-64 -frecord-gcc-switches -fno-optimize-sibling-calls -Wall -W  -Wl,-O1 -Wl,--as-needed -Wl,--defsym=__gentoo_check_ldflags__=0 -o mpi_initfini_c_nolinked mpi_initfini_c.o  
/bin/sh ../../../../libtool  --tag=CC   --mode=link /usr/bin/mpicc  -O2 -pipe -march=x86-64 -frecord-gcc-switches -fno-optimize-sibling-calls -Wall -W  -Wl,-O1 -Wl,--as-needed -Wl,--defsym=__gentoo_check_ldflags__=0 -o mpi_initfini_c_linked mpi_initfini_c.o ../../../../src/tracer/libmpitrace.la 
/bin/sh ../../../../libtool  --tag=FC   --mode=link /usr/bin/mpif90  -O2 -pipe -march=x86-64 -frecord-gcc-switches  -Wl,-O1 -Wl,--as-needed -Wl,--defsym=__gentoo_check_ldflags__=0 -o mpi_initfini_f mpi_initfini_f.o  
libtool: link: /usr/bin/mpicc -O2 -pipe -march=x86-64 -frecord-gcc-switches -fno-optimize-sibling-calls -Wall -W -Wl,-O1 -Wl,--defsym=__gentoo_check_ldflags__=0 -o mpi_initfini_c_nolinked mpi_initfini_c.o  -Wl,--as-needed
libtool: link: /usr/bin/mpicc -DMPIRUN=\"/usr/bin/mpirun\" -O2 -pipe -march=x86-64 -frecord-gcc-switches -fno-optimize-sibling-calls -Wall -W -Wl,-O1 -Wl,--defsym=__gentoo_check_ldflags__=0 -o pass_argument_MPIRUN pass_argument_MPIRUN-pass_argument_MPIRUN.o  -Wl,--as-needed
libtool: link: /usr/bin/mpif90 -O2 -pipe -march=x86-64 -frecord-gcc-switches -Wl,-O1 -Wl,--defsym=__gentoo_check_ldflags__=0 -o mpi_initfini_f mpi_initfini_f.o  -Wl,--as-needed
libtool: link: /usr/bin/mpicc -O2 -pipe -march=x86-64 -frecord-gcc-switches -fno-optimize-sibling-calls -Wall -W -Wl,-O1 -Wl,--defsym=__gentoo_check_ldflags__=0 -o .libs/mpi_initfini_c_linked mpi_initfini_c.o  -Wl,--as-needed ../../../../src/tracer/.libs/libmpitrace.so -L/usr/lib64 -lmpi -lpapi -lunwind -lxml2 -ldl -lz -lrt
/bin/sh ../../../../libtool  --tag=CC   --mode=link /usr/bin/mpicc  -O2 -pipe -march=x86-64 -frecord-gcc-switches -fno-optimize-sibling-calls -Wall -W  -Wl,-O1 -Wl,--as-needed -Wl,--defsym=__gentoo_check_ldflags__=0 -o mpi_commranksize_c mpi_commranksize_c.o  
/bin/sh ../../../../libtool  --tag=FC   --mode=link /usr/bin/mpif90  -O2 -pipe -march=x86-64 -frecord-gcc-switches  -Wl,-O1 -Wl,--as-needed -Wl,--defsym=__gentoo_check_ldflags__=0 -o mpi_commranksize_f mpi_commranksize_f.o  
/bin/sh ../../../../libtool  --tag=CC   --mode=link /usr/bin/mpicc  -O2 -pipe -march=x86-64 -frecord-gcc-switches -fno-optimize-sibling-calls -Wall -W  -Wl,-O1 -Wl,--as-needed -Wl,--defsym=__gentoo_check_ldflags__=0 -o mpi_barrier_c mpi_barrier_c.o  
/bin/sh ../../../../libtool  --tag=CC   --mode=link /usr/bin/mpicc  -O2 -pipe -march=x86-64 -frecord-gcc-switches -fno-optimize-sibling-calls -Wall -W  -Wl,-O1 -Wl,--as-needed -Wl,--defsym=__gentoo_check_ldflags__=0 -o mpi_bcast_c mpi_bcast_c.o  
libtool: link: /usr/bin/mpicc -O2 -pipe -march=x86-64 -frecord-gcc-switches -fno-optimize-sibling-calls -Wall -W -Wl,-O1 -Wl,--defsym=__gentoo_check_ldflags__=0 -o mpi_commranksize_c mpi_commranksize_c.o  -Wl,--as-needed
libtool: link: /usr/bin/mpif90 -O2 -pipe -march=x86-64 -frecord-gcc-switches -Wl,-O1 -Wl,--defsym=__gentoo_check_ldflags__=0 -o mpi_commranksize_f mpi_commranksize_f.o  -Wl,--as-needed
/bin/sh ../../../../libtool  --tag=CC   --mode=link /usr/bin/mpicc  -O2 -pipe -march=x86-64 -frecord-gcc-switches -fno-optimize-sibling-calls -Wall -W  -Wl,-O1 -Wl,--as-needed -Wl,--defsym=__gentoo_check_ldflags__=0 -o mpi_reduce_c mpi_reduce_c.o  
libtool: link: /usr/bin/mpicc -O2 -pipe -march=x86-64 -frecord-gcc-switches -fno-optimize-sibling-calls -Wall -W -Wl,-O1 -Wl,--defsym=__gentoo_check_ldflags__=0 -o mpi_barrier_c mpi_barrier_c.o  -Wl,--as-needed
libtool: link: /usr/bin/mpicc -O2 -pipe -march=x86-64 -frecord-gcc-switches -fno-optimize-sibling-calls -Wall -W -Wl,-O1 -Wl,--defsym=__gentoo_check_ldflags__=0 -o mpi_bcast_c mpi_bcast_c.o  -Wl,--as-needed
/bin/sh ../../../../libtool  --tag=CC   --mode=link /usr/bin/mpicc  -O2 -pipe -march=x86-64 -frecord-gcc-switches -fno-optimize-sibling-calls -Wall -W  -Wl,-O1 -Wl,--as-needed -Wl,--defsym=__gentoo_check_ldflags__=0 -o mpi_gather_c mpi_gather_c.o  
/bin/sh ../../../../libtool  --tag=CC   --mode=link /usr/bin/mpicc  -O2 -pipe -march=x86-64 -frecord-gcc-switches -fno-optimize-sibling-calls -Wall -W  -Wl,-O1 -Wl,--as-needed -Wl,--defsym=__gentoo_check_ldflags__=0 -o mpi_scatter_c mpi_scatter_c.o  
/bin/sh ../../../../libtool  --tag=CC   --mode=link /usr/bin/mpicc  -O2 -pipe -march=x86-64 -frecord-gcc-switches -fno-optimize-sibling-calls -Wall -W  -Wl,-O1 -Wl,--as-needed -Wl,--defsym=__gentoo_check_ldflags__=0 -o mpi_sendirecv_c mpi_sendirecv_c.o  
libtool: link: /usr/bin/mpicc -O2 -pipe -march=x86-64 -frecord-gcc-switches -fno-optimize-sibling-calls -Wall -W -Wl,-O1 -Wl,--defsym=__gentoo_check_ldflags__=0 -o mpi_reduce_c mpi_reduce_c.o  -Wl,--as-needed
libtool: link: /usr/bin/mpicc -O2 -pipe -march=x86-64 -frecord-gcc-switches -fno-optimize-sibling-calls -Wall -W -Wl,-O1 -Wl,--defsym=__gentoo_check_ldflags__=0 -o mpi_gather_c mpi_gather_c.o  -Wl,--as-needed
/bin/sh ../../../../libtool  --tag=CC   --mode=link /usr/bin/mpicc  -O2 -pipe -march=x86-64 -frecord-gcc-switches -fno-optimize-sibling-calls -Wall -W  -Wl,-O1 -Wl,--as-needed -Wl,--defsym=__gentoo_check_ldflags__=0 -o mpi_isendirecv_c mpi_isendirecv_c.o  
libtool: link: /usr/bin/mpicc -O2 -pipe -march=x86-64 -frecord-gcc-switches -fno-optimize-sibling-calls -Wall -W -Wl,-O1 -Wl,--defsym=__gentoo_check_ldflags__=0 -o mpi_scatter_c mpi_scatter_c.o  -Wl,--as-needed
libtool: link: /usr/bin/mpicc -O2 -pipe -march=x86-64 -frecord-gcc-switches -fno-optimize-sibling-calls -Wall -W -Wl,-O1 -Wl,--defsym=__gentoo_check_ldflags__=0 -o mpi_sendirecv_c mpi_sendirecv_c.o  -Wl,--as-needed
/bin/sh ../../../../libtool  --tag=CC   --mode=link /usr/bin/mpicc  -O2 -pipe -march=x86-64 -frecord-gcc-switches -fno-optimize-sibling-calls -Wall -W  -Wl,-O1 -Wl,--as-needed -Wl,--defsym=__gentoo_check_ldflags__=0 -o mpi_isendirecvwaitall_c mpi_isendirecvwaitall_c.o  
/bin/sh ../../../../libtool  --tag=CC   --mode=link /usr/bin/mpicc  -O2 -pipe -march=x86-64 -frecord-gcc-switches -fno-optimize-sibling-calls -Wall -W  -Wl,-O1 -Wl,--as-needed -Wl,--defsym=__gentoo_check_ldflags__=0 -o mpi_sendirecvprobewait_c mpi_sendirecvprobewait_c.o  
/bin/sh ../../../../libtool  --tag=CC   --mode=link /usr/bin/mpicc  -O2 -pipe -march=x86-64 -frecord-gcc-switches -fno-optimize-sibling-calls -Wall -W  -Wl,-O1 -Wl,--as-needed -Wl,--defsym=__gentoo_check_ldflags__=0 -o mpi_sendirecviprobewait_c mpi_sendirecviprobewait_c.o  
libtool: link: /usr/bin/mpicc -O2 -pipe -march=x86-64 -frecord-gcc-switches -fno-optimize-sibling-calls -Wall -W -Wl,-O1 -Wl,--defsym=__gentoo_check_ldflags__=0 -o mpi_isendirecv_c mpi_isendirecv_c.o  -Wl,--as-needed
libtool: link: /usr/bin/mpicc -O2 -pipe -march=x86-64 -frecord-gcc-switches -fno-optimize-sibling-calls -Wall -W -Wl,-O1 -Wl,--defsym=__gentoo_check_ldflags__=0 -o mpi_isendirecvwaitall_c mpi_isendirecvwaitall_c.o  -Wl,--as-needed
libtool: link: /usr/bin/mpicc -O2 -pipe -march=x86-64 -frecord-gcc-switches -fno-optimize-sibling-calls -Wall -W -Wl,-O1 -Wl,--defsym=__gentoo_check_ldflags__=0 -o mpi_sendirecvprobewait_c mpi_sendirecvprobewait_c.o  -Wl,--as-needed
libtool: link: /usr/bin/mpicc -O2 -pipe -march=x86-64 -frecord-gcc-switches -fno-optimize-sibling-calls -Wall -W -Wl,-O1 -Wl,--defsym=__gentoo_check_ldflags__=0 -o mpi_sendirecviprobewait_c mpi_sendirecviprobewait_c.o  -Wl,--as-needed
/bin/sh ../../../../libtool  --tag=CC   --mode=link /usr/bin/mpicc  -O2 -pipe -march=x86-64 -frecord-gcc-switches -fno-optimize-sibling-calls -Wall -W  -Wl,-O1 -Wl,--as-needed -Wl,--defsym=__gentoo_check_ldflags__=0 -o mpi_sendirecvtestwait_c mpi_sendirecvtestwait_c.o  
/bin/sh ../../../../libtool  --tag=CC   --mode=link /usr/bin/mpicc  -O2 -pipe -march=x86-64 -frecord-gcc-switches -fno-optimize-sibling-calls -Wall -W  -Wl,-O1 -Wl,--as-needed -Wl,--defsym=__gentoo_check_ldflags__=0 -o mpi_ibarrier_c mpi_ibarrier_c.o  
/bin/sh ../../../../libtool  --tag=CC   --mode=link /usr/bin/mpicc  -O2 -pipe -march=x86-64 -frecord-gcc-switches -fno-optimize-sibling-calls -Wall -W  -Wl,-O1 -Wl,--as-needed -Wl,--defsym=__gentoo_check_ldflags__=0 -o mpi_ibcast_c mpi_ibcast_c.o  
/bin/sh ../../../../libtool  --tag=CC   --mode=link /usr/bin/mpicc  -O2 -pipe -march=x86-64 -frecord-gcc-switches -fno-optimize-sibling-calls -Wall -W  -Wl,-O1 -Wl,--as-needed -Wl,--defsym=__gentoo_check_ldflags__=0 -o mpi_ireduce_c mpi_ireduce_c.o  
libtool: link: /usr/bin/mpicc -O2 -pipe -march=x86-64 -frecord-gcc-switches -fno-optimize-sibling-calls -Wall -W -Wl,-O1 -Wl,--defsym=__gentoo_check_ldflags__=0 -o mpi_sendirecvtestwait_c mpi_sendirecvtestwait_c.o  -Wl,--as-needed
libtool: link: /usr/bin/mpicc -O2 -pipe -march=x86-64 -frecord-gcc-switches -fno-optimize-sibling-calls -Wall -W -Wl,-O1 -Wl,--defsym=__gentoo_check_ldflags__=0 -o mpi_ibarrier_c mpi_ibarrier_c.o  -Wl,--as-needed
libtool: link: /usr/bin/mpicc -O2 -pipe -march=x86-64 -frecord-gcc-switches -fno-optimize-sibling-calls -Wall -W -Wl,-O1 -Wl,--defsym=__gentoo_check_ldflags__=0 -o mpi_ibcast_c mpi_ibcast_c.o  -Wl,--as-needed
libtool: link: /usr/bin/mpicc -O2 -pipe -march=x86-64 -frecord-gcc-switches -fno-optimize-sibling-calls -Wall -W -Wl,-O1 -Wl,--defsym=__gentoo_check_ldflags__=0 -o mpi_ireduce_c mpi_ireduce_c.o  -Wl,--as-needed
make[5]: Leaving directory '/var/tmp/portage/sys-cluster/extrae-3.8.3-r8/work/extrae-3.8.3/tests/functional/tracer/MPI'
make  check-TESTS
make[5]: Entering directory '/var/tmp/portage/sys-cluster/extrae-3.8.3-r8/work/extrae-3.8.3/tests/functional/tracer/MPI'
make[6]: Entering directory '/var/tmp/portage/sys-cluster/extrae-3.8.3-r8/work/extrae-3.8.3/tests/functional/tracer/MPI'
FAIL: mpi_initfini_c_nolinked_1proc.sh
FAIL: mpi_initfini_c_linked_2proc.sh
FAIL: mpi_initfini_c_linked_1proc.sh
PASS: mpi_initfini_c_linked_4proc.sh
FAIL: mpi_commranksize_f_1proc.sh
PASS: mpi_initfini_f_nolinked_1proc.sh
FAIL: mpi_barrier_c_1proc.sh
FAIL: mpi_commranksize_c_1proc.sh
FAIL: mpi_bcast_c_1proc.sh
FAIL: mpi_scatter_c_1proc.sh
FAIL: mpi_reduce_c_1proc.sh
FAIL: mpi_gather_c_1proc.sh
FAIL: mpi_sendirecv_c.sh
FAIL: mpi_isendirecv_c.sh
FAIL: mpi_isendirecvwaitall_c.sh
FAIL: mpi_sendirecvprobewait_c_2proc.sh
FAIL: mpi_sendirecvtestwait_c_2proc.sh
FAIL: mpi_ibcast_c_1proc.sh
PASS: mpi_ibarrier_c_1proc.sh
PASS: mpi_ireduce_c_1proc.sh
FAIL: mpi_sendirecviprobewait_c_2proc.sh
============================================================================
Testsuite summary for Extrae 3.8.3
============================================================================
# TOTAL: 21
# PASS:  4
# SKIP:  0
# XFAIL: 0
# FAIL:  17
# XPASS: 0
# ERROR: 0
============================================================================
See tests/functional/tracer/MPI/test-suite.log
Please report to [email protected]
============================================================================
make[6]: *** [Makefile:1217: test-suite.log] Error 1
make[6]: Leaving directory '/var/tmp/portage/sys-cluster/extrae-3.8.3-r8/work/extrae-3.8.3/tests/functional/tracer/MPI'
make[5]: *** [Makefile:1325: check-TESTS] Error 2
make[5]: Leaving directory '/var/tmp/portage/sys-cluster/extrae-3.8.3-r8/work/extrae-3.8.3/tests/functional/tracer/MPI'
make[4]: *** [Makefile:1538: check-am] Error 2
make[4]: Target 'check' not remade because of errors.
make[4]: Leaving directory '/var/tmp/portage/sys-cluster/extrae-3.8.3-r8/work/extrae-3.8.3/tests/functional/tracer/MPI'
make[4]: Entering directory '/var/tmp/portage/sys-cluster/extrae-3.8.3-r8/work/extrae-3.8.3/tests/functional/tracer'
make[4]: Nothing to be done for 'check-am'.
make[4]: Leaving directory '/var/tmp/portage/sys-cluster/extrae-3.8.3-r8/work/extrae-3.8.3/tests/functional/tracer'
make[3]: *** [Makefile:649: check-recursive] Error 1
make[3]: Target 'check' not remade because of errors.
make[3]: Leaving directory '/var/tmp/portage/sys-cluster/extrae-3.8.3-r8/work/extrae-3.8.3/tests/functional/tracer'
Making check in merger
make[3]: Entering directory '/var/tmp/portage/sys-cluster/extrae-3.8.3-r8/work/extrae-3.8.3/tests/functional/merger'
Making check in dump-events
make[4]: Entering directory '/var/tmp/portage/sys-cluster/extrae-3.8.3-r8/work/extrae-3.8.3/tests/functional/merger/dump-events'
make  dump-events
make[5]: Entering directory '/var/tmp/portage/sys-cluster/extrae-3.8.3-r8/work/extrae-3.8.3/tests/functional/merger/dump-events'
x86_64-pc-linux-gnu-gcc -DHAVE_CONFIG_H -I. -I../../../..    -I../../../../include -I../../../../src/common -O2 -pipe -march=x86-64 -frecord-gcc-switches -fno-optimize-sibling-calls -Wall -W -c -o dump_events-dump-events.o `test -f 'dump-events.c' || echo './'`dump-events.c
/bin/sh ../../../../libtool  --tag=CC   --mode=link x86_64-pc-linux-gnu-gcc -I../../../../include -I../../../../src/common -O2 -pipe -march=x86-64 -frecord-gcc-switches -fno-optimize-sibling-calls -Wall -W -L../../../../src/tracer -lseqtrace -L/usr/lib64 -R /usr/lib64 -lpapi  -Wl,-O1 -Wl,--as-needed -Wl,--defsym=__gentoo_check_ldflags__=0 -o dump-events dump_events-dump-events.o  
libtool: link: x86_64-pc-linux-gnu-gcc -I../../../../include -I../../../../src/common -O2 -pipe -march=x86-64 -frecord-gcc-switches -fno-optimize-sibling-calls -Wall -W -Wl,-O1 -Wl,--defsym=__gentoo_check_ldflags__=0 -o .libs/dump-events dump_events-dump-events.o  -L../../../../src/tracer /var/tmp/portage/sys-cluster/extrae-3.8.3-r8/work/extrae-3.8.3/src/tracer/.libs/libseqtrace.so -L/usr/lib64 -lunwind -lxml2 -ldl -lz -lrt -lpapi -Wl,--as-needed -Wl,-rpath -Wl,/usr/lib64
make[5]: Leaving directory '/var/tmp/portage/sys-cluster/extrae-3.8.3-r8/work/extrae-3.8.3/tests/functional/merger/dump-events'
make  check-TESTS
make[5]: Entering directory '/var/tmp/portage/sys-cluster/extrae-3.8.3-r8/work/extrae-3.8.3/tests/functional/merger/dump-events'
make[6]: Entering directory '/var/tmp/portage/sys-cluster/extrae-3.8.3-r8/work/extrae-3.8.3/tests/functional/merger/dump-events'
PASS: dump-events.sh
============================================================================
Testsuite summary for Extrae 3.8.3
============================================================================
# TOTAL: 1
# PASS:  1
# SKIP:  0
# XFAIL: 0
# FAIL:  0
# XPASS: 0
# ERROR: 0
============================================================================
make[6]: Leaving directory '/var/tmp/portage/sys-cluster/extrae-3.8.3-r8/work/extrae-3.8.3/tests/functional/merger/dump-events'
make[5]: Leaving directory '/var/tmp/portage/sys-cluster/extrae-3.8.3-r8/work/extrae-3.8.3/tests/functional/merger/dump-events'
make[4]: Leaving directory '/var/tmp/portage/sys-cluster/extrae-3.8.3-r8/work/extrae-3.8.3/tests/functional/merger/dump-events'
Making check in shared-libraries
make[4]: Entering directory '/var/tmp/portage/sys-cluster/extrae-3.8.3-r8/work/extrae-3.8.3/tests/functional/merger/shared-libraries'
make  main_nodebug main_debug
make[5]: Entering directory '/var/tmp/portage/sys-cluster/extrae-3.8.3-r8/work/extrae-3.8.3/tests/functional/merger/shared-libraries'
gcc -fPIC -DPIC fA.c -o libfA_nodebug.so -shared
gcc -fPIC -DPIC fB.c -o libfB_nodebug.so -shared
gcc -g -fPIC -DPIC fA.c -o libfA_debug.so -shared
gcc -g -fPIC -DPIC fB.c -o libfB_debug.so -shared
gcc -g main.c -Wl,-rpath -Wl,../../../../src/tracer/.libs -Wl,-rpath -Wl,. -L ../../../../src/tracer/.libs -L . -lfA_nodebug -lfB_nodebug -lseqtrace -o main_nodebug
gcc -g main.c -Wl,-rpath -Wl,../../../../src/tracer/.libs -Wl,-rpath -Wl,. -L ../../../../src/tracer/.libs -L . -lfA_debug -lfB_debug -lseqtrace -o main_debug
make[5]: Leaving directory '/var/tmp/portage/sys-cluster/extrae-3.8.3-r8/work/extrae-3.8.3/tests/functional/merger/shared-libraries'
make  check-TESTS
make[5]: Entering directory '/var/tmp/portage/sys-cluster/extrae-3.8.3-r8/work/extrae-3.8.3/tests/functional/merger/shared-libraries'
make[6]: Entering directory '/var/tmp/portage/sys-cluster/extrae-3.8.3-r8/work/extrae-3.8.3/tests/functional/merger/shared-libraries'
FAIL: test-shared-library-debug.sh
FAIL: test-shared-library-nodebug.sh
============================================================================
Testsuite summary for Extrae 3.8.3
============================================================================
# TOTAL: 2
# PASS:  0
# SKIP:  0
# XFAIL: 0
# FAIL:  2
# XPASS: 0
# ERROR: 0
============================================================================
See tests/functional/merger/shared-libraries/test-suite.log
Please report to [email protected]
============================================================================
make[6]: *** [Makefile:951: test-suite.log] Error 1
make[6]: Leaving directory '/var/tmp/portage/sys-cluster/extrae-3.8.3-r8/work/extrae-3.8.3/tests/functional/merger/shared-libraries'
make[5]: *** [Makefile:1059: check-TESTS] Error 2
make[5]: Leaving directory '/var/tmp/portage/sys-cluster/extrae-3.8.3-r8/work/extrae-3.8.3/tests/functional/merger/shared-libraries'
make[4]: *** [Makefile:1139: check-am] Error 2
make[4]: Target 'check' not remade because of errors.
make[4]: Leaving directory '/var/tmp/portage/sys-cluster/extrae-3.8.3-r8/work/extrae-3.8.3/tests/functional/merger/shared-libraries'
make[4]: Entering directory '/var/tmp/portage/sys-cluster/extrae-3.8.3-r8/work/extrae-3.8.3/tests/functional/merger'
make[4]: Nothing to be done for 'check-am'.
make[4]: Leaving directory '/var/tmp/portage/sys-cluster/extrae-3.8.3-r8/work/extrae-3.8.3/tests/functional/merger'
make[3]: *** [Makefile:646: check-recursive] Error 1
make[3]: Target 'check' not remade because of errors.
make[3]: Leaving directory '/var/tmp/portage/sys-cluster/extrae-3.8.3-r8/work/extrae-3.8.3/tests/functional/merger'
Making check in callstack
make[3]: Entering directory '/var/tmp/portage/sys-cluster/extrae-3.8.3-r8/work/extrae-3.8.3/tests/functional/callstack'
make  check_Extrae_getcaller_depth2 check_Extrae_user_function
make[4]: Entering directory '/var/tmp/portage/sys-cluster/extrae-3.8.3-r8/work/extrae-3.8.3/tests/functional/callstack'
x86_64-pc-linux-gnu-gcc -DHAVE_CONFIG_H -I. -I../../..    -I../../../include -g -O2 -pipe -march=x86-64 -frecord-gcc-switches -fno-optimize-sibling-calls -Wall -W -c -o check_Extrae_getcaller_depth2-check_Extrae_getcaller_depth2.o `test -f 'check_Extrae_getcaller_depth2.c' || echo './'`check_Extrae_getcaller_depth2.c
x86_64-pc-linux-gnu-gcc -DHAVE_CONFIG_H -I. -I../../..    -I../../../include -I../../../src/common -g -O2 -pipe -march=x86-64 -frecord-gcc-switches -fno-optimize-sibling-calls -Wall -W -c -o check_Extrae_user_function-check_Extrae_user_function.o `test -f 'check_Extrae_user_function.c' || echo './'`check_Extrae_user_function.c
check_Extrae_getcaller_depth2.c: In function â��barâ��:
check_Extrae_getcaller_depth2.c:35:34: warning: format â�-Wformat=][]]
   35 |         printf ("begin = %p v = %p end = %p\n", &&bar_begin, v, &&bar_end);
      |                                 ~^                           ~
      |                                  |                           |
      |                                  void *                      uint64_t {aka long unsigned int}
      |                                 %ld
check_Extrae_getcaller_depth2.c: In function â��fooâ��:
check_Extrae_getcaller_depth2.c:50:34: warning: format â�-Wformat=][]]
   50 |         printf ("begin = %p v = %p end = %p\n", &&foo_begin, v, &&foo_end);
      |                                 ~^                           ~
      |                                  |                           |
      |                                  void *                      uint64_t {aka long unsigned int}
      |                                 %ld
check_Extrae_getcaller_depth2.c: In function â��mainâ��:
check_Extrae_getcaller_depth2.c:67:34: warning: format â�-Wformat=][]]
   67 |         printf ("begin = %p v = %p end = %p\n", &&main_begin, v, &&main_end);
      |                                 ~^                            ~
      |                                  |                            |
      |                                  void *                       uint64_t {aka long unsigned int}
      |                                 %ld
check_Extrae_user_function.c: In function â��barâ��:
check_Extrae_user_function.c:34:34: warning: format â�-Wformat=][]]
   34 |         printf ("begin = %p v = %p end = %p\n", &&bar_begin, v, &&bar_end);
      |                                 ~^                           ~
      |                                  |                           |
      |                                  void *                      uint64_t {aka long unsigned int}
      |                                 %ld
check_Extrae_user_function.c: In function â��fooâ��:
check_Extrae_user_function.c:49:34: warning: format â�-Wformat=][]]
   49 |         printf ("begin = %p v = %p end = %p\n", &&foo_begin, v, &&foo_end);
      |                                 ~^                           ~
      |                                  |                           |
      |                                  void *                      uint64_t {aka long unsigned int}
      |                                 %ld
check_Extrae_user_function.c: In function â��mainâ��:
check_Extrae_user_function.c:66:34: warning: format â�-Wformat=][]]
   66 |         printf ("begin = %p v = %p end = %p\n", &&main_begin, v, &&main_end);
      |                                 ~^                            ~
      |                                  |                            |
      |                                  void *                       uint64_t {aka long unsigned int}
      |                                 %ld
/bin/sh ../../../libtool  --tag=CC   --mode=link x86_64-pc-linux-gnu-gcc -I../../../include -g -O2 -pipe -march=x86-64 -frecord-gcc-switches -fno-optimize-sibling-calls -Wall -W -L../../../src/tracer -lseqtrace -Wl,-O1 -Wl,--as-needed -Wl,--defsym=__gentoo_check_ldflags__=0 -o check_Extrae_getcaller_depth2 check_Extrae_getcaller_depth2-check_Extrae_getcaller_depth2.o  
/bin/sh ../../../libtool  --tag=CC   --mode=link x86_64-pc-linux-gnu-gcc -I../../../include -I../../../src/common -g -O2 -pipe -march=x86-64 -frecord-gcc-switches -fno-optimize-sibling-calls -Wall -W -L../../../src/tracer -lseqtrace -Wl,-O1 -Wl,--as-needed -Wl,--defsym=__gentoo_check_ldflags__=0 -o check_Extrae_user_function check_Extrae_user_function-check_Extrae_user_function.o  
libtool: link: x86_64-pc-linux-gnu-gcc -I../../../include -g -O2 -pipe -march=x86-64 -frecord-gcc-switches -fno-optimize-sibling-calls -Wall -W -Wl,-O1 -Wl,--defsym=__gentoo_check_ldflags__=0 -o .libs/check_Extrae_getcaller_depth2 check_Extrae_getcaller_depth2-check_Extrae_getcaller_depth2.o  -L../../../src/tracer /var/tmp/portage/sys-cluster/extrae-3.8.3-r8/work/extrae-3.8.3/src/tracer/.libs/libseqtrace.so -L/usr/lib64 -lpapi -lunwind -lxml2 -ldl -lz -lrt -Wl,--as-needed
libtool: link: x86_64-pc-linux-gnu-gcc -I../../../include -I../../../src/common -g -O2 -pipe -march=x86-64 -frecord-gcc-switches -fno-optimize-sibling-calls -Wall -W -Wl,-O1 -Wl,--defsym=__gentoo_check_ldflags__=0 -o .libs/check_Extrae_user_function check_Extrae_user_function-check_Extrae_user_function.o  -L../../../src/tracer /var/tmp/portage/sys-cluster/extrae-3.8.3-r8/work/extrae-3.8.3/src/tracer/.libs/libseqtrace.so -L/usr/lib64 -lpapi -lunwind -lxml2 -ldl -lz -lrt -Wl,--as-needed
make[4]: Leaving directory '/var/tmp/portage/sys-cluster/extrae-3.8.3-r8/work/extrae-3.8.3/tests/functional/callstack'
make  check-TESTS
make[4]: Entering directory '/var/tmp/portage/sys-cluster/extrae-3.8.3-r8/work/extrae-3.8.3/tests/functional/callstack'
make[5]: Entering directory '/var/tmp/portage/sys-cluster/extrae-3.8.3-r8/work/extrae-3.8.3/tests/functional/callstack'
PASS: check_Extrae_getcaller_depth2.sh
PASS: check_Extrae_user_function.sh
============================================================================
Testsuite summary for Extrae 3.8.3
============================================================================
# TOTAL: 2
# PASS:  2
# SKIP:  0
# XFAIL: 0
# FAIL:  0
# XPASS: 0
# ERROR: 0
============================================================================
make[5]: Leaving directory '/var/tmp/portage/sys-cluster/extrae-3.8.3-r8/work/extrae-3.8.3/tests/functional/callstack'
make[4]: Leaving directory '/var/tmp/portage/sys-cluster/extrae-3.8.3-r8/work/extrae-3.8.3/tests/functional/callstack'
make[3]: Leaving directory '/var/tmp/portage/sys-cluster/extrae-3.8.3-r8/work/extrae-3.8.3/tests/functional/callstack'
Making check in hw-counters
make[3]: Entering directory '/var/tmp/portage/sys-cluster/extrae-3.8.3-r8/work/extrae-3.8.3/tests/functional/hw-counters'
make  check_Extrae_counters_xml
make[4]: Entering directory '/var/tmp/portage/sys-cluster/extrae-3.8.3-r8/work/extrae-3.8.3/tests/functional/hw-counters'
x86_64-pc-linux-gnu-gcc -DHAVE_CONFIG_H -I. -I../../..    -I../../../include -I../../../src/common -g -O2 -pipe -march=x86-64 -frecord-gcc-switches -fno-optimize-sibling-calls -Wall -W -c -o check_Extrae_counters_xml-check_Extrae_counters_xml.o `test -f 'check_Extrae_counters_xml.c' || echo './'`check_Extrae_counters_xml.c
/bin/sh ../../../libtool  --tag=CC   --mode=link x86_64-pc-linux-gnu-gcc -I../../../include -I../../../src/common -g -O2 -pipe -march=x86-64 -frecord-gcc-switches -fno-optimize-sibling-calls -Wall -W -L../../../src/tracer -lseqtrace -Wl,-O1 -Wl,--as-needed -Wl,--defsym=__gentoo_check_ldflags__=0 -o check_Extrae_counters_xml check_Extrae_counters_xml-check_Extrae_counters_xml.o  
libtool: link: x86_64-pc-linux-gnu-gcc -I../../../include -I../../../src/common -g -O2 -pipe -march=x86-64 -frecord-gcc-switches -fno-optimize-sibling-calls -Wall -W -Wl,-O1 -Wl,--defsym=__gentoo_check_ldflags__=0 -o .libs/check_Extrae_counters_xml check_Extrae_counters_xml-check_Extrae_counters_xml.o  -L../../../src/tracer /var/tmp/portage/sys-cluster/extrae-3.8.3-r8/work/extrae-3.8.3/src/tracer/.libs/libseqtrace.so -L/usr/lib64 -lpapi -lunwind -lxml2 -ldl -lz -lrt -Wl,--as-needed
make[4]: Leaving directory '/var/tmp/portage/sys-cluster/extrae-3.8.3-r8/work/extrae-3.8.3/tests/functional/hw-counters'
make  check-TESTS
make[4]: Entering directory '/var/tmp/portage/sys-cluster/extrae-3.8.3-r8/work/extrae-3.8.3/tests/functional/hw-counters'
make[5]: Entering directory '/var/tmp/portage/sys-cluster/extrae-3.8.3-r8/work/extrae-3.8.3/tests/functional/hw-counters'
FAIL: check_Extrae_PAPI_TOT_CYC.sh
FAIL: check_Extrae_PAPI_TOT_INS_CYC.sh
PASS: check_Extrae_PAPI_TOT_INS.sh
============================================================================
Testsuite summary for Extrae 3.8.3
============================================================================
# TOTAL: 3
# PASS:  1
# SKIP:  0
# XFAIL: 0
# FAIL:  2
# XPASS: 0
# ERROR: 0
============================================================================
See tests/functional/hw-counters/test-suite.log
Please report to [email protected]
============================================================================
make[5]: *** [Makefile:973: test-suite.log] Error 1
make[5]: Leaving directory '/var/tmp/portage/sys-cluster/extrae-3.8.3-r8/work/extrae-3.8.3/tests/functional/hw-counters'
make[4]: *** [Makefile:1081: check-TESTS] Error 2
make[4]: Leaving directory '/var/tmp/portage/sys-cluster/extrae-3.8.3-r8/work/extrae-3.8.3/tests/functional/hw-counters'
make[3]: *** [Makefile:1168: check-am] Error 2
make[3]: Target 'check' not remade because of errors.
make[3]: Leaving directory '/var/tmp/portage/sys-cluster/extrae-3.8.3-r8/work/extrae-3.8.3/tests/functional/hw-counters'
Making check in xml
make[3]: Entering directory '/var/tmp/portage/sys-cluster/extrae-3.8.3-r8/work/extrae-3.8.3/tests/functional/xml'
make  check_Extrae_xml
make[4]: Entering directory '/var/tmp/portage/sys-cluster/extrae-3.8.3-r8/work/extrae-3.8.3/tests/functional/xml'
x86_64-pc-linux-gnu-gcc -DHAVE_CONFIG_H -I. -I../../..    -I../../../include -I../../../src/common -g -O2 -pipe -march=x86-64 -frecord-gcc-switches -fno-optimize-sibling-calls -Wall -W -c -o check_Extrae_xml-check_Extrae_xml.o `test -f 'check_Extrae_xml.c' || echo './'`check_Extrae_xml.c
/bin/sh ../../../libtool  --tag=CC   --mode=link x86_64-pc-linux-gnu-gcc -I../../../include -I../../../src/common -g -O2 -pipe -march=x86-64 -frecord-gcc-switches -fno-optimize-sibling-calls -Wall -W -L../../../src/tracer -lseqtrace -Wl,-O1 -Wl,--as-needed -Wl,--defsym=__gentoo_check_ldflags__=0 -o check_Extrae_xml check_Extrae_xml-check_Extrae_xml.o  
libtool: link: x86_64-pc-linux-gnu-gcc -I../../../include -I../../../src/common -g -O2 -pipe -march=x86-64 -frecord-gcc-switches -fno-optimize-sibling-calls -Wall -W -Wl,-O1 -Wl,--defsym=__gentoo_check_ldflags__=0 -o .libs/check_Extrae_xml check_Extrae_xml-check_Extrae_xml.o  -L../../../src/tracer /var/tmp/portage/sys-cluster/extrae-3.8.3-r8/work/extrae-3.8.3/src/tracer/.libs/libseqtrace.so -L/usr/lib64 -lpapi -lunwind -lxml2 -ldl -lz -lrt -Wl,--as-needed
make[4]: Leaving directory '/var/tmp/portage/sys-cluster/extrae-3.8.3-r8/work/extrae-3.8.3/tests/functional/xml'
make  check-TESTS
make[4]: Entering directory '/var/tmp/portage/sys-cluster/extrae-3.8.3-r8/work/extrae-3.8.3/tests/functional/xml'
make[5]: Entering directory '/var/tmp/portage/sys-cluster/extrae-3.8.3-r8/work/extrae-3.8.3/tests/functional/xml'
FAIL: check_Extrae_xml_envvar_merge.sh
PASS: check_Extrae_xml_envvar_counters.sh
PASS: check_Extrae_xml.sh
============================================================================
Testsuite summary for Extrae 3.8.3
============================================================================
# TOTAL: 3
# PASS:  2
# SKIP:  0
# XFAIL: 0
# FAIL:  1
# XPASS: 0
# ERROR: 0
============================================================================
See tests/functional/xml/test-suite.log
Please report to [email protected]
============================================================================
make[5]: *** [Makefile:970: test-suite.log] Error 1
make[5]: Leaving directory '/var/tmp/portage/sys-cluster/extrae-3.8.3-r8/work/extrae-3.8.3/tests/functional/xml'
make[4]: *** [Makefile:1078: check-TESTS] Error 2
make[4]: Leaving directory '/var/tmp/portage/sys-cluster/extrae-3.8.3-r8/work/extrae-3.8.3/tests/functional/xml'
make[3]: *** [Makefile:1165: check-am] Error 2
make[3]: Target 'check' not remade because of errors.
make[3]: Leaving directory '/var/tmp/portage/sys-cluster/extrae-3.8.3-r8/work/extrae-3.8.3/tests/functional/xml'
make[3]: Entering directory '/var/tmp/portage/sys-cluster/extrae-3.8.3-r8/work/extrae-3.8.3/tests/functional'
make[3]: Nothing to be done for 'check-am'.
make[3]: Leaving directory '/var/tmp/portage/sys-cluster/extrae-3.8.3-r8/work/extrae-3.8.3/tests/functional'
make[2]: *** [Makefile:648: check-recursive] Error 1
make[2]: Target 'check' not remade because of errors.
make[2]: Leaving directory '/var/tmp/portage/sys-cluster/extrae-3.8.3-r8/work/extrae-3.8.3/tests/functional'
Making check in overhead
make[2]: Entering directory '/var/tmp/portage/sys-cluster/extrae-3.8.3-r8/work/extrae-3.8.3/tests/overhead'
make[2]: Nothing to be done for 'check'.
make[2]: Leaving directory '/var/tmp/portage/sys-cluster/extrae-3.8.3-r8/work/extrae-3.8.3/tests/overhead'
make[2]: Entering directory '/var/tmp/portage/sys-cluster/extrae-3.8.3-r8/work/extrae-3.8.3/tests'
make[2]: Nothing to be done for 'check-am'.
make[2]: Leaving directory '/var/tmp/portage/sys-cluster/extrae-3.8.3-r8/work/extrae-3.8.3/tests'
make[1]: *** [Makefile:647: check-recursive] Error 1
make[1]: Target 'check' not remade because of errors.
make[1]: Leaving directory '/var/tmp/portage/sys-cluster/extrae-3.8.3-r8/work/extrae-3.8.3/tests'
make: *** [Makefile:1116: check-recursive] Error 1

add support for libsma

I built and installed https://github.com/Sandia-OpenSHMEM/SOS/ but it doesn't get recognised by the option --with-openshmem because it's library soname is libsma instead of libopenshmem

Googling I found this piece of code that may solve the issue https://fossies.org/linux/tau/src/wrappers/shmem/tau_shmem_wrap line 62

Trace Merging Fails

Hello,

When I try running extrae, the merging of the trace files fails.

merger: Extrae 3.8.3
mpi2prv: Assigned nodes < 0e93bef4c796 >
mpi2prv: Assigned size per processor < 0 bytes >
mpi2prv: File /src/set-0/[email protected] is object 1.1.1 on node 0e93bef4c796 assigned to processor 0
mpi2prv: File /src/set-0/[email protected] is object 1.1.2 on node 0e93bef4c796 assigned to processor 0
etc...
mpi2prv: Time synchronization has been turned off
mpi2prv: Checking for target directory existence... exists, ok!
fopen: No such file or directory
mpi2prv Error: Opening trace file /src/set-0/[email protected]
AddFile_FS: No such file or directory
mpi2prv: Error creating file set
mpi2prv: Error! Some of the processors failed create trace descriptors
mpi2prv: An error has been encountered when generating the tracefile. Dying...

Running mpi2prv manually fails on opening the files as well. The files do exist and are in the location printed, but it seems that mpi2prv is unable to find them. Assistance on this would be very appreciated.

FR: Add support for MPI RMA atomics and passive mode

Dear Extrae Devs,

I am using Extrae and Paraver to analyze the behavior of my application that's entirely based on MPI-3 RMA. I noticed that Extrae does not cover all MPI functions that are relevant for my code, most notable some of the atomic operations (MPI_Fetch_and_op and MPI_Compare_and_swap) as well as passive window synchronization (MPI_Win_flush*). I've been trying to add support myself following the example of MPI_Get_accumulate but seem to run into some memory corruption. Before going down this road further I wanted to ask whether there is already support for these functions available as a patch or in a branch, just to avoid duplicating work. Please let me know if there is, I'm also happy to test development patches.

another build fix

The staging directory shouldn't appear in installed files.

--- extrae-3.7.1/src/others/Makefile.am~        2019-07-05 11:24:32.000000000 +0000
+++ extrae-3.7.1/src/others/Makefile.am 2019-07-22 20:39:14.457120718 +0000
@@ -40,7 +40,7 @@
 
 install-data-hook:
        cp pyextrae/common/extrae.py.in pyextrae/common/extrae.py
-       $(top_srcdir)/substitute $(SED) "@sub_PREFIX@" $(DESTDIR)${prefix} pyextrae/common/extrae.py
+       $(top_srcdir)/substitute $(SED) "@sub_PREFIX@" ${prefix} pyextrae/common/extrae.py
 #if HAVE_PAPI
        $(top_srcdir)/substitute $(SED) "@sub_Extrae_eventandcounters@" Extrae_eventandcounters pyextrae/common/extrae.py
        $(top_srcdir)/substitute $(SED) "@sub_Extrae_neventandcounters@" Extrae_neventandcounters pyextrae/common/extrae.py

Extrae is not instrumenting MPI exclusive scans

Regarding MPI collective operations, or more precisely prefix reductions: While Extrae (version 4.0.3) is correctly instrumenting MPI's inclusive scan function MPI_Scan(), it is not instrumenting its exclusive scan function MPI_Exscan(). Therefore, all exclusive scans don't show up and thus can't be analyzed in e.g. Paraver. Note: This applies to the blocking as well as the nonblocking variant (i.e., MPI_Iexscan).

possible reason: The inclusive scan MPI_Scan() had been there already in MPI version 1.0 from 1994 (see chapter 4.11), but the exclusive scan MPI_Exscan() has been added only later in version 2.1 of the MPI standard from 2008 (see chapter 5.11.2).

importance: The exclusive scan is more general than the inclusive scan, and is getting used more and more often e.g. in efficient load balancing and partitioning algorithms at scale. Therefore, and being part of the MPI standard for 15 years now, it should clearly be instrumented by Extrae. I cross-checked with Score-P, which correctly instrumented both prefix reductions, so that it appears in e.g. Cube.

to-do: Add support for blocking and nonblocking variants of the MPI exclusive scan function(s) to Extrae.

mini example in C to check support for inclusive and exclusive prefix reductions:

#include <mpi.h>

int main(int argc, char *argv[])
{
   int result, input=42;

   MPI_Init(&argc, &argv);

   /* While the inclusive scan gets instrumented by Extrae, */
   MPI_Scan(&input, &result, 1, MPI_INT, MPI_SUM, MPI_COMM_WORLD);
   /* the exclusive scan unfortunately does not get instrumented (yet). */
   MPI_Exscan(&input, &result, 1, MPI_INT, MPI_SUM, MPI_COMM_WORLD);

   MPI_Finalize();
   return 0;
}

Memory corruption in ia32_Initialize

I've been chasing a bug that led to corruption of the buffer containing the configuration file name during startup of extrae, leading to something like:

I/O warning : failed to load external entity "���"
Extrae: Detected GOMP version is 4.5
I/O warning : Extrae: Detected and hooked OpenMP runtime: [Intel KMPC] [GNU GOMP]
failed to load external entity "���"

After some debugging I found that the ia32 clock initialization is corrupting the memory and the reason seems to be the following lines in ia32_Initialize:

  char buffer[ 32768 ];
  [...]
  fp = fopen( "/proc/cpuinfo", "r" );
  bytes_read = fread( buffer, 1, sizeof( buffer ), fp );
  fclose( fp );

  if (bytes_read == 0)
    return;

  buffer[ bytes_read ] = '\0';

Note that if fread reads exactly 32k bytes the write access in the last line is out-of-bounds. Indeed the size of /proc/cpuinfo is larger than 32k on our system:

$ aprun -n 1 cat /proc/cpuinfo | wc
   1297    9469   52636

Changing the line to buffer[ bytes_read-1 ] = '\0'; seems to prevent the memory from getting corrupted and leads to successful parsing of the config file.

Also note that the same problem exists in the ia64 and ppc implementations.

Suspicious address range condition in ObjectTable_GetBinaryObjectAt

The end address in *.sym file entries is not included in the range (sometimes appears as the start address of the next entry), but the condition in ObjectTable_GetBinaryObjectAt uses <=. I think this may find the wrong object if the looked up address is at the start of a range and a range ending at this address appears before, but I didn't encounter this case in my experiments (i.e. I may be missing something).

mpi2prv: merging ends with segmentation fault

Environment:

Extrae-4.0.1 built with GCC 10.2.1, Intel MPI 2021.5, PAPI 6.0.0.1 and Libunwind-1.6.2 on AlmaLinux 8.5

Execution command

mpi2prv -syn -f TRACE.mpits -e <exe> -o <output>.prv

Error log

mpi2prv: Error! File -syn does not contain a valid extension!. Skipping.
mpi2prv: Retrieving hardware counters definitions for ptask 1 from global SYM.
mpi2prv: A total of 6 symbols were imported from TRACE.sym file
mpi2prv: 0 function symbols imported
mpi2prv: 6 HWC counter descriptions imported
merger: Output trace format is: Paraver
merger: Extrae 4.0.1
mpi2prv: Assigned nodes < myhostname >
mpi2prv: Assigned size per processor < <1 Mbyte >
mpi2prv: File /u/vinson3z/Downloads/Gearbox_explicit_20220722/extrae-results/set-0/[email protected] is object 1.37.1 on node myhostname assigned to processor 0
mpi2prv: Time synchronization has been turned off
mpi2prv: Checking for target directory existence... exists, ok!
mpi2prv: Selected output trace format is Paraver
mpi2prv: Stored trace format is Paraver
mpi2prv: Enabling Time Synchronization (Node).
WARNING: TimeSync_CalculateLatencies: Task 0 was not initialized. Synchronization disabled!
mpi2prv: Circular buffer enabled at tracing time? NO
mpi2prv: Parsing intermediate files
mpi2prv: Progress 1 of 2 ... 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% done
mpi2prv: Processor 0 succeeded to translate its assigned files
mpi2prv: Elapsed time translating files: 0 hours 0 minutes 0 seconds
mpi2prv: Elapsed time sorting addresses: 0 hours 0 minutes 0 seconds
mpi2prv: Generating tracefile (intermediate buffers of 6710784 events)
         This process can take a while. Please, be patient.
mpi2prv: Progress 2 of 2 ... 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% done
mpi2prv: Elapsed time merge step: 0 hours 0 minutes 0 seconds
mpi2prv: Resulting tracefile occupies 126494 bytes
mpi2prv: Removing temporal files... done
mpi2prv: Elapsed time removing temporal files: 0 hours 0 minutes 0 seconds
Segmentation fault (core dumped)

Error backtrace (gdb)

0x000000000041a614 in ObjectTable_dumpAddresses (fd=fd@entry=0x5e3210, eventstart=41000001, eventstart@entry=41000000) at ../../../src/merger/common/object_tree.c:294
294                             for (_address = 0; _address < task_info->binary_objects[0].nDataSymbols; _address++)
Missing separate debuginfos, use: dnf debuginfo-install zlib-1.2.11-18.el8_5.x86_64
(gdb) bt
#0  0x000000000041a614 in ObjectTable_dumpAddresses (fd=fd@entry=0x5e3210, eventstart=41000001, eventstart@entry=41000000) at ../../../src/merger/common/object_tree.c:294
#1  0x000000000040d922 in Labels_GeneratePCFfile (name=name@entry=0x7fffffff73a0 "sphflow.pcf", options=options@entry=1041) at ../../../src/merger/paraver/labels.c:1066
#2  0x0000000000410fa0 in Paraver_ProcessTraceFiles (nfiles=1, files=0x5d42f0, num_appl=<optimized out>, NodeCPUinfo=NodeCPUinfo@entry=0x5d5b60, numtasks=numtasks@entry=1,
    taskid=taskid@entry=0) at ../../../src/merger/paraver/trace_to_prv.c:678
#3  0x00000000004046c3 in merger_post (numtasks=numtasks@entry=1, taskid=taskid@entry=0) at ../../../src/merger/common/mpi2out.c:1485
#4  0x0000000000406337 in merger_post (numtasks=numtasks@entry=1, taskid=taskid@entry=0) at ../../../src/merger/common/mpi2out.c:1366
#5  0x0000000000403a6e in main (argc=8, argv=0x7fffffff8d68) at ../../../src/merger/merger.c:69

Info

The traces are generated without any error. The exact same error is also observed using Extrae-3.8.3.

Is there any fix for this problem? Or, is it me who's using it incorrectly?

Extrae build fails with binutils-2.39

OS: Debian GNU/Linux bookworm/sid x86_64
binutils-dev: 2.39-8
gcc (Debian 12.2.0-9) 12.2.0
extrae: 4.0.1

Instalation from Git. steps:

$ ./bootstrap

$ ./configure --enable-openmp --without-dyninst --without-unwind --with-papi=/usr/local/ --with-mpi=/usr/lib/x86_64-linux-gnu/openmpi/

Package configuration for Extrae 4.0.1

Installation prefix: /usr/local
Cross compilation: no
CC: gcc
CXX: g++
Binary type: 64 bits

MPI instrumentation: yes
MPI home: /usr/lib/x86_64-linux-gnu/openmpi/
MPI launcher was not found. It is NOT necessary to compile Extrae but it is necessary to execute the regression tests.
Fortran decoration: 0 underscores
mixed C/Fortran libraries? no
shared libraries? yes
MPI capabilities: 1-sided I/O MPI_Comm_spawn MPI3
GASPI instrumentation: no
OpenMP instrumentation: yes, through LD_PRELOAD
GNU OpenMP: yes
IBM OpenMP: no
Intel OpenMP: yes
OMPT: yes
OpenSHMEM instrumentation: no
pThread instrumentation: yes
Support for pthread_barrier_wait: yes
Support for pthread_cond_* calls: yes
CUDA instrumentation: no
OpenCL instrumentation: no
OPENACC instrumentation: no
Java instrumentation: unsupported

Performance counters: yes
Performance API: PAPI
PAPI home: /usr/local/
Sampling support: yes

PEBS sampling: yes

libbfd available: yes (/usr/lib/x86_64-linux-gnu)
libiberty available: yes (/usr/lib/x86_64-linux-gnu)
zlib available: yes (/usr/local)
libxml2 available: yes (/usr)
BOOST available: no
callstack access: through backtrace (from linux)

Dynamic instrumentation: yes (not found)

Optional features:

On-line analysis: no
Clock routine: POSIX / clock_gettime, but don't need to link against posix clock library explicitly
Heterogeneous support: no
Parallel merge: yes

The error:

make[4]: Entering directory '/home/amvaldesj/programs/extrae/src/merger/parallel'
CC ../common/libmpimpi2prv_la-bfd_manager.lo
../common/bfd_manager.c:228:78: error: unknown type name 'PTR'
228 | static void BFDmanager_findAddressInSection (bfd * abfd, asection * section, PTR data)
| ^~~
../common/bfd_manager.c: In function 'BFDmanager_translateAddress':
../common/bfd_manager.c:300:50: error: 'BFDmanager_findAddressInSection' undeclared (first use in this function)
300 | bfd_map_over_sections (bfdImage, BFDmanager_findAddressInSection, &syminfo);
| ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
../common/bfd_manager.c:300:50: note: each undeclared identifier is reported only once for each function it appears in
make[4]: *** [Makefile:1328: ../common/libmpimpi2prv_la-bfd_manager.lo] Error 1
make[4]: Leaving directory '/home/amvaldesj/programs/extrae/src/merger/parallel'
make[3]: *** [Makefile:1339: all-recursive] Error 1
make[3]: Leaving directory '/home/amvaldesj/programs/extrae/src/merger'
make[2]: *** [Makefile:795: all-recursive] Error 1
make[2]: Leaving directory '/home/amvaldesj/programs/extrae/src'
make[1]: *** [Makefile:1276: all-recursive] Error 1
make[1]: Leaving directory '/home/amvaldesj/programs/extrae'
make: *** [Makefile:1208: all] Error 2

posix_memalign wrapper calls incorrect exit function

The posix_memalign wrapper calls Probe_memkind_malloc_Exit instead of Probe_posix_memalign_Exit.

Bug: MPI_Testsome wrapper calls PMPI_Waitsome

I found that the Extrae wrapper for MPI_Testsome (MPI_Testsome_C_Wrapper) internally calls MPI_Waitsome instead of MPI_Testsome. In most cases, this is probably OK. However, in my specific case this leads to a deadlock because the algorithm relies on test semantics instead of wait semantics.

Given that the wrappers for MPI_Testany and MPI_Testall call the right test functions I assume that this is a bug.

Seen in Extrae 3.5.2.

CUDA builds crash since v4.0.3

I'm trying to build Extrae v4.0.3 with CUDA tracing through the CUPTI interface but I get the following error:

/../../src/common -I/workspace/destdir/include -I/workspace/destdir/cuda/include -I/workspace/destdir/cuda/extras/CUPTI/include  -g -O2 -fno-optimize-sibling-calls -Wall -W -c -o libwrap_cuda_la-cuda_wrapper_cupti.lo `test -f 'cuda_wrapper_cupti.c' || echo './'`cuda_wrapper_cupti.c
libtool: compile:  cc -DHAVE_CONFIG_H -I. -I../../../.. -I../../../../src/common/MPI -I../../../../src/tracer -I../../../../src/tracer/hwc -I../../../../src/tracer/clocks -I../../../../src/tracer/interfaces/API -I../../../../src/tracer/wrappers/API -I../../../.. -I../../../../include -I../../../../src/common -I/workspace/destdir/include -I/workspace/destdir/cuda/include -I/workspace/destdir/cuda/extras/CUPTI/include -g -O2 -fno-optimize-sibling-calls -Wall -W -c cuda_wrapper_cupti.c  -fPIC -DPIC -o .libs/libwrap_cuda_la-cuda_wrapper_cupti.o
cuda_wrapper_cupti.c: In function ‘Extrae_DriverAPI_callback’:
cuda_wrapper_cupti.c:146:29: warning: passing argument 1 of ‘Extrae_cudaLaunch_Enter’ from incompatible pointer type [-Wincompatible-pointer-types]
     Extrae_cudaLaunch_Enter(p->f, p->hStream);
                             ^
In file included from cuda_wrapper_cupti.c:44:0:
cuda_common.h:179:6: note: expected ‘const char *’ but argument is of type ‘CUfunction {aka struct CUfunc_st *}’
 void Extrae_cudaLaunch_Enter (const char*, cudaStream_t);
      ^
cuda_wrapper_cupti.c: In function ‘Extrae_RuntimeAPI_callback’:
cuda_wrapper_cupti.c:167:4: error: unknown type name ‘cudaConfigureCall_v3020_params’
    cudaConfigureCall_v3020_params *p =
    ^
cuda_wrapper_cupti.c:168:7: error: ‘cudaConfigureCall_v3020_params’ undeclared (first use in this function)
      (cudaConfigureCall_v3020_params*)cbinfo->functionParams;
       ^
cuda_wrapper_cupti.c:168:7: note: each undeclared identifier is reported only once for each function it appears in
cuda_wrapper_cupti.c:168:38: error: expected expression before ‘)’ token
      (cudaConfigureCall_v3020_params*)cbinfo->functionParams;
                                      ^
cuda_wrapper_cupti.c:172:8: error: request for member ‘gridDim’ in something not a structure or union
       p->gridDim, p->blockDim, p->sharedMem, p->stream
        ^
cuda_wrapper_cupti.c:172:20: error: request for member ‘blockDim’ in something not a structure or union
       p->gridDim, p->blockDim, p->sharedMem, p->stream
                    ^
cuda_wrapper_cupti.c:172:33: error: request for member ‘sharedMem’ in something not a structure or union
       p->gridDim, p->blockDim, p->sharedMem, p->stream
                                 ^
cuda_wrapper_cupti.c:172:47: error: request for member ‘stream’ in something not a structure or union
       p->gridDim, p->blockDim, p->sharedMem, p->stream
                                               ^
cuda_wrapper_cupti.c:184:4: error: unknown type name ‘cudaLaunch_v3020_params’
    cudaLaunch_v3020_params *p =
    ^
cuda_wrapper_cupti.c:185:7: error: ‘cudaLaunch_v3020_params’ undeclared (first use in this function)
      (cudaLaunch_v3020_params*)cbinfo->functionParams;
       ^
cuda_wrapper_cupti.c:185:31: error: expected expression before ‘)’ token
      (cudaLaunch_v3020_params*)cbinfo->functionParams;
                               ^
cuda_wrapper_cupti.c:189:30: error: request for member ‘func’ in something not a structure or union
     Extrae_cudaLaunch_Enter(p->func, NULL);
                              ^
cuda_wrapper_cupti.c:298:6: warning: initialization from incompatible pointer type [-Wincompatible-pointer-types]
      (cudaMallocArray_v3020_params *)cbinfo->functionParams;
      ^
cuda_wrapper_cupti.c:439:4: error: unknown type name ‘cudaStreamDestroy_v3020_params’
    cudaStreamDestroy_v3020_params *p =
    ^
cuda_wrapper_cupti.c:440:7: error: ‘cudaStreamDestroy_v3020_params’ undeclared (first use in this function)
      (cudaStreamDestroy_v3020_params*)cbinfo->functionParams;
       ^
cuda_wrapper_cupti.c:440:38: error: expected expression before ‘)’ token
      (cudaStreamDestroy_v3020_params*)cbinfo->functionParams;
                                      ^
cuda_wrapper_cupti.c:443:38: error: request for member ‘stream’ in something not a structure or union
     Extrae_cudaStreamDestroy_Enter (p->stream);
                                      ^
make[5]: *** [Makefile:851: libwrap_cuda_la-cuda_wrapper_cupti.lo] Error 1
make[5]: Leaving directory '/workspace/srcdir/extrae-4.0.3/src/tracer/wrappers/CUDA'
make[4]: *** [Makefile:803: all-recursive] Error 1
make[4]: Leaving directory '/workspace/srcdir/extrae-4.0.3/src/tracer/wrappers'
make[3]: *** [Makefile:7073: all-recursive] Error 1
make[3]: Leaving directory '/workspace/srcdir/extrae-4.0.3/src/tracer'
make[2]: *** [Makefile:795: all-recursive] Error 1
make[2]: Leaving directory '/workspace/srcdir/extrae-4.0.3/src'
make[1]: *** [Makefile:1275: all-recursive] Error 1
make[1]: Leaving directory '/workspace/srcdir/extrae-4.0.3'
make: *** [Makefile:1207: all] Error 2

Looks like the the cudaConfigureCall_v3020_params, cudaLaunch_v3020_params and cudaStreamDestroy_v3020_params are defined nowhere. I think CUPTI header do not provide these definitions since CUPTI v12 (CUDA 10?) and commit 388dd3b removed the hardcoded definitions that you had.

bsc-performance-tools / extrae Goto Github PK

extrae's Introduction

Extrae

Optional dependencies

Installation instructions

extrae's People

Contributors

Stargazers

Watchers

Forkers

extrae's Issues

Question

MPI usage

Attempts to trace MPI calls

Environment:

Execution command

Error log

Error backtrace (gdb)

Info

Package configuration for Extrae 4.0.1

Optional features:

Recommend Projects

Recommend Topics

Recommend Org