memtt / malt Goto Github PK

Addr2line had serveral time issues du to ASLR & fPIE activation depending on platforms.

To have a safe fallback in case it append again, could be nice to provide a second symbol extraction using GDB directly.

• prepare a GDB script to extract all symbols
• launch GDB and attach to current process
• run script with output dumping to a file
• read the file when process start again

Should not be defualt mode but could be interesting to have.

Having something that provides a quick comparison between runs from the cli would be a boon.

Would be possible to support other architectures?
Could be possible having a porting how-to in the wiki?

Might be nice to reformat a bit the doc to be able to push them attached to version history in https://readthedocs.org/

Would you like to wrap any pointer data members with the class template “std::unique_ptr”?

Update candidates:

• ProfiledCumulValue
• ProfiledValue

Would be nice to estimate page fault time.

When the tool start we can make a time measurement of page faults from sequential to parallel.

Then we can intercept mmap and count new pages max loaded into the app (will be an estimation because we are not sure it touch it).

Then just multiply by min/max page fault time. We need min/max because we know actually linux kernel page fault don't scale so if they are made in parallel this will kill the perf with high factor.

Need to take care in the GUI to well show this is an estimation and not an explicitly measured cost.

Mix of underscore and camel case. Need to keep underscore as old options, just need to fix the new one in [dump] section.

Build a chart with posix_memalign alignements to be used by the application.

We have "leak size" metric, can also add "leak count" to know how many blocs we leaked.

Add message on stderr in non silent mode to say when file has been dumped.

Also provide a verbose mode to print calls to addr2line.

graphviz is required to draw the call tree ;P

Is there a way to exclude some detected memory leaks by ignoring some symbols ?

For Valgrind, it is called suppression files: http://valgrind.org/docs/manual/manual-core.html#manual-core.suppress
There is often small memory leaks in 3rd libraries which cannot be easily corrected.

Forgot to use macros to get current version.

Upon loading a trace I'm seeing in the browser console:

XML Parsing Error: syntax error
Location: http://kaby:8080/flat.json
Line Number 1, Column 1: flat.json:1:1
XML Parsing Error: syntax error
Location: http://kaby:8080/stacks.json?func=
Line Number 1, Column 1: stacks.json:1:1
XML Parsing Error: syntax error
Location: http://kaby:8080/stacks.json?func=rav1e%3A%3Aencoder%3A%3Aencode_tx_block
Line Number 1, Column 1:
stacks.json:1:1

And in the network tab indeed I see the type marked as xml.

Tested with firefox 67.0b15 and chrome and the behavior seems the same.

To check we nicely extract stacks (mostly to validate fastSkip) and also check symbol extraction portability over distributions which already had several times changes.

Just fix in regexp in JS server script.

Just make a chart with allocated versus mapped/touched.

Would be nice to also support python.

Two ways, use language self inspection as looks possible by https://github.com/asciimoo/memspector/blob/master/memspector.py

Or intercept malloc calls inside python engine and ask via python lib where we are in the python code (not sure this is possible).

Asked by a user.

Can be usefull to profile apps not terminating.

I would see

malt --dump-after X

This can be done with just a timeout. Easy in principle just need to check memory leak side effects and maybe add some mutex as mallocs / no reentrance can append at same time.

I am profiling an application that is tested under a Linux distro that is based on 32-bit (i386) architecture.

The Malt is written assuming that elf object is 64-bit architecture only. It would be more than okay, if the Malt could be automatically configured to the correct 32/64 bitness. Or even dynamically detect 32/64 bitness of the elf object.

I have kind of minimal set of changes that allows the 32-bit Malt to pass Unit tests 100% successfully. Not yet tested beyond unit tests.
patch-malt-32-bits.txt

I'm using GCC 8.3.0 on Ubuntu cosmic and get the following warning when building the tool:

../extern-deps/iniparser/src/iniparser.c: In function ‘iniparser_load’:
../extern-deps/iniparser/src/iniparser.c:701:30: warning: ‘%s’ directive writing up to 1024 bytes into a region of size between 0 and 1024 [-Wformat-overflow=]
             sprintf(tmp, "%s:%s", section, key);
                              ^~            ~~~
../extern-deps/iniparser/src/iniparser.c:701:13: note: ‘sprintf’ output between 2 and 2050 bytes into a destination of size 1025
             sprintf(tmp, "%s:%s", section, key);
             ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

To reproduce:

• check out the library
• mkdir build
• cd build
• cmake .. -GNinja
• ninja

This might be a simple patch just wrapper the rust jemalloc in addition to the standard allocator. Could be an issue if they wrap it with hash in function name, but if not the patch should be trivial.

Add functions to log messages to help user based on verbosity levels instead of using printf direclty and if/else to verbose/silent.

https://travis-ci.org/memtt/malt/builds/546950808?utm_medium=notification&utm_source=email

$ sudo apt-get install -y node libqt5webkit5-dev
Reading package lists... Done
Building dependency tree       
Reading state information... Done
Package node is not available, but is referred to by another package.
This may mean that the package is missing, has been obsoleted, or
is only available from another source
E: Package 'node' has no installation candidate
The command "sudo apt-get install -y node libqt5webkit5-dev" failed and exited with 100 during .
Your build has been stopped.

In additionto get full memory consumption would be nice to also extract memory compared to current cgroup.

First step is the backend, front-end can come later.

Display which allocator is in use. This can be detected with .so file names or/and by symbol names.

When running on background daemon somtimes we do not have anymore stdout/stderr to get info being sure MALT is profiling. Could be nice to have an option to dump those log stream into a requested file instead.

Hi

the default gcc version doesn't seem to be able to rebuild the x86_64.rpm with a plain "rpmbuild --rebuild".

[ 98%] Building CXX object src/qtview/CMakeFiles/malt-qt5.dir/main.cpp.o
cd /home/tru/rpmbuild/BUILD/malt-1.0.0/src/qtview && /usr/bin/c++   -DHAVE_QT5_WEBKIT_WIDGETS -DMALT_PREFIX=\"/usr\" -DQT_CORE_LIB -DQT_GUI_LIB -DQT_NETWORK_LIB -DQT_NO_DEBUG -DQT_WEBKITWIDGETS_LIB -DQT_WEBKIT_LIB -DQT_WIDGETS_LIB -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches   -m64 -mtune=generic  -I/home/tru/rpmbuild/BUILD/malt-1.0.0/src/qtview -I/usr/include/qt5 -I/usr/include/qt5/QtWidgets -I/usr/include/qt5/QtGui -I/usr/include/qt5/QtCore -I/usr/lib64/qt5/./mkspecs/linux-g++ -I/usr/include/qt5/QtWebKitWidgets -I/usr/include/qt5/QtWebKit -I/usr/include/qt5/QtNetwork    -fPIC -o CMakeFiles/malt-qt5.dir/main.cpp.o -c /home/tru/rpmbuild/BUILD/malt-1.0.0/src/qtview/main.cpp
In file included from /usr/include/c++/4.8.2/type_traits:35:0,
                 from /usr/include/qt5/QtCore/qglobal.h:45,
                 from /usr/include/qt5/QtGui/qtguiglobal.h:43,
                 from /usr/include/qt5/QtWidgets/qtwidgetsglobal.h:43,
                 from /usr/include/qt5/QtWidgets/qapplication.h:43,
                 from /usr/include/qt5/QtWidgets/QApplication:1,
                 from /home/tru/rpmbuild/BUILD/malt-1.0.0/src/qtview/main.cpp:11:
/usr/include/c++/4.8.2/bits/c++0x_warning.h:32:2: error: #error This file requires compiler and library support for the ISO C++ 2011 standard. This support is currently experimental, and must be enabled with the -std=c++11 or -std=gnu++11 compiler options.
 #error This file requires compiler and library support for the \
  ^
In file included from /usr/include/qt5/QtCore/qglobal.h:98:0,
                 from /usr/include/qt5/QtGui/qtguiglobal.h:43,
                 from /usr/include/qt5/QtWidgets/qtwidgetsglobal.h:43,
                 from /usr/include/qt5/QtWidgets/qapplication.h:43,
                 from /usr/include/qt5/QtWidgets/QApplication:1,
                 from /home/tru/rpmbuild/BUILD/malt-1.0.0/src/qtview/main.cpp:11:
/usr/include/qt5/QtCore/qcompilerdetection.h:567:6: error: #error Qt requires a C++11 compiler and yours does not seem to be that.
 #    error Qt requires a C++11 compiler and yours does not seem to be that.

If I am using a newer gcc version such as the one provided by devtoolset-6, it builds fine.

On redhat and centos7, seems we skip one level too much to remove MALT internals in call stacks when backtracing, need to check if on all platform or due to some inlining issues in release mode.

Again on gentoo we have troubles due certainly to fPIE approach for every binaries.

There is already a fix in MALT to fallback to .so mode in this case but it seems not working anymore.

On some now old redhat install (7) seems to have an issue with the provided nodejs which does not provide .endsWidth() member on strings used into some C++ function name reshaping in MALT since CERN patches.

Is it possible to catch and to track calls to custom allocators (malloc()/free() like functions) (for detecting memory leaks) ?

In my specific case, I'm using DPDK which provides its own memory management functions (https://doc.dpdk.org/api/rte__malloc_8h.html). For this kind of heavy load application, Valgrind is unusable. MALT seems to work with my application for usual malloc()/free() but I would like to extend its use to the custom allocator.

Can be interesting to get posix_memalign alignement chart

Add a metric to detect when a bloc is freed by another thread, this can create contention on most allocators.

Currenty use array to provide multiple values without repeating field names. But I never placed the array ordered values namings, need to add one inside each section to ease data understanding and future reuse.

There is an issue on Chrome & QT web-kit which misalign the source annotation with a small drift.

There is a bug in wrapper script which instrument SIGUSR1 while we ask for SIGUSR2.

We we indirection via string array to avoid duplicate strings too many time in JSON profile files. But forgot to apply on binary field.

Need to not forget to increment file API version and check we use the right one when loading to not mismatch old/new front-end/back-end.

 "instr":{
                        "0x1":{
                                "function":19
                        },
                        "0x55a9754c31c9":{
                                "file":0,
                                "function":20,
                                "binary":"/home/......./malt/build/src/lib/tests/simple-case-finstr-linked"
                        },
}

In case we forget the debug symbols we might want to run a post mortem tool to recompute the symbols. This can be done in NodeJS directly without too much issues.

Can be called Malt-resolve.

Forgot I stopped making it and might be interesting to finish.

https://github.com/memtt/malt/blob/master/src/doc/file-format.md

Need to remove output from addr2line & ELF warnings:

addr2line -C -f -e /usr/share/locale/en/LC_MESSAGES/kwidgetsaddons5_qt.qm
addr2line -C -f -e /usr/share/locale/fr/LC_MESSAGES/kconfig5_qt.qm
addr2line -C -f -e /usr/share/locale/fr/LC_MESSAGES/kcompletion5_qt.qm
addr2line -C -f -e /usr/share/locale/fr/LC_MESSAGES/kitemviews5_qt.qm

Warning : At /home/sebv/Projects/malt/src/lib/tools/ELFReader.cpp:190 : 
Warning : Cuation, get NULL elf structure, ignore it and skip global variable extraction.

Warning : At /home/sebv/Projects/malt/src/lib/tools/ELFReader.cpp:106 : 
Warning : Invalid kind of elf file, expect ELF (3), get 0

Warning : At /home/sebv/Projects/malt/src/lib/tools/ELFReader.cpp:190 : 
Warning : Cuation, get NULL elf structure, ignore it and skip global variable extraction.

Add a button on bottom right of source GUI to unfold N levels automatically to quickly navigate in call tree.

Add to dev/packaging.sh a check validating manpages has been rebuilt.

Measure memory fragmentation at peak time or on an event based on signals.

We have the list for allocated segments, size, pointer and know the mmap list so we can compute how much we loose before and after each segment.

The real overhead is after or before but this is not strictly exact we can take average or before + after to just get an idea.

The nice thing is we can map on the related line.

Avoid to install thousands JS/CSS.. files for GUI and use grunt to build one file JS/CSS to install only this. Scripts already exist to build, need to call again (was done in the past).

In case we forgot to install debug symbols at running time, we can provide a script afterward to fix the json file to extract the sources in post mordem in order to avoid running again.

Should be easy, maybe make in python or reuse the webserver nodejs code.

Need to check $2 is not empty & error.

Bug reported outside of GitHub by mail. To be checked.

$> malt-webview --no-auth -i malt-pvbatch-0.json 
loading file malt-pvbatch-0.json...
Starting server on http://localhost:8080
Loading file : malt-pvbatch-0.json
Failed to open file malt-pvbatch-0.json, get error : undefined

When looking on leak per location we have the size but also want the number. (already have in backend in theory, just need to export into json profile).