odd cleanup behavior under valgrind about s2n-tls HOT 3 OPEN

My earlier understanding was correct. s2n_examples_test.c (and several others) have been leaking memory, but the Valgrind errors were being suppressed by an overly general valgrind suppression:

This is a wildcard suppression that suppresses all reachable leaks where the call stack starts with a function named main. Since we use ctest, this is effectively all possible memory leaks that we would have.

The reason that the errors in my PR weren't suppressed was because the call stack happened to be too long. We run valgrind with --num-callers=40 which "Specifies the maximum number of entries shown in stack traces that identify program locations".

This stack trace was then long enough that it didn't match the

...
fun:main

wildcard suppression, because fun:main had been dropped off of valgrind's tracking information. We can confirm this be looking at valgrind's auto-generated suppression for the error:

{
   <truncated by me>
   fun:s2n_test_example_negotiate
   fun:s2n_run_self_talk_test
}

We see that the suggested suppression starts with fun:s2n_run_self_talk_test rather than fun:main, which confirms that this was call-stack depth funkiness.

from s2n-tls.

On one hand, this has to be related to attribute/cleanup, because if I remove the DEFER_CLEANUP and just allow the process to exit normally there is no failure.

But on the other hand, this can't be related to attribute/cleanup because those aren't executed at exit : test case.

Perhaps there are some atexit shenanigans? I'll be spinning up an x86 instance and trying to reproduce the failures with the docker image.

from s2n-tls.

Some further investigation. I was able to locally reproduce using the CI docker image.

The other thing killing the process is actually valgrind. the exit code is simply the argument that we pass to valgrind: --error-exitcode=9. Valgrind kills the process because it detects a memory leak.

My current understanding is that my PR did not introduce a memory leak, rather it was some non functional change that made valgrind able to detect the memory leak. s2n_examples_test.c has always been "leaking" memory since DEFER_CLEANUP doesn't run when it's in the same scope as an exit function. But for an unknown reason Valgrind didn't care about that memory. This was confirmed by entirely removing the cleanup function from the server_process, and checking that Valgrind does not see a memory error.

Additionally, If the rest of the tests cases are removed then a memory leak is not reported in s2n_test_example_negoatiate. I tried adding some sleep statements in various places to see if this was related to timing, and that does not seem to be the case.

from s2n-tls.