tcmalloc always dead-lock under IA64 linux. about gperftools HOT 4 CLOSED

sorry, my report have some mistake.

backtrace not use libunwind, use libgcc_s.so.1
(see glibc-2.3.6/sysdeps/ia64/backtrace.c)

and, my suggestion is bad.
because 
o backtrace function use dlopen().
o dlopen() use malloc()

therefore, backtrace() function call from any invoke catastrophe.

example, below stack trace

backtrace()
 +- dlopen()
    +- tcmalloc#malloc()
       +- tcmalloc#GetStackTrace()
          + backtrace() 

Oops, recursive backtrace call.
and, dead-lock.

Yes, glibc's backtrace is not usable on 64-bit systems, for exactly the reason 
you
mention: it might call malloc.  There's the libunwind library that can be used 
instead.

I've just pushed out perftools 0.90, which has much improved 64-bit support.  
The
INSTALL file has instructions and caveats for how to use libunwind for 86-bit
systems.  In my testing on 64-bit machines, it is possible to get things working
without deadlock!

We have been using tcmalloc for a while now. Recently we faced a deadlock 
problem
similar to the one described here.

In short - the problem seems to be inter-library calls between glibc's unwind
routines, and tcmalloc.

After some analysis, we found that in our version of the deadlock, the root 
cause was
that tcmalloc calls GetBackTrace() under a lock - the pageheap lock.

T1: One thread, via dlopen(), is holding some lock within libdl [most probably 
one
within dl_iterate_phdr()] and has called malloc(), which required taking the 
pageheap
lock (because of GrowHeap()) - it is waiting on pageheap lock, held by the 
thread T2
below.

T2: Independently, another thread called malloc(), and maybe due to a sampled
allocation, grabbed the pageheap lock, and then called GetStackTrace() - which 
called
into libdl, and waiting on a lock held by the previous thread T1.

So a workaround for this deadlock could be achieved if the GetStackTrace() call 
can
be done outside the lock. In general, it would be nice to ensure that all
inter-library calls are made without any locks held.

In regular operation (meaning NO profiling turned on), tcmalloc calls 
GetStackTrae()
on two occasions - for sampled allocations and when the heap grows. And the 
reason
for these being called under the pageheap lock is to protect the global lists
"sampled_objects" and "growth_stacks" - am I correct?

We are still using an old version of tcmalloc (v0.7). So I looked up the latest
version v0.96. There, the GetStackTrace() for DoSampledAllocation() has been 
moved
OUT of the lock.

But the call to RecordGrowth() [which in turn calls GetStackTrace()] is still 
under
that lock.

My question is, is there any reason - other than protecting the global lists
mentioned above - why we must have this behavior? Would there be any impact if 
we try
to "fix" this by moving RecrodGrowth() outside the lock? This might need some 
change
in the signatures of the functions - like we could add an output parameter 
"ask" to 
TCMalloc_PageHeap::New(Length n), which would be non-zero IFF GrowHeap() was 
called,
and we actually record the growth doing something like this - (an example) -

void TCMalloc_Central_FreeList::Populate() {

  /* snip */

  Length ask = 0;
  Span* span;
  {
    SpinLockHolder h(&pageheap_lock);
    span = pageheap->New(npages, &ask); <--- GrowHeap does NOT call RecordGrowth()
    if (span) pageheap->RegisterSizeClass(span, size_class_);
  }
  if(ask) {
      RecordGrowth(ask << 12);
  }

  /* snip */
}

void RecordGrowth(size_t bytes)
{
      // no lock held here
      StackTrace* t = stacktrace_allocator.New();
      t->depth = GetStackTrace(t->stack, kMaxStackDepth-1, 3);
      t->size = growth;

      // Take the again (or use some CAS loop)
      SpinLockHolder h(&pageheap_lock);
      t->stack[kMaxStackDepth-1] = reinterpret_cast<void*>(growth_stacks);
      growth_stacks = t; 
}

My questions are -
(1) This is hackish, but will it break anything?
(2) Would the perftool developers consider making this change - i.e. all
inter-library calls, in particular GetStackTrace(), would be done with no locks 
held?

} So a workaround for this deadlock could be achieved if the
} GetStackTrace() call can be done outside the lock. In general, it
} would be nice to ensure that all inter-library calls are made
} without any locks held.

We've looked into this, and it's not really enough to fix things.
Suppose dlopen() calls malloc() and that call ends up causing
heap-growth.  So inside tcmalloc we release all locks and then call
GetStackTrace() which calls back into dlopen().  I don't think there
is anything in dlopen()'s spec that says that it reentrant in this way
(e.g., consider any dlopen data structure invariants that may be
temporarily broken at this point, or some dlopen locks that may be
held right now).

tcmalloc works under the assumption that we can have a GetStackTrace()
that does not need to use malloc.  This means your T2 thread:

} T2: Independently, another thread called malloc(), and maybe due to
} a sampled allocation, grabbed the pageheap lock, and then called
} GetStackTrace() - which called into libdl, and waiting on a lock
} held by the previous thread T1.

should not be calling into libdl if libdl might call malloc (which it
sounds like it might).

The best solution to this is to use libunwind (see the INSTALL file),
which is written to implement GetStackTrace without needing to use
malloc.  I don't know if it has IA64 support yet though; you'd need to
check their website.  The next-best solution is to implement a
perftools-specific ia64 GetStackTrace function, just like we have now
for i386 and x86_64.  If you happen to go that route, let us know!
We'd be glad to apply a patch.

Another possibility -- not as good but will get you working again without 
deadlock --
is to just turn off stack-tracing in tcmalloc entirely.  This means you can't 
do heap
profiling anymore, but you'll still get the advantages of the tcmalloc memory
allocation routines.  To do this, build with -DNO_TCMALLOC_SAMPLES.  Something 
like
this should work (note: untested):
   ./configure CPPFLAGS=-DNO_TCMALLOC_SAMPLES

craig