Memory/leak issue about zetcd HOT 9 OPEN

Great, I'll put some time into that. Thanks so far

from zetcd.

Hmm, do you see the same behavior with v0.0.4?

ref. v0.0.4...v0.0.5

from zetcd.

from zetcd.

It would be best if you can provide reproducible steps. And also try to heap-profile zetcd.

from zetcd.

I'm new to go, could you provide some guidance on enabling heap-profiling?

from zetcd.

@matthewmrichter Please enable profile via zetcd --pprof-addr flag.

And do something like

go tool pprof -seconds=30 http://zetcd-endpoint/debug/pprof/heap

go tool pprof ~/go/src/github.com/coreos/etcd/bin/etcd ./pprof/pprof.localhost\:2379.alloc_objects.alloc_space.inuse_objects.inuse_space.001.pb.gz
go tool pprof -pdf ~/go/src/github.com/coreos/etcd/bin/etcd ./pprof/pprof.localhost\:2379.alloc_objects.alloc_space.inuse_objects.inuse_space.001.pb.gz > ~/a.pdf

Where you need to replace */bin/etcd binaries with the zetcd binary.

I would first try to reproduce without containerization.

from zetcd.

Ok, I think the main offender here may actually be Marathon(https://mesosphere.github.io/marathon/), not Mesos. The usage really shoots up when Marathon starts.

I converted zetcd to run as a service rather than containerized. I took a heap profile - this is shortly after startup. It already blasts up to 5 gig shortly after startup. I will keep an eye on htop for a little while to see if it begins to approach 7+ gig as well and provide another profile.

a.pdf

Steps to reproduce -

1. Build etcd cluster or endpoint (separate autoscaling group behind a load balancer)
2. On mesosmaster server, install zetcd. Configure to point to remote etcd cluster load balancer as etcd endpoint.
3. On mesosmaster server, install mesos, marathon. (marathon::version : '1.4.8-1.0.660.el7', mesos::version: '1.6.0-2.0.4')
4. Configure mesos and marathon to run with localhost:2181 (zetcd port) as zookeeper URL

from zetcd.

I gave it a while, and the process according to htop had gotten up to 6 gigs. Here's a second profile aftertaken at this point, looks mostly the same:

b.pdf

from zetcd.

Here's a qustion, based on the bottleneck being in that "ReadPacket" method..

Currently, I have etcd running on server A and marathon/zetcd on server B. Would it make more sense for zetcd and etcd to live on server A together rather than having zetcd reach out to etcd across the LAN?

from zetcd.