marathon about heapster HOT 22 CLOSED

Heapster currently supports kubernetes and fleet. Extending to add other types is pretty simple. We'd love to have PRs for marathon support and can help with any issues. We won't have time to get to it ourselves for a while though.

@samek would you like to give it a shot?

from heapster.

+1 to what @rjnagal mentioned. It should be straight forward to add a new 'nodes' source. We can help you out if necessary.

from heapster.

@rjnagal Well the way I look it it's either I write it from scratch or We add just the new source to this.

But we would need to discuss the approach part, get some guide lines ..

from heapster.

@samek I was advocating adding a new source. Since the setup uses cadvisor, the only part we need to implement is a watch or poll interface for getting list of nodes from marathon. I'd love to help with any question about approach or guidelines.

from heapster.

@samek: Were you able to make any progress?

from heapster.

What does heapster need? A list of node ips where containers are running?

from heapster.

Yes and no
We've change the source of cadvisor so that it sends enviroment vars to influxdb.
Env vars are what you need.

Sent from my iPhone

On 21 Mar 2015, at 19:44, Vish Kannan [email protected] wrote:

@samek: Were you able to make any progress?

—
Reply to this email directly or view it on GitHub.

from heapster.

Heapster will have to discover the nodes that are part of a mesos cluster.

On Mon, Apr 6, 2015 at 8:10 AM, samek [email protected] wrote:

Yes and no
We've change the source of cadvisor so that it sends enviroment vars to
influxdb.
Env vars are what you need.

Sent from my iPhone

On 21 Mar 2015, at 19:44, Vish Kannan [email protected] wrote:

@samek: Were you able to make any progress?

—
Reply to this email directly or view it on GitHub.

—
Reply to this email directly or view it on GitHub
#136 (comment)
.

from heapster.

ok, then it has to check http://mesosmaster:5050/master/state.json and there the slaves array. It contains the slave ips.

from heapster.

I will start with heapster tomorrow but from what i understand i have to run cadvisor on the nodes and heapster will connect to a list of nodes to gather the metrics?

from heapster.

That's correct @MikeMichel

from heapster.

Hi all,

I was looking to get heapster running on a mesos cluster. I tried to insert some mesos slave running a cadvisor into the external file, but I don't know how the file must look like. By looking into the code I tried following

{ "items": { "name": "mesos-04.development.vaph.be","ip": "193.53.25.110"}}

but that didn't make it.
I got the error :

failed to get information from source - failed to unmarshal contents of file /var/run/heapster/hosts. Error: json: cannot unmarshal string into Go value of type []api.ExternalNode

What should the format of the external file be ?

from heapster.

@exnerd: You are missing [] for items.
Here is an example:

{"items":[ {"name":"host1","ip":"1.2.3.4"},{"name":"host2","ip":"1.2.3.5"} ] }

Here is some test code that generates a list of external hosts on the fly.

Let me know if you have any issues. Ideally, having a native node discovery agent in heapster for mesos might be better for a production setup.
We are happy to provide any help necessary to get that in. You will have to implement a new NodesApi for mesos and examples are under sources/node/kube.go and source/node/coreos.go.

from heapster.

how do i use the external hostsfile? using --source=cadvisor:external add the cmd line is not working. it's not showing in the available option up when doing /usr/bin/heapster --help. tried the latest docker version also the canary one.

from heapster.

Here is what I tried.

$ cat /var/run/heapster/hosts 
{
"items": [ {"name":"host1","ip":"10.240.214.82"},{"name":"host2","ip":"10.240.163.208"} ]
}
$ ./heapster --source="cadvisor:external?cadvisorPort=4194"

Let me know if this works for you.

As I mentioned earlier, it might be much easier to add a node plugin that can programmatically list all mesos nodes.

from heapster.

no, maybe the docker version is outdated. i will try from source tomorrow.

from heapster.

Did anyone get anywhere with this?

from heapster.

i've been playing with this a bit.

I'm just starting to play with mesos and marathon. And I don't find a way to retrieve public and private ip's from mesos.

Mesos REST API returns this information about slaves

{"slaves":[{"active":true,"attributes":{},"hostname":"10.141.141.10","id":"20150913-141231-16842879-5050-1194-S0","pid":"slave(1)@127.0.1.1:5051","registered_time":1442153555.00333,"resources":{"cpus":2,"disk":34068,"mem":1000,"ports":"[31000-32000]"}}]}

Is not a lot, and is not enough to fill nodes.Info.
I've been checking the go library: https://godoc.org/github.com/mesos/mesos-go
But seems to be in really early stages, I don't find anything useful for what I want.

Finally, zookeeper should have this information... so maybe a better approach will be to make requests directly to zookeper. I don't know what is the info inside zk..

Any idea or directions in this regard?

from heapster.

@naxhh: If DNS were to be enabled, we can try to resolve the hostname to an IP. If the convention with mesos is to use the node IP as the hostname, then we can re-use the hostname as IP. But I am not sure if it will work everywhere.

+1 for integrating with zookeeper if that is the most convenient way to move forward.

from heapster.

I've checked the setup at work. And hostname is not always an ip. Resolving the hostname to ip's is an option. I just wanted to make it as simple as possible.

Checking information from Zk.

slave01:

20150902-134310-1560936451-5050-16793�����'"[email protected]:5050*hostname0120.23.0

slave02:

$20150902-134316-4127195142-5050-1887�����'"[email protected]:5050*hostname0220.23.0

This is the information from /mesos/info_0000000001, /mesos/info_0000000002

The information I've found there is pretty much the same I can found in the API (state.json)

Maybe doing it from ZK is more reliable than doing it from the API? In the API we could resolve hostnames or parse pid (pid: "slave(1)@10.0.0.1:5051)

From ZK, I should learn how unserialize that data and parse the master@ip:port thing....

From my POV, resolving hostname to IP's is the simplest way. I don't know if requiring DNS will be a big problem for some users.

Final question: nodes.Info has a few fields. What is really required from there? Public and private ip? or just private ip?

Edit:

By the way. I'm just trying to retrieve the Mesos slaves because someone suggested to do that. But I'm thinking that you could read the apps and retrieve hostnames and ports for cadvisor running apps. At the end that's what heapster needs, right?

v2/apps/cadvisor will have this kind of information:

"tasks": [
 {
   "id": "cadvisor.asdasd",
   "host": "slave1.mesos.com",
   "ports": [ 3001 ],
   "appId: "/cadvisor",
 },
 {
   "id": "cadvisor.asdasd2",
   "host": "slave2.mesos.com",
   "ports": [ 3001 ],
   "appId: "/cadvisor",
 },

]

And invoke this as:
./heapster --source="cadvisor:marathon?marathonUrl=mesos02:8080"

That is enough?

from heapster.

publicIP and internalIP can both be set to the same IP. They are
different in public clouds which have a dedicated internal network. You can
ignore the cpu and memory limits for now.

Is zookeeper used in most mesos clusters? If so, going with zookeeper makes
sense. If not, let's use the Mesos APIs since it should work on all mesos
deployments.

On Tue, Sep 15, 2015 at 6:39 AM, Ignacio Tolstoy [email protected]
wrote:

I've checked the setup at work. And hostname is not always an ip.
Resolving the hostname to ip's is an option. I just wanted to make it as
simple as possible.

Checking information from Zk.

slave01:

20150903-134310-1560936458-5050-16793�����'"[email protected]:5050*hostname0120.23.0

slave02:

$20150903-134316-4127195146-5050-1887�����'"[email protected]:5050*hostname0220.23.0

This is the information from /mesos/info_0000000001,
/mesos/info_0000000002 This id's are auto-incremented so if you delete 01
and add a new slave01 you'll have 02.

The information I've found there is pretty much the same I can found in
the API (state.json)

Maybe doing it from ZK is more reliable than doing it from the API? In the
API we could resolve hostnames or parse pid (pid: "slave(1)@10.0.0.1:5051)

From ZK, I should learn how unserialize that data and parse the master@ip
:port thing....

From my POV, resolving hostname to IP's is the simplest way. I don't know
if requiring DNS will be a big problem for some users.

Final question: nodes.Info has a few fields. What is really required from
there? Public and private ip? or just private ip?

—
Reply to this email directly or view it on GitHub
#136 (comment)
.

from heapster.

support for non-kubernetes deployments is currently suspended.

from heapster.