Hello!

I'm using mr-redis today for the first time (on DC/OS) and this is a great project :) Is there any chance of publishing a 0.0.2 release on GitHub (and DC/OS package Universe ?) ?

The 0.0.1 in March 2016 was a long time ago and many new features have been contributed!

Gavin

hello

hello, I am trying mr-redis to use it in dcos, but when the node that deploys the framework is lost I lose the redis instances when it is deployed in another node.

Any ideas

Hi,

On DC/OS 1.8.1-open I tried to create

curl -X POST http://mrredis.mesos:5656/v1/CREATE/redistyk/256/1/1

I0825 21:25:27.388743  4763 exec.cpp:161] Version: 1.0.0
I0825 21:25:27.403304  4769 exec.cpp:236] Executor registered on agent f17f0016-1054-4e11-8541-6088f5fc435c-S20
I0825 21:25:27.406091  4769 docker.cpp:815] Running docker -H unix:///var/run/docker.sock run --cpu-shares 1024 --memory 134217728 -e MARATHON_APP_LABEL_DCOS_PACKAGE_SOURCE=https://universe.mesosphere.com/repo -e MARATHON_APP_VERSION=2016-08-25T17:24:57.779Z -e HOST=10.102.4.151 -e MARATHON_APP_RESOURCE_CPUS=1.0 -e MARATHON_APP_LABEL_DCOS_PACKAGE_REGISTRY_VERSION=2.0 -e CLUSTER_SIZE=1 -e MARATHON_APP_RESOURCE_GPUS=0 -e MARATHON_APP_LABEL_DCOS_PACKAGE_RELEASE=0 -e PORT_5454=28188 -e MARATHON_APP_DOCKER_IMAGE=paashuawei/mr-redis:latest -e MARATHON_APP_LABEL_DCOS_PACKAGE_NAME=mr-redis -e MARATHON_APP_LABEL_DCOS_PACKAGE_VERSION=0.0.1 -e PORT_5656=28189 -e MESOS_TASK_ID=mr-redis.d9c7ddb2-6ae8-11e6-9cf9-70b3d5800003 -e PORT=28188 -e MARATHON_APP_RESOURCE_MEM=128.0 -e PORTS=28188,28189,28190 -e PORT1=28189 -e MARATHON_APP_LABEL_DCOS_PACKAGE_IS_FRAMEWORK=true -e FRAMEWORK_NAME=mr-redis -e MARATHON_APP_RESOURCE_DISK=0.0 -e MARATHON_APP_LABELS=DCOS_PACKAGE_RELEASE DCOS_PACKAGE_SOURCE DCOS_PACKAGE_REGISTRY_VERSION DCOS_PACKAGE_FRAMEWORK_NAME DCOS_PACKAGE_VERSION DCOS_PACKAGE_NAME DCOS_PACKAGE_IS_FRAMEWORK -e MARATHON_APP_LABEL_DCOS_PACKAGE_FRAMEWORK_NAME=mr-redis -e MESOS_MASTER=zk://master.mesos:2181/mesos -e MARATHON_APP_ID=/mr-redis -e PORT0=28188 -e PORT_2379=28190 -e PORT2=28190 -e LIBPROCESS_IP=10.102.4.151 -e MESOS_SANDBOX=/mnt/mesos/sandbox -e MESOS_CONTAINER_NAME=mesos-f17f0016-1054-4e11-8541-6088f5fc435c-S20.ea40a023-1f13-4b1b-beeb-0fab5e23d623 -v /var/lib/mesos/slave/slaves/f17f0016-1054-4e11-8541-6088f5fc435c-S20/frameworks/e510b87e-cd91-4c50-82e1-c3dc19ad8a43-0001/executors/mr-redis.d9c7ddb2-6ae8-11e6-9cf9-70b3d5800003/runs/ea40a023-1f13-4b1b-beeb-0fab5e23d623:/mnt/mesos/sandbox --net host --name mesos-f17f0016-1054-4e11-8541-6088f5fc435c-S20.ea40a023-1f13-4b1b-beeb-0fab5e23d623 paashuawei/mr-redis:latest
2016/08/25 17:25:28 Configuration file is = {zk://master.mesos:2181/mesos /mrredis/bin/MrRedisExecutor /mrredis/bin/redis-server etcd http://mr-redis.marathon.mesos:2379 stderr mr-redis.marathon.mesos 5454 5656}
2016/08/25 17:25:28 *****************************************************************
2016/08/25 17:25:28 *********************Starting MrRedis-Scheduler******************
2016/08/25 17:25:28 *****************************************************************
2016/08/25 17:25:28 Starting the HTTP server at port 5656
2016/08/25 17:25:28 Hosting artifact '/mrredis/bin/MrRedisExecutor' at 'http://mr-redis.marathon.mesos:5454/MrRedisExecutor'
2016/08/25 17:25:28 Hosting artifact '/mrredis/bin/redis-server' at 'http://mr-redis.marathon.mesos:5454/redis-server'
2016/08/25 17:25:28 host_ip = 10.102.4.151 going to listen and serve
2016/08/25 17:25:28 Scheduler Maintainer is starting
2016/08/25 17:25:28 Not registered previously
2016/08/25 17:25:28 The Framework ID is &FrameworkID{Value:*,XXX_unrecognized:[],} and &FrameworkID{Value:*,XXX_unrecognized:[],}
2016/08/25 17:25:28 Connected to 10.102.4.11:2181
2016/08/25 17:25:28 Authenticated: id=240574411288608991, timeout=40000
2016/08/25 17:25:28 MrRedis Registered &FrameworkID{Value:*f17f0016-1054-4e11-8541-6088f5fc435c-0002,XXX_unrecognized:[],}
2016/08/30 15:04:34 slave lost: &SlaveID{Value:*f17f0016-1054-4e11-8541-6088f5fc435c-S14,XXX_unrecognized:[],}
2016/08/30 15:24:34 slave lost: &SlaveID{Value:*f17f0016-1054-4e11-8541-6088f5fc435c-S13,XXX_unrecognized:[],}
2016/08/31 15:39:35 slave lost: &SlaveID{Value:*f17f0016-1054-4e11-8541-6088f5fc435c-S21,XXX_unrecognized:[],}
2016/08/31 15:59:35 slave lost: &SlaveID{Value:*f17f0016-1054-4e11-8541-6088f5fc435c-S22,XXX_unrecognized:[],}
2016/09/01 12:10:59 slave lost: &SlaveID{Value:*f17f0016-1054-4e11-8541-6088f5fc435c-S24,XXX_unrecognized:[],}
2016/09/01 12:30:59 slave lost: &SlaveID{Value:*f17f0016-1054-4e11-8541-6088f5fc435c-S23,XXX_unrecognized:[],}
2016/09/01 13:33:52 slave lost: &SlaveID{Value:*f17f0016-1054-4e11-8541-6088f5fc435c-S26,XXX_unrecognized:[],}
2016/09/01 13:53:52 slave lost: &SlaveID{Value:*f17f0016-1054-4e11-8541-6088f5fc435c-S25,XXX_unrecognized:[],}
2016/09/07 10:38:03 slave lost: &SlaveID{Value:*f17f0016-1054-4e11-8541-6088f5fc435c-S28,XXX_unrecognized:[],}
2016/09/07 10:58:03 slave lost: &SlaveID{Value:*f17f0016-1054-4e11-8541-6088f5fc435c-S29,XXX_unrecognized:[],}
2016/09/08 10:55:56 Instance Name=redistyk, Capacity=256, masters=1, slaves=1
2016/09/08 10:55:56 Received offer {true 0xc8206ce900 1}
2016/09/08 10:55:56 Created 1 master offers for Instance redistyk
2016/09/08 10:55:56 Received Offer with CPU=0.7 MEM=10323 OfferID=f17f0016-1054-4e11-8541-6088f5fc435c-O13360822
2016/09/08 10:55:56 Launched 0 tasks from this offer
2016/09/08 10:55:56 Received Offer with CPU=0.6 MEM=8737 OfferID=f17f0016-1054-4e11-8541-6088f5fc435c-O13360823
2016/09/08 10:55:56 Launched 0 tasks from this offer
2016/09/08 10:55:56 MrRedis Receives offer
2016/09/08 10:55:57 Received Offer with CPU=0.2 MEM=6177 OfferID=f17f0016-1054-4e11-8541-6088f5fc435c-O13360832
2016/09/08 10:55:57 Launched 0 tasks from this offer
2016/09/08 10:55:57 Received Offer with CPU=0.2 MEM=6177 OfferID=f17f0016-1054-4e11-8541-6088f5fc435c-O13360833
2016/09/08 10:55:57 Launched 0 tasks from this offer
2016/09/08 10:55:57 MrRedis Receives offer
2016/09/08 10:55:58 Received Offer with CPU=0.7 MEM=10529 OfferID=f17f0016-1054-4e11-8541-6088f5fc435c-O13360835
2016/09/08 10:55:58 Launched 0 tasks from this offer
2016/09/08 10:55:58 MrRedis Receives offer
2016/09/08 10:55:59 Received Offer with CPU=1 MEM=10785 OfferID=f17f0016-1054-4e11-8541-6088f5fc435c-O13360847
2016/09/08 10:55:59 Launched 1 tasks from this offer
2016/09/08 10:55:59 Received Offer with CPU=0 MEM=6433 OfferID=f17f0016-1054-4e11-8541-6088f5fc435c-O13360848
2016/09/08 10:55:59 Launched 0 tasks from this offer
2016/09/08 10:55:59 MrRedis Receives offer
2016/09/08 10:56:01 MrRedis Task Update received
2016/09/08 10:56:01 Status={redistyk::cdce9c26-b830-4258-6c4d-89d1789ef761 TASK_RUNNING []}
2016/09/08 10:56:01 Received a Task update from the channel &{redistyk::cdce9c26-b830-4258-6c4d-89d1789ef761 TASK_RUNNING []}
2016/09/08 10:56:01 Error occured 100: Key not found (/MrRedis/Instances/redistyk/Procs/SlaveOf) [33]
2016/09/08 10:56:01 Task redistyk::cdce9c26-b830-4258-6c4d-89d1789ef761 is Running
2016/09/08 10:56:01 Received offer {false 0xc8206ce900 1}
2016/09/08 10:56:01 Created 1 slave offers for Instance redistyk
2016/09/08 10:56:02 Received Offer with CPU=0.2 MEM=6177 OfferID=f17f0016-1054-4e11-8541-6088f5fc435c-O13360863
2016/09/08 10:56:02 Launched 0 tasks from this offer
2016/09/08 10:56:02 Received Offer with CPU=0.6 MEM=8737 OfferID=f17f0016-1054-4e11-8541-6088f5fc435c-O13360864
2016/09/08 10:56:02 Launched 0 tasks from this offer
2016/09/08 10:56:02 MrRedis Receives offer

Hi,

After restarting the scheduler, the mr-redis UI dashboard redis instance list is empty, but the redis instance is still running in mesos slave, and they are also stored in zookeeper. Api v1/STATUS will return empty at this time, after trying to run v1/STATUS/<redis_name> which exist in zookeeper, v1/STATUS can return the value, this is very odd.

eg:

$curl 10.47.70.59:5656/v1/STATUS
[]
$ curl 10.47.70.59:5656/v1/STATUS/zousheng_redis_01
{"Name":"zousheng_redis_01","Type":"S","Status":"RUNNING","Capacity":100,"Master":{"IP":"10.25.234.193","Port":"6382","MemoryCapacity":100,"MemoryUsed":814000,"Uptime":18522,"ClientsConnected":1,"LastSyncedToMaster":0},"Slaves":null}

➜  sched git:(master) ✗ godep go build .
../common/store/etcd/db.go:9:2: cannot find package "github.com/coreos/etcd/Godeps/_workspace/src/golang.org/x/net/context" in any of:
    /usr/local/go/src/github.com/coreos/etcd/Godeps/_workspace/src/golang.org/x/net/context (from $GOROOT)
    /Users/pradeep/src/github.com/mesos/mr-redis/sched/Godeps/_workspace/src/github.com/coreos/etcd/Godeps/_workspace/src/golang.org/x/net/context (from $GOPATH)
    /Users/pradeep/src/github.com/coreos/etcd/Godeps/_workspace/src/golang.org/x/net/context
godep: go exit status 1

Looks like wrong import.

Currently, the actual redis-server is run through the following command:

R.Cmd = exec.Command("./redis-server", "--port", fmt.Sprintf("%d", R.Port))

As far as I'm concerned, this will launch redis with the default settings (except for the specified port of course). It seems that this leads to redis running in "protected mode" which only allows access from the loopback device, rendering all instances secluded from the outside world.

Am I missing an integral step? A redis.conf seems essential for me to properly configure the instances.

Hi,

/v1/status return null after restarting sched, and the fact is that all the redis instances are still running in mesos agents, to fix this, I have to manually run "curl ip:5656/v1/STATUS/sdb_test" to get redis "sdb_test" information, this call returns information, then "curl ip:5656/v1/STATUS" call returns sdb_test redis information, so there is something wrong in the /v1/STATUS call, it's not working as expected.

@Dhilip-Kumar-S; I just saw some code that u updated, and do not see consistency in terms of naming conventions.
Lets stick to common go conventions and some suggestions as below:

• No all caps naming except for constants
• Follow golangs standard in terms of naming structures and member
  if the name starts from caps, then it is public
• packages to be named with all small letters or as the first letter caps, if the name consists of 2 words
• names to be unambiguous and not too long at the same time
  for example I saw,you used "cli" for etcd client package, let it be "client" as cli certainly have a very common different meaning and infact we have another folder with same name.
  @Dhilip-Kumar-S @SrinivasChilveri
• any other suggestions can be added, and lets not have this list as complete now; I will try to update some readme to refer to as we all are just starting to code @this

We have built mr-redis from the latest master and are running it on DC/OS (using zookeeper rather than etcd).

The basics work ok but when the scheduler is restarted the existing redis instances shut down and don't come back.

If you call the /STATUS endpoint it says that the redis instances are up - but looking in mesos they're not running any more

Hi,

Can mr-redis create Mesos-DNS discovery records for the master and slave instances?

Thanks,
Luis

While creating redis instances add attributes to support VIP so that service discovery can be made easy. This is related to #33

I think comparison wise consul gets more point on etcd, for additional features like multiple data centre awareness and better service discovery mechanisms (among the comparison of zookeeper, etcd and consul). but what I found missing is the tree like construct for node values, and watches. Watches might be important for our implementation. Or may be I did not read about it completely.

We can discuss further and conclude.
As of now etcd looks ok for our usage; but if we need service discovery also in future consul might be the right choice.
The API for both are simple and similar to use in go.. :-)

HI @anvithks,

Thanks for the contribution. Could you write a README on how someone could download and setup the UI from scratch please?

Could you also add some screen shot to give a glimse of the UI's look and feel please?

When I try to run cmd$ mrr create help, It gives below error:
No command 'mrr' found, did you mean:
Command 'mrd' from package 'mtools' (main)
Command 'drr' from package 'plastimatch' (universe)
Command 'mrb' from package 'mrb' (universe)
Command 'mtr' from package 'mtr' (universe)
Command 'mtr' from package 'mtr-tiny' (main)
Command 'mr' from package 'myrepos' (universe)
mrr: command not found

OUTPUT:

curl -X "POST" mrredis.mesos:5656/v1/CREATE/redisCluster/100/1/2
2016/11/29 17:21:35 Instance Name=redisCluster, Capacity=100, masters=1, slaves=2 ConfigJson={1}
ZOO CREATE SECTION K=/MrRedis/Instances/redisCluster/
ZOO K=/MrRedis/Instances/redisCluster/Type V=MS
ZOO K=/MrRedis/Instances/redisCluster/Masters V=0
ZOO K=/MrRedis/Instances/redisCluster/Slaves V=0
ZOO K=/MrRedis/Instances/redisCluster/Capacity V=100
ZOO K=/MrRedis/Instances/redisCluster/ExpMasters V=1
ZOO K=/MrRedis/Instances/redisCluster/ExpSlaves V=2
ZOO K=/MrRedis/Instances/redisCluster/Status V=CREATING
ZOO K=/MrRedis/Instances/redisCluster/Mname V=
ZOO K=/MrRedis/Instances/redisCluster/DValue V=1
ZOO CREATE SECTION K=/MrRedis/Instances/redisCluster/Snames/
ZOO CREATE SECTION K=/MrRedis/Instances/redisCluster/Procs/
2016/11/29 17:21:35 [router.go:854] [D] | POST | /v1/CREATE/redisCluster/100/1/2 | 56.157575ms | match | /v1/CREATE/:INSTANCENAME/:CAPACITY/:MASTERS/:SLAVES |
2016/11/29 17:21:35 Received offer {true 0xc4200de360 1}
Request Accepted, Instance will be created.2016/11/29 17:21:35 Created 1 master offers for Instance redisCluster
root@mesos-master1:/usr/local/go/src/github.com/mesos/mr-redis/sched# 2016/11/29 17:21:35 Received Offer with CPU=7.5 MEM=13982 OfferID=215731b7-c3e2-4ad9-8333-c3c7231af861-O5849452
2016/11/29 17:21:35 Launched 1 tasks from this offer
2016/11/29 17:21:35 Received Offer with CPU=8 MEM=14014 OfferID=215731b7-c3e2-4ad9-8333-c3c7231af861-O5849453
2016/11/29 17:21:35 Launched 0 tasks from this offer
2016/11/29 17:21:35 MrRedis Receives offer
2016/11/29 17:21:35 MrRedis Task Update received
2016/11/29 17:21:35 Status={redisCluster::92b8edb7-96ee-4f1e-5f28-58a1a50b11a1 TASK_ERROR 215731b7-c3e2-4ad9-8333-c3c7231af861-S3 []}
2016/11/29 17:21:35 Received a Task update from the channel &{redisCluster::92b8edb7-96ee-4f1e-5f28-58a1a50b11a1 TASK_ERROR 215731b7-c3e2-4ad9-8333-c3c7231af861-S3 []}
2016/11/29 17:21:35 Invalid Key /MrRedis/Instances/redisCluster/Procs/92b8edb7-96ee-4f1e-5f28-58a1a50b11a1, Cannot load
2016/11/29 17:21:35 Invalid Key /MrRedis/Instances/redisCluster/Procs/, Cannot load
2016/11/29 17:21:35 Task redisCluster::92b8edb7-96ee-4f1e-5f28-58a1a50b11a1 is Error

curl hostname:port/v1/STATUS/redisCluster
{"Name":"redisCluster","Type":"MS","Status":"CREATING","Capacity":100,"Master":null,"Slaves":null}

Not able to debug whats wrong I'm doing.Can you please help?

It would be cool for the executors to use constraints. E.g.:

"rack_id", "CLUSTER", "rack-1"
"storage", "CLUSTER", "r6_ssd

this will avoid resource nervous between application and redis (IO/memory)

When I tried to create a single redis instance and didn't intput any field, just click "CREATE" button, it will create a redis with name as null, the request url is v1/CREATE/null/32/1/0, the instance name field actually is required, but it seems not working as expected. Maybe the front end and backend need to change for this null check?

Hi,

I am trying to manage redis in mesos standalone mode, after configure mr-redis, the redis docker instance is not accessable from external, as the port mappings is none, I have to enter the redis docker to get the data. Is there any config parameter to change this, like change the docker network mode from bridge to host?

Please give some adive. Thanks

see below outputs, the PORTS mapping is none:

ONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
3c5b1b79b8b7 redis:3.0 "docker-entrypoint.sh" 37 minutes ago Up 37 minutes

i run the sched process panic。

[root@open-falcon mr-redis]# ./sched
2017/01/10 21:18:56 [config.go:314] [W] open /tmp/mr-redis/conf/app.conf: no such file or directory
2017/01/10 21:18:56 Configuration file is = {127.0.0.1:5050 ./MrRedisExecutor ./redis-server zookeeper 127.0.0.1:2181 stderr 127.0.0.1 5454 5656}
2017/01/10 21:18:56 *****************************************************************
2017/01/10 21:18:56 ***Starting MrRedis-Scheduler
2017/01/10 21:18:56 *****************************************************************
2017/01/10 21:18:56 Starting the HTTP server at port 5656
2017/01/10 21:18:56 [asm_amd64.s:1721] [I] http server Running on :5656
2017/01/10 21:18:56 Hosting artifact './MrRedisExecutor' at 'http://127.0.0.1:5454/MrRedisExecutor'
2017/01/10 21:18:56 Hosting artifact './redis-server' at 'http://127.0.0.1:5454/redis-server'
panic: runtime error: invalid memory address or nil pointer dereference
[signal 0xb code=0x1 addr=0x40 pc=0x4b563c]

goroutine 8 [running]:
github.com/mesos/mr-redis/sched/mesoslib.GetFrameWorkID(0x0, 0x0, 0xc8200f5d50)
/home/ubuntu/gopkg/src/github.com/mesos/mr-redis/sched/mesoslib/mesoslib.go:141 +0x6c
github.com/mesos/mr-redis/sched/mesoslib.Run(0xc8200f5cc0, 0xe, 0xc8200f5d40, 0x9, 0xc8200f5d50, 0x4, 0xc82019c1e0, 0x11, 0xf24a10, 0xe, ...)
/home/ubuntu/gopkg/src/github.com/mesos/mr-redis/sched/mesoslib/mesoslib.go:170 +0x11d
created by main.main
/home/ubuntu/gopkg/src/github.com/mesos/mr-redis/sched/main.go:100 +0x920

goroutine 1 [chan receive]:
github.com/astaxie/beego.(*App).Run(0xc8200f4400)
/home/ubuntu/dev/sched/Godeps/_workspace/src/github.com/astaxie/beego/app.go:171 +0x33d
github.com/astaxie/beego.Run(0xc8200f3af8, 0x1, 0x1)
/home/ubuntu/dev/sched/Godeps/_workspace/src/github.com/astaxie/beego/beego.go:277 +0x17d
github.com/mesos/mr-redis/sched/httplib.Run(0xc8200f5d60, 0x4)
/home/ubuntu/gopkg/src/github.com/mesos/mr-redis/sched/httplib/httplib.go:240 +0x1d8
main.main()
/home/ubuntu/gopkg/src/github.com/mesos/mr-redis/sched/main.go:112 +0x98c

goroutine 17 [syscall, locked to thread]:
runtime.goexit()
/usr/local/go/src/runtime/asm_amd64.s:1721 +0x1

goroutine 6 [syscall]:
os/signal.loop()
/usr/local/go/src/os/signal/signal_unix.go:22 +0x18
created by os/signal.init.1
/usr/local/go/src/os/signal/signal_unix.go:28 +0x37

goroutine 7 [chan receive]:
github.com/golang/glog.(*loggingT).flushDaemon(0x13eaaa0)
/home/ubuntu/dev/sched/Godeps/_workspace/src/github.com/golang/glog/glog.go:882 +0x67
created by github.com/golang/glog.init.1
/home/ubuntu/dev/sched/Godeps/_workspace/src/github.com/golang/glog/glog.go:410 +0x297

goroutine 9 [runnable]:
github.com/mesos/mr-redis/sched/cmd.Creator()
/home/ubuntu/gopkg/src/github.com/mesos/mr-redis/sched/cmd/Creator.go:12
created by main.main
/home/ubuntu/gopkg/src/github.com/mesos/mr-redis/sched/main.go:103 +0x938

goroutine 10 [runnable]:
github.com/mesos/mr-redis/sched/cmd.Maintainer()
/home/ubuntu/gopkg/src/github.com/mesos/mr-redis/sched/cmd/Maintainer.go:12
created by main.main
/home/ubuntu/gopkg/src/github.com/mesos/mr-redis/sched/main.go:106 +0x950

goroutine 11 [runnable]:
github.com/mesos/mr-redis/sched/cmd.Destoryer(0xc8200f5cc0)
/home/ubuntu/gopkg/src/github.com/mesos/mr-redis/sched/cmd/Destroyer.go:15
created by main.main
/home/ubuntu/gopkg/src/github.com/mesos/mr-redis/sched/main.go:109 +0x968

goroutine 12 [IO wait]:
net.runtime_pollWait(0x7fcf4d2ca420, 0x72, 0xc82000a1c0)
/usr/local/go/src/runtime/netpoll.go:157 +0x60
net.(*pollDesc).Wait(0xc82020b020, 0x72, 0x0, 0x0)
/usr/local/go/src/net/fd_poll_runtime.go:73 +0x3a
net.(*pollDesc).WaitRead(0xc82020b020, 0x0, 0x0)
/usr/local/go/src/net/fd_poll_runtime.go:78 +0x36
net.(*netFD).accept(0xc82020afc0, 0x0, 0x7fcf4d2ca518, 0xc82020c100)
/usr/local/go/src/net/fd_unix.go:408 +0x27c
net.(*TCPListener).AcceptTCP(0xc820026208, 0xc820037d58, 0x0, 0x0)
/usr/local/go/src/net/tcpsock_posix.go:254 +0x4d
net/http.tcpKeepAliveListener.Accept(0xc820026208, 0x0, 0x0, 0x0, 0x0)
/usr/local/go/src/net/http/server.go:2135 +0x41
net/http.(*Server).Serve(0xc820018600, 0x7fcf4d2ca4e0, 0xc820026208, 0x0, 0x0)
/usr/local/go/src/net/http/server.go:1887 +0xb3
net/http.(*Server).ListenAndServe(0xc820018600, 0x0, 0x0)
/usr/local/go/src/net/http/server.go:1877 +0x136
github.com/astaxie/beego.(*App).Run.func4(0xc8200f4400, 0xc8201c1b80, 0xc82020af50)
/home/ubuntu/dev/sched/Godeps/_workspace/src/github.com/astaxie/beego/app.go:159 +0x487
created by github.com/astaxie/beego.(*App).Run
/home/ubuntu/dev/sched/Godeps/_workspace/src/github.com/astaxie/beego/app.go:166 +0x9a5

goroutine 14 [runnable]:
github.com/mesos/mr-redis/sched/mesoslib.prepareExecutorInfo.func1(0xc8200f5d40, 0x9, 0xc8200f5d50, 0x4)
/home/ubuntu/gopkg/src/github.com/mesos/mr-redis/sched/mesoslib/mesoslib.go:54
created by github.com/mesos/mr-redis/sched/mesoslib.prepareExecutorInfo
/home/ubuntu/gopkg/src/github.com/mesos/mr-redis/sched/mesoslib/mesoslib.go:66 +0x583
[root@open-falcon mr-redis]#

Instead of distributing the workloads and etcd manully when you start the scheduler make use of their docker images such that it can be downloaded using Unified Conainers feature in Apache Mesos.

When I try to build mrr using cmd$ godep go build -o mrr .
it produces below error:
ubuntu@ip-172-31-44-22:~/gopkg/src/github.com/mesos/mr-redis/cli$ godep go build -o mrr .

github.com/mesos/mr-redis/cli

./mrr.go:24: undefined: conf_file_path

godep: go exit status 2

at line number 24: f, err := os.Open(conf_file_path)
it is trying to open file <conf_file_path>, but <conf_file_path>is not defined anywhere in the code.

The Mrredis framework can add to the mesos,but can't create redis instance?
It is still creating.....

the logs:
2016/10/20 09:59:34 Delta of the previously registered framework is = 9m57.309924424s
2016/10/20 09:59:34 The Framework ID is &FrameworkID{Value:,XXX_unrecognized:[],} and &FrameworkID{Value:,XXX_unrecognized:[],}
2016/10/20 09:59:34 MrRedis Registered &FrameworkID{Value:*69065949-4bdb-45d9-973c-525c0d3bb54f-0008,XXX_unrecognized:[],}

the config file is:
{
"UserName": "ubuntu",
"FrameworkName": "MrRedis",
"Master": "20.26.25.148:5050",
"ExecutorPath": "../exec/MrRedisExecutor",
"RedisImage": "20.26.25.148:5000/redis:2.8",
"DBType": "etcd",
"DBEndPoint": "http://20.26.25.148:2379",
"LogFile": "stderr",
"ArtifactIP": "20.26.25.148",
"ArtifactPort": "5454",
"HTTPPort": "5656"
}

If you save a reds instance to zookeeper with slaves. Then restart the scheduler. The restarted scheduler doesn't know about the slaves (in status you see Slaves: null)

I'm wondering if proxy(multi master) is being developed.

thanks.

@SrinivasChilveri @irfanurrehman

We need to decide on which opensource Licence we should use for this project.

• Redis uses : Three clause BSD license
• Mesos uses : Apache License

should we stick to one of the above as this project is based out of both of them put together or is there a better Opensource Licence we should consider.

Please share your suggestions.

I0610 17:37:12.995946 1965056784 fetcher.cpp:424] Fetcher Info: {"cache_directory":"\/tmp\/mesos\/fetch\/slaves\/acd6d642-dc2c-49f7-becf-1aa2f3dd6904-S1\/root","items":[{"action":"BYPASS_CACHE","uri":{"executable":true,"extract":true,"value":"http:\/\/127.0.0.1:5454\/MrRedisExecutor"}},{"action":"BYPASS_CACHE","uri":{"executable":true,"extract":true,"value":"http:\/\/127.0.0.1:5454\/redis-server"}}],"sandbox_directory":"\/tmp\/mesos\/slaves\/acd6d642-dc2c-49f7-becf-1aa2f3dd6904-S1\/frameworks\/acd6d642-dc2c-49f7-becf-1aa2f3dd6904-0000\/executors\/default\/runs\/3d5e6bce-9de0-4513-a56a-3b65bb54df3c","user":"root"}
I0610 17:37:12.999377 1965056784 fetcher.cpp:379] Fetching URI 'http://127.0.0.1:5454/MrRedisExecutor'
I0610 17:37:12.999409 1965056784 fetcher.cpp:250] Fetching directly into the sandbox directory
I0610 17:37:12.999459 1965056784 fetcher.cpp:187] Fetching URI 'http://127.0.0.1:5454/MrRedisExecutor'
I0610 17:37:12.999474 1965056784 fetcher.cpp:134] Downloading resource from 'http://127.0.0.1:5454/MrRedisExecutor' to '/tmp/mesos/slaves/acd6d642-dc2c-49f7-becf-1aa2f3dd6904-S1/frameworks/acd6d642-dc2c-49f7-becf-1aa2f3dd6904-0000/executors/default/runs/3d5e6bce-9de0-4513-a56a-3b65bb54df3c/MrRedisExecutor'
I0610 17:37:13.167969 1965056784 fetcher.cpp:456] Fetched 'http://127.0.0.1:5454/MrRedisExecutor' to '/tmp/mesos/slaves/acd6d642-dc2c-49f7-becf-1aa2f3dd6904-S1/frameworks/acd6d642-dc2c-49f7-becf-1aa2f3dd6904-0000/executors/default/runs/3d5e6bce-9de0-4513-a56a-3b65bb54df3c/MrRedisExecutor'
I0610 17:37:13.168011 1965056784 fetcher.cpp:379] Fetching URI 'http://127.0.0.1:5454/redis-server'
I0610 17:37:13.168025 1965056784 fetcher.cpp:250] Fetching directly into the sandbox directory
I0610 17:37:13.168042 1965056784 fetcher.cpp:187] Fetching URI 'http://127.0.0.1:5454/redis-server'
I0610 17:37:13.168057 1965056784 fetcher.cpp:134] Downloading resource from 'http://127.0.0.1:5454/redis-server' to '/tmp/mesos/slaves/acd6d642-dc2c-49f7-becf-1aa2f3dd6904-S1/frameworks/acd6d642-dc2c-49f7-becf-1aa2f3dd6904-0000/executors/default/runs/3d5e6bce-9de0-4513-a56a-3b65bb54df3c/redis-server'
I0610 17:37:13.261274 1965056784 fetcher.cpp:456] Fetched 'http://127.0.0.1:5454/redis-server' to '/tmp/mesos/slaves/acd6d642-dc2c-49f7-becf-1aa2f3dd6904-S1/frameworks/acd6d642-dc2c-49f7-becf-1aa2f3dd6904-0000/executors/default/runs/3d5e6bce-9de0-4513-a56a-3b65bb54df3c/redis-server'
F0610 17:37:13.397745   14834 node.go:18] Couldn't determine hostname: exit status 1

Looks like the mesos-go in our godep is old which runs hostname -A command to figure out the hostname, It is fixed in the latest mesos-go which uses hostname -f

Hi,

II am trying to create single redis instance in mr-redis UI (all of mr-redis env is set up in centos 7.2 ), and it failed. I checked the executor and mesos master logs, the scheduler logs indicates the Executor lost on slave, I am a little confused about this, and the mesos-master log "uses less CPUs (None) than the minimum required (0.01)".

I have checked the environment, docker and go env, everyting works fine. Would you please give some advice? thanks

the master logs shows that:

W0809 15:42:37.689164 30062 validation.cpp:920] Executor 'default' for task 'tiantian_1234567890::3fd580ac-dedd-4cc0-4fa8-8b92d30d600d' uses less CPUs (None) than the minimum required (0.01). Please update your executor, as this will be mandatory in future releases.
W0809 15:42:37.689666 30062 validation.cpp:932] Executor 'default' for task 'tiantian_1234567890::3fd580ac-dedd-4cc0-4fa8-8b92d30d600d' uses less memory (None) than the minimum required (32MB). Please update your executor, as this will be mandatory in future releases.
I0809 15:42:37.689831 30062 master.cpp:8270] Adding task tiantian_1234567890::3fd580ac-dedd-4cc0-4fa8-8b92d30d600d with resources cpus():1; mem():100 on agent 0ba4db12-ca49-4e05-95d1-01aed982da90-S5,

and the scheduler log shows:

2017/08/09 16:17:26 MrRedis Task Update received
2017/08/09 16:17:26 Status={tiantian_1234567890::3fd580ac-dedd-4cc0-4fa8-8b92d30d600d TASK_ERROR 0ba4db12-ca49-4e05-95d1-01aed982da90-S5 []}
2017/08/09 16:17:26 Received a Task update from the channel &{tiantian_1234567890::3fd580ac-dedd-4cc0-4fa8-8b92d30d600d TASK_ERROR 0ba4db12-ca49-4e05-95d1-01aed982da90-S5 []}
2017/08/09 16:17:26 Invalid Key /MrRedis/Instances/tiantian_1234567890/Procs/3fd580ac-dedd-4cc0-4fa8-8b92d30d600d, Cannot load
2017/08/09 16:17:26 Invalid Key /MrRedis/Instances/tiantian_1234567890/Procs/, Cannot load
2017/08/09 16:17:26 Task tiantian_1234567890::3fd580ac-dedd-4cc0-4fa8-8b92d30d600d is Error
2017/08/09 16:17:26 executor "&ExecutorID{Value:*default,XXX_unrecognized:[],}" lost on slave "&SlaveID{Value:*0ba4db12-ca49-4e05-95d1-01aed982da90-S5,XXX_unrecognized:[],}" code 9

Currently Mr-redis starts an etcd instance internally when installed in DCOS. Since the scheduler runs on the host network and uses 2379/2380, it causes conflicts when you install the DCOS Etcd universe package. Since DCOS provides Zookeeper, doesn't it make sense to default to using that?

Hi,

I met an error when setting this up in my mac os, the redis instance status is hang at CREATING state, and the scheduler and mesos-slave gives error as below:

[zou[mac ~/go/src/github.com/mesos/mr-redis/cli]$ curl -X POST "http://mac:5656/v1/CREATE/test7/123/1/0"
Request Accepted, Instance will be created.

check status:
[zou[mac ~/go/src/github.com/mesos/mr-redis/cli]$ curl mac:5656/v1/STATUS/test7
{"Name":"test7","Type":"S","Status":"CREATING","Capacity":123,"Master":null,"Slaves":null}

"2017/05/22 16:25:33 Received a Task update from the channel &{test5::0daf90ff-22a2-4f41-66f0-9d55c6b62e3f TASK_FAILED 6b7a6c11-b4ed-4401-806d-02b7f8e2924c-S0 []}
2017/05/22 16:25:33 Invalid Key /MrRedis/Instances/test5/Procs/0daf90ff-22a2-4f41-66f0-9d55c6b62e3f, Cannot load
2017/05/22 16:25:33 Invalid Key /MrRedis/Instances/test5/Procs/, Cannot load
2017/05/22 16:25:33 Task test5::0daf90ff-22a2-4f41-66f0-9d55c6b62e3f is Failed"

mesos-slave error log:
E0522 16:25:32.910143 303992832 fetcher.cpp:558] Failed to run mesos-fetcher: Failed to fetch all URIs for container 'e6c52e85-d228-4840-b304-f1a3e2fa964c' with exit status: 256
E0522 16:25:32.910619 305602560 slave.cpp:4650] Container 'e6c52e85-d228-4840-b304-f1a3e2fa964c' for executor 'default' of framework 6b7a6c11-b4ed-4401-806d-02b7f8e2924c-0000 failed to start: Failed to fetch all URIs for container 'e6c52e85-d228-4840-b304-f1a3e2fa964c' with exit status: 256

Would you please give some advice? thanks

Some users don't currently use etcd and would prefer to stick with the battletested Zookeeper, which Mesos already relies upon. Please consider dropping etcd as a hard requirement so it is easier to setup and use Mr Redis.

Is there a way to specify multiple zookeeper addresses and ports?

I got a 3 master 6 slaves coreos DC-OS cluster set up in Amazon. I ran the mr-redis package in dc-os and the mr-redis application is running in marathon. Now from my local machine i want o configure my mrr utility using

mrr init http://<schedulerIP>:<schedulerPORT>

But my scheduler is running on a slave machine which does not have a public ip , also since it is running with NETWORK=HOST , it cannot be reached through the public slave IP. How can I reach the scheduler.

Hi,

When I issue the following command, I hoped that there would be a high change for the tasks to be spread across multiple Mesos agents:

curl -X POST http://mrredis.mesos:5656/v1/CREATE/redistyk/256/1/3

I was really surprised to see three out of the four total redistyk tasks running on the same agent (even with huge spare capacity across the cluster!)

Can I request the ability to specify the framework equivalent of "constraints": [["hostname", "UNIQUE"]] in a Marathon job?

After installing Mr-Redis and creating an instance (which I can connect to from my redis-cli), I attempt to go to the UI (http:///service/mrredis) and get a 500 error. I know that the proxy is working because I can reach /service/marathon and /service/elasticsearch just fine.

Any ideas why this wouldn't work?