Comments (3)
Actually Mesos supports specific nodes by add resources "ports:[xxxx-xxxx]". For history reason, we need some jobs to run only one or several slaves, we don't want to add more Chronos Master and Mesos Master. Chronos seems has no way to set up cpu/mem/disk/ports resource for one job. So I find a way to fix a code to implement this quickly for presentation.
For example:
Step1:
Slave1 or other slaves work for jobs of Hadnoop:
./mesos-slave.sh --master=zk://172.16.10.100:2181/mesos --resources=mem:2048;cpus:2;ports:[1000-2000] --ip=172.16.10.1xx
Slave2 or others for some jobs like Shark:
./mesos-slave.sh --master=zk://172.16.10.100:2181/mesos --resources=mem:2048;cpus:2;ports:[2000-3000] --ip=172.16.10.2xx
Step2, fix the code as below (please notice that I'm new for scala and the code is only for test):
(/chronos/src/main/scala/com/airbnb/scheduler/mesos/MesosJobFramework.scala)
def buildTask(taskId: String, job: BaseJob, offer: Offer) : (Boolean, TaskInfo.Builder, Offer) = {
...
case Value.Type.SCALAR =>
......
case Value.Type.RANGES =>
if (job.command.indexOf("ports:[") > -1) {
var taskRange = job.command.substring(job.command.indexOf("ports:[") + 7, job.command.length() - 2)
var resourceRangeBegin = x.getRanges.getRange(0).getBegin
var resourceRangeEnd = x.getRanges.getRange(0).getEnd
var resourcePorts = resourceRangeBegin + "-" + resourceRangeEnd
if (resourcePorts == taskRange) {
sufficient("ports") = true;
log.warning("=====>Matched : %s".format(taskRange + ":" + resourcePorts))
} else {
log.warning("=====>Not matched : %s".format(taskRange + ":" + resourcePorts))
}
}
case _ =>
log.warning("=====>: Ignoring offered resource: %s".format(x.getType.toString))
})
if (job.command.indexOf("ports:[") > -1) {
(sufficient("cpus") && sufficient("mem") && sufficient("disk") && sufficient("ports"), taskInfoTemplate, offer)
} else {
(sufficient("cpus") && sufficient("mem") && sufficient("disk"), taskInfoTemplate, offer)
}
Step3, Add a new job in WebUI with command: echo 'Test ports' && echo 'ports:[1000-2000]'. MesosJobFramework will get the ports' range from command to compare with slaves' resources one by one, if matched, then use the slave to run the task. For time limited, I still have no time to fix the WEBUI of Chronos to add "cpu/mem/disk/ports" fields for one job. I think Chronos project team should consider this requirement.
The ports resource seems free to add values based on the format, As my test for only one slave, I directly set the IP inside: ports:[1721610101-1721610101], that's all I learn these days, hope this can help you.
from chronos.
mesos slave has "--attributes" potion to attach some tags to a slave, and this will send to on-top shceduler, like chronos. So the scheduler can filter slave with it's attribute.
However, chronos doesn't support this feature yet, we have plan to implement this feature and like to see accepted by upstream.
To @exnerd , it's generally not a good idea to have your specific job depends on the slaves which in a computing pool, it's better to integrate with Docker container to do that, build your job into a docker image and free to go.
from chronos.
this is kind of a duplicate of #256
from chronos.
Related Issues (20)
- Job arguments get appended on each execution
- Spark jobs priority with Chronos
- What version of Chronos should I use? HOT 5
- Index chronos_job_name_idx_1 is a duplicate of existing index chronos_job_name_idx
- How to use the `taskInfoData` job field HOT 4
- zk auth / framework registration race condition
- Chronos 2.5.1 fails to sustain registered state, flaps (ubuntu 14.04.05 LTS)
- Chronos not respecting role parameter HOT 1
- proxy to leader didn't work with https
- Avoid dumping environment variables in JobNotificationObserver
- Chronos doesn't support custom "ssl ciphers" and "tls version"
- Chronos schedule from Tuesday to Saturday
- how to define a volume as readonly when define a docker job?
- Mesos Chronos: Not able to add response headers
- KeeperErrorCode = ConnectionLoss for
- Chronos Intermittent Issue: Jobs get stuck HOT 9
- Tasks are not started on Windows agent
- Docker Job undocumented field
- Mesosphere Maven repository seems to no longer exist
- Filtering-hacking-sex-in the rubika"*a$10$lM.**QYhhhNamAVuOIEk4s6g.**OAl1cU1****Jqs54uO
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from chronos.