Giter VIP home page Giter VIP logo

clearwaters-container's Introduction

clearwaters-container Container Management for the Clearwaters Cluster

A Brief Description of Docker and Current Goals Docker is a software technology providing operating-system-level virtualization also known as containers. A normal virtual machine exists as a separate system, a sandbox, in userspace that has its own operating system (libraries, file system, etc.). What differentiates Docker from other virtual machines is that the Docker software shares the majority of its operating system features with its host computer, except for its own file system. This allows for the sandbox to be both smaller in memory-size and faster as it can use the host’s operating system to do most of the work. For instance, if you “install” some software in a container that already exists on the host’s memory, then Docker will just link the file in the container to the one in the host, taking up none of the container’s memory and making “installation” speed instantaneous.

For SIGOps, through the magic of hardware, multiple containers have been linked together physically through a head node. The head node connects these containers to the internet and users and vice versa (all jobs will be done through this head node). Docker containers have been built with these hardware pieces but have not yet been linked. It would be equivalent to the concept of each being in their own separate universe and unable to communicate with anywhere else.

Difference between node and container: Each node is an independent computer (separate hardware and everything) and the container is as described above.

Current goal: Able to ping nodes together but cannot access the docker container to ping other containers. Ask Tanshiq what to do. Once we figure that out, start writing Python code that builds network, utilizing ping-ing principles (accessing IP address, etc).

clearwaters-container's People

Contributors

gauravdesale avatar tanishq-dubey avatar therishidesai avatar thomasdriscoll avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

clearwaters-container's Issues

List the docker containers and processes

Make sure the round robin algorithms labels each container formed with an id and name and adds to a log and also documents the size of f for simulation purposes before it actually goes on the cluster

Create daemon for worker nodes

A worker daemon will be created for monitoring the status of worker and executing tasks on the worker. The daemon should be able to complete the following tasks at a minimum.

  • Create a docker container
  • Shutdown a docker container
  • Report which containers are running

Ideally the worker daemon has a config file located somewhere in /etc/ and runs as a service, so that if a worker crashes, it can immediately be restarted.

Create Scheduling Algorithm

Before we can implement the scheduler in code, we should decide on how the jobs will be scheduled.

We have a few options, based on current schedulers, such as round robin, CFS, or even FIFO. However, since we will most likely have a job queue and also are running on multiple systems, we will need to ensure that all tasks are run fairly and also that workers are not overloaded with tasks while some do no work at all.

Simultaneously create and tear down containers on multiple clients.

What we want to do is create containers on all of our worker nodes with a single command from the head node.

The container image we are using is:
registry.gitlab.com/acm-uiuc/sigops/clearwaters-docker/ubuntu-mpich-arm64
Which can be grabbed using:
docker pull registry.gitlab.com/acm-uiuc/sigops/clearwaters-docker/ubuntu-mpich-arm64

I imagine a very simple workflow being the head-node sending a message to all client daemons to create the containers, and the client daemons then responding with the IDs of the created containers.

REST API

Making a REST endpoint to create/delete new docker containers. Similar to gpu-cluster-backend.

Create server manager for workers

As stated in #2 , there will exist worker daemons for managing each worker locally, however a head-node manager must exist as well. The head node manage will do the following tasks:

  • Keep a list of worker nodes
  • Send message to worker daemons for creating/destroying containers
  • Keep a list of container swarms

A communication protocol will need to be decided between the workers and the head-node .

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.