Giter VIP home page Giter VIP logo

microscaling's Introduction

Microscaling Engine

Our Microscaling Engine provides automation, resilience and efficiency for microservice architectures. You can use our Microscaling-in-a-Box site to experiment with microscaling. Or visit microscaling.com to find out more about our product and Microscaling Systems.

This project is no longer being developed. As an alternative we recommend you take a look at Keda.

Docker Image

Build

Build Status

Go 1.6 & 1.7

Microscaling Engine is under development, so we're not making any promises about forward compatibility, and we wouldn't advise running it on production machines yet. But if you're keen to get it into production we'd love to hear from you.

Schedulers

Microscaling Engine will integrate with all the popular container schedulers. Currently we support

  • Docker API
  • Marathon
  • Kubernetes

Support for more schedulers is coming soon. Let us know if there is a particular scheduler you wish us to support.

Metrics

Currently we support scaling a queue to maintain a target length. Support for more metrics is coming soon.

2 queue scaling algorithms are available.

  • SimpleQueue - scales containers up or down by one according to whether the queue is too long or too short.
  • Queue - uses control theory to prevent oscillation.

Queue Types

  • SQS - blog post with more details coming soon.
  • NSQ - see this blog post for more details.
  • Azure storage queues - this blog post describes using the Azure queue as the metric while running microscaled tasks on DC/OS.

Support for more message queues is coming soon. Let us know if there is a particular queue you wish us to integrate with.

Running

The easiest way to run Microscaling-in-a-box is to follow the instructions. The docker run command pulls the latest image of this code from Docker hub.

Running with label-based config

Get scaling parameters from your image metadata by configuring them with the following labels:

  • com.microscaling.is-scalable
  • com.microscaling.priority
  • com.microscaling.max-delta
  • com.microscaling.min-containers
  • com.microscaling.max-containers

Download the compose file and add the following environment variable to the environment settings for the microscaling image:

MSS_CONFIG=LABEL

Building from source

If you want to build and run your own version locally:

  • Clone this repo
  • Build your own version of the Docker image DOCKER_IMAGE=<your-image> make build
  • Specify -it <your-image> instead of -it microscaling/microscaling:latest on docker run so that it picks up your version of the image

Licensing

Microscaling Engine is licensed under the Apache License, Version 2.0. See LICENSE for the full license text.

Contributing

We'd love to get contributions from you! Please see CONTRIBUTING.md for more details.

Contact Us

We'd love to hear from you at [email protected] or on Twitter at @microscaling. And we welcome new issues or pull requests!

microscaling's People

Contributors

aecurrie avatar lizrice avatar ross-makisoft avatar rossf7 avatar russelltrow avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

microscaling's Issues

Docker swarm mode and local images

Hi,
I recently came into a problem using docker 1.12.1 services and swarm mode.
When I want to start a replicated service for a local built image, I get "image not found" errors inside docker logs on the swarm workers. I believe this happens due to the fact that the image was locally built only on master, thus the other workers can't find the local image in order to start the service.
Is there a solution to automatically build an image on all swarm workers ? I know docker compose can build an image and start a service only on the nodes the image was built on.
There is also the solution with image repository, but I want to avoid this, since I am making many image builds and it doesn't make sense to populate a docker repository with all kind of junk data.
Thanks,

Support for docker swarm as a scheduler

I was planning to use this with Swarm for a hackday project I was working on when I realized that it doesn't really support it yet. I ended up writing a quick implementation in python - but I'd rather use this instead because it supports multiple schedulers. My implementation uses swarm that runs haproxy as global front-end services. It uses telegraf to push haproxy connection stats to influxdb. My scaler code fetches these stats from influxdb and compares against service-declared total capacity to determine if the service needs to be scaled up or down. It uses service labels for determining which services need scaling and what's the declared service capaity.

Once swarm support is added, I could start adding haproxy/influxdb support to it.

Problems starting containers with Docker 1.12 RC2

There are problems starting containers with Docker version 1.12.0-rc2 used by the current Docker for Mac and Docker for Windows betas.

HostConfig has been removed from the Container Start API call. So we need to call Container Create with the HostConfig and then call Container Start. This error is also occurring upstream in the Golang Docker API client we're using.

microscaling_1  | ERRO 09:12:09.807:  Couldn't start container ID e1a7c15c2859 for task remainder: API error (400): {"message":"starting container with HostConfig was deprecated since v1.10 and removed in v1.12"}

Building source fails with ws errors in main.go

While following these build section instructions I get the below main.go errors. I don't intend on using the visualization portion of the queue demo so is there a way to build without the websocket feature? My environment won't have connectivity to that endpoint anyway.

# make build
# _/home/foo/github.com/microscaling/microscaling
./main.go:119: cannot use ws (type *"github.com/microscaling/microscaling/vendor/golang.org/x/net/websocket".Conn) as type *"golang.org/x/net/websocket".Conn in argument to getDemandEngine
./main.go:153: cannot use ws (type *"github.com/microscaling/microscaling/vendor/golang.org/x/net/websocket".Conn) as type *"golang.org/x/net/websocket".Conn in argument to getMonitors
./settings.go:174: cannot use ws (type *"golang.org/x/net/websocket".Conn) as type *"github.com/microscaling/microscaling/vendor/golang.org/x/net/websocket".Conn in argument to serverEngine.NewEngine
./settings.go:185: cannot use ws (type *"golang.org/x/net/websocket".Conn) as type *"github.com/microscaling/microscaling/vendor/golang.org/x/net/websocket".Conn in argument to monitor.NewServerMonitor
make: *** [microscaling] Error 2
#

Can we auto-scale our RESTful spring boot docker containers with this ?

Hi, with auto-scaling what I mean is, scale the containers alone and not provisioning of host VM infrastructure. Say, we run various different spring boot micro-services in the docker containers. Can this tool auto-scale and schedule of our docker containers based on the basis of CPU or API performance Latency ?

mesos could be controlled by singularity / aurora, not only marathon

mesos seems to provide load/pending offers metrics so it seems to be possible as well, plus for our case we need to elasticise only some specific racks in mesos

just targeting marathon seems to be a half-measure

yes, this is a FYI anyway, not that someone expects these to be implemented

Producer application never starts

Hey team,

Very new to DC/OS and ACS, and am wanting to see this demo; however the producer application never launches. This is in a newly created ACS (I have redeployed multiple times with the same issue). Stderr gives the error message below. Let me know what additional information is required to troubleshoot or where I can start?

I1205 08:59:20.094852 44087 exec.cpp:161] Version: 1.0.1
I1205 08:59:20.128491 44093 exec.cpp:236] Executor registered on agent 97657c69-5fa5-4466-9a27-a9e418003f78-S1
I1205 08:59:20.129650 44093 docker.cpp:815] Running docker -H unix:///var/run/docker.sock run --cpu-shares 51 --memory 104857600 -e AZURE_LOGGING_QUEUE_TYPE=AzureStorageQueue -e ANALYZER_KEEP_RUNNING=true -e MARATHON_APP_VERSION=2016-12-05T08:53:13.366Z -e HOST=10.32.0.4 -e MARATHON_APP_RESOURCE_CPUS=0.05 -e AZURE_STORAGE_QUEUE_NAME=microscaling-demo -e MARATHON_APP_RESOURCE_GPUS=0 -e MARATHON_APP_DOCKER_IMAGE=rgardler/acs-logging-test-simulate:latest -e PORT_10001=24755 -e MESOS_TASK_ID=producer.1bcd97b8-bac9-11e6-ba43-70b3d5800001 -e PORT=24755 -e MARATHON_APP_RESOURCE_MEM=100.0 -e SIMULATION_ACTIONS=0 -e SIMULATION_DELAY=1 -e PORTS=24755 -e AZURE_STORAGE_ACCOUNT_NAME=XXXX -e AZURE_STORAGE_ACCOUNT_KEY=XXXX -e MARATHON_APP_RESOURCE_DISK=0.0 -e MARATHON_APP_LABELS= -e SLACK_WEBHOOK=XXXX -e ANALYZER_SLEEP_TIME=0.1 -e MARATHON_APP_ID=/producer -e PORT0=24755 -e LIBPROCESS_IP=10.32.0.4 -e MESOS_SANDBOX=/mnt/mesos/sandbox -e MESOS_CONTAINER_NAME=mesos-97657c69-5fa5-4466-9a27-a9e418003f78-S1.477f2fff-2127-4ae1-bf83-5e60c5bd24dc -v /var/lib/mesos/slave/slaves/97657c69-5fa5-4466-9a27-a9e418003f78-S1/frameworks/97657c69-5fa5-4466-9a27-a9e418003f78-0000/executors/producer.1bcd97b8-bac9-11e6-ba43-70b3d5800001/runs/477f2fff-2127-4ae1-bf83-5e60c5bd24dc:/mnt/mesos/sandbox --net host --name mesos-97657c69-5fa5-4466-9a27-a9e418003f78-S1.477f2fff-2127-4ae1-bf83-5e60c5bd24dc rgardler/acs-logging-test-simulate:latest
WARNING: Your kernel does not support swap limit capabilities, memory limited without swap.
Traceback (most recent call last):
File "src/simulatedLogs.py", line 95, in
simulate()
File "src/simulatedLogs.py", line 40, in simulate
notify.info(msg)
File "/src/notify.py", line 19, in info
send(msg, config.SLACK_INFO_CHANNEL)
File "/src/notify.py", line 13, in send
"channel": "#" + channel,
TypeError: Can't convert 'NoneType' object to str implicitly

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "src/simulatedLogs.py", line 99, in
log.error("Unable to simulate logging", exc_info=True)
TypeError: error() got an unexpected keyword argument 'exc_info'

image download failure

Hi,

I am doing a microscailing test on DCOS and it seems like the container image on quay.io has a problem.

Pulling repository quay.io/rossf7/microscaling
docker: Error: Status 403 trying to pull repository rossf7/microscaling: "{"error": "Permission Denied"}".

Would you help to fix the issue?

Thanks,

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.