This project is no longer maintained by Turbine Labs, which has shut down.
Rotor is a fast, lightweight bridge between your service discovery and Envoy’s configuration APIs. It groups your infrastructure into Envoy clusters and defines simple routes for each service. Instances are gathered directly from your service discovery registry, and clusters are created by grouping together instances under a common tag or label. This instance and cluster information is served via Envoy’s endpoint and cluster discovery services, CDS and EDS.
Rotor also sets up Envoy’s routing and listeners (RDS and LDS) to serve these clusters and send stats to common backends like Prometheus, statsd, dogstatsd, and Wavefront.
Rotor is a great starting point for quickly integrating Envoy with your existing environment. It provides a simple service mesh for many applications out of the box. If you have a more complex application, you can modify it to your needs or add an API key to unlock traffic management with Houston.
Rotor connects two types of information:
- Service Discovery information is collected from your existing registry, such as Kubernetes or Consul.
- Envoy Configuration is served over Envoy’s discovery services: EDS, CDS, RDS, and LDS.
- (optionally) Configuration via UI and API is provided by the Turbine Labs API, with an API key.
Without an API key (“standalone mode”), Rotor will serve Envoy configuration as follows:
- Endpoints (EDS) are mirrored from your service discovery.
- Clusters (CDS) are created by grouping endpoints based on labels on your
nodes/pods/hosts. The label format depends on the service discovery; typically
tbn_clusterortbn-cluster. - Routes (RDS) are created from your clusters. Each cluster is exposed via
a single domain with the same names as the cluster, and a single catch-all
route (
/). This is similar to Consul or Kubernetes DNS discovery for service registries. - Listeners (LDS) are statically configured. Rotor configures Envoy to listen on port 80 and sets up ALS to collect stats on the routes served via RDS.
- Access Logging (ALS) is configured, and Rotor can send request metrics to Prometheus, statsd, dogstatsd, and/or Wavefront.
For better control over the routes and listeners that Rotor sets up, there are two options:
- Define static listeners and routes in a config file via the
ROTOR_XDS_STATIC_RESOURCES_FILENAMEenvironment variable. Rotor will serve these "dynamic" routes over RDS and LDS. To change these routes, update the file and restart Rotor. More information and examples in this blog post. - Add an an API key for Houston. This allows Rotor to pull routes from Houston, provides full control over domains, routes, and cluster behavior via Houston's UI and API. For more information, see the section below on adding an API key.
For control over cluster features like circuit breaking, gRPC, and static
clusters, Rotor can also read either a set of static clusters or a cluster
template from the same ROTOR_XDS_STATIC_RESOURCES_FILENAME file. See an
example of setting up gRPC and static clusters on
the Turbine Labs blog.
The simplest way to start using Rotor is to use the docker image. You’ll need to configure it to point to your service discovery, then configure Envoy to read xDS configuration from Rotor. How you setup Envoy will depend on your environment, though you can see a simple example in the section on Envoy configuration.
Rotor supports the following service discovery integrations:
- Kubernetes
- Consul
- AWS/EC2
- AWS/ECS
- DC/OS
- (experimental) Envoy v1 CDS/SDS
- (experimental) Envoy v2 CDS/EDS
Additionally, Rotor can poll a file for service discovery information. This provides a lowest-common-denominator interface if you have a mechanism for service discovery that we don't yet support. We plan to add support for other common service discovery mechanisms in the future, and we'd love your help.
Note: you must label or tag the instances you want Rotor to collect! The name of this label is configurable, and the exact configuration option depends on your service discovery registry. To see the flags available for your SD, run:
docker run turbinelabs/rotor:0.19.0 rotor <platform> --helpwhere <platform> is one of: aws, ecs, consul, file, kubernetes, or marathon.
Kubernetes requires a number of RBAC objects to be created before running Rotor. The easiest way to create all these is via the YAML file in this repo:
kubectl create -f https://raw.githubusercontent.com/turbinelabs/rotor/master/examples/kubernetes/kubernetes-rotor.yamlRotor discovers clusters by looking for active pods in Kubernetes and grouping them based on their labels. You will have to add two pieces of information to each pod to have Rotor recognize it:
-
A
tbn_cluster: <name>label to name the service to which the Pod belongs. The label key can be customized. -
An exposed port named
httpor a customized name. A pod must have both thetbn_clusterlabel and a port namedhttpto be collected by Rotor.
Rotor will also collect all other labels on the Pod, which can be used for routing.
An example of a pod with labels correctly configured is included here. An example Envoy-simple yaml is also included.
Consul requires the datacenter and the host/port of your Consul server.
docker run -d \
-e "ROTOR_CMD=consul" \
-e "ROTOR_CONSUL_DC=<your datacenter>" \
-e "ROTOR_CONSUL_HOSTPORT=<consul ip address>:8500" \
turbinelabs/rotor:0.19.0To mark a Service for Rotor, add a tag called tbn-cluster. See
examples/consul for a working example.
Rotor can collect labels from the AWS API on EC2 instances.
docker run -d \
-e 'ROTOR_AWS_AWS_ACCESS_KEY_ID=<your aws access key>' \
-e 'ROTOR_AWS_AWS_REGION=<your aws region>' \
-e 'ROTOR_AWS_AWS_SECRET_ACCESS_KEY=<your secret access key>' \
-e 'ROTOR_AWS_VPC_ID=<your vpc id>' \
-e 'ROTOR_CMD=aws' \
-p 50000:50000 \
turbinelabs/rotor:0.19.0You need to tag instances with the service name and port it exposes
by adding a tag of the format tbn:cluster:<cluster-name>:<port-name>.
Instances that serve more than one port can be tagged multiple times.
For example, to expose two services from a single instance on ports
8080 and 8081, you can tag the instance by running:
aws ec2 create-tags \
--resources <your instance id> \
--tags \
Key=tbn:cluster:your-service-name:8080,Value= \
Key=tbn:cluster:your-other-service:8081,Value=ECS integration uses the AWS API, similar to EC2.
docker run -d \
-e 'ROTOR_AWS_AWS_ACCESS_KEY_ID=<your aws access key>' \
-e 'ROTOR_AWS_AWS_REGION=<your aws region>' \
-e 'ROTOR_AWS_AWS_SECRET_ACCESS_KEY=<your secret access key>' \
-e 'ROTOR_CMD=ecs' \
-p 50000:50000 \
turbinelabs/rotor:0.19.0You can run this inside or outside of ECS itself, as long as your Envoy instances have access to the container on port 50000.
ECS tags indicate the service name and exposed port, and they
are located with dockerLabels on the container definition:
{
"dockerLabels": {
"tbn-cluster": "your-service:8080"
}
}Rotor runs as an app inside DC/OS. Save this as rotor.json:
{
"id": "/tbn/rotor",
"cpus": 1,
"mem": 128,
"container": {
"type": "DOCKER",
"docker": {
"image": "turbinelabs/rotor:0.19.0",
"forcePullImage": true
}
},
"env": {
"ROTOR_CMD": "marathon",
"ROTOR_MARATHON_DCOS_ACS_TOKEN": "<your dc/os access token>",
"ROTOR_MARATHON_DCOS_URL": "<your dc/os admin URL>",
"ROTOR_MARATHON_DCOS_INSECURE": "<true if admin URL is not HTTPS>"
},
"healthChecks": []
}Deploy the app with:
dcos marathon app add rotor.jsonTo have rotor pick up services, add tbn_cluster labels to each container
definition with the service name.
Rotor can read from flat files that define clusters and instances. To
specify the format of the file, use the --format flag (or the
ROTOR_FILE_FORMAT environment variable). Possible values are "json"
and "yaml", and the default value is "json".
docker run -d \
-e 'ROTOR_CMD=file' \
-e 'ROTOR_FILE_FORMAT=yaml' \
-e 'ROTOR_FILE_FILENAME=/path/to/file/in/container' \
-p 50000:50000 \
turbinelabs/rotor:0.19.0The format defines clusters and the associated instances:
- cluster: example-cluster-1
instances:
- host: 127.0.0.1
port: 8080
- host: 127.0.0.1
port: 8081
- cluster: example-cluster-2
instances:
- host: 127.0.0.1
port: 8083Once Rotor is running, you can configure Envoy to receive EDS, CDS, RDS, and LDS configuration from it. You can put together a bootstrap config based on the Envoy docs, or you can use envoy-simple, a minimal Envoy container that can configured via environment variables.
docker run -d \
-e 'ENVOY_XDS_HOST=127.0.0.1' \
-e 'ENVOY_XDS_PORT=50000' \
-p 9999:9999 \
-p 80:80 \
turbinelabs/envoy-simple:0.19.0You may have to modify the host and port, depending on where you have Rotor deployed.
If you are not using envoy-simple, you will have to set
--service-cluster=default-cluster and --service-zone=default-zone flags on
your Envoy. With a Houston API key, Rotor is capable of serving many different
Envoy configurations, depending on which Envoy is asking. In standalone mode,
all Envoys are assumed to be part of the same Zone and Cluster, so you must make
sure these values are passed to Rotor. envoy-simple passes default-cluster
and default-zone by default. To serve multiple configs, either run multiple
Rotors, fork Rotor and add your own config, or see
Using with Houston.
You can verify that Rotor and Envoy are working correctly together by curling the admin interface to Envoy to see the routes that have been set up:
curl localhost:9999/config_dumpIf everything is working, you should see a JSON config object with routes for all your services.
Global flags for Rotor can be listed with
docker run turbinelabs/rotor:0.19.0 rotor --help. Global flags can be be
passed via upper-case, underscore-delimited environment variables prefixed
with ROTOR_, with all non-alpha characters converted to underscores. For
example, --some-flag becomes ROTOR_SOME_FLAG.
Per-platform flags can be listed with
docker run turbinelabs/rotor:0.19.0 rotor <platform> --help. Per-platform
flags can be similarly passed as environment variables, prefixed with
ROTOR_<PLATFORM>. For example --some-flag for the kubernetes platform
becomes ROTOR_KUBERNETES_SOME_FLAG.
Note Command-line flags take precedence over environment variables.
Rotor can be configured to periodically log a leaderboard of non-2xx requests
to stdout. This functionality is controlled by selecting the number of
responses to track (ROTOR_XDS_GRPC_LOG_TOP or --xds.grpc-log-top) and the aggregation period
(ROTOR_XDS_GRPC_LOG_TOP_INTERVAL or --xds.grpc-log-top-interval). These are global flags and, if being passed
on the command line, should come before platform configuration. As with any
flag they may also be specified via environment variable.
When viewing Rotor logs the request leaderboard is recorded in the following format:
[info] <timestamp> ALS: <number of requests>: <HTTP response code> <request path>There are a few ways to figure out what's going on with Rotor.
You can make Rotor's logging more verbose by adding ROTOR_CONSOLE_LEVEL=debug
to the environment, or by setting the --console.level flag if running the
binary by hand.
You can dump the full configuration that Rotor serves by running
rotor-test-client within the running Rotor docker container:
docker exec <container id> rotor-test-clientIf you've set ROTOR_XDS_DEFAULT_CLUSTER or ROTOR_XDS_DEFAULT_ZONE, you'll
need to correspondingly set them as arguments:
docker exec <container id> rotor-test-client --zone=<zone> --cluster=<cluster>If you're running the binaries by hand, and you've passed the --xds.addr flag
to rotor, you'll need to pass the same value in the --addr flag to
rotor-test-client.
For development, running tests, or custom integration, you may want to run Rotor locally.
- Go 1.10.3 or later (previous versions may work, but we don't build or test against them)
The rotor project depends on these packages:
The tests depend on our test package, and on gomock, and gomock-based Mocks of most interfaces are provided.
The Rotor plugins depend on many packages, none of which is exposed in the public interfaces. This should be considered an opaque implementation detail, see Vendoring for more discussion.
It should always be safe to use HEAD of all master branches of Turbine Labs open source projects together, or to vendor them with the same git tag.
go get -u github.com/turbinelabs/rotor/...
go install github.com/turbinelabs/rotor/...mkdir -p $GOPATH/src/turbinelabs
git clone https://github.com/turbinelabs/rotor.git > $GOPATH/src/turbinelabs/rotor
go test github.com/turbinelabs/rotor/...Please see Versioning of Turbine Labs Open Source Projects.
Patches accepted! In particular we'd love to support other mechanisms of service discovery. Please see Contributing to Turbine Labs Open Source Projects.
Rotor is also a part of a paid subscription to Houston. By adding an API key to Rotor, you unlock traffic management for your whole team, including:
- An easy-to-use UI for creating and modifying routes
- Full configuration of all of Envoy’s features: advanced load balancing, health checking, circuit breakers, and more
- Automatic collection of Envoy’s metrics for routes, clusters, and more, with easy integration into statsd, Prometheus, and other common dashboards
Specifically, instead of the static routes described in Features, an API key allows more flexible configuration of routes, domains, listeners, and clusters through Houston. You can also run multiple Rotor processes to bridge, e.g. EC2 and Kubernetes, allowing you configure routes that incrementally migrate traffic from one to the other. Houston also collects all the additional metadata on your instances, allowing you to route traffic based on custom tags and labels.
If you already have an API key, see the Turbine Labs docs for how to get started.
All Turbine Labs open-sourced projects are released with a Contributor Code of Conduct. By participating in our projects you agree to abide by its terms, which will be carefully enforced.
React
Typescript
Django
Laravel
Facebook
Microsoft
Google
Alibaba
D3
Tencent