A kubectl-like helper for interacting with multiple-clusters concurrently.

• Overview
• Getting started
• Usage
  • Commands
  • Types
  • Multi-cluster usage
  • Flags
  • Logs
  • Config
  • Cron
  • Adhoc Jobs
• Setup and configuration
  • Labels
  • Hiding clusters
  • Default columns

Ctl is a tool that helps with using multiple Kubernetes clusters simultaneously.

You should have kubectl set up before using ctl. At the least, you should have a kubeconfig file. For wish, see setting up k8s envs.

You can compile the binary by running make. Move the binary created in bin to your /bin folder.

Example:

$ ctl get pods bye --k8s_env=dev
CONTEXT               NAMESPACE  NAME                  READY  STATUS     RESTARTS  AGE      K8S_ENV  AZ
app-05-dev.k8s.local  default    bye-1565193600-trc9d  0/1    Succeeded  0         2h17m4s  dev      us-west-1a
app-05-dev.k8s.local  default    bye-1565197200-hq2z6  0/1    Succeeded  0         1h17m5s  dev      us-west-1a
app-05-dev.k8s.local  default    bye-1565200800-zg44s  0/1    Succeeded  0         17m6s    dev      us-west-1a

The commands get, describe and log function and look similarly to that in kubectl. Get prints out a table of found resources. Describe prints out details about each found resource.

The main commands follow the format

$ ctl [command] [TYPE] [NAME] [flags]

The two main operations are currently get and describe. Get prints out a table with found resources and describe prints out details about each found resource.

Currently, ctl supports the following resource types (with shorthand in parentheses):

• pods (po)
• cronjobs
• jobs
• deployments (deploy)
• replicasets (rs)
• configmaps (cm)
• k8s_env

*k8s_env list all possible k8s_env for the clusters

Ctl operates across multiple cluster, but it may not be desired to work on all of them. For the advanced user, you may directly specify which cluster(s) to use via the context flag:

--context=app-05-dev.k8s.local,test-cluster.k8s.local

Howevever, for most users, we have added functionality to make it easier to specify and narrow down clusters.

When configured, new labels can be added to all clusters. For example, at Wish ctl now has three global flags: k8s_env, az and region. You may use any combination of these to narrow down which clusters to query on.

$ ctl get pods --k8s_env=dev

Additional columns are added to get output from labels. These columns are set by the config. See configuration for more details.

Most objects can be scoped to a namespace. By default, ctl will operate on the specified namespace in your ctl config for adhoc jobs but otherwise operates on all namespaces, but you may specify a single namespace with the -n, --namespace flag. For more expensive operations, specifying contexts and namespaces can improve the performance.

Objects can be filtered by labels. The cluster-level labels above are shorthands for these labels filters (--k8s_env=dev is equivalent to -l k8s_env=dev). Use flag -l, --label. You can set any number of labels together: -l a=b,b=c -l c!=d is valid.

There are three types of label filters:

Equal
a=b. The value of label a must equal b.

Not equal
a!=b. The value of label a cannot be b.

Set in
a in (b,c,d). The value of label a must be one of b, c or d.

For the get command, you can add columns in the table output that corresponds to the value of such label.

Logging syntax is quite different from that of other commands. As logging is only for pods, the syntax is just ctl logs [NAME].

Ctl by default logs from the first container of a pod instead of asking to specify a container. If you want logs from a container that is not the first, then you must use the -c, --container flag.

A big feature of ctl is the support for multiple logs at a time. As all of ctl uses regex search, if multiple pods are matched, then logs from all of them are printed.

Additionally, this feature supports the -f, --follow option which constantly streams logs.

The run command is a wrapper over kubectl apply. It requires cluster config to set up.

The syntax is ctl run APPNAME [flags]. The label flags can be used to narrow down which cluster to run on. If there are multiple clusters in which a run is specified on, ctl randomly picks one.

Ctl prints out the command before running.

Cluster level configs are cached. To update this cached data, run ctl config fetch. Sometimes you may need to delete the ctl config folder. This is normally at $XDG_CONFIG_DIR/ctl or ~/.config/ctl.

Most cronjob features can be accessed through the base ctl commands. The ctl cron command allows for diect manipulation of k8s cronjobs. You may see ctl help cron for more details.

There are 4 main commands associated with running adhoc jobs, ctl up, ctl down, ctl login, and ctl cp in/out.

ctl up APPNAME will check the ctl-config for the APPNAME and run kubectl apply to its associated manifest. The user can supply 4 flags to this command:

• --deadline=<your deadline>, will check the manifest file for the string "{ACTIVE_DEADLINE_SECONDS}" and replace it with what was set in the flag. This flag determines the activeDeadlineSeconds of the job. When the job life the exceeded the deadline set, the job along with its associated resource will be deleted. The current default is 4 hours.
• --cpu=<your cpu>, will check the manifest file for the string {CPU} and replace it with what was set in the flag. If no value is set, it will check ctl-config for a default value and if none is found, it will use 0.5 as a default.
• --memory=<your memory>, will check the manifest file for the string "{ACTIVE_DEADLINE_SECODS}" and replace it with what was set in the flag. If no value is set, it will check ctl-config for a default value and if none is found, it will use 128Mi as a default.
• --user=<your user>, will check the manifest file for the string {USER} and replace it with what was set in the flag. This flag is used for spawning the job and finding the ad hoc pods associated with your name. The default name used is the user's hostname.
• --image=<your image>, will check the manifest file for the container specified by --container and override this container's image with your image
• --container=<your container>, will check the manifest file for your container and update it's image with the image specified by --image

Note that the TTLAfterFinished feature must be enabled on your kubernetes cluster

ctl down APPNAME will simply look through all of the clusters and namespaces for the users' jobs created through ctl up and delete them.

• --user=<your user>, will check for jobs spawned by the user. The default name used is the user's hostname.

ctl login APPNAME will run a kubectl exec command on your following job. the command to be run will be defined in the ctl-config configmap and will use the pod associated with the job (if there is one). If there is no existing pod, ctl up APPNAME will be run before this command is executed on the newly created pod.

Also, when running login, it will give the name of the pod spawned by the job so the user can use ctl cp

• --user=<your user>, will check for jobs spawned by the user. The default name used is the user's hostname.
• --container=<your container>, will check the pod for the following container and run the command on that container. If no container is specified it will use the first one found if there are multiple containers.
• --python=<your python script>, will run your python script if specified or start a python shell on login

ctl cp in/out APPNAME SOURCE will copy files into the adhoc job pod for the given app name(ctl cp in) or out of the pod (ctl cp out) using kubectl cp

• --out=<your output destination>, the destination of the copied files. If no destination is set it will default to /tmp/ctl
• --container=<your container>, will check the pod for the following container and run the command on that container. If no container is specified it will use the first one found if there are multiple containers.

This section refers to the optional setup on the server-side of configuration ctl for all users. To use the optional features of ctl (such as ctl up, ctl down, and ctl login), a ConfigMap should be added to clusters.

This ConfigMap should be located in namespace ctl and have name ctl-config.

To use the ad hoc job feature above, the ctl-config must be configured in a certain way. This example will use jsonnet to make the config file. This jsonnet can generate json and yaml files. Anything with <text> wrapped you should change to your specific needs.

[
    {
        apiVersion: 'v1',
        kind: 'ConfigMap',
        metadata: {
            namespace: 'ctl',
            name: 'ctl-config',
        },
        data: {
            k8s_env: <YOUR ENVIRONMENT>,
            region: <YOUR REGION>,
            _hidden: <TRUE OR FALSE>,
            _run: std.toString({
                "<YOUR APPNAME>": {
                    resources: {
                        cpu: "<YOUR DEFAULT CPU>",
                        memory: "<YOUR DEFAULT MEMORY>"
                    },
                    active: <TRUE OR FALSE>,
                    login_command: "<YOUR LOGIN COMMAND HERE>",
                    manifest: std.manifestJson(
                        <YOUR JOB MANNIFEST HERE>

                        for example 

                        {
                            apiVersion: 'batch/v1',
                            kind: 'Job',
                            metadata: {
                            name: '<YOUR APPNAME>-{USER}',
                            namespace: 'ctl-oneoff',
                            },
                            spec: {
                                activeDeadlineSeconds: "{ACTIVE_DEADLINE_SECONDS}",
                                ttlSecondsAfterFinished: 0,
                                template: {
                                    metadata: {
                                        labels: {
                                            name: '<YOUR APPNAME>-{USER}',
                                            tier: 'merchant-oneoff-pod',
                                        },
                                        namespace: 'merchant-oneoff',
                                    },
                                    spec: {
                                        restartPolicy: Never,
                                        containers: [
                                            {
                                                name: '<YOUR APPNAME>-pod-{USER}',
                                                command: ['/bin/bash', '-c', '--'],
                                                args: ['while true; do sleep 30; done;'],
                                                image: perl,
                                                resources: {
                                                    requests: {
                                                        cpu: '{CPU}',
                                                        memory: '{MEMORY}',
                                                    },
                                                },
                                            }
                                        ],
                                    }
                                },
                            }
                        }
                    ),
                }
            })
        },
    }
]

All fields in the ConfigMap that are not prefixed with an underscore are set as cluster-level labels. These labels will be propagated down to the objects on each cluster. Setting these labels are useful for describing and filtering the clusters.

A cluster can be hidden from users by setting _hidden to be true.

You can set default label columns to be printed with get output via _default_columns. Set this to be a comma separated list in string format. E.g. "_default_columns":"col1,able".