Reference:

https://kubernetes.io/docs/concepts/configuration/assign-pod-node/#node-affinity-beta-feature

We need that to complement taints/tolerations.

Example from our monitoring pods:

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  labels:
    app: prometheus
  name: prometheus
  namespace: prometheus
spec:
  replicas: 1
  selector:
    matchLabels:
      app: prometheus
  template:
    metadata:
      name: prometheus
      labels:
        app: prometheus
    spec:
      affinity:
        nodeAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            nodeSelectorTerms:
              - matchExpressions:
                - key: "monitoring-dedicated"
                  operator: In
                  values: ["true"]
      tolerations:
      - key: "dedicated"
        operator: "Equal"
        value: "monitoring"
        effect: "NoSchedule"
      containers:
      - name: prometheus
        image: quay.io/coreos/prometheus:v1.1.1
        ports:
        - name: web
          containerPort: 9090

This deployment will tolerate the taint dedicated:monitoring=NoSchedule while also only scheduling pods into nodes that has the selector monitoring-dedicates=true

Taints and tolerations alone can't guarantee in which node the pods will be scheduled.

maybe report it to redis or to a statsd gauge

The problem is at https://github.com/topfreegames/maestro/blob/master/plugins/grpc/forwarder.go#L124

The forwarder always tries to cast the received metadata fields to string without checking their type first

it would be nice to have a good visualization of everything that's happening

today I tried to update a scheduler configuration and got the following:

➜  fpsmultiplayer git:(maestro_master) ✗ maestro update docker/schedulerv2.yaml
Status: 500
Response: {"code":"MAE-003","description":"timeout during room deletion","error":"error when deleting old rooms. Maestro will scale up, if necessary, with previous room configuration.","success":false}

taking a look here:

➜  fpsmultiplayer git:(maestro_master) ✗ kprod get pods -n overkill-v2
NAME                   READY     STATUS    RESTARTS   AGE
overkill-v2-34a5d12d   1/1       Running   0          1m
overkill-v2-56f719c6   1/1       Running   0          1m
overkill-v2-77c7cd73   1/1       Running   0          1m
overkill-v2-9dd97f7b   1/1       Running   0          1m
overkill-v2-cd7ea664   1/1       Running   1          10h
overkill-v2-d6f6762b   1/1       Running   0          47m
overkill-v2-df7d6a5e   1/1       Running   0          10h
overkill-v2-e46e5780   1/1       Running   1          10h
overkill-v2-e72c7149   1/1       Running   0          5h
overkill-v2-ebcc99a9   1/1       Running   0          1m

it seems that maestro killed some pods but also left others running and didn't even tried to delete them.

we could use some form of self-healing here

Maestro is not behaving as expected when it chooses a hostPort that is not available in any of the kubernetes nodes, in this cases it should edit the pod to change the chosen hostPort or delete it and create a new one to trigger another random port get

Events:
  Type     Reason            Age                From               Message
  ----     ------            ----               ----               -------
  Warning  FailedScheduling  49s (x26 over 8m)  default-scheduler  0/10 nodes are available: 1 Insufficient cpu, 1 PodFitsHostPorts, 1 PodToleratesNodeTaints, 9 MatchNodeSelector.

we need a way to scale the gru's manually, it's useful and I don't see why not.

Summary

Hey everyone!
We're deprecating the current version of maestro (v9) in favor of a new major change (v10).
The v10 is a revamp of Maestro, having a totally different codebase and architecture. Although some endpoints have correlations, the structure is different and this new version is incompatible with the current one at almost all levels.

Related libs

• Maestro client is not affected by this release;
• Maestro cli is affected, having a new version too.

v10 adoption

After being released, the v10 will still miss some already planned features, like autoscaling. That said, depending on your need, it's better to wait for these features to come on new minor versions through Q2 and Q3, although we still recommend using v10 if it's possible.

v9 end of life

The v9 code will still be hosted in the "v9" branch in the repository.

We will keep the v9 support after the v10 is released (more details below). However, after that period, the version will no longer receive any updates.

Here is the complete timeline of the v9 next steps:

Issues and pull requests

All issues regarding v9 will need to be labeled. Then, a contributor member will review them to see if they will be accepted following the v9 status (Deprecated, Inactive, and Read-only).

It would be useful to cache room address and port so not access kubeAPI on ping/status/roomEvent

It would be better to start a goroutine for each scheduler on controller to parallelize deployment.

During scheduler update, pods must be rescheduled no new nodes (deleted and recreated) when affinity and toleration changes.

Just add new case on function MustUpdatePods in controller/controller.go

We need to replace them

Yes, we all know that a good demo would make our potential users learned what it is and how to use it. Just like agones project has created some demo games, e.g. paddle-soccer, and a series of blog posts, I wish our projects could also have sth like this!

Thanks.

ErrImagePull should stop right away and return error message, not wait for timeout.

On commit b26ed73, when updating scheduler with new occupiedTimeout, it is not updated on the watcher.

Maybe save it on Redis and always read it from there.

Maestro needs a force delete of schedulers

On models/pod.go, "hostNetwork" is set on podYaml variable in the wrong place. It must not be on containers array, but in the same level as containers key.

In the case of correcting it, it is important to test if it won't affect the communication with other services because it cannot access kubernetes services and internal IPs.

Maestro should have a cancel route to cancel update and create operations

maestro needs to be secure so that we can expose the api through the internet