Feature Request: tolerations and nodeAffinity about assemblyline HOT 12 CLOSED

From what I can tell, Field and Label selectors cannot be used to directly dictate scheduling behaviors such as tolerations or node affinity.

from assemblyline.

Looks like the configuration for the linux_node_selector should translate to a nodeAffinity for the service pods:
https://github.com/CybercentreCanada/assemblyline-core/blob/14bfcd276ac904882724c4eb9075fe0350cf9f3f/assemblyline_core/scaler/controllers/kubernetes_ctl.py#L140

So we would just need to add a configuration for tolerations.

from assemblyline.

You can test with this development release to make sure you can configure the system as necessary when we merge this into stable (along with the helm-chart changes).

from assemblyline.

Sorry, not seeing the tolerations or the nodeAffinity in the service pods.

scaler:
      linux_node_selector:
        label:
          - key: dedicated
            operator: In
            values:
              - assemblyline
      cluster_pod_list: true
      cpu_overallocation: 2
      service_defaults:
        backlog: 10
        min_instances: 0
        growth: 30
        shrink: 10
        environment:
          - name: "SERVICE_API_HOST"
            value: "http://service-server:5003"
        tolerations:
          - effect: NoSchedule
            key: dedicated
            operator: Equal
            value: assemblyline

from assemblyline.

Would the scaler.linux_node_selector configuration work in this case as you would just need to specify the selector (whether it be based on fields or labels) that correspond to the node?

https://cybercentrecanada.github.io/assemblyline4_docs/odm/models/config/#scaler

from assemblyline.

@cccs-rs I pulled the latest dev images and updated my helm charts, but wasn't able to get the nodeAffinity and tolerations on Services. They did show up on all the core services. Here are my respective value.yaml config.

# An affinity to be applied to all core (non-service) pods not provided by imported charts.
# https://kubernetes.io/docs/reference/kubernetes-api/workload-resources/pod-v1/#NodeAffinity
nodeAffinity:
  requiredDuringSchedulingIgnoredDuringExecution:
    nodeSelectorTerms:
    - matchExpressions:
      - key: dedicated
        operator: In
        values:
        - assemblyline

# The tolerations to be applied to all core (non-service) pods not provided by imported charts.
# https://kubernetes.io/docs/reference/kubernetes-api/workload-resources/pod-v1/#tolerations
tolerations:
  - effect: NoSchedule
    key: dedicated
    operator: Equal
    value: assemblyline
    
configuration:
  core:
    scaler:
      linux_node_selector:
        requiredDuringSchedulingIgnoredDuringExecution:
          nodeSelectorTerms:
          - matchExpressions:
            - key: dedicated
              operator: In
              values:
              - assemblyline

from assemblyline.

configuration:
  core:
    scaler:
      linux_node_selector:
        requiredDuringSchedulingIgnoredDuringExecution:
          nodeSelectorTerms:
          - matchExpressions:
            - key: dedicated
              operator: In
              values:
              - assemblyline

This would have to be updated to (based on docs):

configuration:
  core:
    scaler:
      linux_node_selector:
        label:
          - key: dedicated
            operator: In
            values: 
              - assemblyline

from assemblyline.

from assemblyline.

Hmm... did Scaler read in the new configurations? Can confirm by checking the config.yml that's mounted

or:
python -c "from assemblyline.common import forge; print(forge.get_config().core.scaler.linux_node_selector);"

from assemblyline.

assemblyline@scaler-665f754877-p2nlr:~$ python -c "from assemblyline.common import forge; print(forge.get_config().core.scaler.linux_node_selector);"
<Selector {"field": [], "label": [{"key": "dedicated", "operator": "In", "values": ["assemblyline"]}]}>

I tested by deleting the scaler pod so that it was recreated with the latest. Then I deleted a service pod and inspected the toleration values after it was recreated.

from assemblyline.

To set the tolerations for services, then you'll need to configure core.scaler.service_defaults.tolerations per https://github.com/CybercentreCanada/assemblyline-base/pull/1676/files.

The configuration mentioned earlier should only set the affinity in the podSpec.

from assemblyline.

This should be featured in the 4.5.0.28 release.

from assemblyline.