k8s queue customization through CRIC about harvester HOT 8 CLOSED

This is deployed and you can start customizing the queues, but please keep an eye in case there are any issues. You can also tell me to to do the first CRIC configurations.

The namespace is already taken from CE queue.

from harvester.

With regard to adding a k8s cluster to AGIS as a CE, it is useful to know this to set the flavour:

the flavour is editable field in the form, so it's possible to select any option from the drop-down menu (for the flavour field), press the backspace key and then enter new flavour value, in our case "K8S".

The clusters are also in GOCDB but I'm not sure if that is pertinent.

from harvester.

Hi Ryan,

I would like to extend the topic a little bit and offer more flexibility on customization. The things I believe you are currently interested in changing:

• turn off SCORE affinity: I would not turn it off, but it's your choice after all
• turn off SCORE-MCORE anti-affinity: your setup could indeed work better without it
• namespace
• enable memory limits: I recommend to be careful with this one on the big queue

So, my proposal is to use the "Associated Parameters" in CRIC (look at for example https://atlas-cric.cern.ch/atlas/pandaqueue/detail/MWT2/). I could define multiple fields there, for example:

• k8s.scheduler.score_affinity: bool
• k8s.scheduler.score_mcore_antiaffinity: bool
• k8s.namespace: string
• k8s.memory.limits: bool
  and others we come up in the future. The names are first suggestions, if you have better suggestions about any of those fields, it's OK.

CRIC checks that those fields are defined, i.e. it should not let you add an unexisting k8s.wrong field. CRIC also checks that the value you entered is e.g. a valid bool.

Then harvester would read those fields and act accordingly. If you really want to get the namespace from the CE, it's also possible, but no one else is using the CE and it offers less flexibility. I prefer to have everything together.

Please let me know if this works for you.

from harvester.

Hi Fernando,

Nice, the overall idea sounds good to me and should improve k8s operations. I am not familiar with the CRIC parameters or what naming scheme would be suitable, but in general it is preferable to have adjustable settings in CRIC that we can manage ourselves without having to make manual Harvester config changes or impose requests on you.

CRIC checks that those fields are defined

Would CRIC also list which fields are available and provide a brief description? If not probably we should make some simple documentation.

k8s.memory.limits implies the size of the RAM limit but since it is a bool, maybe "set_limit", "apply_limit" or "enforce_limit" would be better.

Having a CE defined for k8s is very useful when used in production for a T2 and not as a prototype (e.g. being able to declare downtimes).
And the k8s namespace seems like the most direct equivalent of a CE queue so that makes the most logical sense to me; the CRIC "CE" represents the k8s cluster.
The concept of a 'queue' (namespace) is also applicable for APEL publishing, e.g. the SubmitHost is kermes.cloud.computecanada.ca:6443/harvester , borrowing the same APEL string format that describes a CE and cluster and using it with the same logical intention.

Having the namespace as a parameter of the Panda queue seems less conceptually tidy to me, but that being said I don't know much about how queues and CEs are defined and associated with each other in CRIC, or what the "CEQueue" objects are for, e.g. https://atlas-cric.cern.ch/core/cequeue/detail/403/

So I don't know; it is not really clear to me either way, but we should do whatever is reasonably pragmatic but hopefully also conform somewhat to how CRIC is used for other systems, to hopefully avoid idiosyncrasies for k8s.

Thanks!

from harvester.

Would CRIC also list which fields are available and provide a brief description? If not probably we should make some simple documentation.

The admin that creates the parameter (in this case me) needs to provide type, description, allowed values.

You can then see the available parameters on https://atlas-cric.cern.ch/ > ATLAS (in top menu) > PanDA Queue parameters. I'm showing you with the k8s filter.

I interpret you are OK with the proposal, but prefer the namespace in the CE. Let me start implementing these things and I'll see how difficult the CE-namespace part is.

from harvester.

I actually implemented the changes today, pending to be tested. If you don't like any parameter name, please let me know ASAP.

Regarding the namespace, the CE queue will have preference over the associated parameter. You currently have it set to 'atlas' and it should be 'harvester' I believe. Can you change it on your 2 queues, please?

Unless I get blocked from testing or something goes wrong, I'll deploy the changes on Monday. Then you can start using the associated parameters to configure your queues.

from harvester.

prefer the namespace in the CE

It's not clear enough to me to say anything firm either way so I'll leave it up to you, whatever you think is appropriate.

Okay I'll ask Asoka to update CRIC.

from harvester.

#108

from harvester.