add auto-shrink auto-grow feature about circus HOT 9 CLOSED

Maybe we could imagine a scale policy / show

policiy could be a list of conditionns:

"grow": { 
   "cpu": {"if": "> XX" # % of use, "while": "XX" # seconds}, 
   "ram": { "if": "> XX" # bytes } 
 }

And the same for shrink.

Then we could have a parameter named scale which could take the maximum number of workers you want to scale. This value and eventually the policy could be set dynamically too:

 circusctl scale <showname> 5
 circusctl set_policy <showname> grow.cpu.if "> 60%"

something like it. What do you think?

from circus.

so, are all these options at the Show class level ?

if so can you describe the new constructor interface.

For the configuration part I think we want two things:

• being able to use the Show class programmatically - so the trainer needs to pass the options
• define new options in the ini file -- so I think those are flat options

last, we need to define how this is going to work. I suspect the Show class needs a heartbeat thread that collects the cpu value of each Fly to decide whether to add or remove one.

from circus.

The constructor would have nothing special, this is just the show with
more options :

class Show(object):

    def __init__(self, name, cmd, num_flies=5, warmup_delay=1.,
                 working_dir=os.getcwd(), shell=False, uid=None,
                 gid=None, send_hup=False, scale=None, policy={}):
        ...

Configuration in an ini file (for those who are using it) :

[policy:showname]
cpu.if = ..
cpu.while = ...

I agree we need a way to check each flies info and eventually kill or
spawn them until the scale limit. I think it need to be done its own
thread or process whatever to make the work in parallel. We could either
launching a thread or if we want to be fancy a gevent process.

The trainer is passing to this circus director the ipc endpoint to
communicate with the controller. Each scale order (ttin, ttou) are sent
to the controller. First time the controller is launched it get the
shows config, if a show is scalable (scale is not None) then it get
flies info using the controller. then X mn like the trainer it will get
flies info & . On reload the show configuration is reloaded.

We also need to make sure that this director is alive which could be
handled by the trainer. In this case we need to know if the
thread/gevent process is still alive and not to busy.

If we are OK for this scenario I can provide an implementation soon.

from circus.

what's a director ? the thread that gets the process stats ? other than that the plan looks good to me

from circus.

yes I called it the director. Sounded good at this point...

from circus.

I wonder if we should not create the tests environment before adding new features. Things are getting hairy :)

from circus.

I the priority should be the security, then right we should add new features :)

from circus.

This should be done as a plugin.

from circus.

we don't seem to see the need for a generic auto-shrink/grow plugin at this point

from circus.