Dynamic semaphore initialization from ticket quota about semian HOT 4 CLOSED

Had a chat with @sirupsen and I think that the approach we've settled on boils down to:

Have a new semaphore based on the maximum number of tickets that tracks the tickets per worker (needs to be unique per process or thread  -probably parent pid or pid_threadid).

The difference between this and the configured global maximum is the number of tickets currently available for that resource.

Based on the updates to this, we can dynamically update the number of available RPC tickets, based on some fraction. If the floor of this is different from the previous floor, do a thread-safe update of the ticket count. 

from semian.

@csfrancis @casperisfine any comments on this approach? Namely the newest comment from Dale

from semian.

@csfrancis asked me:

in k8s are sysv semaphores shared across the entire physical host? that seems strange

and this brings up a point of clarification: the reason why any of this is necessary is because we have to use hostIPC for logging. Because we are logging to SySV MQ, we are forced into using the host IPC namespace.

from semian.

K, @csfrancis had a short conversation and he's onboard with the solution of dynamically adjusting ticket counts. Some excellent point Scott made:

• We should call the configuration option quota, not tickets to not confuse the two. ArgumentError should be raised if both are set.
• When workers unregister themselves (they're killed or stop and SEM_UNDO does its thing), the worker count needs to be adjusted by something. We can cache the worker count in a semaphore in the semaphore set for the resource. On #acquire if it's different, we call update_ticket_count. It seems better for this reason to do it at #acquire time rather than #register time.
• A problem with the alternative approach of Semian["#{Process.ppid}_#{Thread.id}_#{resource_name} that Scott pointed out is that you'll basically have to GC it. By default, this limit is 32,000 on Linux. We have about a 100 in Shopify. If we run, say 10 pods per host, it'll take 32,000 / (100 * 10) = 32 deploys before we exhaust this space.

from semian.