allow to force Hindsight to shutdown when an essential sandbox dies about hindsight HOT 7 CLOSED

Something equivalent to Heka behavior would be great: http://hekad.readthedocs.org/en/latest/config/index.html?highlight=can_exit#configuring-restarting-behavior

from hindsight.

Hindsight is designed to always be running (eventually it will be daemonized). It should be thought of as a service waiting for a job to be submitted (that is why it can run without any plugins). A job (plugin) failure should never cause the service to shut-down since it may be running other peoples jobs in addition to yours. This type of functionality would not be part of the infrastructure itself but could be bolted on much like the pruning/file cleanup is (you could monitor for sandbox terminations, restart them as desired or shutdown the HS process).

from hindsight.

Ok, I see.
How would you technically monitor the sandbox terminations?

from hindsight.

Currently you could monitor the HS error log output (stderr). There are some open options/todo's around that i.e.,possibly generating Heka protobuf message and dumping them on the input queue, or creating a dedicated input to manage them, etc

You could also monitor the hindsight.tsv (stats) file but that is only updated once a minute so it would be a a little less timely.

from hindsight.

Once a minute is fine with me, but how do you know a sandbox has been terminated by monitoring hindsight.tsv? This information is not available in the tsv file, isn't it?

from hindsight.

The fact that it is not there is exactly how you would detect it.

from hindsight.

Ok, I see.
Thanks.

from hindsight.