Giter VIP home page Giter VIP logo

supervisord-slow-reload-fix's Introduction

Hacking supervisor slow reload/restart

Supervisord slow reload/restart demo.

TLDR

Due to supervisor signalling behavior separate programms get signalled one at a time. Groups on the other hand share signals between all workers simultaniuosly. Supervisord will wait for script to exit before starting another task, signalling to other programm or starting another one.

Test results

Reloading 30 tasks w/o group Reloading 30 tasks with group
1m 32.5s 4.85s

Some code references

supervisorctls controller plugins do_signal separate process names and group names and signall them differenly.

RPC on the other side on receiving signal proceeds incoming signal to group processes via make_all_func SignalProcessGroup.

Hacked solution

As of supervisor 4.2.4 supervisorctl reload does not accept arguments. So we cannot use all, like with supervisor restart.

But we can leverage group notification mechanics.

Simply creating group (not called all for unexpected conflicts with supervisorctl hardcoded all) containing all of the programms controlled by supervisord. This insures that all processes will recieve SIGTERM simultaiously upon reload/restart.

Benchmark

This repository contains:

  • simple python script imitating some real service behavior, upon receiving SIGTERM script will sleep for 3 seconds then exit with rc0.
  • dockerized supervisord/supervisorctl.
  • sample config imitating 30 different realworldish commands

I've included some supervisorctl reload logs with group enabled and without as a worst case scenario.

Running benchmark

docker compose up -d
docker compose exec supervisor supervisorctl reload

upon completion you can examine logs via:

docker compose exec supervisor cat /var/log/supervisord.log

Now can comment group, restart supervisor container, rerun benchmark and see results for yourself.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.