[Question] Scalability about ndscheduler HOT 2 CLOSED

Why don't you use a distributed task queue such as Huey, RQ, or Celery for this? Or you can use a combo of a task queue and NDS (though to me, it seems like an overkill)

from ndscheduler.

So I think you could do this, but if you have jobs that will only run once, you probably want a distributed task queue for that. There's not really any reason generating 30K schedules a day wouldn't work, but we certainly haven't tested it.

If you want things to run at certain times, I'd recommend instead having a job that starts up periodically during the day and checks some other distributed queue (e.g. SQS, RabbitMQ) to see what work needs to be done.

from ndscheduler.