reynoldsbd / async-jobs Goto Github PK

User should be able to set a max CPU threshold using a "builder pattern" method on Scheduler. When threshold is provided, Scheduler should wait until system CPU usage is below threshold before scheduling new concurrent jobs.

It is possible for a job to have side effects which necessitate reloading the entire job graph. Add a new Outcome::Reload variant (see #4) representing this scenario.

When a job returns the reload outcome, stop executing jobs and load a new Schedule based on the originally submitted job. Then restart execution.

As an optimization, it would be interesting if we could somehow preserve the state of the old (pre-reload) schedule and map it onto the new reloaded schedule. Intent of this would be to prevent unnecessary/duplicate execution of jobs after a reload.

This feature has two parts:

1. Allow individual jobs to report an unchanged outcome
2. If all of a job's dependencies reported unchanged, then skip execution of that job and set its outcome to unchanged as well

For part 1, create a new enum:

#[non_exhaustive] // so we can add more outcomes in the future
pub enum Outcome {
    Success,
    Unchanged,
}

Change return type of Job::run to Result<Outcome, Self::Error>

Update JobState::Succeeded variant to include outcome value

For part 2, update Schedule::mark_complete to accept the full Result of job run and store job outcome appropriately. Also add logic to recursively set unchanged outcome for dependent jobs

Modify scheduler to start and run jobs concurrently where possible.

Will probably be implemented using a FuturesUnordered:

https://docs.rs/futures/0.3.6/futures/stream/futures_unordered/struct.FuturesUnordered.html

User should be able to configure the maximum number of concurrently running jobs by calling a "builder pattern" method on Scheduler before running their jobs.

See if there are any features of dependency crates that can be disabled

Add type parameter to Job trait and Scheduler struct representing a Context that is passed by reference to all Job methods. Intent is to eliminate the need for each Job instance to carry its own references to shared state.

If possible, default Context type should be the unit struct ().

User should be able to configure how the scheduler responds to a job failure:

• halt - immediately drop all actively running jobs
• finish - allow actively running jobs to finish, but do not schedule any more jobs
• continue - continue scheduling as many jobs as possible