The requester node attempts to schedule work on disconnected nodes resulting in the job never running about bacalhau HOT 4 CLOSED

What is the proposal here? I believe the default option now is to auto-approve nodes, and only schedule on approved and connected nodes. Is any of that still missing?

from bacalhau.

Yeah the only scheduling on connected and approved nodes is missing. Currently we schedule on disconnected node for some job types and ignore their approval state for other job types. frankly it's a bit of a mess:

• we need to address this TODO: https://github.com/bacalhau-project/bacalhau/blob/main/pkg/orchestrator/scheduler/batch_service_job.go#L158
• modify this section: https://github.com/bacalhau-project/bacalhau/blob/main/pkg/orchestrator/scheduler/daemon_job.go#L93
• modify this section: https://github.com/bacalhau-project/bacalhau/blob/main/pkg/orchestrator/scheduler/ops_job.go#L110

Or rather than modify, allow these aspects of scheduling to be configured.

Further we need to ensure that worked scheduled on an offline node runs when the node comes back online which we will need #3772 to do. e.g. the orchestrator could listen for connected events and create an evaluation to execute the work.

from bacalhau.

Another point to consider:
How can we allow users to define different scheduling heuristics for compute nodes. e.g. nodes in a data center ought to have a more strict requirement on connectedness than nodes that are expected to go offline for longer periods of time (e.g. submarine compute nodes)

from bacalhau.

@wdbaruni previously suggested adding another (a third) timeout in future which allows nodes to be offline for that long before being considered dead.

from bacalhau.