Feature request: Do not try to recover output checkpoints for plugins that don't use it about onyx HOT 3 OPEN

@jgerman it sounds like your issue is unrelated to missing lookups when loading checkpoints.

Check the back-pressure on your tasks. It is the most frustrating and hard to identify cause of random peer timeouts. Despite the brief section in the user guide, onyx 0.14 does not handle backpressure all that well. When the preceeding task is pushing data out faster than the downstream tasks can take, it can stop processing heartbeat messages.

If. you have a lot of back-pressure, consider writing to an intermediate kafka topic to decouple the imbalance between producer/consumer speed. Tuning the idle-sleeps in the config as the user guide suggests is actually not that effective in my experience.

http://www.onyxplatform.org/docs/user-guide/0.14.x/#backpressure

from onyx.

We actually see this issue in our long running jobs even when no changes to the job have been made. The stack trace is almost identical other than you're using S3 and we're using ZK.

I'm pretty sure ours is due to a task rebooting but haven't figured out why it happens, or why it appears to consistently be the same output task.

from onyx.

Thanks! Any suggestions for measuring back pressure? I've only indirectly looked at it via consumer lag on our kafka topics.

I've definitely seen issues with checkpoints not being written which I assumed was due to heartbeats (barriers?) not being processed.

from onyx.