Lodestar stopped keeping up with head on mainnet about lodestar HOT 25 OPEN

Hey @nflaig . We've had this happen on two nodes in our Lido clusters. Yes, we run multiple CL:EL, the VC is a Vouch.

Lodestar started syncing after we restarted it. We didn't want to wait so we checkpoint synced to get back on track.

So, this is definitely a thing, it happened on different blocks on two nodes.

from lodestar.

Two observations from the logs

1. seems like your EL was still syncing before Lodestar started to fall behind

Jun-06 15:41:41.000[]                 �[32minfo�[39m: Synced - slot: 9238706 - head: 0x4901…b19e - exec-block: syncing(20033746 0x9174…) - finalized: 0x6724…7027:288707 - peers: 66
Jun-06 15:41:29.000[]                 �[32minfo�[39m: Synced - slot: 9238705 - head: 0xf8f2…6ddd - exec-block: syncing(20033745 0x538c…) - finalized: 0x6724…7027:288707 - peers: 62
Jun-06 15:41:17.000[]                 �[32minfo�[39m: Synced - slot: 9238704 - head: 0x165e…6111 - exec-block: syncing(20033744 0x40c5…) - finalized: 0x6724…7027:288707 - peers: 67
Jun-06 15:41:05.000[]                 �[32minfo�[39m: Synced - slot: 9238703 - head: 0x1a85…16df - exec-block: syncing(20033743 0x52c6…) - finalized: 0x6724…7027:288707 - peers: 68

2. somehow the validator client is still submitting attestations even though your node was no longer synced, any chance you have multiple beaocon nodes running?

From the logs, it's hard to tell why it started to fall behind, it's not a general issue for this block as my node is still synced, might require debug logs to figure out what's going on.

If your node is still not back to head, you might wanna force a checkpoint sync it by passing

--checkpointSyncUrl <url> --forceCheckpointSync

from lodestar.