osd stuck after hearbeat failure about rook HOT 6 OPEN

Restarting of the OSD was successful?

If the OSD process exits, the pod will restart. So if the OSD did not restart after that error, the ceph-osd process must not have exited.

from rook.

@travisn Yes after restarting osd pod it was showing up
before that it was showing out in ceph status but pod was running but it was stuck.
In osd pod we can see below logs :
"
handle_connect_message_2 accept replacing existing(lossy) channel (new one lossy = 1)
no message from osd.x
osd not healthy; waiting to boot
osd is healthy faluse - only 0/12 up peers(less than 33%)
set_numa_affinity unable to identify public interface"

In between there were logs
"feature acting upacting
transitioning to stray"

Lastly it was showing
: /var/lib/ceph/osd/osd-x/block close
fbmap shutdown

but osd pod wasn't restarted
@Rakshith-R can you please help on above to find root cause ?

from rook.

@akash123-eng Was there any active client IO in the cluster? If the OSD's device was closed, the OSD may not notice until it tries to commit the IO. At that point, then it should fail and restart.

from rook.

@travisn yes there was active client io in the cluster
Other osd were working fine

from rook.

@travisn yes there was active client io in the cluster Other osd were working fine

Ok, then not sure. This just happened once, or has it happened multiple times?

from rook.

@travisn yes it happened once for now. but wanted to get behind its root cause
so we should fix it

from rook.