XOR and EC goals are not safe, even when just single disk fails about lizardfs HOT 5 OPEN

The inevitable did happen today. Lost files and data even with EC goals coverage and a single disk failure.

One of my hard drive went bad with bad sectors, lizardfs reported damaged disk and I had to remove it from the system. And a lots of chunks and files shown up as unavailable even though the goal was EC(8,2).

I spot checked a few smaller missing files with mfsfileinfo and they only has 7 out 10 copies. So either lizardfs put all 3 missing copies on this single failed disk or it only generates 8 copies and one of them is on this particular disk.

I'm not sure how to have a "clear reproduction scenario". The system is in a definitely bad and precarious state and we can certainly debug from here and hopefully has a way to fix it. But not sure how to get to this state from a clean install.

Best thing at the moment is to have a way to detect files that are at risk and way to force it to redistribute.

from lizardfs.

Hi, @aletus this looks like a very interesting bug - thank you for sharing it.
Can we have a video chat about it, please?
If yes can you send me your contact data to [email protected], please?
after we got more data and a clear reproduction scenario I will gladly update this issue with an explanation for the fix.

from lizardfs.

I've observed this for EC goals too. @aNeutrino do you have a code pointer where the logic for spreading the chunks out is?

from lizardfs.

I wonder if it could be behind the reason Moosefs doesn't delete discarded junks until a week later.

from lizardfs.

@aletus, which version are you using? I did notice uneven distribution on only a few files/cases throughout 3+ years quite intensive using of 3.12.0, but nothing that would threaten my data - and believe me, you wouldn't probably entrust your data with the drives I did ;-)

I'm using 16+2, 8+2, 4+2, 4+1, 2+1 and x2, x3, x4 replication modes mostly, and if one drive goes south, I can see several chunks that have 2 copies to replicate, but on the other hand they do not appear in "endangered chunks" list, so I didn't wonder what is the reason of this behavior.

from lizardfs.