Fighting BITROT and URE with every last breath

I have always been a huge proponent of typical RAID. Back in the day of 2TB drives, disks were quite reliable. Even when a disk failed, I never had a problem doing a RAID recovery/resync onto a new disk. The RAID would keep running for days or weeks no problem even with 1 disk missing, up until the point a new disk could be inserted and replace the bad disk. From that perspective, the RAID system was very reliable. Fast-Forward to today where 8TB drives are ubiquitous and the probability of doing a RAID5 recovery a 9x8TB array is between 0.3% and 56.2%. I had to learn about URE percentages the hard way, so here we are.
I propose that the fundamentals of RAID are still good (including the software), but doing a full recovery on a LARGE array is no longer a realistic option given the bit error rates of current drives (i.e. RAID5 on a 72TB array). I choose to make many small "microraids" to encapsulate my data. This will keep the recovery percentages very high for each array. Each microraid is backed by a set of disk images, placed anywhere on any disk. For each microraid you can choose a different level of redundancy, even though they are stored on the same set of physical disks. microraids gives the flexibility to have any number of RAID 0/1/4/5/6 arrays as long as available drive space will allow it. Also each microraid can be checked independently for integrity and consistency in multiple ways.

• Disk capacities do not have to match
• Integrity issues can be identified before recovery
• Recovery probabilities are significantly increased
• You can put multiple raid types (i.e. 0/1/4/5/6) on the same disk

• calc
• Standard Utilities: sgdisk / hdparm / dd / losetup / mdadm

• Step 1: Prepare your physical disks
• Step 2: Create your microraid
• Step 3: Start/Stop your microraid
• Step 4: Check your microraid
• Step 5: Replace a faulty image
• Step 6: Tuning your microraid (Optional)
• Step 7: Grow your microraid (Optional)

• RapidLinux
• Slackware
• Ubuntu

• Cooler Master Elite 110
• ASRock J3455B-ITX / J4005B-ITX / J4105B-ITX / J4025B-ITX / J4125B-ITX
  • These ASRock Mini-ITX boards support PCIe in x2 mode
• Ableconn PEX-SA130
  • ASM1062 chipset supports 2x Port Multiplier
• 2x Mediasonic ProBox HF2-SU3S2
  • Each ProBox connects 4x 3.5" drives of any size via eSATA
• 8x 4TB HGST Ultrastar 7K4000 REFURB
  • Chosen to show software reliability/recovery on very inexpensive hardware

• SYBA SI-PEX40072
  • This Marvell Chipset also supports 2x Port Multiplier
• Ableconn PEX-SA134
  • 2xASM1062 chips support 4x Port Multiplier; requires PCIe x4 lanes

• bitrot and atomic cows
• Raid Rebuild Probability Calc
• URE in RAID
• RAID questions