USB IO Errors and RAID vs ZFS

About 8 years ago, I created a RAID-5 with 4 hard disks.  Two of the disks were IDE and two where USB external disks.  Each disk was 250GB, giving me 750GB.  Everything seemed to work, so I moved a lot of my data onto the RAID.  At some point, the RAID disappeared.  I looked at the logs and the two USB external disks got flagged as being bad.  Because you can't "lose" 2 disks in a RAID-5, I lost ALL of the data.  Here is the thing: the disks weren't bad!  That event is when I learned that USB hard disks return random IO errors.  This IO errors are very small and are recoverable.  The RAID software interpreted the IO errors as a bad sector, though.  Once one of the sectors was bad, the entire hard disk was flagged as bad because it didn't have the same capacity as the other disks.  It turns out backups are important.

After that incident, I played around with EVMS to create a pool out of the disks.  The volume management allowed me to see some interesting effects.  Whenever an IO error occurred with one of the USB disks, the available pool size shrank.  Since the volume was smaller than the pool size, however, I didn't lose any data like I did with a raw RAID.

I am now playing with ZFS.  I have a RAID-Z with SATA hard disks.  I started getting IO errors on one of my hard disks.  These are real errors related to my brand new 3TB WD drive going bad.  I had a refurb 2TB Seagate but I had to plug it in via USB.  I remembered my problems with USB and RAID-5, but this was only going to be one disk, not two.  I plugged in the disk and replaced the files.  Like 8 years ago, everything seemed to work fine.  After about a day, the first IO error occurred.  ZFS handled it just fine.  The zpool status command tells you how many IO errors have occurred on the various disks.  Because of the CRC capabilities, ZFS is able to handle the problem without marking the entire disk as bad.  ZFS gives you the option to either ignore the errors or replace the bad disk.  That is what makes a good solution!