Suspicious disk failures

2012-10-16 hardware monitoring home-lab

The disks in my home server have been acting up lately: Two of them (which I bought at once a few years ago) deliver different results when I hash a file multiple times, another just randomly hangs sometimes for a few minutes. The SATA and power cables seemed fine, replacing them had no effect on the problem. So, why not check what wisdom S.M.A.R.T. has to offer?

root@holo:~# smartctl --all /dev/sdc
smartctl 5.43 2012-06-05 r3561 [x86_64-linux-3.2.0-4-amd64] (local build)
Copyright (C) 2002-12 by Bruce Allen, http://smartmontools.sourceforge.net

=== START OF INFORMATION SECTION ===
Model Family:     Hitachi Deskstar 7K1000.C
Device Model:     Hitachi HDS721010CLA332
##### snip #####

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

Hm, looks like the disk is fine. So it’s the SATA controller’s fault after all… wait, what’s that?

ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
  1 Raw_Read_Error_Rate     0x000b   098   098   016    Pre-fail  Always       -       131075
  2 Throughput_Performance  0x0005   135   135   054    Pre-fail  Offline      -       98
  3 Spin_Up_Time            0x0007   137   137   024    Pre-fail  Always       -       284 (Average 270)
  4 Start_Stop_Count        0x0012   099   099   000    Old_age   Always       -       4808
  5 Reallocated_Sector_Ct   0x0033   100   100   005    Pre-fail  Always       -       0
  7 Seek_Error_Rate         0x000b   100   100   067    Pre-fail  Always       -       0
  8 Seek_Time_Performance   0x0005   130   130   020    Pre-fail  Offline      -       35
  9 Power_On_Hours          0x0012   099   099   000    Old_age   Always       -       10556
 10 Spin_Retry_Count        0x0013   100   100   060    Pre-fail  Always       -       0
 12 Power_Cycle_Count       0x0032   100   100   000    Old_age   Always       -       1257
192 Power-Off_Retract_Count 0x0032   096   096   000    Old_age   Always       -       4869
193 Load_Cycle_Count        0x0012   096   096   000    Old_age   Always       -       4869
194 Temperature_Celsius     0x0002   187   187   000    Old_age   Always       -       32 (Min/Max 15/60)
196 Reallocated_Event_Count 0x0032   100   100   000    Old_age   Always       -       0
197 Current_Pending_Sector  0x0022   100   100   000    Old_age   Always       -       0
198 Offline_Uncorrectable   0x0008   100   100   000    Old_age   Offline      -       0
199 UDMA_CRC_Error_Count    0x000a   200   200   000    Old_age   Always       -       0

I am no authority on hard disks, and making sense of those numbers is hard. But I can imagine what “Pre-fail” could mean. Running a quick Internet search about this condition didn’t really yield any definitive information. It was suggested that this information shouldn’t be taken at face value, since every manufacturer does their own thing. Still, it’s a little bothering.

Anyway, two new disks, 3TB each, will arive tomorrow or the day after. That should take care of the faulty reads - if not, I can probably scrap that motherboard or buy a PCIe SATA controller.

Moral of the story🔗

  • Monitor all the things, so you can replace the disk while you can still read un-corrupted data from it.
  • Go redundant, so even a failing disk doesn’t do any harm. I think I’ll go with ZFS.
  • If the summary says “Hey, no need to worry. Everything is fine, I got this.”, ignore the summary and at least skim the details.