[Bioclusters] Urgent advice on RAID design requested

[Joe Landman]

George N. White III wrote:\

> MTBF is a statistical measure based on failure rates for a large
> number of fresh units.  You may have a component with 10 year MTBF

Absolutely.

> that whose mechanical bits will wear out in 5 years.  Vendors have

USB2 disks are a great example.  Even if the disk MTBF is in the 0.5M 
hour region, other components in the product aren't demonstrating this. 
  Especially the (very cheap) brick power supplies.

> become very adept at designing hardware that wears out a couple days
> after the warranty expires.  I hear horror stories about the 2nd disk

Hmmm...  I find that statement suspect at best.

> failing while rebuilding a RAID, but how many sites have a schedule to
> replace drives before they actually fail in service rather that
> waiting until the first one fails.  I've experienced too many cases

This is actually not a bad idea.  It is done with machine tools, and CNC 
parts.  Each moving/cutting part has a known statistical lifetime, and 
they are usually arranged so that you are "constantly" (in some 
definition of this word) replacing the cutting parts.

> where a number of identical parts (disks, power supplies, fans) in
> workstations purchased at the same time all fail at roughly the same
> time.  Sometimes there is a trigger event (A/C failure) that stresses
> systems within limits that they would handle when new, but after 2-3
> years cooling fans are less effective due to dust buildup, added
> components have increased heat production in the machine room, etc. so
> you get a cluster of failures.  Rebuilding a RAID is also a stressor.

Agreed.  Many things go into the MTBF estimation.  24x7 load times don't 
often take into account power cycling.  Or other stresses.

> 
>> The point being, if you are going to bet your life, or your data on
>> something, it makes sense to go with hard data as compared to 
>> speculation.
>>
>> The cheapest drives around, Maxtors and their ilk have seen failure
>> rates higher than 3-4% in desktop and other apps.  Sure, you will save a
>> buck or two on the front end (acquisition).  Unless you can tolerate
>> data loss, do you want to deal with the impact on the back end?  Without
>>   trying to FUD here, how much precisely is your data worth, how many
>> thousands or millions of dollars (or euros, or ...) have been spent
>> collecting it?  Once you frame the question in terms of how much risk
>> you can afford, you start looking at how to ameliorate the risk.
>>
>> There are simple, (relatively) inexpensive methods.  N+1 supplies adds
>> *marginal* additional cost to a unit.  Using better drives (notice I
>> didn't say FC/SCSI/SATA), adds minute costs to the unit.  Using
>> intelligent redundancy (RAID6 with hot spares, mirrored,...) reduces
>> risk at an increase in cost.
> 
> So does a sensible schedule to replace older units before they fail.

In combination with a RAID system, yes, this could be quite beneficial. 
     But as with everything you need to balance the cost of the solution 
against the benefit it gives.

> For organizations where unscheduled downtime is expensive, the
> benefits include being able to schedule  replacements to minimize
> disruptions.

Again, see machine tool operators.  They have been doing this for a while.

> 
>> We are not talking about EMC costs here.  Or NetAPP.  If you are
>> spending north of $2.5/GB of space you are probably overspending, though
>> this is a function of what it is and what technology you are buying.
>>
>> > separate machines with cheap components (chapest SATA drives with 
>> single
>> > power supply) is better that one expensive machine (higher quality hard
>> > drives, redundant power supply). What you Gurus say?
> 
> You have given an ill-posed question.  The answer is very sensitive to

The question had been asked in other posts.  What you buy should be a 
function of what you need to do with it as much as it is constrained by 
your budget.

> the I/O profile of your workload. There can be a big performance hit
> for the I/O it takes to replicate the data between the boxes.  Some

This is a function of box design as much as it is IO workload.

> workloads will have low I/O windows where replication can be done.
> How robust is your processing if the the separate machines get out of
> sync? One approach is to keep the filesystem metadata on a small
> highly reliable machine.

  ...  so if the meta data machine, through some terrible ABC 
(adminstration before coffee) accident goes offline ...

Aside from that, this design is a single point of information flow, 
which you wish to strenuously avoid if at all possible in high 
performance systems.

> 
>> I believe that you can save money at the most appropriate places to do
>> so.  Im not sure this is it.  Its your data, and you have to deal
>> with/answer for what happens if a disk or machine demise makes it
>> un-recoverable.  People whom have not had a loss event usually dont get
>> this (e.g. it hasnt bitten them personally).  If you have ever lost data
>> due to a failure, and it cost you lots of time/energy/sweat/money to
>> recover or replicate this, you quickly realize that the "added" cost is
>> a steal, a bargin in comparison with your time.  Which you should value
>> highly (your employer does, and rarely do they want you spending time on
>> data recovery, unless this is your job, as compared to what you are paid
>> to do).
> 
> There are usually people (who won't be around when the problems
> appear) telling management "cheap, secure, and reliable? -- no
> problem!".  In large organizations, the time/energy/sweat includes

Yup.

> sitting in the committees make the recommendations to management.
> Many large organizations have people running spreadsheets to look at
> the cost of data storage/processing in various sites. The results are
> then used to require every site to use the approach that looks
> cheapest -- often without appropriate consideration of the risks or of
> differences in workloads.

Yup.  At the end of the day it is $$$ (or insert your favorite currency).

> 

-- 
Joseph Landman, Ph.D
Founder and CEO
Scalable Informatics LLC,
email: landman at scalableinformatics.com
web  : http://www.scalableinformatics.com
phone: +1 734 786 8423
fax  : +1 734 786 8452
cell : +1 734 612 4615