[Bioclusters] A set of notes on how to do RAID optimization on a compute node

Joe Landman bioclusters@bioinformatics.org
Mon, 12 Aug 2002 16:29:00 -0400 (EDT)


Hi Allen:

  I have seen advantages to the RAID cards (specifically the speed of 
operations, and no need for additional software/setup), and disadvantages 
(speed of particular operations).

  I have heard good things about the 3ware controllers 
(http://www.3ware.com/) for RAID0 and RAID5, but I havent had the luxury 
of playing with one of the 7500 cards.

  Software RAID will limit much of your throughput via the PCI bus, and 
the individual drivers.  Hardware RAID (such as the well done 3ware cards) 
will parallelize many of the operations that the CPU would do (hence a 
lower load on the CPU and memory).  The PCI bus will still be your 
limiting factor.  It depends upon your specific needs how much better if 
at all, a hardware RAID0 could do.

  Software RAID requires that that CPU divide up the page buffers and 
decide which disk gets which page, sending the right pages in the right 
order to the right disk.  This is an overhead, though on todays fast 
machines, it is generally not that significant in most operating cases 
(corner cases of extreme memory pressure could compound the IO problem, 
possibly leading to deadlocks, but you have to be working hard to crash 
the machine at that point by memory oversubscription).  Hardware RAID 
simply treats the device as a single disk to the driver, so the CPU need 
only set up a DMA transfer.  The hardware does the rest.

  There are other issues though, one specific one is buffering, which 
suprisingly, the software RAID seems to interact with somewhat better.

  Short answer is that your mileage may vary, and try to borrow the card
you want to try out first.  Chris D. would probably have recommendations
or thoughts on the IO, RAID, configs, etc.  side.  I can recommend some
specific BLASTing, or other codes if you want to measure real performance
impact.

Joe

On Mon, 12 Aug 2002, Allen Day wrote:

> Joe,
> 
> Have you had any experience setting up a RAID with one of the IDE 
> RAID controllers?  I'm considering setting up a RAID 0, and want to know 
> if I can benefit from using one of these controllers rather than just 
> putting in an ATA card.
> 
> -Allen
> 
> On Mon, 12 Aug 2002, Joe Landman wrote:
> 
> > Hi Simon:
> > 
> > On Mon, 12 Aug 2002, Vsevolod Ilyushchenko wrote:
> > 
> > > Joe,
> > > 
> > > This is a very informative article. Thank you. A couple of followup 
> > > questions:
> > > 
> > > 1. How would I do a software raid with more than two disks. That is, 
> > > what PC hardware configuration to have more than two independent 
> > > controllers. The only way to get them that I know of is through a RAID 
> > > card, but the card will be the bottleneck then.
> > 
> > The PCI bus will be your limiting factor for IDE.  If you build a SCSI 
> > based system, using Ultra 160 SCSI on a PCI-66 (266 MB/s bus), you 
> > normally would achive about 200 MB/s sustainable bandwidth (the budget you 
> > have to work with).  If each disk can talk at 40 MB/s (10k RPM disks can 
> > do this), then you can in theory have 5 disks in a RAID0 stripe (just add 
> > more 
> > 
> > 	device	    /dev/disk
> > 	raid-disk     N+1
> > 
> > lines in your /etc/raidtab.  The /dev/disk is the disk device, and the N+1 
> > is the next raid disk number (starting from 0).  So if you have 5 scsi 
> > disks, sda to sde, all using partition 2 for the file system, your 
> > /etc/raidtab file would have the device section looking more like this:
> > 
> >         device      /dev/sda2
> >         raid-disk   0
> >         device      /dev/sdb2
> >         raid-disk   1    
> >         device      /dev/sdc2
> >         raid-disk   2    
> >         device      /dev/sdd2
> >         raid-disk   3    
> >         device      /dev/sde2
> >         raid-disk   4    
> > 
> > Of course, your chunk-size parameter will need to be optimized, and you 
> > will need to change the nr-raid-disks to 5.
> > 
> > This will give you RAID0, 5 way striping.
> > 
> > For IDE, you can purchase an inexpensive ATA100 IDE card for PCI, and put 
> > 2 disks on it (one on each channel).  Together with another disk on the 
> > main IDE channels, you can get a 3 way stripe, which should put you pretty 
> > close to 100 MB/s.
> > 
> > > 2. In general, how much of a speedup do you expect if you use 2-disk and 
> > > 5-disk software raid, compared to the same disk drive used standalone?
> > 
> > Raither hard to answer a general question like this:  Which version of 
> > RAID ... RAID 0?  1, 3, 5?  Which file system, what type of access pattern 
> > (large sequential block reads, versus smaller random reads and writes).
> > 
> > This quickly gets into a discussion of how to tune for a specific range of 
> > applications.  My company, Scalable Informatics (formed after I left 
> > MSC.Software), could help answer some of these.
> > 
> > Let me know if you want to talk.  Thanks Simon!
> > 
> > Joe
> > 
> > > 
> > > Thanks,
> > > Simon
> > > 
> > > Joe Landman wrote:
> > > > Hi folks:
> > > > 
> > > >   Chris had asked about methods to to RAID optimization on compute
> > > > nodes.  The idea being that he could get good/better performance out of
> > > > his system if he tuned various options in the software RAID0 under
> > > > linux.
> > > > 
> > > >   The following URL is a short writeup I had been working on for a while
> > > > now.  
> > > > 
> > > > http://scientificappliance.com/scalable_fs_part_1.html
> > > > 
> > > > If you have any questions or comments, please feel free to send them to
> > > > me.   Thanks!
> > > > 
> > > 
> > > 
> > > 
> > 
> > 
> 
> _______________________________________________
> Bioclusters maillist  -  Bioclusters@bioinformatics.org
> https://bioinformatics.org/mailman/listinfo/bioclusters
> 

-- 
Joe Landman,
email: landman@scientificappliance.com
web  : http://scientificappliance.com