[Bioclusters] Tool to benchmark disk IO?

Dan Bolser bioclusters@bioinformatics.org
Sun, 26 Sep 2004 12:57:44 +0100 (BST)


On Sat, 25 Sep 2004, Joe Landman wrote:

>
>
>Dan Bolser wrote:
>
>>NFS
>>      ------Sequential Output------ --Sequential Input- --Random-
>>      -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
>> Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
>>   8G  5291  29  6375   3  4563   2 11364  66 11250   3 267.3   1
>>      ------Sequential Create------ --------Random Create--------
>>      -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
>>files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
>>   16   223   0  2322   4   339   0   229   0  2460   3   323   0
>>
>>  
>>
>
>Yup.  Looks ok.  Could tune the server and client a bit, but this looks 
>pretty good.

Yeah, unless I could easily >1.5x the performance with some optimization I
don't think I will bother.

>
>
>>IDE
>>      ------Sequential Output------ --Sequential Input- --Random-
>>      -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
>> Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
>>   7G 17383  95 39662  24 18243   9 17345  89 51922  14  91.1   0
>>      ------Sequential Create------ --------Random Create--------
>>      -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
>>files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
>>   16  1431  98 +++++ +++ +++++ +++  1392  99 +++++ +++  4225  99
>>
>>
>>  
>>
>
>This looks about right.   Sequential input looks low for a Seagate, is 
>this a Western Digital or an IBM/Fujitsu? 

:) Fujitsu.


>>SCSI
>>      ------Sequential Output------ --Sequential Input- --Random-
>>      -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
>> Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
>>   7G 16634  95 32241  22 13841   6 13413  69 37512  10 168.7   0
>>      ------Sequential Create------ --------Random Create--------
>>      -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
>>files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
>>   16  1383  97 +++++ +++ +++++ +++  1257  95 +++++ +++  3566  95
>>
>>
>>  
>>
>
>Interesting.  If this is a 10k or 15k RPM SCSI device, then I am not 
>sure how well it is tuned.  You should be able to hit about 70MB/s from 
>a 15kRPM unit (with a little tuning).

I always have problems with big data transfers onto this device - my
machine hangs for a short time proportional to the amount of data
transferred. Don't know the RPM.

>>Test machine memory:	4124716k
>>
>>NFS
>>      ------Sequential Output------ --Sequential Input- --Random-
>>      -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
>> Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
>>   8G  4826  68 10614   7  7645   6  7297  99 37843  10 348.7   1
>>      ------Sequential Create------ --------Random Create--------
>>      -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
>>files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
>>   16   240   0  2961   5   366   1   246   0  2938   1   351   0
>>
>>
>>Underlying NFS export is ext3 raid system (RAID5 I think) with 8 disks.
>>  
>>
>
>Ok, this might explain some of the numbers.  RAID5 is expensive on writes.
>
>>One 'issue' IT had with XFS was its performance under NFS, from what you
>>have said I guess that isn't an issue? i.e. NFS exported XFS is the same
>>as local XFS except for the network connection for all the IO? They were
>>worried that NFS would hide any benifits of the underlying FS.
>>  
>>
>
>Actually NFS tends to stress the underlying IO system, not to mention 
>the network stack, the kernel and a few other things.  So if there are 
>bottlenecks in code paths (ext3 journaling, especially with soft-RAID5), 
>you are going to pay a serious performance penalty.  If you are running 
>this as a software RAID, you are going to run into memory pressure and 
>buffer cache interactions under high loads.  Can make for exciting times 
>:(  XFS is an excellent file system under heavy load.
>
>I have found that RAID1's tend to be overall better performers, and you 
>can stripe the RAID1's to get RAID 10's.  You lose storage capacity, but 
>the performance is quite nice under load (writes are not nearly as 
>expensive as under RAID5, and reads are often load balanced between drives).

Dosn't this happen with raid5 too (balanced reads across disks)? I think
we have hardware raid.

Anyway, thanks very much for all the information. I will be hard pressed
to get just one XFS 'trial' partition past the IT guys, so they will never
change the raid system at my request. I understand their point of view,
which is the classic - if it aint broken, don't fix it. Naturally from my
point of view - faster is nicer, but I can't rigorously justify my
'feeling' that things should be working better. My main argument for XFS
is that it handles millions of files per directory, which ext3 fails to
do. 

Thanks again,
Dan.



>>Thanks,
>>
>>Dan.
>>
>>
>>
>>
>>On Sat, 25 Sep 2004, Joe Landman wrote:
>>
>>  
>>
>>>Hi Guy:
>>>
>>> With version 3 of NFS, you might try
>>>  
>>>   rsize=32768,wsize=32768,tcp,intr,hard,nfsvers=3
>>>
>>>Version 3 lets you do 32k rsize/wsize.  TCP can be (but isn't always) a 
>>>win. You might experiment with the noatime, and some of the other 
>>>switches.  There are some other "tricks" as well to tuning NFS.  It 
>>>isn't too difficult to hit wire speed over 100 base T, it is a bit 
>>>harder to hit it over gigabit.  Lots more to tune (including disk).  I 
>>>would recommend XFS or JFS as the underlying exported file system.  Ext3 
>>>still has some non-optimized serializing code paths in its journaling 
>>>system that make it hard to get very good performance out of it.
>>>
>>>Joe
>>>
>>>Guy Coates wrote:
>>>
>>>    
>>>
>>>>One thing which makes a big difference in IO speed is tweaking the various
>>>>mount options:
>>>>
>>>>For your NFS mounts, the following is a good start (maybe proto=tcp if
>>>>your server supports it):
>>>>
>>>>rsize=8192,wsize=8192,hard,intr,vers=3,proto=udp
>>>>
>>>>
>>>>For ext3, mount  with "data=writeback"[1], and for reiserfs mount  with
>>>>"notail" options.  If you've got it compiled into your kernel, XFS is
>>>>worth trying too. It is quite speedy.
>>>>
>>>>Guy
>>>>
>>>>
>>>>[1] Check the man page for what this does; if you machine goes down in the
>>>>middle of a file write you stand an increase chance of that file getting
>>>>mangled; the filesystem itself should be OK.
>>>>
>>>>--
>>>>Dr. Guy Coates,  Informatics System Group
>>>>The Wellcome Trust Sanger Institute, Hinxton, Cambridge, CB10 1SA, UK
>>>>Tel: +44 (0)1223 834244 ex 7199
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>_______________________________________________
>>>>Bioclusters maillist  -  Bioclusters@bioinformatics.org
>>>>https://bioinformatics.org/mailman/listinfo/bioclusters
>>>> 
>>>>
>>>>      
>>>>
>>>    
>>>
>>
>>
>>_______________________________________________
>>Bioclusters maillist  -  Bioclusters@bioinformatics.org
>>https://bioinformatics.org/mailman/listinfo/bioclusters
>>  
>>
>
>