On Sat, 25 Sep 2004, Joe Landman wrote: > > >Dan Bolser wrote: > >>NFS >> ------Sequential Output------ --Sequential Input- --Random- >> -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks-- >> Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP >> 8G 5291 29 6375 3 4563 2 11364 66 11250 3 267.3 1 >> ------Sequential Create------ --------Random Create-------- >> -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete-- >>files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP >> 16 223 0 2322 4 339 0 229 0 2460 3 323 0 >> >> >> > >Yup. Looks ok. Could tune the server and client a bit, but this looks >pretty good. Yeah, unless I could easily >1.5x the performance with some optimization I don't think I will bother. > > >>IDE >> ------Sequential Output------ --Sequential Input- --Random- >> -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks-- >> Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP >> 7G 17383 95 39662 24 18243 9 17345 89 51922 14 91.1 0 >> ------Sequential Create------ --------Random Create-------- >> -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete-- >>files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP >> 16 1431 98 +++++ +++ +++++ +++ 1392 99 +++++ +++ 4225 99 >> >> >> >> > >This looks about right. Sequential input looks low for a Seagate, is >this a Western Digital or an IBM/Fujitsu? :) Fujitsu. >>SCSI >> ------Sequential Output------ --Sequential Input- --Random- >> -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks-- >> Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP >> 7G 16634 95 32241 22 13841 6 13413 69 37512 10 168.7 0 >> ------Sequential Create------ --------Random Create-------- >> -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete-- >>files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP >> 16 1383 97 +++++ +++ +++++ +++ 1257 95 +++++ +++ 3566 95 >> >> >> >> > >Interesting. If this is a 10k or 15k RPM SCSI device, then I am not >sure how well it is tuned. You should be able to hit about 70MB/s from >a 15kRPM unit (with a little tuning). I always have problems with big data transfers onto this device - my machine hangs for a short time proportional to the amount of data transferred. Don't know the RPM. >>Test machine memory: 4124716k >> >>NFS >> ------Sequential Output------ --Sequential Input- --Random- >> -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks-- >> Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP >> 8G 4826 68 10614 7 7645 6 7297 99 37843 10 348.7 1 >> ------Sequential Create------ --------Random Create-------- >> -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete-- >>files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP >> 16 240 0 2961 5 366 1 246 0 2938 1 351 0 >> >> >>Underlying NFS export is ext3 raid system (RAID5 I think) with 8 disks. >> >> > >Ok, this might explain some of the numbers. RAID5 is expensive on writes. > >>One 'issue' IT had with XFS was its performance under NFS, from what you >>have said I guess that isn't an issue? i.e. NFS exported XFS is the same >>as local XFS except for the network connection for all the IO? They were >>worried that NFS would hide any benifits of the underlying FS. >> >> > >Actually NFS tends to stress the underlying IO system, not to mention >the network stack, the kernel and a few other things. So if there are >bottlenecks in code paths (ext3 journaling, especially with soft-RAID5), >you are going to pay a serious performance penalty. If you are running >this as a software RAID, you are going to run into memory pressure and >buffer cache interactions under high loads. Can make for exciting times >:( XFS is an excellent file system under heavy load. > >I have found that RAID1's tend to be overall better performers, and you >can stripe the RAID1's to get RAID 10's. You lose storage capacity, but >the performance is quite nice under load (writes are not nearly as >expensive as under RAID5, and reads are often load balanced between drives). Dosn't this happen with raid5 too (balanced reads across disks)? I think we have hardware raid. Anyway, thanks very much for all the information. I will be hard pressed to get just one XFS 'trial' partition past the IT guys, so they will never change the raid system at my request. I understand their point of view, which is the classic - if it aint broken, don't fix it. Naturally from my point of view - faster is nicer, but I can't rigorously justify my 'feeling' that things should be working better. My main argument for XFS is that it handles millions of files per directory, which ext3 fails to do. Thanks again, Dan. >>Thanks, >> >>Dan. >> >> >> >> >>On Sat, 25 Sep 2004, Joe Landman wrote: >> >> >> >>>Hi Guy: >>> >>> With version 3 of NFS, you might try >>> >>> rsize=32768,wsize=32768,tcp,intr,hard,nfsvers=3 >>> >>>Version 3 lets you do 32k rsize/wsize. TCP can be (but isn't always) a >>>win. You might experiment with the noatime, and some of the other >>>switches. There are some other "tricks" as well to tuning NFS. It >>>isn't too difficult to hit wire speed over 100 base T, it is a bit >>>harder to hit it over gigabit. Lots more to tune (including disk). I >>>would recommend XFS or JFS as the underlying exported file system. Ext3 >>>still has some non-optimized serializing code paths in its journaling >>>system that make it hard to get very good performance out of it. >>> >>>Joe >>> >>>Guy Coates wrote: >>> >>> >>> >>>>One thing which makes a big difference in IO speed is tweaking the various >>>>mount options: >>>> >>>>For your NFS mounts, the following is a good start (maybe proto=tcp if >>>>your server supports it): >>>> >>>>rsize=8192,wsize=8192,hard,intr,vers=3,proto=udp >>>> >>>> >>>>For ext3, mount with "data=writeback"[1], and for reiserfs mount with >>>>"notail" options. If you've got it compiled into your kernel, XFS is >>>>worth trying too. It is quite speedy. >>>> >>>>Guy >>>> >>>> >>>>[1] Check the man page for what this does; if you machine goes down in the >>>>middle of a file write you stand an increase chance of that file getting >>>>mangled; the filesystem itself should be OK. >>>> >>>>-- >>>>Dr. Guy Coates, Informatics System Group >>>>The Wellcome Trust Sanger Institute, Hinxton, Cambridge, CB10 1SA, UK >>>>Tel: +44 (0)1223 834244 ex 7199 >>>> >>>> >>>> >>>> >>>> >>>> >>>>_______________________________________________ >>>>Bioclusters maillist - Bioclusters@bioinformatics.org >>>>https://bioinformatics.org/mailman/listinfo/bioclusters >>>> >>>> >>>> >>>> >>> >>> >> >> >>_______________________________________________ >>Bioclusters maillist - Bioclusters@bioinformatics.org >>https://bioinformatics.org/mailman/listinfo/bioclusters >> >> > >