[Bioclusters] OS X and NFS

[Tim Cutts]

On 15 Jul 2005, at 1:53 pm, Tim Cutts wrote:

>
> On 14 Jul 2005, at 11:37 pm, John H. Lee wrote:
>
>
>>
>> Since Juan's system is OS X, can anyone offer a suggestion for OS  
>> X?  Cluster filesystems like Lustre and PVFS sound promising for  
>> Linux, but what options do we have on Macs?  Any experience with  
>> Xsan?  With a fibre channel limit of 64 endpoints, can Xsan even  
>> be considered for systems as large as Tim's?
>>
>> We have a 16-node Xserve cluster with one Xserve RAID.  The RAID  
>> is connected via FC to one server, which exports the volumes via  
>> AFP and NFS over gigabit ethernet.  As expected, the AFP/NFS  
>> server is a bottleneck.
>>
>
> I believe SGI's cluster filesystem, CXFS, offers client access for  
> Mac OS X.  It's probably in the works for Lustre too (OS X's  
> filesystem plugin architecture should make an open source effort  
> possible - I just don't know whether the Lustre guys have done it)
>

I should have said - to answer the first parts of your question, I  
don't know what the scaling limits of most of these filesystems are.   
Currently, we keep them quite small; 14 or 28 nodes per cluster  
filesystem, using GPFS.  That still means we only have to distribute  
a dozen copies of the data for our first IBM farm, and those 12  
copies are then shared by 168 dual CPU machines.  That's a lot better  
than having to use 168 separate copies, or attempting NFS.

I have a feeling Xsan's limits are quite small - 16 nodes rings a  
bell, but someone from BioTeam will probably know the answer to that  
question.

Great things are promised for Lustre, and I guess we'll find out over  
the next few months just how far we can push it.  We intend to use it  
for cluster-global scratch filesystems first.  (a) these get battered  
most heavily (by concurrent writes, too, the worst of all things), so  
will be a good test, and (b) it's scratch, and the users know it can  
go at any minute, so we'll be OK in case of filesystem disaster in  
this new and unknown territory.

Tim

-- 
Dr Tim Cutts
Informatics Systems Group, Wellcome Trust Sanger Institute
GPG: 1024D/E3134233 FE3D 6C73 BBD6 726A A3F5  860B 3CDD 3F56 E313 4233