On 7/15/05 9:05 AM, "John H. Lee" <johnhlee at berkeley.edu> wrote: > Since Juan's system is OS X, can anyone offer a suggestion for OS X? > Cluster filesystems like Lustre and PVFS sound promising for Linux, > but what options do we have on Macs? Any experience with Xsan? With > a fibre channel limit of 64 endpoints, can Xsan even be considered > for systems as large as Tim's? > > We have a 16-node Xserve cluster with one Xserve RAID. The RAID is > connected via FC to one server, which exports the volumes via AFP and > NFS over gigabit ethernet. As expected, the AFP/NFS server is a > bottleneck. As I mentioned, we're running Xsan in our OS X cluster, and also for our department-wide fileserver (also OS X). This is absolutely an improvement, as we can distribute the file sharing load over multiple NFS servers, all of whom can export the same volume. For our 120 node Xserve cluster, this is an immense help, as now each NFS server responds to only 30 clients, rather than trying to serve 120. Initial testing we did (see http://www.bioitworld.com/archive/121504/box.html) indicated that the reduction in load is nearly linear (i.e. 4 NFS servers gives us a 4x increase in performance over a single NFS server for all 120 cluster nodes). This is really great news, and we could do even better by adding additional NFS servers, but setting up the Xsan array properly is a bit complicated, and not as inexpensive (to do things properly) as might be initially thought (we were pretty early adopters of this technology, and so found out the hard way what not to do). In particular, its important to have a private network for the metadata communications, and its best to isolate metadata to its own dedicated storage pool (as Xsan really hits the metadata storage containers hard and generates a lot of communication traffic). This can bump up costs somewhat. We played with PVFS and other similar shared filesystems (MFS, GFS) a while back on a smaller (48 processor) linux cluster, but never really found anything that was as robust (as NFS) for processes that did a lot of I/O and clusters with lots of nodes. I imagine the situation is somewhat better now, but I don't know which (if any) of these have been ported to OS X? -- M. Michael Barmada, Ph.D. Associate Professor of Human Genetics Graduate School of Public Health, University of Pittsburgh ===================================================================== There are three kinds of people in this world: those that can count, and those that can't... The requirements said: Windows 2000 or better. So I got a Macintosh. To know the mighty works of God; to comprehend His wisdom and majesty and power; to appreciate, in degree, the wonderful working of His laws, surely all this must be a pleasing and acceptable mode of worship to the Most High, to whom ignorance can not be more grateful than knowledge. ~Copernicus =====================================================================