Tim Cutts wrote: > > [...] > Most very large clusters in the HPC world don't allow NFS at all, or > minimise it. > > Our 1000-node cluster does allow some NFS, but this is to scratch > directories, and *not* to all users' home directories, in general. Hello, Tim. We only have a 'small' 64-node cluster here :-) However, I've opted to use BOBCAT architecture: http://www.epcc.ed.ac.uk/bobcat/ Although the original EPCC BOBCAT no longer exists, it's spirit lives on in our RRI/BioSS cluster: http://bobcat.rri.sari.ac.uk The important thing is to have TWO completely separate private network fabrics: One for DHCP/NFS, the other for IPC. The main problem we have is that IPC (i.e. Inter Process Communication) can swamp the bandwidth of a single network fabric and you rapidly lose control of the cluster. I think there are some MYTHS about NFS and clusters around because of the bandwidth contention on a single network fabric. The NFS network traffic on our cluster is completely segregated from the IPC traffic which is throttled by the bandwidth of its own network fabric. The switches on the two network fabrics are NOT connected in any way... Tony. -- Dr. A.J.Travis, | mailto:ajt at rri.sari.ac.uk Rowett Research Institute, | http://www.rri.sari.ac.uk/~ajt Greenburn Road, Bucksburn, | phone:+44 (0)1224 712751 Aberdeen AB21 9SB, Scotland, UK. | fax:+44 (0)1224 716687