[Bioclusters] OS X and NFS

[Joe Landman]

Yes you can run TCP stacks atop the low latency interconnects.  You will 
still have to pay to time cost of the stack traversal (not zero time 
unfortunately).  In the past, GM (Myrinet) had some issues running both 
a TCP/ip stack and a normal gm process together.

That said, there are many issues associated with getting low latency, 
and the value of using these for NFS and its ilk may be hard to quantify 
compared to the overall cost.  The issue for NFS et al is usually 
pipe/server bandwidth:  how much data can the server shovel off disks, 
and how wide is the bandwidth pipe into the server, so that for N 
requestors, Bandwidth/N is still a reasonable number (this is 
incidentally why NFS is such a bad choice when N gets large, and your 
Bandwidth remains constant).

We usually advise that end users separate out their file systems into a 
fast internal file system for scratch/local space, and an enterprise 
class file system for long term backed up storage.  The latter gets 
expensive quickly, and isn't that high in overall performance (NFS like, 
NAS like, SAN like).  The former can get expensive if you want 
appliances or commercially supported software.  The two are built to 
very different objectives, and it is rare that you find a single product 
that does both well.  It is always a compromise.  Most of the latter are 
quite expensive, and some of the former are.

Lustre, PVFS, GFS, et al seek to change the economics of the fast file 
system.  Though I know some people do, I am not sure I trust them yet 
for permanent data storage.  Their design goals are vastly different 
from the enterprise class file systems.  That needs to be kept in mind 
when designing systems.

Joe

M. Michael Barmada wrote:
> On 7/15/05 9:05 AM, Tim Cutts <tjrc at sanger.ac.uk> wrote:
> 
> 
>>On 14 Jul 2005, at 11:24 pm, David Kramer wrote:
>>
>>
>>>Respectfully, and at the risk of sounding, ridiculously naive --
>>>why not
>>>consider upgrading the I/O switching technology to Myrinet or
>>>Infiniband for
>>>higher-bandwidth and ultra-low latency, before buying more servers?
>>
>>Depends on the application.  Gigabit ethernet is reasonable
>>bandwidth, it's just not very good latency.  So if you do a lot of
>>properly parallel MPI stuff where fast job turnaround is important,
>>then the low latency interconnect might be a winner.  But for those
>>of us (and Sanger is an example) where throughput is paramount, and
>>individual job turnaround less so, the embarrassingly parallel single-
>>threaded job approach is king; in such scenarios low latency
>>interconnects cost a lot of money for relatively little gain, and
>>increasing node count gives a better return.
> 
> 
> 
> So now its my turn - respectfully (and at the risk of sounding ridiculously
> naïve myself) - can you use the low latency interconnects (myrinet or
> Infiniband) for TCP or UDP bound services (like NFS or AFP)? My
> understanding was that they were only for things like MPI...?
> 

-- 
Joseph Landman, Ph.D
Founder and CEO
Scalable Informatics LLC,
email: landman at scalableinformatics.com
web  : http://www.scalableinformatics.com
phone: +1 734 786 8423
fax  : +1 734 786 8452
cell : +1 734 612 4615