[Bioclusters] Re: Rsync and NCBI and bio-mirror.net

[Lee Watkins]

Aaron Darling <darling at cs.wisc.edu> wrote:

>Anyways, my point is that it would be great if NCBI would set up a bittorent server themselves, but I wouldn't hold my breath waiting for their leadership on the issue.
>

People interested in high-performance file transfer might check out 
multicast data distribution tools from Micah Beck's "Logistical 
Computing and Internetworking Lab"  http://loci.cs.utk.edu/ 
<http://loci.cs.utk.edu/projects/index.php>  Collectively they call it a 
"Logistical Distribution Network" (LoDN - "lowdown").  I've seen demos 
at Internet2 and Supercomputing meetings (mainly, like BitTorrent, for 
sucking down huge digital video files) and it's quite impressive.  Data 
streams from multiple storage sites ("depots") simultaneously, 
aggregated and presented to the client as a single "exNode" ... 
basically it treats high-speed internet connections as a virtual 
backplane using a specialized protocol/middleware.   There is no single 
point-of-failure and no one site has to bear the entire burden of 
providing data resources.  It's sort of the same mentality as the LOCKSS 
project in the digital library/archival community -- "Lots Of Copies 
Keeps Stuff Safe" -- except oriented towards highly dynamic rather than 
archival data, so for instance there are mechanisms to keep data 
current.  And unlike bittorrent and other p2p content sharing services, 
control is retained over data stored on depots. Whether it would work 
well for our purposes or NIH/NCBI would even go for something like this 
is uncertain, but the NSF and US Dept. of Energy have been interested 
enough to fund it for several years.

-- 

Lee Watkins, Jr. 
Director of Bioinformatics 
Center for Inherited Disease Research (CIDR) 
Institute of Genetic Medicine 
Johns Hopkins Bayview Research Campus 
333 Cassell Drive, Triad Bldg. Suite 2000 
Baltimore, MD 21224

lwatkins at jhu.edu | www.cidr.jhmi.edu 
410-550-7042 ofc | 410-903-2989 cell 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://bioinformatics.org/pipermail/bioclusters/attachments/20060202/e011e0ff/attachment.html