[Bioclusters] Re: Rsync and NCBI and bio-mirror.net

[Steve O]

Hi,
After messing around for a while trying to optimize the ncbi downloads, 
I realized independent of the rsync costs that simply FTPing the 
formatted files was the best bet.  Even that was a bit tricky since the 
files might change while you're downloading them.  I no longer have to 
deal with this problem, but what I wished at the time was that some 
brave soul, perhaps bio-mirror, would offer a bit torrent of, say, 
Monday's snapshot of a file.  Then every subsequent day, a bit torrent 
would be offered of the diffs of the unpacked file vs. the previous 
unpacked version.  Sites that wished to keep up with daily changes could 
apply the patches, while less critical applications could just get the 
weekly distribution.  Running a bit torrent client would require server 
resources, but the actual load on your servers should be reduced 
significantly.

-steve