[Bioclusters] what are people using to mirror large ftp repositories

Jeremy Mann bioclusters@bioinformatics.org
Wed, 10 Sep 2003 15:30:49 -0500 (CDT)

> Hi folks,
> I've turned a bunch of Seagate 160gb IDE disks into a large software 
> RAID5 volume and am trying to mirror the raw fasta data from 
> ftp://ftp.ncbi.nlm.nih.gov/blast/db/FASTA/ for use on a personal 
> development project.
> The 'wget' utility (Redhat 9 on a dual PIII system w/ ext3 filesystem) 
> is bombing out on a few of the remote files which even when compressed 
> are greater than 2gb in size.
> My kernel and ext3 filesystem support large filesizes but 'wget' or my 
> shell seem to have issues.

Ah yes, I spent a few minutes on getting ours to work. ncftpget is what 
you want to use. Support +2gig files. Command usage example:

ncftpget -d download.log ftp://ftp.ncbi.nlm.nih.gov/blast/db/FASTA/igSeqNt.Z
ncftpget -d download.log ftp://ftp.ncbi.nlm.nih.gov/blast/db/FASTA/igSeqProt.Z

Jeremy Mann
University of Texas Health Science Center
Bioinformatics Core Facility
(210) 567-2672