<br><font size=2 face="sans-serif">We have many databases to distribute so I have written a few scripts to benchmark different distribution methods, udpcast seems to work best for us currently. I have a script that checks the size of each file in the database directory on each node, and them makes a list of files that need to be udpcast-ed from the master copy on the distribution node and to which nodes to cast the file to. It then starts up a listener on the appropriate nodes for each file and sends it out. Since UDP cast is slow for a smaller number of clients, it also checks to see how many clients need the file, and if it is smaller than your set break point, it uses NFS (could be a command line switch to rsync) to distribute the file at the same time as the udpcast is going on. There is also a setpoint for the filesize to decide whether to use udpcast or NFS.</font>
<br>
<br><font size=2 face="sans-serif">We use filesize as an indicator as it takes 4 hours just to do the checksum on all our files each night, and this time will be growing with our databases. This could be easily made a command line switch to choose what method to use. udpcast has different data checking in it's protocol to cover for UDP.</font>
<br>
<br><font size=2 face="sans-serif">I have also written a script that uses a treed rsync to distribute the data, but rsync was using way too much overhead with the size of our databases, and these will be growing.</font>
<br>
<br><font size=2 face="sans-serif">I was planning on updating the script to do checksums weekly, but I found a problem with our kernel I had to solve first. I will be starting to develop the scripts again this coming week. </font>
<br>
<br><font size=2 face="sans-serif">Is anybody interested in such a project?</font>
<br>
<br><font size=2 face="sans-serif">Well I am out of wind,</font>
<br>
<br><font size=2 face="sans-serif">Jason</font>
<br>
<br>
<br>
<table width=100%>
<tr valign=top>
<td>
<td><font size=1 face="sans-serif"><b>Jan van Haarst <jvhaarst@gmail.com></b></font>
<br><font size=1 face="sans-serif">Sent by: bioclusters-bounces+jason.calvert=pharma.novartis.com@bioinformatics.org</font>
<p><font size=1 face="sans-serif">04/30/2005 03:55 AM</font>
<br><font size=1 face="sans-serif">Please respond to jan; Please respond to "Clustering, compute farming & distributed computing in life science informatics"</font>
<br>
<td><font size=1 face="Arial"> </font>
<br><font size=1 face="sans-serif"> To: jeremy@biochem.uthscsa.edu, "Clustering, compute farming & distributed computing in life science informatics" <bioclusters@bioinformatics.org></font>
<br><font size=1 face="sans-serif"> cc: (bcc: Jason Calvert/PH/Novartis)</font>
<br><font size=1 face="sans-serif"> Subject: Re: [Bioclusters] NCBI updates and how you do them</font></table>
<br>
<br>
<br><font size=3 face="Times New Roman">Hi,</font>
<br><font size=3 face="Times New Roman"> </font>
<br><font size=3 face="Times New Roman">On our cluster we use UDPcast ( </font><a href=http://udpcast.linux.lu/><font size=3 color=blue face="Times New Roman"><u>http://udpcast.linux.lu/</u></font></a><font size=3 face="Times New Roman"> ) to push the data to the nodes, and rsync afterwards to double check the transfer.</font>
<br><font size=3 face="Times New Roman">The way I understood it, rsync and the (non FASTA) blast databases don't work well together, you end up sending the complete database through rsync, which isn't the best solution if you want to push data to a lot of nodes at the same time. </font>
<br><font size=3 face="Times New Roman">But maybe that isn't the case anymore, what do you see when you update the database through rsync ?<br>
</font>
<br><font size=3 face="Times New Roman">UDPcast works by broadcasting the data to the nodes, on which listeners pick up the data. </font>
<br><font size=3 face="Times New Roman">There are other ways to distribute data form one to many, but UDPcast works fine for us.</font>
<br><font size=3 face="Times New Roman"> </font>
<br><font size=3 face="Times New Roman"> </font>
<br><font size=3 face="Times New Roman">Kind regards,</font>
<br><font size=3 face="Times New Roman">Jan<br>
</font>
<br><font size=3 face="Times New Roman">2005/4/26, Jeremy Mann </font><a href=mailto:jeremy@biochem.uthscsa.edu><font size=3 color=blue face="Times New Roman"><u>jeremy@biochem.uthscsa.edu</u></font></a><font size=3 face="Times New Roman">: </font>
<br><font size=3 face="Times New Roman"><br>
Is rsync the way to push to all nodes? If not, what other alternatives exist?<br>
<br>
--<br>
Jeremy Mann </font><font size=3 color=blue face="Times New Roman"><u><br>
</u></font><a href=mailto:jeremy@biochem.uthscsa.edu><font size=3 color=blue face="Times New Roman"><u>jeremy@biochem.uthscsa.edu</u></font></a><font size=2 face="Courier New">_______________________________________________<br>
Bioclusters maillist - Bioclusters@bioinformatics.org<br>
https://bioinformatics.org/mailman/listinfo/bioclusters<br>
</font>
<br>
<br>