[Bioclusters] Ethernet Performance

[Chen Peng]

--Apple-Mail-13--286395712
Content-Transfer-Encoding: 7bit
Content-Type: text/plain;
	charset=US-ASCII;
	format=flowed


--
Chen Peng <chenpeng@tll.org.sg>
Senior System Engineer
Temasek Life Sciences Laboratory
On 13-Jul-04, at AM 11:11, Joe Landman wrote:

> PCP and similar codes build tree structures out of their connections.  
> Each node in the tree has an incoming connection to its parent, and 
> outgoing connections (2 or possibly more) to neighbors.  Each bucket 
> (not packet, but container of data), is moved along the tree, stored 
> at a node, and retransmitted to its leaves (if any).  This code and 
> similar codes effectively diffuse the data to the edges of the tree.
>

That's exactly what we did for our cluster. However, we made it at at 
higher leverl with rsync. Basically when a node finishes 
synchronization, it can serve others as the golden copy. We automated 
the process and paralleled rysnc across the cluster. For our 64 nodes 
cluster, it generally speeds up by 800% - 900%.

As  you have explained, there is a limitation set by the switch. We are 
using gigabit ethernet and one 700 MB file can be "parallel sync-ed" to 
the 64-node cluster in 10-12 minutes, where the average speed is 
70-80MB/sec.


> If you measure the total amount of data moved to the nodes of the 
> network, and divide by the total transfer (or diffusion) time, you 
> will get the transfer rate.  This rate increases as the size of the 
> network increases.  At some point, the rate of data transfer may 
> become comparible to the switch backplane bandwidth( the amount of 
> data you can push through the switch per unit time).
> Joe

--Apple-Mail-13--286395712
Content-Transfer-Encoding: 7bit
Content-Type: text/enriched;
	charset=US-ASCII



<fixed>--

Chen Peng <<chenpeng@tll.org.sg> 

Senior System Engineer

Temasek Life Sciences Laboratory</fixed>

On 13-Jul-04, at AM 11:11, Joe Landman wrote:


<excerpt>PCP and similar codes build tree structures out of their
connections.  Each node in the tree has an incoming connection to its
parent, and outgoing connections (2 or possibly more) to neighbors. 
Each bucket (not packet, but container of data), is moved along the
tree, stored at a node, and retransmitted to its leaves (if any). 
This code and similar codes effectively diffuse the data to the edges
of the tree.


</excerpt>

That's exactly what we did for our cluster. However, we made it at at
higher leverl with rsync. Basically when a node finishes
synchronization, it can serve others as the golden copy. We automated
the process and paralleled rysnc across the cluster. For our 64 nodes
cluster, it generally speeds up by 800% - 900%.


As  you have explained, there is a limitation set by the switch. We
are using gigabit ethernet and one 700 MB file can be "parallel
sync-ed" to the 64-node cluster in 10-12 minutes, where the average
speed is 70-80MB/sec. 



<excerpt>If you measure the total amount of data moved to the nodes of
the network, and divide by the total transfer (or diffusion) time, you
will get the transfer rate.  This rate increases as the size of the
network increases.  At some point, the rate of data transfer may
become comparible to the switch backplane bandwidth( the amount of
data you can push through the switch per unit time). 

Joe

</excerpt>
--Apple-Mail-13--286395712--