[Bioclusters] gigabit ethernet performance

[Chen Peng]

--Apple-Mail-3--416262076
Content-Transfer-Encoding: 7bit
Content-Type: text/plain;
	charset=US-ASCII;
	format=flowed

Hi all,

Thank all of you for valuable suggestions and being involved in the 
discussion.

We have finally figured out where is the bottleneck for our gigabit 
ethernet. To put it simple, the FTP is limited to 12MB-13MB/s is 
because of disk IO. Our main FTP server got some file system problem 
and disk is getting slower than it should be. After fixing the disk 
problem, the FTP speed restores to be around 20MB/s.

Surprisingly, the SCP performance is heavily bound to CPU power. In the 
following table we compared performance among four different machines. 
Note that this is NOT benchmark, we use this table only to understand 
where is the bottleneck.

All the operation is against a 100MB file.

Model           Powerbook       Xserve G4       IBM X440        Sun 
Fire 280R
CPU             1GHZ G4         2x1.25G G4      4x2.0G Xeon     1.2GHZ 
SPARC
Disk            IDE 4200rps     ATA-133 IDE     SISC-RAID 1     SCSI

Copy            5.710 sec       2.523 sec       4.390 sec       3.187 
sec
Speed           17.51 MB/s      39.63 MB/s      22.78 MB/s      31.38 
MB/s

SCP             17.575 sec      10.135 sec      8.787 sec       26.680 
sec
Speed           5.690 MB/s      9.868 MB/s      11.37 MB/s      3.748 
MB/s

FTP             6.586 sec       5.288 sec       ---             ---
Speed           15.18 MB/s      18.91 MB/s

Copy            cp ./dummy.100M ./dummy.100M.2
SCP             scp *******:/tmp/dummy.100M ./dummy.100M.2
FTP             ncftpget -u*** -p*** ftp://******/dummy.100M

It is clear that disk IO is not the bottleneck for all the models. The 
more powerful is the CPU, the better is the scp performance. In 
addition, we compared FTP performance. As FTP involves much less CPU 
workload, the speed instantly boosts to 19MB/s, while SCP is only 
10MB/s for the same Xserve G4.

Therefore, to tune the network performance, we should focus on each 
connection point instead of only looking at the switch or NIC. CPU and 
HD speed is also important. Traffic from machine A to B involves:

       (1)        (2)        (3)        (4)
A(HD) --> A(NIC) --> SWITCH --> B(NIC) --> B(HD)

All connection from 1 to 4 need to be checked and benchmarked carefully.

Cheers,
--
Chen Peng <chenpeng@tll.org.sg>
Senior System Engineer
Temasek Life Sciences Laboratory

--Apple-Mail-3--416262076
Content-Transfer-Encoding: 7bit
Content-Type: text/enriched;
	charset=US-ASCII

<fontfamily><param>Optima</param><bigger>Hi all,


Thank all of you for valuable suggestions and being involved in the
discussion.


We have finally figured out where is the bottleneck for our gigabit
ethernet. To put it simple, the FTP is limited to 12MB-13MB/s is
because of disk IO. Our main FTP server got some file system problem
and disk is getting slower than it should be. After fixing the disk
problem, the FTP speed restores to be around 20MB/s.


Surprisingly, the SCP performance is heavily bound to CPU power. In
the following table we compared performance among four different
machines. Note that this is NOT benchmark, we use this table only to
understand where is the bottleneck.


All the operation is against a 100MB file.

</bigger></fontfamily><fixed><color><param>3332,3332,3332</param><x-tad-bigger>

Model           Powerbook       Xserve G4       IBM X440        Sun
Fire 280R

CPU             1GHZ G4         2x1.25G G4      4x2.0G Xeon     1.2GHZ
SPARC

Disk            IDE 4200rps     ATA-133 IDE     SISC-RAID 1     SCSI</x-tad-bigger></color><x-tad-bigger>

</x-tad-bigger><color><param>3332,3332,3332</param><x-tad-bigger>

Copy            5.710 sec       2.523 sec       4.390 sec       3.187
sec

Speed           17.51 MB/s      39.63 MB/s      22.78 MB/s      31.38
MB/s</x-tad-bigger></color><x-tad-bigger>

</x-tad-bigger><color><param>3332,3332,3332</param><x-tad-bigger>

SCP             17.575 sec      10.135 sec      8.787 sec       26.680
sec

Speed           5.690 MB/s      9.868 MB/s      11.37 MB/s      3.748
MB/s</x-tad-bigger></color><x-tad-bigger>

</x-tad-bigger><color><param>3332,3332,3332</param><x-tad-bigger>

FTP             6.586 sec       5.288 sec       ---             ---

Speed           15.18 MB/s      18.91 MB/s</x-tad-bigger></color><x-tad-bigger>

</x-tad-bigger><color><param>3332,3332,3332</param><x-tad-bigger>

Copy            cp ./dummy.100M ./dummy.100M.2

SCP             scp *******:/tmp/dummy.100M ./dummy.100M.2

FTP             ncftpget -u*** -p*** ftp://******/dummy.100M</x-tad-bigger></color><x-tad-bigger>


</x-tad-bigger></fixed><fontfamily><param>Optima</param><bigger>It is
clear that disk IO is not the bottleneck for all the models. The more
powerful is the CPU, the better is the scp performance. In addition,
we compared FTP performance. As FTP involves much less CPU workload,
the speed instantly boosts to 19MB/s, while SCP is only 10MB/s for the
same Xserve G4.


Therefore, to tune the network performance, we should focus on each
connection point instead of only looking at the switch or NIC. CPU and
HD speed is also important. Traffic from machine A to B involves:

</bigger></fontfamily><fixed><color><param>3332,3332,3332</param><x-tad-bigger>

      (1)        (2)        (3)        (4)

A(HD) --> A(NIC) --> SWITCH --> B(NIC) --> B(HD)</x-tad-bigger></color><x-tad-bigger>


</x-tad-bigger></fixed><fontfamily><param>Optima</param><bigger>All
connection from 1 to 4 need to be checked and benchmarked carefully. 


Cheers,

</bigger></fontfamily><fixed>--

Chen Peng <<chenpeng@tll.org.sg> 

Senior System Engineer

Temasek Life Sciences Laboratory</fixed>


--Apple-Mail-3--416262076--