Home » Forums » MythDora » General Discussion

Network Speed Question

Submitted by liels on May 19, 2009 - 4:26am

This is not specifically an MD issue.

I have an MD be with 4xSegate 1.5 TB drives that will read/write at 400+ MB/s (yes, MB not Mb) and a workstation with the same array used to back it up from time to time. I've been backing up using rsync over gigabit Ethernet (125 MB/s raw nominal, less overhead) and have been disappointed with the throughput (about 45-50 MB/s sustained). STFW suggests the bottleneck is the lane connecting the onboard nics (broadcom IIRC) are attached to the chipset. But I'm not 100% sure of that.

My thought is that for $40 each I can pick up an Intel PCI-e CT NIC (PCI-e x1 nominally 250 MB/s)

http://www.amazon.com/Intel-EXPI9301CT-Gigabit-Desktop-Adapter/dp/B001CXWWBE/ref=sr_1_1?ie=UTF8&s=electronics&qid=1242732084&sr=8-1

and configure it for 9k Jumbo frames over the DIR-655 switch.

Do any of you have thoughts or experience on this? If I can cut my backup time in half I'd be happy.

Thanks

‹ Firewire Recording Feature for Mythdora 11... ›

What are you using to

Submitted by pisani on May 19, 2009 - 1:47pm.

What are you using to benchmark your disk performance? Is that a hardware raid card employed there? Is the 40-45MB/s your rsync performance? Is there a lot of smaller files?

»

40-50MB is what rsync over

Submitted by liels on May 20, 2009 - 3:22am.

40-50MB is what rsync over ssh is reporting as sustained for large files. The small files are not numerous enough to make a big difference.

No hardware raid software raid 0; The disks are supposed to have between 70 and 120 MB/s each and the SATA channels are supposedly 3 Gb/s each(!) to the south bridge, 2 GB/s north bridge to south bridge and something like 6 GB/s north bridge to memory. So I believe 460 MB/s array performance.

Note that when I was using JFS I didn't quite get this level of performance ... XFS has been much better for me in that regard. (I followed the advice on http://www.mythtv.org/wiki/Optimizing_Performance w/r/t xfs optimizations)

Throughput measured with dd if=/dev/zero:

/usr/bin/time -f %E -o chunk.txt bash -c 'dd if=/dev/zero of=/storage/bigfile bs=1M count=10240; sync'
10240+0 records in
10240+0 records out
10737418240 bytes (11 GB) copied, 25.1968 s, 426 MB/s
[mythtv@mythbe storage]$ cat chunk.txt
0:26.45

This is with the disk half full ... it was more like 460 MB/s when empty.

»

Wow... I'm quite impressed

Submitted by pisani on May 20, 2009 - 6:15am.

Wow... I'm quite impressed with the software raid performance. I guess RAID0 is the ticket.. I didn't think sustained throughput would be that high..

What is the target disk that you're rsyncing from? Is it a single disk? Have you tested the read performance there? If it's a single disk the 45 MB/s is pretty typical for reads probably.

For example my laptop:

[root@arp089-laptop moto-ubuntu]# hdparm -t /dev/sda

/dev/sda:
Timing buffered disk reads: 170 MB in 3.03 seconds = 56.17 MB/sec

»

I'm backing up to an

Submitted by liels on May 20, 2009 - 7:29am.

I'm backing up to an identical array (although I've got JFS on it at the moment).

I've become a big fan of RAID-0

My first try at mythtv invoved 4x320GB seagates in a RAID 5 with XFS. The performance was terrible (I probably had sub-optimal raid chunk / xfs parameters and was probably doing 2 reads and 2 writes per write ...). RAID-0 was worlds better ...

I also then came to the realization that raid isn't backup and wouldn't save me from something like

sudo rm -rf /* ~

or a mistake in an rsync script ....

The only thing raid-5 would buy me is a little extra availability but so far I've not needed it for that (I have had to swap a drive but I saw it going bad by watching smartctl which I also recommend).

After STFW I'm pretty sure that the network is my bottleneck. What I'm not sure of is if the intel nics & jumbo frames will open it up to more like 80-100MB/s. I suppose I should use pcattcp to find out for sure before I buy anything. Still those intel nics are relatively inexpensive ...

For completeness, here's what I did to get my performance:

$sudo mdadm --create --verbose /dev/md0 --level=0 --raid-devices=4 /dev/sdb /dev/sdc /dev/sdd /dev/sde
$sudo mdadm --examine /dev/md0
$sudo mdadm --detail --scan
$sudo mdadm --detail --scan > mdadm.conf
$sudo cp mdadm.conf /etc/mdadm.conf 

$sudo mkfs.xfs -d -sunit=128,swidth=512 /dev/md0

$sudo /sbin/blockdev --setra 65536 /dev/md0


$grep xfs /etc/fstab
/dev/md0		/storage		xfs	defaults,noatime,nodiratime,nosuid,nodev,allocsize=512m,logbufs=8 0 0

$sudo mount /storage

$/usr/bin/time -f %E -o chunk.txt bash -c 'dd if=/dev/zero of=/storage/bigfile bs=1M count=10240; sync'

$cat /etc/rc.d/rc.local
#!/bin/bash
# This file allows you to execute things at the end of boot
touch /var/lock/subsys/local
/sbin/mke2fs /dev/ram0
/bin/mount -t ext2 /dev/ram0 /swapram
/bin/dd if=/dev/zero of=/swapram/sw bs=1024
/sbin/mkswap /swapram/sw
/bin/chmod 0600 /swapram/sw
/sbin/swapon /swapram/sw
/sbin/blockdev --setra 65536 /dev/md0

»