10G nics not operating at 10G, why?

szboardstretcher · 06-25-2014, 10:17 AM

I have two computers, identical, with 10GB nics in them. I have them -directly- connected and addressed. Driver is installed. Ethtool says it is in 10000Mb/s mode full duplex.

But doing any transfers, I only get 129MB per second.

How would you do further troubleshooting on this?

Ethtool:

Code:

Settings for eth2:
        Supported ports: [ TP ]
        Supported link modes:   100baseT/Full
                                1000baseT/Full
                                10000baseT/Full
        Supports auto-negotiation: Yes
        Advertised link modes:  100baseT/Full
                                1000baseT/Full
                                10000baseT/Full
        Advertised pause frame use: No
        Advertised auto-negotiation: Yes
        Speed: 10000Mb/s
        Duplex: Full
        Port: Twisted Pair
        PHYAD: 0
        Transceiver: external
        Auto-negotiation: on
        MDI-X: Unknown
        Supports Wake-on: d
        Wake-on: d
        Current message level: 0x00000007 (7)
        Link detected: yes

rsync:

Code:

sending incremental file list
testing.file
      5536.35M  75%   128.77MB/s    0:02:36

dd write test:

Code:

dd if=/dev/zero of=/home/caldera/testing.file bs=50k count=50000
50000+0 records in
50000+0 records out
2560000000 bytes (2.6 GB) copied, 20.5084 s, 125 MB/s

hdparm:

Code:

 hdparm -Tt /dev/sda

/dev/sda:
 Timing cached reads:   13676 MB in  2.00 seconds = 6842.34 MB/sec
 Timing buffered disk reads: 520 MB in  3.00 seconds = 173.33 MB/sec

business_kid · 06-25-2014, 10:31 AM

If they're directly connected, are you using a crossover cable? What's the distance between them? Is the cable rated for 10G? I'd like to be in optic fibre at that speed.

TobiSGD · 06-25-2014, 10:32 AM

Seems to be pretty obvious that you are limited by your disk.

szboardstretcher · 06-25-2014, 10:35 AM

Crossover and regular.

5 Feet.

Cat 6e and Cat 5 tested.

The cards are Ethernet, fibre isnt an option unfortunately.

Quote:

ixgbe 0000:01:00.0: eth2: NIC Link is Up 10 Gbps, Flow Control: RX/TX
kernel/drivers/net/ixgbe/ixgbe.ko

Thats the driver being used.

jefro · 06-25-2014, 03:20 PM

It shouldn't make any difference patch or crossover. All modern nic's correct for cable. If ethtool reports it is in 10G then it is. It is possible that a bad cable or nic or connector could cause a bad quality of service issue.
Cat 5 won't do usually.

It would have to be a friggin monster of a system with a raid ssd to offer 10G speeds I'd think. You'd have to have an equal system on other end also.

One thing that may help is top. If you find cpu usage is way high then see if checksum offload is enabled on both nic's.

We also may have a bit and byte issue here.

I'd create a contiguous binary file and transfer that for a test.

You could possibly make a loop back cable and test it on a single system but again you'd have to have a double huge system.

See what top says on this transfer on both sides.

szboardstretcher · 06-25-2014, 03:35 PM

Thanks guys. The bottleneck is the drive.

I can't get the drive to write faster than 140MB/s, and others have suggested that 10G is best used between a computer and a SAN or other array capable of writing as fast as the card.

jefro · 06-25-2014, 09:42 PM

I don't get much more than about half or less of a 1G nic in general systems.

In some weak systems using fast nic's also eat up cpu time. There is always a lot of chatter on nics and the cpu may need to work more on the now fast nic.