Odd TCP Behavior

Rawjoe · 12-14-2006, 08:50 AM

I have a TCP connection from a Linux server that is acting weird, unexpected. Basically it seems to be waiting for an ACK from the client before continuing the stream, maybe like it's in congestion control algorithm. But nothing leads me to believe it should be avoiding congestion, there were no TCP Retransmissions. Below is my wireshark capture (modified a little to protect the innocent). Maybe I just don't have enough of an understanding of how TCP works, but I will explain what I am thinking as I go along.

Code:

//command sent from client to server
13 *REF*       client   server   2459 > 12345 [PSH, ACK] Seq=96 Ack=270 Win=65265 Len=39
//data as a result of command from server to client
14 0.028881    server   client   4990 > 54321 [PSH, ACK] Seq=0 Ack=0 Win=730 Len=1072
15 0.030718    server   client   4990 > 54321 [ACK] Seq=1072 Ack=0 Win=730 Len=1460
//windows client ACKs every other packet
16 0.030741    client   server   54321 > 4990 [ACK] Seq=0 Ack=2532 Win=65535 Len=0
//more data from server
17 0.031026    server   client   4990 > 54321 [PSH, ACK] Seq=2532 Ack=0 Win=730 Len=684
18 0.032650    server   client   4990 > 54321 [ACK] Seq=3216 Ack=0 Win=730 Len=1460
//windows every other ack
19 0.032670    client   server   54321 > 4990 [ACK] Seq=0 Ack=4676 Win=65535 Len=0
//more data from server
20 0.032932    server   client   4990 > 54321 [PSH, ACK] Seq=4676 Ack=0 Win=730 Len=684
21 0.034538    server   client   4990 > 54321 [ACK] Seq=5360 Ack=0 Win=730 Len=1460
//windows every other ack
22 0.034557    client   server   54321 > 4990 [ACK] Seq=0 Ack=6820 Win=65535 Len=0
//more data from server
23 0.034818    server   client   4990 > 54321 [PSH, ACK] Seq=6820 Ack=0 Win=730 Len=684
24 0.036340    server   client   4990 > 54321 [ACK] Seq=7504 Ack=0 Win=730 Len=1460
//windows every other ack
25 0.036362    client   server   54321 > 4990 [ACK] Seq=0 Ack=8964 Win=65535 Len=0
//data from server, plus server finally acks command sent from client
26 0.036625    server   client   4990 > 54321 [PSH, ACK] Seq=8964 Ack=0 Win=730 Len=684
27 0.039278    server   client   12345 > 2459 [ACK] Seq=270 Ack=135 Win=6651 Len=0
//a whole 120 ms later, the ack from windows comes in, makes sense because there weren't two packets
//on the stream to force an immediate ack from windows
28 0.158178    client   server   54321 > 4990 [ACK] Seq=0 Ack=9648 Win=64851 Len=0
//the last bit of data from the server.  this data i thought would be sent back up with the other data
//the client reported a big enough window to receive the data, so why did it wait for an ack to send
29 0.158567    server   client   4990 > 54321 [PSH, ACK] Seq=9648 Ack=0 Win=730 Len=1072

Debug output on the server shows that all the data was written to the socket within 8ms, so why does it take 130ms to get to the client? I've tried all the fixes I've seen on the web, turning off Nagle, changing the congestion algorithm, turning off SACK, increasing tcp_wmem, but nothing seems to work.

I don't know if this will help, but my response to the command is 10 different socket writes each 1072 in length. I've seen it on the 2.6.17 and the 2.6.15 kernels. I appreciate anyones help and understanding with this.

Rawjoe · 12-14-2006, 10:53 AM

Ok, maybe I wasn't actually turning off Nagle. I turned off Nagle with another command and it seems to work as expected. Can anyone tell me the difference between:

old way: setsockopt(respsock, IPPROTO_TCP, TCP_NODELAY, (char*) &flag, sizeof(flag));
new way: setsockopt(respsock, SOL_TCP, TCP_NODELAY, (char*) &flag, sizeof(flag));

I can't seem to find the difference between IPPROTO_TCP and SOL_TCP as levels in the setsockopt function. Both values are defined on my machine.