Perhaps a lot of this will not apply to your situation, but some will.

Here are some of my checklist items for when you are presented with assertions of poor network performance, in no particular order:

• Is *any one* CPU on either end of the transfer at or close to 100% utilization?  A given TCP connection cannot really take advantage of more than the services of a single core in the system, so average CPU utilization being low does not a priori mean things are OK.


• Are there TCP retransmissions being registered in netstat statistics on the sending system?  Take a snapshot of netstat -s -t from just before the transfer and one from just after, and run it through beforeafter from ftp://ftp.cup.hp.com/dist/networking/tools:
      netstat -s -t > before
      transfer or wait 60 or so seconds if the transfer was already going
      netstat -s -t > after
      beforeafter before after > delta
  


• Are there packet drops registered in ethtool -S statistics on either side of the transfer?  Take snapshots in a manner similar to that with netstat.


• Are there packet drops registered in the stats for the switch(es) being traversed by the transfer?  These would be retrieved via switch-specific means.


• What is the latency between the two end points?  Install netperf on both sides, start netserver on one side and on the other side run:
      netperf -t TCP_RR -l 30 -H
  and invert the transaction/s rate to get the RTT latency.  There are caveats involving NIC interrupt coalescing settings defaulting in favor of throughput/CPU utilization over latency:
  ftp://ftp.cup.hp.com/dist/networking/briefs/nic_latency_vs_tput.txt
  but when the connections are over a WAN, latency is important and may not be clouded as much by NIC settings.

This all leads into:

• What is the *effective* TCP (or other) window size for the connection?  One limit to the performance of a TCP bulk transfer is:
      Tput <= W(eff)/RTT
  
  The effective window size will be the lesser of:
    a) the classic TCP window advertised by the receiver (the value in the TCP header's window field shifted by the window scaling factor exchanged during connection establishment (why one wants to get traces including the connection establishment...)
  
      this will depend on whether/what the receiving application has requested via a setsockopt(SO_RCVBUF) call and the sysctl limits set in the OS.  If the application does not call setsockopt(SO_RCVBUF) then the Linux stack will "autotune" the advertised window based on other sysctl limits in the OS.
  
    b) the computed congestion window on the sender - this will be affected by the packet loss rate over the connection, hence the interest in the netstat and ethtool stats.
  
    c) the quantity of data to which the sending TCP can maintain a reference while waiting for it to be ACKnowledged by the receiver - this will be akin to the classic TCP window case above, but on the sending side, and concerning setsockopt(SO_SNDBUF) and sysctl settings.
  
    d) the quantity of data the sending application is willing/able to send at any one time before waiting for some sort of application-level acknowledgement.  FTP and rcp will just blast all the data of the file into the socket as fast as the socket will take it.  scp has some application-layer "windowing" which may cause it to put less data out onto the connection than TCP might otherwise have permitted.  NFS has the maximum number of outstanding requests it will allow at one time acting as a defacto "window" etc etc etc

  Copyright © 2010 rick jones