Context

I was recently at a customer who just installed an APS unit in their Data Center.

Their infrastructure was fairly simple:

• they had all their production servers in the data center,
• their users were located either in the headquarters (where the data center is located) or in one of their nearly 100 remote sites; these sites were connected to the headquarters through an MPLS network.
• they are part of a larger group, whose data center is connected to the same MPLS cloud, that provides access to some central services like DNS, mail (Lotus Notes) and  access to a secured Internet gateway.

Illustration 1: Simplified network topology

Reminders

Retransmission

Packets being resent, after having been either lost or damaged.

Packet Retransmission is identified thanks to their TCP sequence and acknowledgement numbers, and checksum values. Only packets with a non-null payload are checked.

Retransmission Delay

Time Delay between a packet and the last retransmission.

RD stands for Retransmission Delay. RD is defined as the time between a packet and its last retransmission.

Illustration 2: Retransmission delay calculation

Retransmission Rate

Ratio of retransmitted packets to the total number of packets.

RR stands for Retransmission Rate. RR is defined as the ratio of retransmitted packets to the total number of packets in a conversation.

Illustration 2 : Retransmission Delay Calculation / Calcul du retard du à la retransmission

What we observed

The customer’s network manager was first referring to a complaint from an end user regarding a slow access from their remote site to an application located in their data center.

By looking at the application performance chart, we saw a significant amount of Retransmission Delay (from server to client) and no other change apart from a slight increase in the DTT (Data Transfer Time) for the application in question.

Illustration 3 : Example of Application Performance Chart / Exemple de graphique de performance d'application

We extended our scope of investigation by looking at the network performance chart for all applications (for clients located in the remote sites and servers in the data center): we  observed that the retransmission delay was high, regardless of which application was being used. Also, that the RD occurred mostly in the direction from Server to Client.

Illustration 4 : Example of Network Performance Chart / Exemple de graphique de performance réseau

I made the hypothesis that there might be some congestion between the data center and the remote site in the direction DC → remote site. So we looked at the bandwidth graph of the APS for the traffic between the data center and the remote site: we  observed a peak of traffic (mainly due to Windows SUS – which told me a lot about the lack of control of their network & system administration traffic flows) reaching 1.2Mbps.

The customer found the value of 1.2Mbps interesting, although not enough at first glance for him to be convinced that there was a congestion… because the bandwidth available on the remote site end was 2Mbps (and the maximum bandwidth available on the DC end was 80Mbps, and only a very low amount of this maximum bandwidth was being used).

So we decided to check on the SNMP graphs provided by their telecom operator for the remote site router… and the bandwidth graph was showing a flat line at 2Mbps for 30 minutes for the incoming traffic.

Conclusion on the meaning of retransmissions

Retransmissions are significant and you should have a look at retransmissions to determine:

• whether they are intermittent  or continuous
• what is the perimeter where you can observe them (for which client zone(s), for all servers or one)

What they tell us in the end is that some packets are not reaching the other hosts or that  acknowledgment packets are not getting back to the sender.

The direction of the retransmission (server → client or client → server) may not be so significant as a congestion in one way may induce retransmission in both ways (for example, congestion from server to client would generate some retransmissions from server to client – the packets sent by the server do not get acknowledged fast enough and the server retransmits them- and from the client to the server – because even though the packets from the client to the server reach the server fairly fast, the acknowledgment packets from the server to the client suffer from the congestion and the client retransmits the original packets). You should also keep in mind that the balance of retransmission between client → server and server → client also depends on the balance of traffic between both directions.

Conclusion on the impact of the retransmission delay on the User Experience

In 2 ways Retransmission Delay is a good indicator of the impact of network degradation on the Quality of Experience of network users:

• Retransmission delay in one way is driven by the quantity of data sent in that direction; this value corresponds to the additional time required for a user to get all their data.
• Retransmission has a secondary consequence on the time required to receive  data: when there is a retransmission, the host resets its TCP window and the size of the buffer to its minimum default size. This means that each time there is a retransmission, the throughput is going back to a very low level and then start increasing again progressively. If retransmissions are frequent, then the throughput often goes back to a minimum level  and never reaches an optimal level. This means a much larger Data Transfer Time, because the throughput to transfer the applicative response remains very low. This phenomenon is what is usually called a TCP Slow-Start (see http://en.wikipedia.org/wiki/Slow-start)