Previous

Content

Next 

1.0.- One TCP flow  

Warning: this page has a bundle of png and gif pictures; it will last a little if you have an slow Internet connection.
The best way to initiate this study is to have a look to how TCP works when it is alone and happy, having for it all the network resources and not having, yet, some other TCP or UDP flows to compete with them. We will present first the simple topology where these experiments will be advanced. It is a simple link connecting two hosts.

 
In both hosts we have TCP. Host A has an FTP source that always has a packet to send, as soon as the network can accept it. The packets are 1000-byte size. Host B is in charge only to acknowledging the packets it receives. Ack packets are 40-byte size. The link is a 2 Mbps link having a propagation delay of 10 milliseconds. The buffer size is 20 packets, then our queue can have a maximum of 20 packets. The queue is DropTail (just a FIFO queue). Most of the actual routers and hosts in the Internet use this kind of queue. The TCP implementation is Reno.
 
The tcl code to feed the ns-2 simulator is in tcp-1flow.tcl.
 
At time 0.1 seconds the FTP source in A starts to send packets, and at time 10.0 seconds it stops to send packets.
 
To get information from the simulation we use, besides the ns-2 simulator, the awk program to parse the ns-2 output, and the gnuplot utility to plot the results.
 
For each test we are going to get the following information:
 
  1. Instant throughput in kbps
  2. System throughput in kbps
  3. Latency in ms
  4. Jitter in ms
  5. Bytes trasferred
  6. Losses
To filter noises we use EWMA (Exponential Weighted Moving Average) to smooth the outputs using α = 0.9, before ploting.
 
   

 
The first graph obtained from the test is instant throughput in kbps:

Very nice graph. TCP increase its rate softly during the first 0.5 seconds until reaching almost the link bandwitdh capacity of 2 Mbps. Then, the rate is maintained very steady. The maximun link capacity can't be reached because some resources are needed to transport the ack packets from the receiver B to the sender A. The network throughput is, more or less:

1000/1040 * 2000 kbps ~ 1923 kbps
Observe that no one is telling TCP how many packets it can send; it, very intelligently, adapts itself to the available link bandwidth and use as much of this resource as is possible.
Let's see now the system throughput. The instant throughput is calculated dividing the bytes of each packet (1000 bytes) by the time required to send the packet. The system throughput is calculated dividing the total bytes sent up to a given time by the total time transcurred in seconds. In fact, the system throughput is what really have importance because it says what is the power of your transmission, this means, how many bytes you are transfering by unit of time.

Okay, as expected, the system throughput increase even more slowly until reaching an steady state of almost 2 Mbps.
Now is time to see the latency. How much a packet last for going from the sender to the receiver.

After reaching the steady state the latency is a constant value of 72-73 ms. If we were to transport real-time traffic using this scenario (just one TCP flow), we could have used TCP as the transport protocol. Almost an steady state rate and an stable latency of 73 ms. A little more about real-time traffic is here where we talk about VoIP.
Let's see now the jitter, this means, how the latency change from packet to packet.

Well, as expected. At the beginning, some random values, but when the flow stabilizes, the jitter is just zero. Very nice quality to transport real-time traffic.
What about the total bytes transferred? ; here we have:

KB sent: 2421.289062
Calculating the system throughput (st) from this data we have:

st = 2421 KB * 8 bits/byte / 9.9 seconds = 1956 Kbps
As we are expecting. Finally let's see the losses. We calculate losses dividing the number of packets that are dropped by the total number of packets that are sent; this time, we have:

   Packets sent:  2385
Packets dropped:     0
   Total losses:     0.000000%
Losses are zero. TCP works as a champion protocol. It uses as much as is possible from the available link bandwidth, generating an steady state rate, low latency, zero jittering, and no losses.
Okay, all very beautiful, all very nice, all work fine. But, let's start to change the parameters of the test. Our first change will be to select another packet size. For example, what will be the result of our test if the packets are small packets of 100-byte size, instead of big packets of 1000-byte size?. Let's start having a look to the instant throughput for this new scenario:

Whep...  Even using EWMA to smooth the output we have some noise (an oscillation) around the 1800 kbps level. The oscillation must be very large because the smoothed output is a little wide. This effect should be noted (surely...) in the system throughput; let's see this graph:

Oh, my god... We lost almost one-half of the system throughput. What happened?
Well, our first answer would be: packets are now 100-byte size and ack packets are 40-byte size. The efficiency of the transmission has now to be lower. Let's have some estimation:

100/140 * 2000 kbps ~ 1428 kbps
But, even taking into account this de-rating our throughput is even more lower. Then, some part of the de-rating is due to the larger oscillation too. Smaller packets generate a more unstable flow and waste link bandwidth resource.
But, what will be the latency, now?

Fine, fine. At least we have a better latency, enough better. The packets are smaller and travel faster. This is good for real-time services. Don't forget that the time required by a packet to travel from one end to the other (having no queues) is B/R + P, where B is the packet size, and R and P the bandwith and propagation delay of the link. Our link is a 10ms link. Then, most of the latency in this case is caused by the propagation delay of the link being used.
Latency is very stable, then the jitter should be almost zero, let's see this:

Good, very good. No jittering at all. Now is the time to see what happen with the total bytes transferred; because of the poorer system throughput, the number of bytes transferred should be a lot less. Here we have:

KB sent: 1297.500000
It was really a very bad idea to reduce our packet size. TCP lost almost one-half of its capacity to transfer packets from A to B. Finally let's see the losses; what ocurred?

   Packets sent:  9491
Packets dropped:     0
   Total losses:     0.000000%
Again, despite we have now a bundle of packets, losses remain in zero.
Hey, guys... You have the graphs. The simulation are taken with a proved tool like the ns-2 simulator (using by expert people like Van Jacobson, Sally Floyd, Matthew Mathis and some others to make their tests). Then, have your own conclusions because I have mine. The goal of this document is to show you how TCP and UDP protocols behave when they are transporting packets through links. Having these ideas very clear in your mind I'm sure you are better prepared to have better decisions.
But, I can't resist the feel to tell you something: some people, without any solid arguments to support their mind tell me... no Leonardo, TCP is very slow, I prefer to use something faster like UDP or IPX/SPX, please, sit down here, I'm going to transfer this file from this computer to both (almost the same capacity) servers, the first using TCP and the next using IPX/SPX, have a look, did you see it? IPX/SPX is almost twice as faster...
Okay, but do not forget that you are transfering this file in an almost unloaded network; if the network is not congested, any aggresive protocol like UDP or IPX/SPX works faster than TCP, I must agree with this. But, when you have many flows competing by the network resource, and these flows are very aggresive and do not apply any congestion control mechanism, the fight is not won by any one; everyone is trying to send its packets as fast as it can without lending attention to the network condition; then you have big oscillations, flows can not reach higher throughput and everyone suffer. Then, you are doing exaclty the contrary what you want. It is like a congested cross-road with a damage traffic light and everyone trying to cross the first. What do you have? Don't believe me? Have a look to this graph taken by me from one of my customers, let's call them the company X;  they happen to have a 10 Mbps LAN.

They use IPX/SPX, TCP and UDP protocols in its network. In the graph TCP is the red one (they have an 1.5 Mbps ADSL connection with Internet); blue one is IPX/SPX and green one is UDP. Have a look to the big and fast oscillations generated by the flows sometime. And, do not forget that in this graph the outputs are smoothed using a factor α = 0.95. Even with this smoothing the oscillations are very high. Do you want to see the graph when no smoothing is done? Here you have:

How goes it? Everyone is oscillating. Do you know what are the consequences? Easy, a 10 Mbps network where the system throughput is much lower. Let me show you:

Did you see it? Data from the same sample taken in octuber 15, 2003 during 247 seconds. A network having peaks up to 9000 Kbps or even higher values, but having a system throughput of only 695 Kbps (less than one tenth of the peak values). Then, what's the matter with having very fast and aggresive protocols?
BTW, this samples, graphs and summaries were taken and made using a simple SuSE Linux box and the tools: tcpdump, awk and gnuplot. No need to spend a lot of money (using Cisco RMON, for example) to get very good information of what is going on with the network. I told you this before in the document Buying additional bandwidth is always the solution?  Have a look to this document; it clears you a lot your understanding when needing to make new purchases.
An additional observation: have a look to the number of TCP and UDP connections. Very high number having into account the number of bytes being transferred, aren't they? Do you know what is going on? They are using P2P connections (Kazaa, E-mule, etc.); some really very bad to the health of the network. Router and switch resources are very quickly exhausted for this kind of virus connections. I hope to talk a little more about this problem in the near future. An advance is here.
Well, having into account that a bigger packet size was good for our system throughput, why don't try now using a really big packet to see what happen; our next experiment will be to repeat the tests using a 4000-byte packet size. The first graph using the new packet size is the instant throughput:

Well, excluding a longer delay to reach the steady state condition, there isn't really much more difference with the 1000-byte packet size test. Let's have a look to the system throughput:

Same thing, more or less the same behavior. I note some smaller value for the steady state condition, but really I'm not sure. Checking later the number of packets transmitted we will be sure enough.
Let's see the latency:

Poow!!  Things are a lot harder here. Latency increased too much. Now it is over the 300ms threshold, something really very bad to transport real-time traffic. For bulk-data traffic it doesn't matter this delay. Even some service like telnet, ssh, and www will not be very happy having this latency. Fortunately, these services don't use these big size packets.
Nevertheless, the latency is very stable. Jitter should be almost zero; let's see this:

Yes, as expected. Initially, some random values to stabilize later on zero value. No problem by this side. Let's see the number of bytes transferred to check our first feel about the system throughput:

KB sent: 2481.640625
Well, really we were wrong. Bytes transferred is a little more than when using 1000-byte size packets. We have 60.35 KB more transferred for an increase of 2.4925%; is not too bad, is it? Using larger size packets we improve our system throughput.
Finally, let's check what about losses; here we have:

   Packets sent:   630
Packets dropped:     0
   Total losses:     0.000000%
Great, less packets but losses remain in zero. Well, fellows..  Increasing the packet size we got a little increase in the system throughput but the latency was seriously damaged. Then, it's just a matter to evaluate your needs to select the right choice for you.
Following our investigation we will try to see now what happen when the link bandwidth is reduced. How TCP will react? Best way to know this is to adjusting the link bandwidth to 1 Mbps instead of the original value of 2 Mbps. The propagation delay will be kept in 10ms. Here we have the instant throughput for this case:

Nice. The curve is very similar to the original one for a 2 Mbps link, but this time TCP adjusts its throughput to a value a little bit less than 1000 Kbps. Again, no one is telling TCP what the maximum link capacity is, but it adjusts itself to the environment conditions. Really, it doesn't matter because it couldn't deliver more bandwidth than the maximum permitted by the link being used. If it does this without generating additional losses it would be working exactly as we want.
Let's see the system throughput:

The system throughput is as expected. The protocol adapts itself to the link capacity. What about latency? Will it suffer when reducing the link bandwidth? The latency graph is here:
 

 
The latency increase from 73ms to close to 155ms. Something to be expected because having the same packet size (1000-byte) and half the link bandwidth, the packets will last at least double of the time to be transmitted from end-to-end in this case.
 
But the jitter should be very small when the flow stabilizes. Let's see the graph:
 
   

 

Okay. The bytes transferred should be reduced by half, more or less. Here we have:

KB sent: 1221.835938
Just 50.43%. Exactly as we are expecting. Finally the fire's test: could TCP transport this load without having additional losses? Let's see the losses:

   Packets sent:  1204
Packets dropped:     0
   Total losses:     0.000000%
  Great, really great!!  TCP is enough intelligent to adapt itself to the network condition transporting packets as much as is possible (according to the link bandwidth) generating no losses. It will be very interesting to compare this behavior with the UDP protocol that does not implement any kind of congestion avoidance mechanism. The man-hour of very specialized people that TCP carries in its back is really very, very large and impresive. An enormous effort has been done and is being done to have a transport protocol like TCP. TCP algorithms have been studied and polished by many people working hard to have what we have. TCP is the infrastructure where the Internet succesful is based.
Now we are going to study what will be the system response when we change the buffer size in the link. Our buffer is actually 20 packets. What about if we take this value to 50 packets?
First step: to see the instant throughput:

It's really very similar to our base environment. Nothing being perceptible. Let's see the system throughput:

Same story, almost (if not exactly equal..) the same to our first test using a buffer of 20 packets. Let's see what about latency:

Really incredible!! Exactly the same latency of our first test using a 20-packets buffer. One would be tempted to think that because the buffer is larger the latency should be also larger. What's happening? An explanation could be found having a graph of what is occurring with the queue. Could we monitor the queue length? Yes, why not? Let's see the queue behavior for our first test using a buffer length of 20 packets.

Humm... very, very interesting. Observe that despite we have a 20-packets buffer, this never becomes full. The curve is smoothed using EWMA to purge noises but the graph tells us that the average queue length is some value between 14 and 15 packets, being closer to the 14 packets. The average queue length indicates to us that the queue stays most of the time with 14 packets and some time with 15 packets. This could be the explanation why we don't have more latency when the buffer is increased to 50 packets. We could have a larger buffer but if it is not filled, because the flow is well controlled by TCP, it doesn't matter, in this case, for the latency measuring. Let's confirm this idea. Here we have the same graph for the 50 packets buffer:

Eureka!! Our idea was right. The graph is exactly the same, this means, no having congestion, because TCP controls very well the traffic rate, it doesn't matter to have a larger buffer because it will not be never filled.
Having this information about the queue behavior we could try to calculate latency given these conditions. What will be the calculated latency? Could this calculation be confirmed with the simulation? Having 1000-byte packets, a queue of 14-15 packets and a link with an available bandwidth of 2Mbps, it is very easy to calculate the estimated latency; here we have:

                14.5 packets/queue * 1000 bytes/packet
Queue latency = -------------------------------------- = 55.31 ms
                 2 Mbits/sec * 1024*1024/8 bytes/Mbit
55.31 ms is the time that a packet last in the queue. But we have also the packet own delay, this means, the time we need to transmit a 1000-byte packet over a 2Mbps link. This value is:

                               1000 bytes/packet
Packet own latency = ------------------------------------ = 3.81 ms
                     2 Mbits/sec * 1024*1024/8 bytes/Mbit
Finally we have the propagation delay of the link; this is 10ms. Then, the total calculated latency will be:

Mean latency = 55.31 + 3.81 + 10 = 69.12ms
Okay, we are almost right because simulation indicates a latency of 72-73ms. I'm not very sure if ns-2 simulator add some random value to consider time of forwarding at the routers. It could be the reason of the little difference. But, in any case, the important thing is to understand the concept we are trying to explain. To finish with the overbuffering test let's see now the jitter graph:

As you see the behavior is very similar to the test with a buffer of 20 packets.  Now is the time to check the bytes transferred. Here we have:

KB sent: 2421.289062
Exactly the same value obtained with the buffer of 20 packets.  Finally, the losses:

   Packets sent:  2385
Packets dropped:     0
   Total losses:     0.000000%
Again, exactly the same. Well, increasing our buffer length doesn't affect the TCP behavior for this particular condition (do not forget this); being the link uncongested the performance was the same for both cases. But, now, it is impossible not to ask ourselve what would be the TCP performance if, instead of having an overbuffered system, we had an underbuffered system. Let's then to reduce the buffer size to 5 packets, instead of the original value of 20 packets of our first test. The instant throughput graph is as follows:

Well, well... TCP is having problems, isn't it?  The curve now is not steady state, but instead, packet dropping fire the TCP congestion avoidance mechanism provoking serious throughput oscillations. This situation indicates to us that having the correct buffer length is very important for the TCP health. Overbuffered routers increase the latency when congestion is present (only when the longer buffers are filled), but underbuffering routers are even worst because packet dropping generate throughput oscillations. Let's see what happen with the system throughput for this case:

Very bad, really bad.. After 10 seconds the system throughput is yet below 1700 Kbps. A serious de-rating when it is compared with the ideal situation of a 20 packets buffer. Being the buffer shorter we should expect a lower latency because packets should last less time in the buffer queue. The latency graph is this:

Latency has been reduced but the cost is too high. Observe also that now the latency is not steady state; it has variations along the time. Some applications are even more sensible to jittering that to the latency phenomenon itself. Let's have a look to the jitter:

It wasn't a good idea to reduce the buffer length, was it? Let's see the bytes transferred; this value should have a serious de-rating; here we have:

KB sent: 2088.164062
We lost almost 400 kbps (16,67%) of our initial throughput. What about losses? Can TCP maintain no losses in this situation? Here we have:

   Packets sent:  2097
Packets dropped:    40
   Total losses:     1.907487%
It was not possible!!  Underbuffering affects the excelent TCP score of no losses transmissions. The curiosity to know how things are going on in the buffer queue itch my hands. Here we have the graph:

Whao... A lot of problems here. Instead of having an steady state queue occupancy as we had above when using a 20 packets buffer, the queue occupancy in this case is very unstable. To get higher performance we have to avoid as much as is possible any kind of oscillations. The system must be tunned to get the most from it.
Our last test for this environment (One TCP flow) will be to see what happen when the link is longer, i.e., the propagation delay is higher. Next tests will be done using a propagation delay of 100ms instead of 10ms, almost a ten times longer link. The instant throughput is:

Damn!! We have oscillations again. Longer links are not good for TCP behavior. Let's see how the system throughput is affected:

Horrible!! We lost 60% of the system throughput. After 10 seconds the system throughput has not reach even though 750 kbps. A very bad performance.
Sally Floyd, Matthew Mathis, Jeffrey Semke and Jamshid Mahdavi studied TCP performance and developed a TCP response function. They discovered that TCP bandwidth distribution is a complex matter where several factors take place, such as phase effects, link's roundtrip time, packet size, packet loss, number of routers, and router's packet dropping randomness. The steady-state TCP performance function can be expressed as follows:
 

 
Where, for the TCP connection, BW is the maximum bandwidth that can be reached, MSS is the maximum segment size, RTT is the mean roundtrip time and p, the packet loss probability of the link. C is a constant which value is SQRT(3/2) when the receiver acknowledges any received packet, and SQRT(3/4) when the receiver uses delayed acknowledgment, i.e., an ACK is sent every other received packet. This function bounds the maximum bandwidth throughput that a TCP connection can reach. It applies for the Reno TCP version (being used in our simulations), but is not as good for the Tahoe TCP version.
 
As can be seen analyzing this function, the maximum throughput that a TCP connection could reach, in steady-state operation, is directly proportional to MSS and inversely proportional to RTT and p. Let's suppose that we are working with equal size packets. Then, connections whose ends are farther (have longer roundtrip time) are in disadvantage when competing with connections whose ends are closer.
 
Also, the packet loss probability affects the TCP behavior. And this term is closely related to the end-to-end link conditions. Which factors increment p and decrease the maximum reachable throughput? Some of them could be:
 
  • The congestion condition. If the links are highly congested the routers are biased to drop more packets trying to alleviate the congestion; the drop probability increases accordingly.
  • The quality of link or media. Nevertheless, this effect is marginal, except for those cases of some media with high losses, like satellite links.
  • The type of traffic. Routers and firewalls are configured to be biased against some kind of traffic. For example, UDP traffic, because its unresponsive behavior, could find more problems to cross a router or firewall than TCP traffic (a firewall because it is connectionless). An aggressive flow could be treated by a router when it is detected, in disadvantage when comparing with other less aggressive flows.
I, intentionally, ignore the packet size above when talking about RTT and p. But the packet size is a very important variable. Being the throughput directly proportional to the traffic's mean packet size, traffic with very small packets, like VoIP, is in disadvantage when trying to compete with bulk traffic, like those containing big packets generated by FTP file transfers.
Walking back to our 100ms link propagation delay test, let's see how the latency is affected in this case. It should be enormous this time.

Well, yes, but not too much. Certainly it is higher but I did a mental calculation thinking that if having a 10ms link propagation delay the latency was 72ms, now having a 100ms link propagation delay the new latency should be something like 160ms. But no, it is just 104ms. Where this difference was saved? Most of the previous 72ms latency was spent in the buffer queue (a 14 packets queue). Let's see the queue graph for this test:

What a surprise!! Increasing the link propagation delay affected the queue behavior. Now the queue is almost empty. This curve is smoothed but it is indicating that the queue sometime has one packet and sometime is empty. The no-smoothed curve could confirm our appraisal.

Exactly as we thought. The queue oscillates between zero and one packet. Then, the 14 packets queue delay does not apply in this case and this is the reason of the saving in latency. Let's see how the jitter bothers in this case:
 

 
 
Well, not too much, is more or less equal to zero. The bytes transferred should be a lot lower; here we have:
 

KB sent: 924.257812
 
A de-rating of 61.83%, really very bad. What about losses? We should expect an increase in losses. The output is:
 
     Packets sent:   911
Packets dropped:     0
   Total losses:     0.000000%		  
 
No, no losses!! Working in this difficult environment is really unexpected no having losses. Packets sent were reduced significantly but the transmission was done with no losses. TCP surpasses even this test.
 
Well, folks, with this we finished the One TCP flow study. Our next step will be to repeat the same tests but using the UDP protocol. Let's go ahead.
   

Previous

Content

Next