Two TCP flows

Previous

3.0.- Two TCP flows

Warning: this page has a bundle of png and gif pictures; it will last a little if you have an slow Internet connection.

Continuing with our study we will see now how two TCP flows compete for network resources. The topology will be the same we used before for One TCP flow; this time we have two FTP sources in the host A.

In both hosts we have TCP. Host A has two FTP sources that always have packets to send. The packets are 1000-byte size. Host B is in charge to acknowledging the packets it receives. Ack packets are 40-byte size. The link is a 2 Mbps link having a propagation delay of 10 milliseconds. The buffer size is 20 packets and the queue is DropTail (just a FIFO queue). The TCP implementation is Reno. Some other conditions apply the same for these tests that before. Have a look to One TCP flow.

The tcl code to feed the ns-2 simulator is in tcp-2flow.tcl.

At time 0.1 seconds the FTP sources in A start to send packets (both at the same time), and at time 10.0 seconds they stop to send packets (again, both at the same time).

The first graph obtained from this test is instant throughput in kbps:

Here things go different than when TCP was alone and happy having for it all the available network resources. Now it's time to compete with some other flows. Now there are oscillations provoked by the competition for resources. After the initial slow-start phase both flows suffer some loss that pull them back just before the first half second. Congestion window reduce to half and TCP continue with a new phase based on the congestion avoidance algorithm. This algorithm takes control over the two flows. The known figure, some kind of saw tooth, can be represented for a well known TCP performance function. These oscillations affect the effective throughput that can be reached.

To check this, let's see now the system throughput. The instant throughput is calculated dividing the bytes of each packet (1000 bytes) by the time required to send the packet. The system throughput is calculated dividing the total bytes sent up to a given time by the total time transcurred in seconds. In fact, the system throughput is what really have importance because it says what is the power of your transmission, this means, how many bytes you are transfering by unit of time.

Whao.., observe how the loss episode affect the system throughput of both flows during the starting phase. Then, after congestion avoidance algorithm takes control, the system throughput raises little by little trying to reach its steady state. Observe also, how both flows act as good citizen trying to share the available resources by equal slices. Being the total link bandwidth available about 2Mbps, each flow tries to get for itself one half of this availability. Don't be foolish by the short time. If you leave to run the time, both flows try to reach the steady state sharing like good friend the food. Next figure shows how things go after 60 seconds.

Very nice, isn't it? Don't push me, there are bread for everyone!!. Okay, now it's time to see the latency. How much a packet last for going from the sender to the receiver when two flows compete.

Okay, there are some large deviation due to some packets lost and recovery after almost 1.4 seconds. But the latency, in general terms is below 100ms. Red flow can't be checked as well because, I suppose, gnuplot plots first red points and later green points and much of them coincide in place and time. Then, what can we say? Just that latency is practically the same for both flows.

Let's see now the jitter, this means, how the latency change from packet to packet.

Well, as expected. Despite some random values, the jitter for both flows is almost equal and very close to zero.

What about the total bytes transferred for each flow? ; for flow #1 we have:

KB sent: 1109.101562

For flow # 2 we have:

KB sent: 1069.492188

Total bytes transferred are then:

KB sent: 2178.593750

This figure compared with the bytes transferred when just one flow was running on the network, i.e., ~ 2421.29 KB, implies a reduction of about 10.02% on the number of bytes transferred with respect to only one flow. Let's check losses to see if this reduction could be due to this type of problem.

For flow # 1 losses are as follows:

   Packets sent: 1108
Packets dropped:    15
   Total losses:     1.353791%

and for flow # 2 they are:

   Packets sent: 1074
Packets dropped:    20
   Total losses:     1.862197%

Well, we have some losses due to the initial flow competition to share fairly the available resources. With 2182 packets sent and 35 packets dropped, the consolidated losses are 1.60%, enough lower than the previous value above. But, observe also that number of packets sent suffered a significant reduction. When just one flow was running on the network the number of packets sent were 2385, and now we have only 2182. The loss episode before the first one half second that triggered the first congestion window reduction on both flows should be the reason of this de-rated throughput.

Okay, to continue with our study let's start changing some parameters of the test. We will select now small packets of 100-byte size instead of big packets of 1000-byte size. The instant throughput graph for this new scenario is as follows:

Here we have again the problem of green points overwriting red points. But the graph shows us that both flows experiment almost the same behavior. The oscillation is more dense because packet size is smaller. But a very interesting observation is that we don't have here the loss episode we had before at start up when using larger packets. The start up was better in this case and perhaps using smaller packets we have avoided to loss some packets. Let's see how this type of oscillation affect the system throughput:

It is very interesting this graph. If you recall when we were studying the TCP behavior with one flow and 100-byte size packets, the reached system throughput was only a little bit more that one half of the link capacity. Have a look here to see the graph for this scenario. Now the situation is the same for each individual flow, but, having now two flows we can reach the maximum link bandwidth capacity.

But, what will be the latency, now?

Same as before when trying with one flow, latency is a lot better. Packets are smaller and travel faster. Most of the latency in this scenario is caused by the propagation delay of the link being used, i.e., 10ms.

Latency is very stable; jittering must be almost zero; let's confirm this:

Nice, almost no jittering.

Next step will be to see the total bytes transferred; because of the poorer system throughput, the number of bytes transferred should be a lot less. For flow #1 we have:

KB sent: 1201.523438

For flow # 2 we have:

KB sent: 1199.062500

Total bytes transferred are then:

KB sent: 2400.585938

The number of KB sent for each connection is almost the same; the link bandwidth resource is being shared as fair as possible. But even better, in this case the number of total KB sent is higher than when 1000-byte size packets are used. The increased throughput is ~ 10.21%. When we used a single TCP flow the results were exactly the inverse; i.e., throughput was lower when using smaller packets than when using larger packets. It seems that smaller packets behave better when we increase the number of concurrent flows. This, of course, should be proved using some other scenarios having three, four or more flows.

The better response is achieved this time because we have avoided the start up loss episode by using smaller packets.

Well, now is time to check losses; for flow # 1 losses are as follows:

   Packets sent: 8789
Packets dropped:     0
   Total losses:     0.000000%

and for flow # 2 they are:

   Packets sent: 8771
Packets dropped:     0
   Total losses:     0.000000%

Exactly as we supposed. Reducing the packet size we avoided the start up loss episode and this time we don't have any losses. Using two flows the system behavior is improved if we reduce the packet size. It would be very interesting to prove if this advantage is maintained even when using three, four, or more flows. I leave this test for the curious lector to be advanced.

Well, our next step will be to repeat the previous tests using a 4000-byte packet size. The first graph using the new packet size is the instant throughput:

Now we have more than one noticeable start up loss episode. Perhaps up to four or five looking at the spikes in the graph above. At time 1-second the loss episode is really serious. Do not forget that this curves are smoothed using EWMA and the big fall close to the time 1-second is almost no transmission at all. We will check this later when having a look to the system losses.

Let's have a look to the system throughput for this scenario:

Well, it's true. The loss episode close to the time 1-second was really serious. TCP backoff suspends packets flow trying to help the network to recover its stable state. For the short testing time of 10 seconds this event will affect system throughput severely.

Let's see now the new latency:

Latency increases noticeably in this scenario and being worst is not stable. We will have jittering too. Let's check this.

Comparing these two last graphs with the same graphs when using 1000-byte size packets indicates that bigger packets induce more latency and jittering as it was expected. But also latency and jittering are higher and more unstable when comparing with the same packet size with only one TCP flow on the link. Latency was very stable at 320ms and jittering was almost zero when just one flow was in the link. Now the competition between the two TCP flows increases both, latency and jittering, and has a more unstable response.

Let's see now the number of bytes transferred to check our first feel about the system throughput; the repeated loss episodes should affect noticeably the number of bytes transferred:

For flow #1 we have:

KB sent: 1175.742188

For flow # 2 we have:

KB sent: 1025.820312

Total bytes transferred are then:

KB sent: 2201.5625

Again the number of bytes transferred is a little bit more than when using 1000-byte size packets. We have 23 KB more transferred for an increase of 1.06%. Despite of the observable loss episodes using larger size packets we improve our system throughput.

Finally, let's check losses in each flow; here we have losses for flow #1:

   Packets sent:   309
Packets dropped:    10
   Total losses:     3.236246%

and for flow # 2 they are:

   Packets sent:   272
Packets dropped:    11
   Total losses:     4.044118%

Consolidated losses are 21 packets lost over 581 packets sent for a percentage value of 3.61%. Comparing with the scenario of 1000-byte size packets (1.60% of consolidated losses) the response is worse now by more than double. Comparing with the scenario of the same packet size but having just one flow the response is also worse. Now we are transmitting a total of 581 packets lossing 21 of them. Having one TCP flow we transmitted 630 packets with no loss. The derating is -11.11%.

TCP has the advantage that it adjusts itself to the current link bandwidth trying to share this as fair as is possible between competing flows, but the problem is that to reach some stable state some tests must be done by each of these flows competing for resources. These continuous tests brings the network close to congestion and packet losses are inevitable. Also, the flow rates can not be stable. Competition, tests, congestion, and loss episodes to indiuce backoff on transmission generates oscillations in the network that affect the useful system throughput (called goodput).

I'm going to write something that perhaps some people could say I'm really crazy, but I envision Internet future as not using TCP protocol at all, but instead some constant bit rate protocol as UDP but having reliability. Cisco wrote something like this called Reliable UDP. This company is so smart when it's business what we are talking about, that it's not strange that they are preparing themselve for this future. But, beware, it's not as easy. To do without TCP we have to prepare before the Internet's infrastructure to implement a reliable global admission control environment and truly respect for resource utilization. This respect has to be enforced by an strong and reliable QoS infrastructure which can support any kind of people and/or organization trying to use the global network. I really think that new technologies based on QoS routing (constraint routing and policy routing) and Traffic Engineering will bring us, sooner than later, this ambitious environment. MPLS and MPLS with Differentiated Service support and admission control using RSVP or LDP is going in this direction. I hope god will give me enough live to see this with my own eyes. Okay, let's stop with wishes and come back again to our study.

Next step in our investigation will be to see what happen when the link bandwidth is reduced having two flows competing. How TCP will react? We will adjust the link bandwidth to 1 Mbps instead of the original value of 2 Mbps. The propagation delay will be kept in 10ms. Here we have the instant throughput for this case:

Okay. The curve is very similar to the original one for a 2 Mbps link, but this time TCP adjusts its throughput to a value a little bit less than 1000 Kbps respecting the environment condition.

Let's see the system throughput:

Again very similar to the graph for a 2Mbps link but reduced to one half. The worst loss episode occurred at 1200kbps when link bandwidth is set to 2Mbps and at 600kbps in this case when the link bandwidth is set exactly to one half, i.e., 1Mbps. In both cases TCP tries to share the available resources as fair as possible. Let's see now the latency:

Really no noticeable differences. Let's see now jittering:

Same behavior with some dispersion. Link bandwidth reduction affected flow's bandwidth but in this case has almost no effects over latency and jitter. It seems that as long as there is some reasonable available link bandwidth, latency and jitter are more affected by the link propagation delay than by the link bandwidth.

Next we will check the total bytes transferred for each flow; for flow #1 we have:

KB sent: 552.539062

For flow # 2 we have

KB sent: 539.335938

Total bytes transferred are then:

KB sent: 1091.8750

Again we have a de-rating of about -10.64% when we compare this response with the response for only one flow. Interaction between different TCP flows has the effect of reducing the total system throughput that can be reached. It would be interesting to repeat this test using many TCP flows to see what happen in this case. However, bandwidth utilization and distribution can't be better. The utilization is now exactly 50.11% that was before having a link bandwidth of 2Mbps. Then available bandwith is being utilized now as well as it was before. And bandwidth distribution is 50.60% for flow #1 and 49.39% for flow #2. The resource is being shared between flows as fair as is possible.

Finally for this test let's check losses; for flow #1 we have:

   Packets sent:   557
Packets dropped:    12
   Total losses:     2.154399%

And for flow #2 we have:

   Packets sent:   546
Packets dropped:    14
   Total losses:     2.564103%

Well, we have more losses now. Number of packets sent are now 50.55% of the number of packets sent when we used a 2Mbps link, but the number of lost packets are now not as lower. We lost 15/20 packets using the 2Mbps link but now, having a 1Mbps link, we lost 12/14, i.e., a relation of 80%/70%. Link bandwidth reduction has affected TCP behavior by increasing the loss rate. Again, it would be very interesting to check what happen with the loss rate if we have several TCP flows instead of just two as in this example.

Okay, following with our study let's check now what happen if we increase the buffer size in the link. Our buffer is actually 20 packets. What about if we take this value to 50 packets? The instant throughput is:

First good new we observe with this scenario is that the startup is softer now. Having more buffering the initial loss episodes are avoided or at least reduced. This is good because probably we are going to have less losses in this case. Of course, something has to be payed. More buffering could imply more packets in the queues and an increased latency too.

There are some investigations done in this area of optimal buffering selection. When there are too many TCP flows competing for network resources we can reach a condition where only a good selection of the queue buffers can save our network to become almost stalled. Have a look to the paper TCP Behavior with Many Flows by Morris to see what we are talking about. People who like Kazaa, E-mule and P2P should read this paper too to know the devasting effect of many TCP flows competing in their networks. A good recomendation is having buffers set to 200ms of traffic at the available bandwidth rate. In our example we have a 2Mbps link, then the optimal buffer should be 51200 bytes. Assuming packets of 1000-byte size the optimal buffer is just 51 packets, very close to the buffer selection for our current test. But beware; this is just a recommendation not a rule. Each situation should be analized before having a decision. For example, 200ms is too much for some sort of very sensible realtime flows.

Let's see how the system throughput runs this time:

Okay, as expected we don't have the terrible loss episode close to the time 1-second that brought the throughput back to 200kbps when the buffer was only 20 packets. We have oscillation here but they are mitigated by the larger buffer. In general if you make some graphs of your traffic and you see too much oscillations, increasing buffering can help to have softer response and a higher system throughput; again paying in more time for latency. At An study of network load in this site you can have a look to some graphs we took some time ago showing this particular behavior.

Okay, because we are talking about latency let's see it for this environment:

Very nice; compare this graph with we saw before when running with a buffer of 20 packets. Now dispersion disappears and latency has an steady state behavior. Of course, previous maximal latency was not more than 100ms and now we have almost 160ms, but this time we have less dispersion and no jittering at all after the steady state condition is reached.

It's time to see the jitter's graph:

Definitively it was a very good idea to increase the buffer size to have better response when having two flows. More flows means more fights to catch resources and an increased oscillatory response of the network. More buffering means to open spaces to make easy the interaction between flows.

It is interesting to compare latency in this test with the same metric but having only one TCP flow. With one flow latency was almost constant at 72ms and practically the same that was obtained using a 20-packets buffer. This time having two flows latency increases from 100ms to 160ms but its oscillatory behavior was smoothed. Let's have a look to the queue for this case:

The queue ocillates at 16 packets. Sometime it must be higher than 20 packets (don't forget that the graph is smoothed) but it is very improbable that it reaches 50 packets. Then we can say that 20 packets is not enough for this scenario but also that 50 packets is too much because the queue never will reach these values. Let's have a look to the queue for a buffer size of 20 packets:

This queue shows a very oscillatory behavior. It tells us that the amount of buffering is not enough. Using a buffer of 50-packets improve the behavior but it is too much as we saw above in the previous graph..Some value above 20 packets would be the right choice for the environment we are studying this time.

To finish this test let's see now the number of bytes transferred; for flow #1 we have:

KB sent: 1228.945312

And for flow #2 we have:

KB sent: 1216.757812

Total bytes transferred are then:

KB sent: 2445.703124

Comparing with the number of bytes transferred having a buffer of 20-packets (2178.593750KB) we've got now an increased value of 12.26%. Using more buffering increased the system throughput. But, let's compare this value with the same value when only one flow was flowing throughout the network. We've got before 2421.289062KB (using one flow and 50-packets buffering); now we have 2445.703124KB and bingo!! A little bit increase, but better is something than nothing, of 1.008%, but having now two flows competing for resources. The larger buffers improved TCP behavior at the cost of some additional latency but permiting more space for both flows to accommodate themselve into the network.

Good, very good, but lacks to check losses. For flow #1 we have:

   Packets sent: 1211
Packets dropped:     0
   Total losses:     0.000000%

And for flow #2:

   Packets sent: 1199
Packets dropped:     0
   Total losses:     0.000000%

Nice!! Using just additional buffering, something that it is very simple to configure in any router, we've had success in reducing losses to zero.

Okay, fellows, to continue with our tests let's now to reduce the buffer size to 5 packets. This new scenario was terrible for one TCP flow and should be even worst for two TCP flows. The instant throughput graph is as follows:

Comparing with the same scenario but having a buffer of 20-packets we observe here two things: a.- Now flows oscillate between 1400 and 1600 kbps and with a 20-packets buffer flows oscillated between 1600 and 1800 kbps. We have then some reduction of the system throughput. b.- Now the oscillation is of some higher frequency and curve are less interlaced. Green flow has just a little higher throughput than red flow. Using 20-packets buffer both flows were more interlaced and throughput seemed to be the same.

If we compare this scenario with the same but having one flow, have a look here, now flows are more complicated having higher oscillations and worser loss episodes. It is possible to verify how this new environment affected the flows behavior checking the system throughput and later, the loss rates. Let's begin with system throughput:

As we can see the startup loss episodes are worse this time and bandwidth distribution is less fair. Green flow got better throughput almost 200kbps higher that red flow. Reducing the queue's buffers we have a less fair environment and lower throughput perhaps due to the worse loss episides.

If some reduction in buffering got an adverse effect in system throughput, perhaps we can have some improvement in latency. Let's check this metric:

Certainly latency is now lower and indeed more stable. Packets are now having to make less queue in buffers. Let's check jitter this time:

Well, some dispersion at +/- 5ms and even less at -10ms but most of the packets have zero jittering. Again desired behavior is a compromise. We can have lower latency and jittering but paying with more losses and lower throughput.

Let's check now the number of bytes transferred; we can expect here some noticeable reduction. Bytes transferred for flow #1 are:

KB sent: 940.507812

And for flow #2:

KB sent: 1180.195312

Total bytes transferred are then:

KB sent: 2120.703124

Then we have a reduction of -2.66% in system throughput comparing with the same but having buffers of 20-packets. Also, the fairness is less now. Flow distribution is 44.34%-55.66% in this ocasion against 50.92%-49.08% with 20-packets buffering.

Finally let's check losses for this environment. For flow #1 we have:

   Packets sent:   978
Packets dropped:    51
   Total losses:     5.214724%

And for flow #2 we have:

   Packets sent: 1210
Packets dropped:    47
   Total losses:     3.884298%

Losses have increased considerably. Consolidated losses are 98 packets/2188 packets = 4.48%, which is 280% higher than losses when a 20-packets buffer was used (1.60%). Buffer reduction was not convenient because marginal latency and jittering reduction was not justified by the increasing loss rate. Then, as long as we are talking about TCP it is better to have enough buffering to soften protocol oscillations and improve system throughput. Issue is then trying to find the right choice to have enough buffering without increasing too much latency and jittering response.

Our last test will be to see what happen when we select a propagation delay of 100ms instead of 10ms, almost a ten times longer link. The instant throughput is:

Very interesting behavior. Observe that oscillations seem to be softer in this case when we compare with the 10ms propagation delay link. According to the TCP response function if we have more propagation delay, TCP system throughput will be reduced. That's true but also is true that having a longer link is like having more buffering. And we said just before that buffering tends to soften TCP oscillations. The 10ms-2Mbps link has a capacity to store 2.56KB (2 1/2- 1000-byte size packets), but a 100ms-2Mbps link has a capacity to store 25.6KB, i.e., up to 25- 1000-byte size packets. Then having a longer link is like having more buffering. This helps TCP to share a little better the available resources. But, and always we have a but, it's required to pay having some reduction in the effective system throughput according to the TCP response function. Then, let's check system throughput for this case:

Compare this figure with the same but using a 10ms propagation delay link. In this test, again, the response is soften, in fact, we don't have the harmful loss episode before time 1-second. However, in this case and after 10 seconds the system throughput is just a little more than 700 kbps while before, at that time, the system throughput was a little more that 800 kbps. Having a longer link we have more buffering to soften TCP behavior but also TCP throughput is reduced following the TCP response function.

Let's see the latency for this case; we have to expect a higher latency because the link is really a lot longer:

Latency is now always above 100ms, just 104ms, but more stable. The longer link seems to have a marginal contribution (4ms when comparing with a 10ms link, but really?) to the latency; let's check the queue to confirm or not this asseveration:

Umm..., very interesting. The queue is now almost empty, i.e., packets don't queue in the buffer. Then latency is not caused by buffer delay. Latency is on the link. But have a look to the queue when the link propagation delay was 10ms just above. Observe that for this case the queue was approximately 6 packets. It seems now like the queue was translated from the node's buffer to the link. And in fact it was what occurred. 6 x 100-byte size packets are ~6KB and the 100ms propagation delay link has a capacity of 25.6KB, more than is required to enqueue incoming packets at this rate. Flow of packets fits in the link then the buffer node is almost empty. We have the traditional premise. Nothing is free. Reducing buffering, in the buffer node or the link, reduce RTT, reduce latency and should increase TCP throughput (according to the TCP response function), but because loss episodes are more frequent the advantage is lost because the "p" parameter in the equation increases too. Increasing the buffering, in the buffer node or increasing the link propagation delay, we have less loss episodes, behavior is soften but we have to pay with more latency and perhaps a reduction in the maximum system throughput that is reachable. It is very important to have these concepts enough clear when we design and configure our networks.

Let's have a look to the jitter. It should be almost zero because latency is very stable.

Well, as expected. Very low being almost zero. Let's check now the total bytes transferred. For flow #1 we have:

KB sent: 924.257812

For flow #2 we have:

KB sent: 908.007812

Total bytes transferred are then:

KB sent: 1832.265624

As you can see we have a reduction in KBs sent of -15.89% with respect to the test using a 10ms propagation delay link and even more if we compare with the test having just one TCP flow. Nevertheless, when we increased only the buffer but the link was kept in 10ms the system throughput was increased in 12.26%. Then it was better to use buffering rather than a longer link to soften and improving TCP behavior. Longer link caused some additional problems being the worst the noticeable reduction in bytes transmitted per second.

Okay, it's time to check losses. For flow #1 we have:

   Packets sent:   911
Packets dropped:     0
   Total losses:     0.000000%

And for flow #2 we have:

   Packets sent:   895
Packets dropped:     0
   Total losses:     0.000000%

As was expected increasing the link length we have eliminated totally the losses in a similar way we did before increasing buffering. But we have payed too much now because the number of packets successfully transmitted were reduced from 2147 packets to 1806 packets ((15.88% less packets). If we use more buffering and the same link propagation delay the reduction is even worst, from 2410 packets to 1806 packets, this means -25.06% less packets. It has been shown again that longer links (longer propagation delay) are not good for TCP behavior. It was seen before when dealing with one TCP flow and now with two TCP flows. Longer links soften TCP behavior but the price to pay for this is too much high. As long as is possible is better to work with TCP using short links or media having short propagation delays.

Well, with this we have finished the Two TCP flows study. Our next step will be to repeat the same tests but using the UDP protocol and two flows. I invite you to continue reading.

Previous

Content

Next