Abstract

Future computer networks are expected to carry bursty traffic. Shortest-path routing protocols such as OSPF and RIP have the disadvantage of causing bottlenecks due to their inherent single-path routing. That is, the uniformly selected shortest path between a source and a destination may become highly congested even when many other paths have low utilization. We propose a family of routing schemes that distribute data traffic over the whole network via bounded randomization; in this way, they remove bottlenecks and consequently improve network performance. For each data message to be sent from a source s to a destination d, each of the proposed routing protocols randomly choose an intermediate node e from a selected set of network nodes, and routes the data message along a shortest path from s to e. Then, it routes the data message via a shortest path from e to d. Intuitively, we would expect that this increases the effective bandwidth between each source-destination pair. Our simulation results indicate that the family of proposed load-balanced routing protocols distribute traffic evenly over the whole network and, in consequence, increases network performance with respect to throughput, message loss, message delay and link utilization. Moreover, implementing our scheme requires only a simple extension to any shortest-path routing protocol.


Introduction
In a wide -area store -and-forward computer network, such as the Internet, routing protocols are essential. They are mechanisms for finding an efficient path between any pair of source and destination nodes in the network and for routing data messages along this path. The path mus t be chosen so that network throughput is maximized and message delay and message loss are reduced as much as possible .
There are mainly two types of routing protocols: source routing and shortest-path routing (destination routing). In source routing, a s ource node determines the path that a data message must take [11]. In shortest-path routing, each node uses its routing table to store a preferred neighbor to each destination.
Thus, the routing table specifies only one hop along the path from the current node to the destination. In a stable state of the protocols, the path consisting of consecutive preferred neighbors for a given destination is assumed to be a shortest path to the destination.
Shortest-path routing protocols are classified into two types of routing protocols: distance-vector routing [17], for example, used in the RIP Internet protocol [13], and link-state routing [14], for example, used in the OSPF Internet protocol [15].
In the distance -vector routing protocol, each node maintains a rout ing table and a distance vector, which contain, respectively, a preferred neighbor for the shortest path to each destination in the network and the distance of the path to the destination. Each node has incomplete knowledge of the network topology and know s only its neighboring nodes. From these neighbors, the node chooses the closest neighbor to each destination. Each node periodically sends its distance vector to each of its neighbors to inform it of any distance changes to any destinations. The node dete rmines which neighbor is the closest to each destination by comparing the distance vectors of its neighbors ( [13], [17]).
Link-state routing protocols require each participating node to maintain complete network topology information. Each node actively te sts the status of the links between itself and its neighbors. Then, it periodically broadcasts the local link status information to all other nodes. Since each node receives the local link status information from all other nodes, it is able to build a grap h of the whole network topology and to compute the shortest path from itself to every other node ( [14], [15]).
Shortest-path routing protocols suffer performance degradation because all data messages are routed via the same shortest path to the destinatio n as long as the routing tables remain unchanged. The problem with these routing protocols is that there are no mechanisms for altering the routing other than updating the routing tables. The shortest path may be highly congested, even when many other path s to the destination have low link utilization. This congestion may trigger the loss of valuable data messages due to buffer overflow at some node . Using a single path to the destination limits the maximum throughput possible between the source and the destination to be at most the minimum capacity of any link along the shortest path from the source to the destination.
Maximizing network throughput is an important goal in the design of routing protocols. If the network uses shortest-path routing protocols t o carry bursty traffic, then many of these data messages might be dropped due to the limited buffer space of each node when these shortest paths are congested. In this paper, we want to minimize the message loss due to the buffer overflow at each node. We also want to maximize the network throughput. Our approach increases the effective bandwidth between the source and the destination so that more data messages can be delivered. A result in network flow theory, known as the max -flow min -cut theorem [8], sho ws that distributing the traffic load over all available paths between a source and a destination in the network, instead of using only one path of minimum cost, may increase the effective bandwidth up to the capacity of the minimum cut separating these tw o nodes. Suppose that node a wants to send data messages to node f. Suppose that we use the hop count in order to calculate t he cost (length) of a path in the network. Then the effective bandwidth between node a and node f is 30, while the effective bandwidth of the shortest path ( a-h-g-f) from node a to node f is 5. Now we can see that there exists an unused effective bandwidth between each pair of nodes in a shortest -path routing protocol, which we could use productively.
These routing protocols improve performance by routing data messages via multiple paths to the destination. They provide alternate paths to distribute data traffic when the selected shortest path to the destination becomes congested. We mention Shortest Path First with Emergency Exits [20] based on linkstate routing, Multiple Disjoint Paths [18] based on distance -vector routing, and Dynamic Multi -path Routing [2] based on source routing. The disadvantages of these techniques are that they require considerable processing overhead, need significant storage space, or significantly increase the complexity of the routing algorithms. Several randomized multiple-path routing schemes ( [7], [16] , [19]) have been proposed for regular network topologies , such as mesh, torus , and butterfly, but these schemes are not suitable for the Internet, which has an irregular network topology.
In a recent paper [3], we proposed a load-balanced routing scheme that improves network performance by randomly distributing the traffic load over a set of paths to the destination. The routing protocol was formulated for an IP (Internet Protocol) network with an irregular flat network topology.
We proposed an improved method that implements our routi ng approach in order to maximize the number of messages that reach their destination and to minimize the number of messages that are dropped due to buffer overflow at each network node [4] . We applied our routing algorithm to a hierarchical network topology [5].
In this paper, we simulate a family of our proposed routing protocols ( [3], [4]) more comprehensively.
We increase the number of connections in our simulation network topology up to n •(n-1), where n is the number of the nodes in the entire network. Also, we use two different network topologies for the experiments.
The rest of this paper is organized as follows. Section 2 gives an overview of the max -flow/min-cut theorem. Section 3 sketches the load-balanced routing protocol. Section 4 introduces the protocol notation to give a formal version of our routing protocol, which is given in Section 5. In Sections 6 and 7, we present the simulation model and our results. Sections 8 and 9 outline future work and draw conclusions, respectively. Appendix includes additional simulation results.

The Max-Flow/Min-Cut Theorem
We sketch the max-flow/min-cut theorem in this section [8].

Networks and Flows
A network G(N, A) is defined to be a set N of nodes and a set A of pairs of distinct nodes from N called arcs. An arc (i, j) is an ordered pair, to be distinguished from the pair (j, i). A path P in a network is a sequence of nodes ( n 1 , n 2 , …, n k ) with k ≥ 2 and a corresponding sequence of k -1 arcs such that the ith arc in the sequence is ( n i , n i+1 ). Nodes n 1 and n k are called the source node and the destination node, respectively. Each arc ( i, j) is assigned a non -negative number c( i, j), called the capacity of the arc ( i, j). A flow x ij is the rate of traffic transmitted on the arc ( i, j) in the network. For every arc (i, j), 0 ≤ x ij ≤ c(i, j).

Maximum Flow and Minimum Cut in a Network
The total flow F of a node is the sum of the flows into the node. In the max-flow problem, two nodes are distinguished: the source ( s) and the destination ( d). The objective is to pus h as much flow as possible from s to d while observing the capacity constraints. There exist cuts between node s and node d in a network. A set with th e minimum capacity among all sets of arcs in a network whose deletion destroys all directed paths from s to d is called the minimum cut.   Max-Flow/Min-Cut Theorem [8]: Every network has a maximum total flow F, which is equal to the capacity of the minimum cut.

Overview of the Load-Balanced Routing
In this section, we informally sketch how our method, called Load-Balanced Routing (LBR), routes data messages to the destination. Each node creates data messages and receives data messages from its neighbors. The node should forward these data messages to its neighbors so that the number of links (the cost of a path) traversed by each data message is as small as possible, while at the same time attempting to distribute these data messages evenly throughout the network to avoid congestion and increase network throughput.  Therefore, the operation of the protocol is as follows. Initially, the source node s sends the message with b = 0 and routes it to node e. As long as b = 0, the me ssage keeps being routed along the network until it reaches node e. At node e, b is updated to 1, and the message is routed towards node d. As long as b = 1, the message keeps being routed along the network until it reaches node d, where it is delivered.
This technique distributes the traffic load over many more paths between a source and a destination in the network than the non -randomized routing schemes and increases the effective bandwidth up to the capacity of the minimum cut separating these two nodes , which is the upper bound on the available bandwidth between these two nodes [8].
As an example, consider Figure 1 again. Suppose that node a (source) wants to send data messages to node f (destination). For load balancing, node a should distribute the da ta messages uniformly over all possible paths to node f. Node a may accomplish this by selecting at random an intermediate node, say node c, among a set of nodes in the network whenever node a sends a data message to node f, routing it to the intermediate node c via the shortest path between node a and node c and then routing it to destination f via the shortest path between node c and node f.
The question to be considered next is how to select a set of candidates for the intermediate node.

Load-Balanced Routing via Full Randomization
We can use simply all the network nodes for a set of candidates for an intermediate node. In this case, our routing scheme is called Load -Balanced Routing via Full Randomization (LBR -FR). LBR-FR may increase the effective bandwidth between each source-destination pair up to the capacity of minimum cut separating the pair, which is the upper bound on the effective bandwidth [8]. LBR-FR has a shortcoming for pairs of nodes of short distance . It is possible that a data message is routed to the destination via a very long path, much longer than a shortest path from the source to the destination.
For example, in Figure 1, suppose that node a wants to send a data message to node b and it randomly chooses node f as the intermediate node. As a result, the algorithm routes the data message to node f via the shortest path ( a-h-g-f) and then routes it to node b via the sh ortest path (f-e-c-b). Although there is a path of length 1 between node a and node b, the proposed scheme results in the use of a path of length 6.
Clearly, routing paths that are excessively long will waste network resources.

Load-Balanced Routing via Bounded Randomization
To remedy the above mentioned problem of LBR -FR, we introduce a parameter k, with the goal of excluding nodes that are "too far away" from the source from being candidates for an intermediate node .
The value of the parameter k will be a distance from the source node. The set of candidates is restricted to all the nodes whose distance from the source is at most k.
The value chosen for k affects delay, path length, load balancing, and network throughput. If k is zero, the length of th e path is minimized because our routing protocol becomes a conventional shortest -path routing protocol, and thus the data message will be routed via a shortest path to the destination. On the other hand, if k is non -zero, a larger number of routing paths m ay be available, which generally will alleviate congestion and increase the effective bandwidth between these two nodes, but at the expense of possibly increasing the length of the traveled path. If k is INFINITY, the proposed algorithm is LBR -FR (see Sect ion 3.1). This may increase the effective bandwidth up to the capacity of the minimum cut separating these two nodes.
Choosing an appropriate value for k is crucial for the performance of the algorithm. Choosing too small a value may exclude nodes that are far away from the source from being candidates for an intermediate node , but it will increase the likelihood of a bottleneck. On the other hand, choosing too large a value may waste network resources by routing messages via excessively long paths , but it may increase the effective bandwidth up to the capacity of the minimum cut separating each pair of nodes . To reach a compromise between these two extremes, the parameter k may be chosen to be the average of the distance to each node reachable from the sour ce (LBR-BR1) [3]: where d i is a node in the network and s is the source node.
This value is a constant for the source s, since each link is considered to have a cost of 1. This value, however, has shortcomings. It limits the effective bandwidth between ea ch pair of the nodes in the network to less than the capacity of the minimum cut separating the pair. This static value of k may be too strong a restriction for a pair of nodes with a long path length and too weak a restriction for a pair of nodes with a short path length.
To remedy this problem of a static value for the parameter k, we may choose the value of the parameter k dynamically so that the distance between source and destination is factored in (LBR -BR2) [4]: where d i is a node in the network, s is the source node and d is the destination node.
This value of the parameter k changes dynamically according to the length of the shortest path from the source node s to the destination node d. Since the factor of dist(s, d) is less than 1, the length of a ny path chosen in this way is strictly less than three times the length of the shortest path (which is of course dist(s, d) ), assuming that the distance function behaves like a true metric.

Protocol Notation
In this paper, we use a simple notation to d efine our routing protocol.
( 1 neighboring processes is joined by a communications channel. Henceforth, we use the term p rocess and node interchangeably.  . If an action with this receive guard is chosen for execution, then, bef ore its command is executed, the data message is removed from the channel and its field is copied into the local variable var. Furthermore, variable j is set to the identity of the neighbor from whom the message is received.

Specification of the Load-Balanced Routing Protocol
In this section, we present a formal version only for LBR-BR2. Formal versions of LBR-FR and LBR-BR1 [3] can be derived in a straightforward manner from that of LBR-BR2. Each process has a constant n with the number of the processes in the network and an input set N with the identities of its neighbors. In the third action, the process receives a data message and then routes the messa ge to a neighbor using statement RTMSG. In the fourth action, the process sends a copy of its distance vector to each of its neighbors. In the last action, the process receives a copy of the distance vector from one of its neighbors, and updates its routin g

Simulation Model
Our simulation studies were done on the Maryland Routing Simulator (MaRS) [1], which is a network simulator developed at the University of Maryland. MaRS was designed to provide a flexible platform for the evaluation and comparison of network routing algorithms. MaRS is a discrete-event simulator. It allows th e user to define a network configuration, control its simulation, log the values of selected parameters, and save, load, and modify network configurations. A network configuration consists of a physical network, a routing algorithm , and a workload.
The physical network consists of link components and node components. A node component models the "physical" aspects of a store -and-forward entity. It is characterized by parameters such as buffer space, processing speed, message queuing discipline, and failure and repair  [17], which uses a shortest-distance path for each pair of source and destination nodes. LSR is a link -state routing protocol [14], where each node calculates and broadcasts the costs of its outgoin g links periodically and Dijkstra 's shortest path algorithm [10] is applied to the view of the network topology to determine next hops. To get a better understanding of the proposed LBR protocols, we compare the performance of our three LBR protocols against that of DVR and LSR. In our simulation, the assumed physical network is the NSFNET topology given in Figure 4.
All links have a bandwidth of 1.5 Mbits/sec. We assumed that there are no link or node failures.
Each node has a buffer space of 50,000 bytes. The processing time of a data message at each node equals 1 µsec. In order to calculate the cost of a path in the network, we use the hop count.
When we use the hop count as a link cost, the cost of each link is 1. The propagation delay of each link is 1 msec.
The workload consists of FTP (file transfer protocol) and telnet connections. A connection is a communication session established between end -user applications at sourc e and destination nodes. All FTP and telnet connections have the following parameters: the data message length equals 512 bytes, the inter -message generation time is 1 or 10 msec, and the window size is 500 messages. Traffic was introduced into the network by the FTP and/or telnet connections at the nodes they were attached to. Network traffic consisted of data messages sent from the source of a connection to the destination and response messages sent from the destination of the connection to the source. Fu rther, each source and /or destination node send s acknowledgments for data messages received. Also present in the network are routing messages, which are sent periodically to update the state of the network. Connections start when the simulation begins and they are considered never-ending.
We consider the performance measures of throughput, message delay, message loss, and link utilization. The measurement interval of each simulation is 100,000 msec.
• Throughput: The total number of data bytes acknowledged du ring the measurement interval divided by the length of the measurement interval. • Message delay: The total delay of all data messages acknowledged during the measurement interval divided by the number of data messages acknowledged during the measurement interval.
• Message loss: The total number of messages dropped during the measurement interval.
• Link utilization: The data service rate divided by the link bandwidth.    . Th e LBR -BR2 protocol has a low average delay both when the number of connections is high and low. LBR-FR has the highest average delay at most times during the measurement interval when the number of connections is high. Figure 8 shows throughput versus numb er of connections in the hot spot. In this scenario, a special node, called hotspot, has many more connections than the other nodes in the network. All the LBR protocols have better throughput than DVR and LSR do until the number of hotspot connections is 9. LBR-FR has the highest throughput when the number of connections is in the range from 1 to 5, while LBR-BR2 has the highest throughput when the number of connections is more than 5.  Table 1 shows the average link utilization during the measurement interval when the number of connections is 8. By inspecting the link utilization over the whole network, we can see that the LBR protocols distribute the data messages more uniformly over the whole network than DVR and LSR. Also, the total and the average of the link utilization indicate that the LBR protocols use network resources more productively than DVR and LSR (see Figure 4).

Link
Link In Figures 9 and 10  We shall now discuss our observations and the interesting insights obtained the results.
The performance of LBR -BR1 and LBR -FR is getting worse as the number of connections increases. LBR -FR exhibits a good performance with respect to throughput, message loss and message delay when the number of connections is low (in the range from 1 to 19); however, it exhibits the worst performance among the consi dered routing protocols when the number of connections is high (in the range from 20 to 182). LBR-FR will increase the effective bandwidth up to the capacity of the minimum cut separating these two nodes, but it has a much better chance than LBR-BR1 and LBR-BR2 that a data message is routed to the destination via a path that is much longer than a shortest path from the source to the destination. LBR -BR1 also exhibits a good performance with respect to throughput and message loss when the number of connections is low (in the range from 1 to 25), while it exhibits a performance worse than DVR and LSR when the number of connections is high (in the range from 34 to 182). LBR-BR1 will decrease the effective bandwidth less than the capacity of the minimum cut sepa rating these two nodes, while it reduces the chance that a data message is routed to the destination via a path that is much longer than a shortest path from the source to the destination. LBR -BR2 exhibits good performance with respect to throughput and me ssage loss when the number of connections is in the range from 1 to 100. The performance of the shortest -path routing protocols (DVR and LSR) is getting better as the number of connections increases. In particular, DVR and LSR exhibit better performance with respect to throughput and message loss when the number of connections approaches 182. This is to be expected, since even a single path routing algorithm would tend to distribute load over the entire network (from a global perspective), as the number of connections approaches n•(n-1), where n is the number of the network nodes.

Future Work
We proposed a family of load -balanced routing protocols for balancing load. In contrast to LBR-FR, LBR-BR1 and LBR -BR2 use a parameter k in order to exclude nodes that are too far away from the source from being candidates for an intermediate node. In this paper, we studied the proposed load-balanced schemes and compared them with conventional shortest-path routing protocols. From our sim ulation results, we see that LBR-BR2 has the best performance among all the routing protocols as the number of connections is in the range of from 1 to 80 . The curves tend to converge as the number of connections approaches 90. From this point on, the performances of DVR and LSR are getting better than those of the load -balanced routing protocols. A direction of future work is to determine the value of the parameter k to delay the cross point of the curves. Specifically, we will try to determine the value o f the parameter k to improve the performance of our schemes when the number of connections is high. For example, using the distance (or cost) from the particular destination for the parameter k may give better performance because then an intermediate node closer to the destination may be chosen.

Conclusions
We presented a family of load -balanced routing protocol s to distribute the data traffic, via bounded randomization, over all available paths to a destination in the network for data load balancing. Our simulation results show that one of the proposed schemes (LBR -BR2) has good performance with respect to throughput, message loss, message delay and link utilization, compared with DVR and LSR, which are conventional destination routing protocol s. The LBR schemes are simple and suffer from little control overhead . LBR-BR2 has the best performance when the number of connections is in the range of 1 to 80. It randomly chooses one node within k hops as an intermediate node. This value of the parameter k dynamically changes according to the length of the shortest path from the source node to the destination node. Since our routing scheme may use a path that is more expensive than the shortest path, the gain from load balance may be smaller than the cost resulti ng from a longer delay in a network with very balanced traffic. From the result s of our simulation, we conclude that our new routing scheme (especially LBR-BR2) has the following advantage over existing schemes except when the number of connections approaches n•(n-1): Better utilization --The improved network throughput, the lower message loss and traffic load balancing achieved by using the proposed LBR scheme s is indicative of better utilization than that of the conventional shortest-path routing schemes.

A. Simulation Model
In this simulation, the assumed physical network is the simulation network topology given in Figure   12. All links have a bandwidth of 1.5 Mbits/sec. We assumed that there are no link or node failures.
Each node has a buffer space of 50,000 bytes. The processing time of a data message at each node equals 1 µsec. In order to calculate the cost of a path in the network, we use the hop count. When we use the hop count as a link cost, the cost of each link is 1. The propagation delay of each link is 1 msec. The workload consists of FTP (file transfer protocol) and telnet connections. All FTP and teln et connections have the following parameters: the data message length equals 512 bytes, the inter -message generation time is 1 or 10 msec, and the window size is 500 messages. Connections start when the simulation begins and they are considered never-ending.
The routing algorithms used in our simulations are shortest path routing scheme ( LSR) and a proposed load-balanced routing scheme (LBR-BR2). LSR is a link -state routing protocol. We compare the performance of our LBR-BR2 protocol against that of LSR.
We consider the performance measures of throughput, message delay, and message loss. The measurement interval of each simulation is 100,000 msec.