Stability of Interior Routing and Load Balancing Techniques

The consequences of deploying a marginally stable solution can be a sudden and catastrophic transition to an unstable state. Events related to instability vary in severity. Network meltdowns are often documented only in standards forum discussions or operational forum discussions and therefore being otherwise undocumented become the topic of Internet Folklore. Route oscillations can have a very detrimental effect on the network performance perceived by end users. In the mid-1990s, severe route oscillations could cause routers to fail, resulting in more route oscillation and eventually a provider meltdown. While router software today is more robust and therefore meltdown is very unlikely, performance degregation due to route oscillation as perceived by user traffic can be quite severe. Slow route oscillation, also known as route slosh, results in performance degregation though not as severe. This document outlines what the routing community knows and doesn't know about the topic of route stability in the types of networks deployed today and in potential deployment based on existing IETF work. A brief section also mentions areas for research (). This document tries to avoid straying from well understood routing protocol behaviors.

Related terms are grouped together. The terms listed below describe undesireable routing behaviors. The terms are listed in order of decreasing severity. A routing meltdown is a sustained period when a very substantial number of routers either repeatedly crash and reboot or repeatedly lose routing adjacencies. Route instability is a sustain period when route change has a cascading effect and causes widespread transient problems such as route black holes or routing loops. A route loop occurs when a cycle exists in forwarding entries. Some protocols are prone to creation of transient route loops. A black hole exists when forwarding entries do not get packets to their destination. Transient black holes may occur during route change in some routing protocols. Route oscillation is a condition where routes keep changing in the absense of change in any conditions which should legitimately cause route change. Route slosh is a proper subset of route oscillation. When route slosh occurs routes keep switching among two or more states where limited brief stability occurs between state changes due to protocol timers or other factors. Route slosh is essentially just slow route oscillation. The set of terms below describe normal routing behaviors and other routing terms. Related terms are grouped together within this list. Route change is as the name implies when routes and the corresponding forwarding entries change. Route change may occur for many reasons. An SRLG is is a group of links which due to a shared physical resource are likely to fail simultaneously or nearly simultaneously. A network fault occurs when a node or link fails. Node protection is modeled using SRLGs covering all of the links adjacent to a given node. A multiple fault refers to the failure of multiple resources causing links that are not covered by a common SRLG to fail. Interior gateway protocols (IGPs) commonly use link metrics, which is synonymous with link costs, to determine a best route. See and for discussion of IGPs. Traffic engineering (TE) involves capacity information as well as the simple link metrics used in link state protocols. The primary focus here is the type of traffic engineering commonly practiced in RSVP-TE networks. See and for discussion of MPLS TE. Route recovery refers to getting traffic flowing after some event such as a network fault disrupts forwarding. Route recovery may involve protection and/or restoration. Route protection refers to a form of recovery using an alternate path, generally involving no signaling after the fault to begin using the alternate path, and involving either local detection of the fault or a simple proactive connectivity verification. In MPLS TE, the use of protection paths with zero bandwidth allocation is common. This is discussed in . Route restoration refers to recovery that involves exchange of routing information and/or some form of signaling to set up an alternate path. A definition of the term "multipath" can be found in . Common forms of multipath include Equal Cost Multipath (ECMP), Ethernet Link Aggregation, MPLS Link Bundling. See for discussion of existing forms of multipath. A local decision is one in which a single network element is involved in the decision. A distributed decision is one where the independent decisions of multiple network elements have an effect on the outcome. Offline route computation is where a centralized global routing decision with the results of the decision passed to all affected network elements. Global optimization is often cited as its advantage, though in IP networks it is seldom used due to very slow route restoration.

This document focuses primarily on routing stability of the interior protocols used in service provider (SP) networks and on load balancing techniques used in SP networks. Service providers almost exclusively use link state (LS) routing protocols as interior gateway protocols (IGPs) rather than distance vector (DV) protocols. The two link state routing protocols commonly used are OSPF and IS-IS . The MPLS LDP protocol creates MPLS forwarding entries which either follow IGP routes or behaves in a manner similar to a DV protocol. Techniques such as IPFRR and LDPFRR alter the transient behavior of the IGPs or LDP and are discussed briefly. MPLS traffic engineering (TE) as practiced using RSVP-TE makes use of IGP extensions based on to carry TE information. MPLS-TE has inherently different behavior than IGP routes alone or MPLS using LDP. There has been activity in the IETF related to using secondary link metrics related to delay, jitter, and loss . This includes use of these secondary metrics for MPLS TE path selection . Load balancing using hashing techniques date back to at least the mid-1980s, but were not formally documented until and though focusing solely on already outdated use of hashing in load balancing. It was not until documents such as and that these load balancing techniques were well documented. There is also interest in dynamic load balancing which will also make use of delay, jitter, and loss information, either locally available or IGP advertised . The topic of stability with regard to this type of work is also discussed. Much has been written about the stability of the BGP routing protocol. However, since the focus of this document is on interior routing stability, discussion of BGP stability is out of scope. There is some brief discussion of the impact of interior routing stability on BGP routes in and . Load balancers typically used by enterprise and to some extent in data centers are not within scope. This type of load balancer is not used in SP networks are the balancing algorithms are largely proprietary.

The greatest concern regarding stability and oscillations has been in service provider (SP) networks, simply because any instability which occurs more readily at large scale has in the past affected SP networks first. It may be that today's very large data centers now pose unique scalability and stability problems as well, but the focus remains on SP networks. Some forms of instability can occur in enterprise networks and even very small networks, however common practice has evolved to make such occurances acheivable only through severe misconfiguration. This section provides a brief introduction to a set of protocols in use in SP networks today. The focus is mainly on link-state routing protocols and LDP, on MPLS TE, and on multipath load balancing. These are the common underpinnings of SP networks and a recent topic of stability concerns due to proposed extensions to today's common practice.

Interior routing protocols are commonly referred to as interior gateway protocols (IGPs). IGP is somewhat of a historic term but still widely used. There are two types of IGPs in widespread use in IP networks. There are numerous alternate techniques described in research, but with no deployment. The two common types are Distance Vector (DV) and Link State (LS). In later sections only the behavior of LS protocols is discussed since DV protocols are seldom if ever used in large SP networks.

Distance vector (DV) routing protocols are rarely if ever used by service providers. Distance vector routing protocols exchange distances to remote destinations. One of the earliest DV protocols in widespread use was Routing Information Protocol (RIP). The first version is now known as RIP-1 . RIP-1 was used in the 1980s and 1990s but replaced by RIP-2 due to RIP-1 shortcomings, most severe at the time being lack of CIDR support. RIP-ng provides an extension to RIP-2 which supports IPv6. RIP is generally only used in small networks such as homes, small offices, and small enterprise. RIP's advantage is simplicity. RIP's disadvantages are many. Among them are slow convergences as networks become large. Another DV routing protocol is Enhanced Interior Gateway Routing Protocol (EIGRP). EIGRP is a much more advanced DV routing protocol than RIP, eliminating most if not all of RIP's shortcomings. EIGRP was a Cisco proprietary protocol roughly until 2013 when Cisco gave permission for others to use EIGRP, recognizing that no SP would use EIGRP due to the vendor lock-in. Yet another DV routing protocol is Babel which is well suited for wired and wireless network and is particular well suited for ad-hoc wireless networks. Babel is not expected to see any deployment on SP networks, but could perhaps displace RIP on small networks such as home, small office, and enterprise.

Link state (LS) routing protocols build a topology database based on each router advertising a set of neighbor adjacencies and providing information about the router's capabilities and about each adhacency. This topology database is known as a link state database (LSDB). Among the advantages of LS protocols is the ability to carry additional topology information useful for other purposes, such as MPLS traffic engineering (TE) (see ). The two major LS protocols are open shortest path first (OSPF) , defined by the IETF, and intermediate system to intermediate system (IS-IS) , defined by ISO but extended by IETF for IP use and for MPLS TE use. Differences between OSPF and IS-IS have sparked heated discussion, though most experts concede that use of either is more a matter of preference than strong technical merit. OSPF sends a single router link state advertisement (LSA) per router with multiple opaque LSAs carrying additional information. IS-IS sends a single link state protocol data unit (LSPDU) per router with TLV packed into multiple LSPDU fragments. Multiple LSPDU fragments are needed because the size of the complete LSPDU often far exceeds the link MTU. Arguments regarding the technical merit of one of these two LS protocols over the other are usually focused on information packing and flooding efficiency. In both protocols the link state database (LSDB) can become very large in large networks, particularly those using MPLS TE. Nearly identical MPLS TE extensions exist in OSPF and ISIS, with information exchanged in relatively small type-value-length tuples (TLVs) extensions carried in OSPF opaque LSA or in the IS-IS LSPDU fragments. In OSPF related TLVs are packed into LSA. In IS-IS many related and unrelated TLVs are packed into a given LSDB fragment. The behavior of LS protocols is discussed in detail in .

MPLS traffic engineering (TE) makes use of additional information carried in the link state (LS) routing protocols OSPF and IS-IS (see ). MPLS label switched paths (LSPs) are set up using TE extensions to RSVP, known collectively as RSVP-TE. The base RSVP-TE protocol specification is . The behavior of MPLS TE is discussed in detail in .

The MPLS Label Distribution Protocol (LDP) can either make use of the best path as computed by the IGP or act as an independent DV protocol. IETF has decided not to add TE extensions to LDP . MPLS provides a light weight tunneling encapsulation and LDP provides a relatively light weight protocol to support MPLS forwarding and therefore to efficiently carry pseudowires (PW) and virtual private network (VPN) services. LDP creates point to multipoint (P2MP) LSPs directed to specific forwarding equivalence class (FEC) rather that the point to point (PTP) LSPs commonly used in RSVP-TE.

IP/LDP Fast Reroute (IP/LDR FRR) is also applicable in MPLS networks. ECMP and Loop-Free Alternates (LFA) are well established IP/LDP FRR techniques and were the first methods to be widely deployed. Work on IP/LDP FRR is ongoing within the IETF RTGWG. Two topics actively discussed in RTGWG are microloops and partial coverage of the established techniques in some network topologies. covers the topic of IP/LDP Fast Reroute microloops and microloops prevention. RTGWG has developed additional IP/LDP FRR techniques to handle coverage concerns. RTGWG is extending LFA through the use of remote LFA . Other techniques that require new forwarding paths to be established are also under consideration, including the IPFRR "not-via" technique defined in and maximally redundant trees (MRT) . ECMP, LFA (but not remote LFA) and MRT swap the top label to an alternate MPLS label. The other methods operate in a similar manner to RFC 4090 facility backup and push an additional label. IP/LDP FRR methods which push more than one label have been suggested but are in early discussion. The IPFRR techniques apply to LDP fast reroute (LDPFRR). Some techniques make use of tunneling to improve coverage. The tunneling could in principle be provided by GRE but in practice MPLS with LDP is most often used to avoid a potential failure of a specific next-hop or specific SRLG. The techniques which use tunneling require that the router known the path that other routers will be using to know that the potential failure will be avoided and therefore require the use of a LS IGP rather than a DV IGP or no IGP and use of LDP as a DV protocol.

Network protection and restoration are collectively know as recovery. Getting traffic to flow again after a fault is network recovery, regardless of how it is accomplished. Route protection is often handled largely in hardware or forwarding card firmware rather than control plane software and can acheive recovery within a few tens of milliseconds. While protection is fast, protection against multiple faults is highly impractical. Restoration may involve flooding updated routing information and may involve signaling. Therefore restoration is inherently slower than protection but covers all variations of multiple faults except those multiple faults which cause network partitioning and are inherently irrepairable. The practice of zero bandwidth protection in MPLS FRR relies on traffic class (TC) markings and diffserv to provide very fast recovery for high priority traffic with no cost due to resource reservation of the protection paths. This protection of high priority traffic may occur at the expense of transient congestion for lower priority traffic. When combined with restoration, particularly if restoration is relatively fast, any congestion of lower priority traffic is eliminated or at least minimized shortly after protection goes into effect. A very cost effective deployment, and therefore quite common, is to use MPLS FRR for very fast (tens of msec) single fault protection and to optimize router code for fast (well under a second) restoration. Zero bandwidth reservation on the protection paths is commonly and perhaps exclusively used today. The fast restoration covers arbitrary multiple faults and relieves congestion after initial zero bandwidth FRR protection. This type of deployment is aimed at providing low cost and high availability for preferred services and also providing low cost and reasonably short disruptions for lower priority services when faults occur.

Multipath load balancing in use today includes Equal Cost Multipath (ECMP), Ethernet Link Aggregation , and MPLS Link Bundling . In each case the load balancing involves a local decision and generally uses static load balancing. In all of these cases the load balancing is based on IP header and MPLS label stack information as described in . Ethernet Link Aggregation using Ethernet MAC addresses is entirely useless in SP networks or any routed network. Therefore Ethernet Link Aggregation as implemented by routers also uses IP header and MPLS label stack information rather than Ethernet MAC addresses. Static load balancing, being static, is inherently stable. Dynamic load balancing is still a local decision, but where measurements are made and the load balance is adjusted based on those measurements. Measurements can be based on output link utilization, queue occupancy or queuing delay, or queuing loss including active queue management (AQM) loss, or some combination of these. Dynamic load balancing has the potential to be unstable but has been successfully deployed in SP networks providing existance proof that stable implementations are possible. The behavior of multipath load balancing is discussed in .

Rarely if ever are service provider (SP) network deployments exactly alike, but most have a great deal in common. Nearly all major SP (possibly all) use either OSPF or IS-IS. Some use no MPLS at all, though this subset is shrinking. Many use LDP but not RSVP-TE. Some use both LDP and RSVP-TE. Some use RSVP-TE but not LDP though this may be less common than a decade ago. The use of MPLS FRR is thought to be more common than IPFRR or LDPFRR at least among larger providers. Protection is not always provided. Fast restoration is considered essential even where protection is used due to the high occurance of multiple faults despite efforts to account for shared resources using SRLG. All large SP use multipath techniques, typically ECMP and Link Aggregation. Static load balancing has generally been used. Though dynamic load balancing has been successfully deployed, today static load balancing may be exclusively used due to the disappearance of the one core router supplier supporting dynamic load balancing. Given only these set of tools and given fixed link metrics, it is difficult to build a network which is unstable. It is possible to build an unstable network, even with fixed link metrics, if scaling limitations described in are ignored and exceeded, but as mentioned in , with today's common practices this situation is acheivable only through severe misconfiguration.

Various existing routing protocols exhibit different transient behaviors and different tendencies toward specific problems. Any problem which is transient is generally considered less severe than a problem that is persistent. This section describes some common problems that new protocols, implementations, and deployments need to avoid where possible. , , and discuss the tendency of some routing protocols or techniques to exhibit these various problems.

After a fault a transient black hole at the failed resource may be unavoidable. Some routing protocols have a tendency to form transient black holes elsewhere when a topology change occurs and can also form transient loops in some circumstances. Transient loops can cause severe congestion within the loop, affecting traffic which should not have been impacted from the initial fault.

The most severe long convergence time problems are almost always due to poor implementations or badly designed networks. However, even among competent routing protocol implementations and competent network designs, some routing protocols are more prone to long convergence times than others. While IGP protocols are not entirely immune to long convergence times, MPLS TE is particularly prone to long convergence times unless care is taken to avoid scaling issues. The shortest path first (SPF), sometimes called shortest path tree (SPT) computation in a LS protocol is time complexity order NlogL, where N is the number of nodes and L is the number of links. For MPLS TE, one CSPF computation is needed for each LSP and each CSPF computation is order NlogL. A fault may affect the majority of the LSPs for which a given LSR is ingress and therefore may involve order N^2logL (N squared log L) computation for that LSR.

There are many situations in which the most common mode of failure or a common mode of failure causes simultaneous failure of multiple nodes or more often multiple links. For example, many links may traverse a common physical fiber or fiber segment using wave division multiplexing (WDM). These shared resources can be taken into account using shared risk link groups (SRLG) when computing protection paths. A multiple failure is one where network elements (nodes and/or links) which do not share any common SRLG fail simultaneously. Multiple failure occur on a regular basis. These occur as a result of a potential shared resource failure which was either unknown or overlooked or considered so improbable as to make it more practical to ignore it. Examples of completely unavoidable multiple failures are fibers on the same earthquake fault line, fibers crossing the same bridge (where only bridge collapse would cause simultaneous failure), and fibers on both sides of railroad tracks (where only a train derailment would cause simultaneous failure), widespread flooding. There are many other less dramatic failure scenarios which are more common. Protection does not cover multiple faults. Covering even double faults using protection adds significant network complexity. Where protection requires resource reservation such as in transport networks, the need to reserve a multiple of the resource being protected make covering multiple faults prohibitably expensive in most cases. Using protection covering only single faults and having no restoration capability is unacceptable for IP networks and unacceptable for many of today's services. Protection covering only single faults with no restoration is common practice for transport protected services largely accounting for steep decline in use of protected transport services. Using restoration with excessively long convergence times is also unacceptable for IP and for many of today's services.

An extremely severe problem is persistent route oscillations. Other than buggy implementations or exceeding implementation scaling limits, producing oscillations in today's SP networks would require misuse of link metrics which vary with traffic load or some other form of positive feedback (see ). When protocol timers or other protocol features limit oscillation to a fairly long period, the resulting low frequency route oscillation is known as route slosh. Route slosh is also a severe problem if it is persistent. If damped such that the slosh impact fairly quickly reduces to zero, then some slosh may be acceptable.

Any system with positive feedback is unstable. For example, consider a simple electronic amplifier. With DC positive feedback, any small signal would cause a saturated output. Amplifiers employ negative feedback to improve linearity. However if the feedback is delayed a little, then positive feedback can occur at high frequencies which can result in a fried amplifier. The classic case of positive feedback in routing is the delay based link metrics used briefly in the 1980s . Various forms of damping are discussed in the paper describing this use of delay based link metrics, some with limited success, though in the end ARPANET discontinued using delay based link metrics. Use of delay, jitter, or loss has the potential for creating a situation where a positive feedback loop exists and therefore where oscillation occurs. This problem is discussed in , and . A similar feedback may occur in MPLS TE if an "auto-bandwidth" feature is used, though occurance unlikely. This feature varies the requested LSP resource reservation based on observed traffic measured at the LSP ingress. Increasing the LSP resource reservation could put an LSP on a longer delay path. The additional delay may reduce the offerred load slightly and allow the LSP to be moved back to its original path. In practice this problem is avoided by using only very long term filtered measurements, though the result may be very low frequency route slosh.

Routers have finite resources. Scaling problems can occur in the following situations. When protocols are poorly designed and inherently consume excessive resources. When specific types of deployments are used with protocols which consume excessive resources in those configurations. When implementations provide inadequate resources to support what today are considered normal deployments. Resources which may become exhausted include specialized buffering, storage, and computational resources. Specific examples of scaling problems are given in and .

Link state (LS) protocols are simple and effective. Transient behavior during route change is somewhat prone to route loops but with reasonably good implementations convergence is fast and therefore transient problems are short lived. The two prominent link state (LS) protocols, OSPF and IS-IS, exhibit very similar behavior. Only where there is a significant difference will either of the two protocols be mentioned in this section.

Transient problem arise from the hop by hop nature of forwarding with direct application of LS protocols. Each router makes a decision to change routes independently and the order in which routers learn of a LSDB change and make a decision based on changes can vary. Transient situations can arise where routers in a subset of the topology have installed routes based on a slightly different LSDB and either route traffic in a two router loop which is suppressed resulting in a black hole, or in a multirouter loop causing a forwarding loop. As long as convergence is fast, the impact of transient problems is small. A forwarding loop can create very severe but temporary congestion. The temporary congestion created by a forwarding loop will often affect traffic to destinations that should not have been affected by the route change. LS convergence, including flooding, routing decision, and installation of forwarding entries today can be well under a second but does not rival the few tens of milliseconds acheived by protection mechanisms.

Links are considered in an "up" or "down" state depending on whether some criteria is met. The most simple criteria, successful delivery of routing protocol hello packets, is among the most problematic. Under some conditions, a link may undergo excessive state changes unless the implementation takes some action to prevent this situation. For example, hysterisis in the criteria may be used or link state may be held in a "down" state as a result of frequent change. Excessive transitions may occur when link state "down" transitions are quickly triggerred by some number of errors and are quickly reverted to the "up" state if the error rate falls below the target rate. This can be solved by one or more of a number of means: Hold the link in the down state for a minimum time period which is long compared to the measurement interval. Make use of hysteresis, such as a long period of completely error free operation before changing link state back to "up". Make use of a technique which progressively keeps the link down for longer periods depending on the rate at which the error threshholds are crossed. For example, can be applied to IGP link state at the originating router as well as applied to BGP. If link state changes are frequent, even just for a single link, then a network running a LS protocol may spend a significant amount of time in a transient state where route loops and black holes are possible.

A problem unique to IS-IS is related to LSPDU fragement packing and shifting boundaries. If an implementation maintains a sorted order among TLVs, maintains tight packing of TLVs into LSPDU fragements, and moves TLVs from one LSPDU fragment to another to maintain the tight packing, then a transient condition can occur where the LSPDU fragement from which a set of TLV is withdrawn arrives first and an SPF is triggerred before the change to the LSPDU fragement where the set of TLVs have been added is learned. A worst case implementation strategy tries to pack the set of LSPDU fragments as tightly as possible and in keeping them tightly packed frequently moves TLVs across LSPDU boundaries when changes occur. The boundary shift due to a link changing may result in erroneous withdrawl of a set of unrelated TLVs. In addition, a larger set of LSPDU fragments have to be advertised if boundaries are shifted than if less tight packing is targeted. There are numerous strategies to avoid moving fragment boundaries and to eliminate the erroneous withdrawl of TLVs. One strategy is to set a target in which LSPDU fragments are packed within a fractional range of full. For example a target 1/2 to 3/4 might be used. A second strategy is to separate the action of updating TLVs from the action of moving boundaries to keep within the target packing range and when moving boundaries delay sending the withdrawl of the TLVs relative to their addition in the adjacent fragment. Other strategies to avoid this problem are possible.

Link state (LS) convergence time includes flooding, routing decision (SPF plus changes to inter-area or inter-level routes plus changes to external routes), and forwarding information base (FIB) update. The forwarding entries used by external routes carried within the IGP and by other routing protocols such as BGP are often affected by the underlying IGP intra-area or intra-level routes. This second set of routes may have to be changed, though with multistage forwarding lookups used in many routers today most secondary changes are completed automatically. Operational experience has shown that flooding should be prioritized over route recomputation. The SPF computation on fast hardware with a small topology may occur in under 10 msec, and in some cases under 1 msec, however on slower hardware and with larger topologies, an SPF may exceed 100 msec. If reflooding at each hop is delayed by SPF time, the overall convergence is slowed. In addition, a single resource fault may result in multiple topology changes (see for background). Prioritizing reflooding and slightly delaying SPF helps ensure that the next SPF will be based on the full set of changes. With a decision to prioritized reflooding and to delay SPF the question arises as to how long to delay SPF. For most networks an approximate worst case fault detection time is known. This detection time often has a lower bound dictated by speed of light propogation delay (about 1 msec per 200 km or 125 miles of fiber) and a practical upper bound that is close to the lower bound. For long haul terestrial networks a lower bound of 10-20 msec is common. For trans-oceanic fiber this can be 50-100 msec. A common practice is to set a timer when the first IGP change is received, reflood as quickly as possible, and perform an SPF when the timer elapses. With a timer that is a small multiple of worst case fault detection time the SPF will tend to occur with a full set of LSDB changes rather than a partial set of changes. In a multithreaded control processor architecture, it is possible to continue reflooding while an SPF is in progress, as long as newly arriving changes do not affect the SPF in progress. In such a case a delay in starting an SPF is often useful, but the delay can be made shorter, particularly with a very fast SPF relative to worst case fault detection time. With a very fast SPF, the limiting factor may be forwarding entry install time. Convergence time is also impacted by scaling issues discussed in .

Scaling issues result from exceding the practical capacity of a given resource. Some of the affected resources include the following: Severe scaling problems can occur in full mesh or very dense mesh networks due to limited short term buffering of flooded data. See . Computation issue seldom occur except with poor implementations or networks containing a very large single OSPF area or IS-IS level. See . An extremely large number of external routes carried by a LS protocol can cause scaling problems. See . The amount of time required to install an extremely large number of forwarding entries can cause problems. See . Any of these problems can be made slight better by an exceptionally good implementation or much worse by an unusually bad implementation. Each of the subsections that follow illustrate a type of scaling that is capable of overwhelming even the best implementations through careless network design. Particularly bad implementations can have their own unique scaling limitations. The set of scaling issues described here is almost certain to be incomplete.

A physical fault may affect a number of neighbor adjacencies advertised by a number of routers. This is generally a relatively small number, perhaps tens of adjacencies and a smaller number of routers. In OSPF each router advertises a new router LSA plus it may advertise opaque LSAs with additional information about the affected links. In IS-IS the equivalent of the router LSA is generally packed into one LSPDU fragement. One or more additional LSPDU fragements may carry the TLVs with additional information about the affected links. IS-IS passes each LSPDU fragment as a full MTU packet, which for SP core networks is often in excess of 8 KB. Each router originating a set of changes passes this updated information, perhaps a few 10s of KB, to each remaining neighbor. If the information is not already known from some other source, then any given neighbor will then reflood to each of its neighbors. Most SP networks have an average of about 2.5-4 core neighbors per core router. For this type of topology, known as a sparse mesh, LS protocol flooding (absent any bad implementation) is fast, efficient, and free of scaling problems. A very dense mesh network or a full mesh network represents a pathologic case. Attempts to use an overlay model often result in a full mesh or a very dense mesh. In a full mesh of N routers, each router originating a set of changes is adjacent to every other router, or N-1 routers. Each of the N-1 routers will reflood to its N-2 neighbors. The original 10s of KB origninated as a result of a single fault is multiplied by a factor that is order N^2 (N squared). Any given LSR may receive N feeds of the same information. For example, if N is 1000 and 40 KB of change was originated, a given router may receive 40 MB of route updates when a single link changes state. This can overflow the memory allocated for socket buffers and cause a lot of this flooded information to drop and have to be retransmitted, lengthening convergence. Any part of the feed that is dropped at one router from one neighbor but successfully received from another neighbor will be sent the other way toward the router from which the information was dropped. If a router fails in a full mesh network of N nodes, the situation is much worse than for a link failure. That one router is adjacent to N-1 other routers and each must originate at least one LSA or LSPDU fragement and likely a few. We'll call the number of LSAs or LSPDU fragments originated per router K. Each of these N-1 routers originate K LSAs or LSPDU fragments and send these to their N-2 neighbors which will reflood to N-3 of their neighbors. The total number of LSAs or LSPDU fragements sent is order K*N^3. Each router could receive a feed of K*N-1 LSAs or LSPDU fragments from each of N-2 neighbors in a short time period. For example, for K=5, the MTU is 8KB (IS-IS), and N=1000, a router can receive a set of 40 MB feeds totaling about 40 GB. No router today or in the forseeable future has that amount of socket buffer space. Flooding provides a fast way to distribute route changes and works extremely well on a typical SP network with a sparse mesh. Flooding simply does not work well in large full mesh topologies or even large very dense mesh topologies. For example, some attempts to overlay IP networks with a LS protocol over an ATM full mesh in the mid 1990s yielded severely unstable (and therefore short lived) networks. Full mesh or dense mesh topologies should be avoided.

The shortest path first (SPF) computation used in LS protocols requires time order NlogL, where N is the number of nodes and L is the number of links. In network designs with a node degree that is relatively independent of N, for example a node degree of 3-5 on average, computation time can be expressed as KNlogN, where K is a constant. A rule of thumb which provided a reasonable estimator around the year 2000 used KNlogN as an estimator with K=1usec. This was actually much faster than most SPF of that day, but today K might be smaller. For K=1usec and N=1000, KNlogN would be about 10,000 usec or 10 msec. From the prior example, it is clear that a network with close to 1,000 nodes in a single OSPF area or IS-IS level was supportable at that time (circa 2000). Some five to ten years earlier and with slower processors and some poorly written and therefore slower SPF code, a rule of thumb of 400 nodes was commonly cited as the upper limit for the size of an OSPF area. The figure 400 was said to be emperically determined. Though processors are getting faster and SPF code gradually improves as well, there will always remain a practical limit to the number of nodes that can be supported in a given OSPF area or IS-IS level. At some point convergence times become excessive. In extreme cases protocol timers of other protocols such as BGP (configured to use faster timers than the BGP defaults) can expire due to excessive convergence times and cause further network disruption.

Due to OSPF and IS-IS periodic route refresh and due to the way in which both protocols implement flooding, a very large number of external routes should not be carried by the IGP. Implementations have been known to tolerate up to tens of thousands of external routes, but generally will become unstable with a BGP global routing feed or worse yet, multiple BGP feeds into the IGP.

Operational experience supports implementations installing forwarding entries for OSPF intra-area or ISIS intra-level routes first, then installing forwarding entries for any inter-area routes or routes from another level, and then external routes. External routes may be carried in the IGP or carried by another protocol, most commonly BGP. BGP may carry a very large number of routes. By installing intra-area or intra-level forwarding entries first, accurate forwarding with an area or level is better ensured. By giving the next priority to inter-as or inter-level routes, accurate forwarding within the infrastructure of the autonomous system (AS) is better ensured. This ensures that protocols that depend on this infrastructure will continue to operate in time of high route change. Many forwarding engines allow either two stage, multistage, or fully recursive forwarding lookup. Supporting multiple stages allows the forwarding for external routes which depend on internal routes to automatically change if only the internal route has changed. This can dramatically decrease the number of forwarding entries that need to be changed and reduce effective convergence time In a particularly bad implementation if forwarding entry installation is slow, and if only single stage forwarding entries are supported, and if IGP routes are not prioritized, forwarding entries can remain incorrect long enough for IBGP peerings to go down (particularly if BGP timers are reduced from the BGP default values). In such cases it is possible for the route change resulting from IBGP peerings to cause further churn. Not since the late 1990s has there been implementations with severe forwarding entry install time problems deployed in SP networks. The problem is only worth mentioning in hopes that new vendors don't repeat these mistakes.

The behavior of MPLS using LDP is very similar to the behavior of LS protocols. Some DV protocols support loop suppression through maintaining a sequence of hops and this too is supported by LDP. The LDP loop suppression avoids creating transient loops in LDP when transient loops occur in the LS routing protocol. LDP therefore has better transient behavior than a LS routing protocol alone. Services which are carried only by LDP, such as PW and VPN services, cannot loop. When IP is carried by LDP and where there is a fallback to IP forwarding, transient loops will remain a problem for that IP traffic.

IPFRR and LDPFRR are described in . IPFRR provides protection against single faults. IPFRR can include SRLG consideration. Timer mechanisms help avoid microloops by delaying SPF based on distance from a fault, in that way sequencing the transition from protection to restoration . If multiple faults occur IPFRR has the potential to be slightly counterproductive. IPFRR may have a greater potential to create transient loops in the event of a multiple fault. For example, LDP without LDPFRR can effectively suppress loops, but with tunnels and label stacking can create transient loops. In addition any loops that form when a multiple fault occurs may be made to persist longer due to the timers introduced to suppress microloops. The protection against single faults offered by IPFRR and LDPFRR greatly outways any transient misbehavior that may occur when multiple faults occur.

Past use of delay link metrics in LS protocols has resulted in oscillations (see for background). The impact of path change due to link metric change has a major effect on traffic levels and therefore an effect on the link metric. The interdependency of path, traffic levels, and link metric creates the potential for strong positive feedback and severe oscillations. Oscillations due to the use of link metrics based on delay, jitter, or loss in a LS protocols may be unavoidable. This is discussed further in .

MPLS traffic engineering (TE) forwarding is not decided hop by hop. MPLS TE Label Switched Path (LSP) path selection is determined entirely by the ingress LSR and nearly always include only strict hops in the explicit route object (ERO) which dicates the LSP path in RSVP-TE signaling . The ingress LSR makes its path decision based on estimated bandwidth requirements of the LSP and link bandwidth availability advertised in the IGP. MPLS TE traffic cannot flow on an LSP until the LSP is setup using RSVP-TE signaling. At setup time, the estimated bandwidth requirements serves as a bandwidth reservation. Due to the bandwidth accounting and the ability for ingress LSR to direct traffic, the behavior of MPLS TE is significantly different than that of LS protocols. MPLS TE makes use of resource accounting, avoiding attempts to utilize more capacity than any part of the topology can support. In this way networks using RSVP-TE differs from those using LDP. In addition, some MPLS TE implementations allow multiple LSP between a given ingress and egress, each potentially taking different paths, with load split across these LSP in proportion to their reserved bandwidth. This is a form of static load balancing (see and ).

MPLS TE LSP are created using an explicit path from ingress LSR to egress LSR. There is no opportunity for differences in the LSDB to cause forwarding loops. MPLS TE therefore has better transient response than LS protocols in that it is always loop free. MPLS FRR provides protection against single faults. Since it is initiated at the point where the fault is detected, known as the point of local repair (PLR), MPLS FRR is very fast, typically a few tens of milliseconds. This provides an excellent response to single faults, with restoration time often only slightly longer than the fault detection time. When multiple faults occur, MPLS FRR may provide only partial protection, unable to protect LSP where the fault affects both the primary path and protection path. In this case recovery relies entirely on restoration. In either cases of single fault or multiple fault, restoration is used to return to a close to optimal traffic layout and restore protection to LSP for which protection is needed. Restoration times can be long if care is not taken in network design or when using implementations which have not been optimized for very fast restoration.

MPLS TE relies heavily on fast restoration, though fast restoration is not easily achieved. Only the best implementations acheive restoration times in the low number of seconds for regions of MPLS TE deployment employing a full mesh of LSP with a hundred to a few hundred nodes. The reason that acheiving fast MPLS TE relies restoration is difficult is the computational complexity of TE restoration. This is discussed in . Providers can subdivide their topology into multiple interconnected regions with a full mesh of LSP within each region, greatly improving convergence time, even if the topology is served by a single OSPF area or IS-IS level. For example, an OSPF area of 1,000 nodes can be roughly subdivided into 10 regions of about 100 nodes, plus a core which interconnects the 10 regions using two or more LSR from each region and where the core may have well under 100 nodes. Being ten times smaller, the convergence in each region can be expected to be more than a hundred times faster. If edge-to-edge LSP are needed, as they often are, either LDP can be run over the RSVP-TE LSP or RSVP-TE hierarchy can be used. Further compounding MPLS TE convergence time problems is the possibility of crankback. Generally proposals in IETF which give an ingress LSR too little information to compute a path and rely too heavily on crankback are rejected for that reason. However in a situation where a fault has occurred and multiple ingress LSR may try to make use of a next best resource, crankback is likely to occur if the resource is too limited to accommodate all of the LSP setups. The impact of crankback can be reduced by implementations which sequence LSP setups, setting up the larger LSP first and delaying slighly those LSP setup for which a viable path still exists. The amount of crankback that occurs can be greatly reduced by these small delay between non-urgent LSP setup requests, prompt origination of LSDB changes particularly for links which are approaching capacity depletion, and quick reflooding. This short list only touches on the types of optimizations that have been implemented. A full treatment of the topic of MPLS-TE optimizations is beyond the scope of this document.

The most common scaling problem in RSVP-TE is due to the CSPF computational complexity described in and . A less common problem is overload of resources related to flooding similar to the scaling problems described in but potentially made more severe by overly aggressive origination of LSDB changes, in the worst case sending an LSDB change for each LSP setup. The scaling limitations of RSVP-TE are dramatically impacted by network topology. Providers can cause scaling problems through poor network design. The result of poor network design will be highly excessive convergence times. MPLS regions supporting a full mesh of LSP should generally not exceed 100-200 nodes, and keeping well under 100 where possible is advisable. Building a RSVP-TE full mesh over a lower layer full mesh overlay (for example, a layer-2 overlay) will cause scaling problems related to flooding. If the overlay is large, CSPF related flooding problems will also occur.

The problems with link metrics based on delay, jitter, and loss are not as severe with MPLS TE as they are with link state (LS) protocols (see and ) if individual LSP are slow to move relative to readvertisement. One or perhaps a few of many LSP may change a path to avoid a congested link, unlike LS protocols where a change in metric causes all traffic for which another path has become a lower cost to move. Where there is only a single LSP from any ingress to egress the problems of coarse granularity of traffic adjustment remains. This is discussed further in .

Multipath load balancing is briefly introduced in . The minimal requirement for multipath load balancing is that packet order is maintained within any given microflow, as called for in the diffserv architecture . MPLS-TP adds additional requirements that make it necessary to treat an MPLS-TP LSP as a microflow. Carrying MPLS-TP LSPs within an MPLS LSP is discussed in .

Common practice today involves static multipath load balancing based on the assumption that the hash that is performed on incoming traffic will evenly distribute the traffic over the hash space. For the case where a very large number of small microflows exists, this is a valid assumption and under 5% imbalance is common. In static load balancing a hash is performed over the input traffic and the hash space is divided proportionally to the capacity of component links. uses the term flow rather than microflow to acknowledge that the hash used in load balancing quite often is not based on the defined characteristics of an IP microflow in . Often only label stack information is used, which only works well if entropy labels or PW flow labels are used. Using a hash does not generally identify single flows but rather identifies sets of flows which all hash to the same value. Static load balance has an advantage of being simple and also being inherently stable. When a small number of very large flows exist, usually mixed with a very large number of very small flows, static load balancing will perform poorly. These large flows may be due to large PW flows, high capacity IPsec gateways or other applications. Dynamic load balancing can accomodate these large flows but has the potential to be unstable. Dynamic load balance stability is discused further in and .

In dynamic load balancing the same problem that exists for failed prior techniques but with significant differences. In both cases a reaction to excessive load moves traffic to another part of the network and in both cases moving traffic away from a congested part of the network changes the parameters being meaasured. There are differences between dynamic load balancing used in multipath and the load shifts in delay based link metrics in a LS protocol. These differences make it possible to ensure stability in dynamic load balancing where stability may not be possible for delay based link metrics in a LS protocol. Briefly some differences between delay based link metrics in a LS protocol and dynamic load balancing are the following. In a LS routing protocol a change to a link metric moves a lot of traffic. In dynamic load balancing based on hashing, the amount of traffic that can be moved at a time can be as little as the traffic in one hash bucket. In dynamic load balancing it is possible to determine the amount of traffic that will be moved before making an adjustment if counters are available in the load balancing hardware. When dynamic load balancing is used solely in local decisions, the measurement and adjustment is made within the same router. No protocol exchange outside the router is required. Stability in dynamic load balancing when used in local decisions has been successfully deployed and is more easily provably stable than dynamic load balancing used in a distributed decision. See . The topic of preventing feedback problems in dynamic load balancing is discussed further in .

Delay, jitter, loss, and link utilization based link metrics used in LS protocols or used in MPLS TE and multipath load balancing are all forms of load balancing. All of these forms of load balancing can be categorized in term of the following characteristics. Static load balancing is inherently stable. Dynamic load balancing is not inherently stable but neither is it inherently unstable. Dynamic load balancing granularity may be uncontrolled such as LS routing protocols or controlled. A controlled load balancing granularity may be very coarse such as MPLS TE where entire LSP are moved, or fine such as hash based multipath techniques. The amount of traffic moved in an adjustment to dynamic load balancing may be unknown or known. For example, a link metric change in a LS protocol forces a load adjustment but the magnitude of the adjustment is unknown until after the adjustment is made. When moving an LSP, the amount of traffic moved is known. When an adjustment is made in a hash based technique, the amount of traffic moved is unknown if there is no measurement available to determine the amount of traffic falling within a group of hash values, but can be known if counters have been provided in the forwarding hardware. The units of measurement and the units of load balance traffic adjustment may not be entirely compatible. For example, if delay, jitter, and loss are measured at the point of congestion and the measurements at the point of traffic adjustment are offerred traffic load, it is hard to predict how movement of traffic measured in Mb/s or Gb/s will impact delay, jitter, or loss, particularly given the behavior of TCP congestion avoidance. Measurements may be made locally and traffic adjusments may be applied locally in which case no protocol extensions are needed. Prior implementations of dynamic multipath operated in this way and were deployed successfully. Measurements may also be made and advertised by a LS protocol such that decisions made by numerous other nodes affect the traffic at the links being measured. Stability is more difficult to ensure with this type of distributed load balance decision. Any dynamic load balancing technique has timing factors that can impact stability. For example, the frequency of measurements and any filtering or hysteresis applied to measurement can be dictated by specifications or made advisory and modified in specific deployments. The frequency of adjustments to load balance and changes to the adjustment amounts can also be dictated by specifications or made advisory and modified in specific deployments. Each of the above characteristics affect whether a specific load balancing technique is stable. Since static load balancing is inherently stable, the remainder of this section focuses on dynamic load balancing. The following subsections focuses on the characteristics listed above. discusses the interaction of these characteristics and discusses factors in protocol design which affect the tendency toward feedback problems and potential instability.

It is not possible to ensure stability in a technique which uses uncontrolled load balancing, such as LS routing protocols. It may be possible to introduce timing changes or other changes which tend toward stability or reduce the impact somewhat (such as long timers). This seemed to be a goal of . With no way to ensure stability and very limited success in mitigating it, prior efforts to load balance in LS routing protocols were abandoned and future attempts strongly discouraged. Load balancing can generally acheive better results with a fine granularity than a coarse or very coarse granularity. However techniques which use a coarse granularity can be made stable. MPLS TE is inherently a load balancing technique which is somewhat static by virtue of the normally configured bandwidth reservations and with the coarse granularity of any given LSP bandwidth reservation. Stability is ensured if LSP remain on their current path even if better paths become available, subject to relatively long timers preferably with timer jitter (timer jitter is a random component included in the timer value). Hash based load balancing is a fine granularity load balancing with the granularity determined by the number of usable bits in the hash. For example, entropy labels or PW flow labels are limited to 20 bits of usable hash. This allows dividing traffic into potentially a million groups of flows. Most implementations today make use of tables which are much smaller than 20 bits and therefore provide a somewhat more coarse granularity.

The amount of traffic that will be moved when an adjustment to dynamic load balancing is made may be either unknown or known prior to making the adjustment. Dynamic load balancing with an unknown traffic adjustment amount has been deployed successfully. This success relies on there being a reasonably good correlation between the fraction of the hash space moved and the fraction of total traffic which falls into that hash space. When the traffic adjustment amount is unknown, very large flows occasionally cause a large change in traffic when only a small change was intended. In such cases a single hash value may have a disproportionately large amount of traffic. A pathologic behavior would be to continuously move a small range of hash values back and forth as a result of this. This problem is avoidable. A far better situation exists when the traffic adjustment amount is known. For MPLS TE LSP, the reservable bandwidth is known and most LSR can measure traffic on a per LSP basis. For hash base multipath few existing implementations have measurements over the hash space for a particular multipath. Making such measurement is not difficult but requires that the designer of the forwarding hardware had the forsight to provide counters in the appropriate places.

Measurements of delay, jitter, and loss at a point of congestion cannot be readily translated into measurements of utilization at a point where an adjustment to traffic load balance is made. For example, one or more large TCP microflows may be in the TCP slow start phase, where traffic flow increases exponentially until a loss or ECN notification is detected. Prior to congestion, rising utilization can be measured directly. After the onset of congestion, utilization hits 100% and can no longer be used as a measure of loading at the point of congestion. The amount of rise in delay and jitter or the rate of rise in delay cannot easily be used to determine the amount traffic to move. If loss is occurring, the amount of loss cannot easily be be used to determine the amount traffic to move but increasing loss may be correlated to the degree of overutilization. The measurement of delay and jitter is a measure of time, generally measured in milliseconds (msec). Loss is measured as a fraction of traffic. There is no reliable means to determine how much traffic should be moved from one link with a given queuing delay to another link with a different amount of queuing delay. The same applies to two links with different amounts of loss or one link with queuing delay but no loss and another link with loss. Jitter is the most problematic of these measures. Jitter is a function both of the link utilization and the burstiness of traffic sources. A rise in jitter can occur at very low link utilizations. Jitter has never been used as the basis of load balancing and questions remain about the feasibility of doing so.

Delay, jitter, and loss based link metrics used in LS protocols or used in MPLS TE and multipath load balancing are all forms of load balancing. Load balancing can be applied in two ways. Some types of load balancing can be applied locally, using only measurement made at local interfaces to impact the proportion of traffic going out those local interfaces. Load balancing can be applied in a distributed fashion where measurements are advertised, usually using additional information carried by a LS protocol and action is taken by many routers which make use of the resources for which measurements have been made. Some techniques are inherently local or inherently distributed. Delay, jitter, and loss based link metrics used in LS protocols or used in MPLS TE are inherently distributed. Ethernet Link Aggregation is a multipath technique which is inherently local. Other forms of multipath such as MPLS Link Bundling or any multipath technique which can make use of MPLS LSP as component links are neither inherently local or inherently distributed.

It is easier to ensure stability when load balancing is entirely a local decision. There is no protocol exchange required to do this sort of load balancing and there are no interoperability issues with neighbor routers. Therefore there has been no need for IETF to dictate how this is done. Common practice has evolved which is well known within the industry, widely deployed, and very similar across implementations. It has remained poorly documented in the IETF, being a local decision. Load balancing in Ethernet Link Aggregation is inherently a local decision. As commonly practiced today, ECMP and Link Bundling also make use of local decisions.

Few attempts to make use of distributed load balancing have reached the IETF. A prerequisite for distributed load balancing is a means to advertise the measurement results. This prerequisite is met by and but no means of applying these metrics has been defined. A very early attempt to make use of distributed load balancing was OSPF-OMP, ISIS-OMP, and MPLS-OMP . These drafts were abandonned due to provider lack of interest at the time. This was partially due to the difficulty in providing mathematical proof of stability. There has been a renewed though cautious interest in distributed load balancing as evidenced by and related work. Any use of and are distributed load balancing techniques. With these extensions with measurements based on delay may optionally include queuing delay. Measurements of jitter are inherently queuing based. Measurements of loss are also queuing based except for lossy links such as wireless links. The authors of these documents have intentionally put stability concerns out of scope due to the difficulty in ensuring stability and even greater difficulty of mathematically proving stability.

In any system with feedback careful attention to timing is needed to ensure that the system remains stable. For load balancing this may be easier to accomplish where measurements and traffic adjustments are entirely local. The following timing considerations are related to making measurements and propogating measurement results: Utilization, delay, jitter, and loss measurement interval for a direct measurement. Type of filtering of direct measurements, if any, applied to produce a reportable measurement. Amount of positive or negative change to reportable measurement needed before updating the reportable measurement. This amount should be a function of the time since the last reported measurement and a function of the importance of making an adjustment at that measurement level. The time necessary to propogate changes must be considered when trying to determine if a technique is stable. This may be a small but non-zero time. Upon receiving an updated measurement result, any traffic adjustment should be deferred for a target time period with jitter applied to the timer. The target time period should be a function of the following: the percent contribution to the traffic at the link or path where traffic would be moved from, the difference between the conditions at the link or path from which traffic would be moved from and the link or path where the traffic would be moved to, any other factor that the protocol specification deems relevent or that the implementation deems relevant and that is allowable by the protocol. Upon receiving an updated measurement result, any traffic adjustment amount should be a function of the following: the amount of traffic that needs to be moved from all traffic sources to acheive balance, the amount of traffic that can be moved from all traffic sources to acheive balance, the percent contribution to the traffic at the link or path where traffic would be moved from, the difference between the conditions at the link or path from which traffic would be moved from and the link or path where the traffic would be moved to, any other factor that the protocol specification deems relevent or that the implementation deems relevant and that is allowable by the protocol, any prior traffic adjustment between the set of links or paths and the time that has elapsed since that adjustment. It is important to note that for a given update there may be multiple more lightly loaded links or paths for which traffic can be moved to and there may be multiple more heavily load links or paths for which traffic can be moved from. The amount of traffic for which there is already a scheduled adjustment from or to a given link or path must be taken into consideration when computing the amount of a new traffic adjustment.

The goal of this section is not to produce an algorithm for traffic adjustments. The goal is simply to point out factors that will either increase or decrease the tendency toward feedback problems and potential instability. The most severe problem is positive feedback at some frequency (or time period) causing a growing oscillation until saturation is reached. Also very sever is an undamped oscillation, which may not grow but will continue forever. Oscillations often occur due to overshoot in making an adjustment. Overshoot can be caused in a distributed load balancing environment by multiple traffic sources each making an adjustment that individually would bring the network closer to balance but together cause overshoot. This is why the contribution of a given traffic source to the total traffic must be considered. If there is a no tendency for overshoot at all, there is a tendency to have no oscillation at all. If in the event of overshoot, subsequent adjustments are always less aggresive, then there will be a damped oscillation, known as a "ringing". Attempts to respond quickly to a traffic imbalance may result in some overshoot. Damping subsequent responses will reduce subsequent overshoot and reduce ringing. Small overshoot and damped ringing is undesirable but might be acceptable if there was some other gain such as faster reaction time. What follows are some suggestions to help ensure stability and improve damping. If there was a prior traffic adjustment between two links or paths, then the following type of bounds should be applied. If the new traffic adjustment is in the same direction, the adjustment amount should be no more than the prior adjustment amount multiplied by some factor (greater than one) that is determined to by the protocol to help ensure stability. For example, if the prior traffic adjustment was "recent" this factor might be two or less (a doubling in the next adjustment or less) and rise proportionately if this elapsed time was above the "recent" threshhold. If the new traffic adjustment is in the opposite direction, the adjustment amount should be no more than the prior adjustment amount multiplied by some fraction that is determined to by the protocol to adequately damp any ringing. For example, if the prior traffic adjustment was "recent" this factor might be one half and rise proportionately with time since last adjustment if this elapsed time was above the "recent" threshhold, reaching unity at some very long time where prior history may be less relevant. When an updated measurement result and a traffic adjustment is already scheduled, the computation of timer value to be used to defer taking action and computation of adjustment amount should be repeated. The updated adjustment amount should be used. The lesser of the remaining time on the prior timer and the new computed timer value should be used. Analysis of traffic adjustment scheduled time and traffic adjustment amount should be repeated each time there is a change to either side of an already scheduled traffic adjustment. When traffic adjustment is strictly local, this should not impose a high computational burden. For distributed load balancing, the computational burden in a large mesh network will have to be considered when considering protocol timer default value guidance and safe bounds of timer values. It is very difficult to mathematically prove that the specifications for a particular dynamic load balancing method is stable. So far, at best an argument can be made that specific provisions within a particular dynamic load balancing method strongly tend toward stability. See for further discussion.

So far there is no mathematic proof of stability for any given dynamic load balancing method. Having such a proof would go a long way toward encouraging deployment of new methods. Absent a mathematic proof of stability, some dynamic load balancing methods for which a strong argument could be made for a tendency toward stability have been deployed. Once deployed, successful deployments provide an existance proof that networks can be stable, but this existance proof falls short of ensuring that all deployments will always be stable. The discrte nature of traffic adjustment events rather than continuous small adjustment prevent the use of linear systems analysis. This is unfortunate since stability proofs for linear systems (or proof of instability) are relatively simple. The mesh topology and analog nature of traffic adjustment amount make it difficult or impossible to apply techniques based on discrete states, such as Markov chain. Proof of stability may be easier for dynamic load balancing applied as a local decision rather than a distributed decision (see ). For a local decision, a set of inputs to a node are balanced over a set of outputs. Absent routing loops, the output will not affect the inputs. For dynamic load balancing with distributed decisions a proof must be applicable to an arbitrary mesh topology. This is a more difficult problem than the problem of local decision. There may be characteristics of the mesh that make a proof for dynamic load balancing with distributed decisions somewhat less difficult. It may help that there exists a non-cylic relationship between move heavily loaded links and less loaded links, where given that all routers have the same information, traffic is only moved from more heavily loaded links to less loaded links. Other characteristics, such as non-cyclic nature of the traffic flow, may help as well. The problems of providing stability proofs for existing and proposed mechanisms are clearly difficult and not yet solved. As such this remains a good topic for researh.

While there was interest in OMP in the 1990s there was some discussion of stability on the OSPF, ISIS, and MPLS WG mailing lists as well as off list. Particularly helpful was discussion with Matt Mathis (PSC) and Teunis Ott (Bellcore).

This memo includes no request to IANA.

This document discusses both the known stability or instability of previously deployed routing and load balancing techniques and the unknown stability or instability of techniques which may be deployed in the future. The difficulty in proving stability is also discussed. Given the scope of the document, with no proposal for new protocols and only discussion of factors that impact stability, there are no security concerns raised by this document.