Destination Independent Exit Policies

The simplest exit policies do not consider the destination in deciding which exit to use. They're most often used in networks served by only one ISP. They certainly can be used when multiple ISPs are availble, but their simplicity leads to poor exit choices as shown above in the section called Exit Selection Scenario with Multiple ISPs.

Exit Selection and Static Default Routes

When your network has only a single Internet connection, it's common for some or all of the routers and hosts to use a static default route to reach that connection. This exit policy could be described as "all exit traffic goes to interface x." Static default routes are simple to understand and easy to administer. And there appears to be little downside—what's the harm in sending traffic toward a link that's down when there's no alternative?

However, strange things can start to happen in response to link failures when static default routes are used in a network with more than one Internet connection. Said differently, this simplest of exit policies functions poorly when a choice among exit paths is available. This section discusses some of the perils of using static default routes in such networks. Figure 7-9 shows a simple network with a single Internet connection using static default routes.

Figure 7-9. Static Default Route and One Internet Connection

Hosts on LANC have static default routes pointing at RouterC. RouterC has a static default route pointing at RouterA. RouterA has a static default route pointing toward the Internet. All networks have Internet connectivity as long as all links and routers are working.

But consider what might happen after a second Internet connection has been added at RouterB as shown in Figure 7-10. The static default route in RouterC sends traffic toward RouterA even when RouterA's Internet connection is down. In this case, RouterA ends up redirecting or forwarding the traffic to RouterB before it can exit. This is at the very least wasteful since the traffic could've gone directly from RouterC to RouterB.

Figure 7-10. Failed Internet Connection Causing Redirection

Also consider what would happen if RouterA had a path to RouterB only through RouterC as shown in Figure 7-11. This could lead to a routing loop where Routers A and C hand a packets destined for the Internet back and forth till their TTL expires.

Figure 7-11. Failed Internet Connection Causing Routing Loop

Many Network Administrators will've removed static default routes from their networks before implementing BGP because they cause other problems. For example, consider what would happen to the original network of Figure 7-9 if the line between RouterA and RouterC went down as shown in Figure 7-12. Users on LANC would lose Internet connectivity even though the Internet could've been reached via RouterB. Note that connectivity to other networks within your AS (e.g. LANA) might've been maintained if an IGP had identified an alternate route (e.g. LANC -> RouterC -> RouterB -> RouterA -> LANA).

Figure 7-12. Failed Internal Network Causing Loss of Internet Connectivity

It's not that static default routes must be purged from your network once you deploy BGP. Even with BGP, it's still ok to use static default routes in cases where there's only one path toward all exits from your AS. Consider Figure 7-13.

Figure 7-13. Where Static Default Routes Won't Cause Trouble

Only Interior RouterA and Interior RouterD have a single path to the Internet. All the other routers in Figure 7-13 cannot have static routes and maintain reliable Internet connectivity.

Routers that have more than one path to a destination and use static routing are likely to lose traffic in the face of some link failures. If you're concerned enough about reliability to be considering redundant Internet connections, then you probably should just bite the bullet and remove all static routes from your network when two or more paths are available.

In a network that connects to the Internet at only one point, the IGP's job can be as simple as choosing the best path to networks that are internal to the AS. Static default routes are often adequate in such networks (though they will cause problems as shown in Figure 7-12 above). But once you attach your network to the Internet at multiple points, you'll have much better reliability if you give your IGP an additional job: distributing default routing information throughout your AS so that intelligent exit choices can be made.

Exit Selection with "Floating" Static Default Routes

This exit policy could be described as "all exit trafic goes to interface x unless it's down when it goes to interface y."

Network administrators who've been burned by one of the failures induced by static routes as described above often try adding "floating" static routes to their network. These are just additional, lower priority static routes to the same destination ( in this case since we're discussing static routes). A floating static route has a higher administrative distance (and hence lower priority) than the primary static route. All traffic follows the static route with the lowest administrative distance (i.e. highest priority). Normally that's the primary static route. Traffic only follows a floating static route when all higher priority static routes are suppressed.

So how does a static route get suppressed? The answer is important. In its subtleties lie the reasons why floating static routes are more reliable than lone static routes, but also the reason that they're not as reliable as upper-layer alternatives (e.g. IGPs and BGP) that I'll discuss later.

How Do Floating Static Routes Work?

As a very crude form of dynamic routing, Cisco routers suppress static routes that lead to interfaces that're down. Let's return to the network of Figure 7-12 and consider how a floating static route could help. We saw above how Internet connectivity for LANC was lost when the link between RouterC and RouterA failed. A floating static route in RouterC pointed at RouterB would have prevented this as shown in Figure 7-14.

Figure 7-14. Floating Static Route Example

In the normal case, traffic from RouterC follows the primary static route toward RouterA. But if RouterC detects the failure of the link to RouterA, then the primary static route is suppressed and traffic follows the floating static route toward RouterB. Floating static default routes seem to've saved the day! But let's look more closely at how this happened so we can see when floating static routes help and when they don't.

How Can Floating Static Routes Fail?

The key to understanding the limitations of floating static routes is knowing how an interface is marked as "down." The details vary depending on the media used, but in general, an interface is considered down when signaling at the link layer is lost.

For example, an Ethernet interface is marked down when the link integrity signal is lost. Similarly, a serial interface is marked down when the link-layer framing signal is lost.

The presence of link-layer signals might sound like a good way to know if a link can deliver data, but it's only a necessary condition for delivering data to the desired destination, not a sufficient one. It's often a rude awakening when you discover that it's possible (sometimes even common) for link-layer signals to be present even though the destination is unreachable. Such occurrences are unlikely when point-to-point copper or fiber connects router interfaces directly to each other. The likelihood goes up dramatically as the complexity of the interconnecting (link-layer) equipment goes up. It's easiest to see such failures by looking closely at the equipment used to connect pairs of routers by Ethernet or serial port. See Figure 7-15.

Figure 7-15. Ethernet and Serial Interface Interconnection Equipment

If RouterA has a primary static route pointing toward RouterB, that route would indeed be suppressed if the Ethernet cable connecting RouterA to the hub failed. But what if the Ethernet cable connecting the hub to RouterB failed? The Ethernet link integrity signal would still be present on RouterA's Ethernet interface so the interface would not be marked as down and the primary static route would remain active even though RouterB was unreachable. A floating static route would never come into play because RouterA's Ethernet interface would never be marked as down and the primary route would never be suppressed.

Similarly, if RouterC has a primary static route pointing toward RouterD, that route would indeed be suppressed if the serial cable connecting RouterC to the DAX at the CO failed. But what if the DAX failed or were accidentally configured to connect your time slots to somebody else's circuit? Link-level framing signals would still be present on RouterC's Serial interface so the interface would not be marked as down and the primary static route would remain active even though RouterD was unreachable. A floating static route would never come into play because RouterC's Serial interface would never be marked as down and the primary route would never be suppressed.

Floating Static Routes Summary

If there's anything other than point-to-point copper or fiber connecting two interfaces, there's a good chance that the link between them can fail to pass data even though the interfaces at both ends are marked "up." Such failures are not avoided by floating static routes, though they are avoided by IGPs or BGP. But of course, exit selection via an IGP or BGP is more difficult to configure than floating static routes. Consider the likelihood that links in your network can fail when the interfaces remain up and your tolerance for complexity carefully when deciding if floating static default routes are right for your network.

ISP Network Structure

The next two sections discuss policies that interact with your ISP's networks in different ways. In order to best understand the differences between these policies, it is helpful to understand how most ISP's have structured their networks. This structure impacts the reliability of their network and yours.

Most top-tier ISP's maintain reliability in the face of line and equipment failure up to (but not including) their border routers. This is achieved by building a network "core" that interconnects these border routers and maintains connectivity to other ISPs. The core contains redundant routers and communication lines. This core equipment is often referred to as the ISP's "backbone."

Toplogically, the core routers are typically interconnected in a partial mesh. The communications lines interconnecting these routers run between the core network "points of presence" or "POPs." See Figure 7-16.

Figure 7-16. Typical ISP POPs and Core Router Interconnects

Border routers are coloacated at these core network POPs. Ideally, there is more than one core router at each POP and redundant connectivity between the border routers and these core routers. See Figure 7-17.

Figure 7-17. ISP POP with Good Redundancy

Note that there's no technical requirement for this structure. Such a structure is common among top-tier ISPs because it represents a good tradeoff between cost and reliability. This redundancy in the core network affords an ISP the flexibility to take core network routers and lines out of service for maintenance and upgrading without loss of service to customers.

In practice, many growing ISPs gradually build up to this level of redundancy as the number of customers served by a POP grows. As an ISP is first moving into an area, there may be only a single border router for the area and it may have only a single connection to the ISP's core. Of course, these are both single points of failure impacting the customers served by the border router.

With this background, let's look at a policy that's aware of your network's connectivity to your ISP's border router but not aware of connectivity to your ISP's core network.

Use Closest Interface with ISP Border Router Connectivity

Alternate Title: Exit Selection with Default Routes from Your ISP

Following paragraph needs to be merged

Some ISPs refuse to send you a default route via BGP. But if yours are willing, then your border routers can simply redistribute these default routes into your IGP. Needs more flesh. Needs figure showing ISP's border router injecting default route and failure of border-core link.

This policy results in your interior and border routers being aware of their dynamic connectivity to your ISP(s) border routers. They choose the best path to an ISP's border router. This awareness comes not from manual configuration by a network administrator, but instead from a dynamic connection between your ISP(s) border routers and your border (and perhaps indirectly your interior) routers.

The dynamic awareness of this connectivity improves the reliability of this policy significantly over policies with simpler implementation techniques. For example, failures at the link layer are detected by this policy whereas they are not by floating static routes.

However, this policy requires more complex implementation techniques in your network and in your ISP's. In particular, your border routers and your ISP's border routers must run BGP so that they can "peer" to exchange reachability information. Further, some mechanism is often used so that this reachability information flows throughout your network to all interior routers as well.

The techniques behind this policy are conceptually simple: instead of configuring your interior and border routers each with its own default exit route, you ask your ISP(s) to originate a BGP default route over each of your connections. These default routes can be redistributed into your IGP so that they flow throughout your AS. Hence interior routers can dynamically choose the best default exit path available. Alternatively, all your interior routers can run BGP to learn of their best default route.

There is no strict requirement that these default routes propagate beyond your border routers. But as we saw above in the section called Exit Selection and Static Default Routes, interior routers with more than one exit path should avoid static default routes. Propagating default routes received from your ISPs is an excellent alternative to static default routes.

Fix bad segue

Need to say that there is no default route on the Internet backbone. All routers with full BGP routing tables are aware of all reachable networks that're part of the Internet. Consequently, there's no need for them to have a default route. Hence your ISP has to originate a BGP default route if you're to receive one. This is generally done within the ISP's border router. (See the section called Originating a BGP Default Route in Chapter 8 for configuration information.) ISPs generally wouldn't want such a default route rattling around their own BGP routing tables. Many ISPs refuse to originate such default routes, instead preferring to propagate their core network routes. (See the following section for more on this.)

A limitation of this policy is that it only makes your network aware of connectivity to your ISP's border routers. Any failure of border-core connectivity within your ISP's network will go unnoticed. This may be immaterial if you're connected into a POP with good border-core redundancy as shown above in the section called ISP Network Structure. But if there are one or more single points of failure between your border router and your ISP's core network, then the policy described in the next section is likely to yield better reliability.

(Maybe this is a good sidebar?) When introducing this policy, I said that nearest exit information came from your ISPs instead of being configured into your routers. In practice, it's probably best to use information from your ISP to override a static exit configruation rather than using it in place of a static configuration.

Here's why: BGP can detect link-layer failures since link-layer connectivity is a necessary condition for maintaining a BGP session. However, the converse is not true: the absence of a BGP session is not a sufficient condition for concluding that the link layer has failed. For example, your ISP may one day decide to reconfigure BGP in some way that causes your peering session to fail even though the line between you is fine. Hence any exit policy that depends on information from your ISP should probably be backed up with high administrative distance ("floating") static exit routes.

Use Closest Interface with ISP Core Router Connectivity

Alternate Title: Exit Selection with Backbone Routes from Your ISP

Following paragraph needs to be merged

Your ISP will probably offer to send you BGP advertisements for her backbone routes. This is a small group of routes to network from which the ISP has numbered interfaces on her backbone routers. The theory goes that following the lowest cost path to these networks will lead you on the lowest cost path to the ISP's backbone. You configure your routers so that instead of routing default traffic toward an address (as you would with a static default route), you route your traffic toward these "default networks." Your IGP will handily carry advertisements for these routes as injected by your border routers and assign metrics so that the optimal exit will be selected. Your IGP won't be overloaded since it's a small number of routes.

Add reference and index entry for ip default-network configuration command.

Maybe the following paragraphs should be the segue at the end of the previous section?

Let us suppose that you are served by an ISP that does not have reliable border-core connectivity at your POP. Recall that their border router originates the default route you receive. You will continue to receive this default route even if there is a failure in border-core connectivity at your ISP. Hence you could end up delivering traffic to a POP that had lost Internet connectivity and your traffic would be dropped. This would be tragic if you also had connectivity through another ISP who was able to deliver your traffic.

Admittedly, the chances of the above scenario happening to you may be remote, but I have seen it happen. (The temporary fix was to drop the BGP session to the failed POP so that exit traffic found another route till the POP was repaired.) No matter how remote the possibility, there's an alternative policy that avoids ISP border-core connectvity failures and is only a little more complex.

What if your ISP consistently numbered interfaces connecting their core rouers to each other from an IP network numbering range dedicated to this purpose? What if this range were used for no other purpose? See xxxx.

If core-core interfaces are numbered in this way, then the presence of a BGP route to the block of addresses from which core-core interfaces are numbered is a very good indication of core connectivity. Hence many ISPs who've taken the trouble to number their interfaces in this way very much prefer to advertise a BGP route to one or more core network numbers instead of advertising a BGP default route. These "backbone routes" serve the role of a default route and provide you with a very reliable indication of Internet connectivity.

Mention importance of ISPs carefully numbering their border-border, border-core, and core-core interfaces.

Need figure showing core interface numbering and core network routes propagating via BGP from ISP's core to ISP's border to your border.

As with receiving default routes via BGP, there's no requirement that these "backbone routes" propagate beyond your border routers, but it's a good idea. xxxx

A floating static still a good idea.

Copyright © 1999-2000 by Robert A. Van Valzah