The previous post covered the policy model and the access policy chain. That’s everything inside the fabric. This one is about the boundary, which is where a lot of ACI designs turn out to be won or lost.
The good news is that this is the part of ACI that behaves most like a network you already know. Route peering and static routing happen per VRF on leaf switches, in a way that’s directly comparable to VRF-lite on a traditional routing platform. Leaf switches carrying an L3Out are called border leaf switches. If you can configure BGP on a Nexus, you can configure it here.
The bad news is that the routing works long before the policy does, and the failure mode is an adjacency that comes up perfectly while no traffic passes. That part gets its own post, because there’s too much of it to fit here. This post covers the physical and protocol side: the object hierarchy, border leaf design, interface types, floating L3Out, protocol selection and convergence.
Anatomy of an L3Out
An L3Out defines the routing relationship between a tenant VRF and an external routing domain, whether that’s a campus core, a WAN, an MPLS VPN cloud or a data centre interconnect.
It’s built from a consistent set of objects, and knowing the hierarchy makes troubleshooting much faster.
| Object | Purpose |
|---|---|
| L3Out | The container. Bound to a VRF and to an external routed domain |
| Logical node profile | Selects the border leaf switches, assigns router IDs and loopbacks |
| Logical interface profile | Defines the interfaces: routed, subinterface, SVI or floating SVI |
| Protocol policies | BGP peer, OSPF interface, EIGRP interface, BFD and timer configuration |
| External EPG | Classifies external subnets for contract enforcement, carries route control flags |
| Route profile | Route map applied to import or export, for filtering and attribute manipulation |
Same principle as the access policy chain from the last post. A fault is almost always a missing relationship rather than a missing feature.
Dedicated or shared border leaf
A border leaf can be dedicated to external connectivity, or it can carry compute workloads at the same time. Shared compute and border leaf has been supported since ACI 3.2, and it needs second-generation hardware, meaning EX, FX or later.
I’d use dedicated border leaf switches in production, and the argument is policy TCAM. On a shared border leaf, TCAM is consumed by both compute policy and external policy, and external policy is the one that grows unpredictably when somebody adds a few thousand prefixes to the external network. Separating them means the two failure modes stay separate.
There’s a second reason. Where first-generation and second-generation leaf switches are mixed, remote endpoint learning should be disabled on the border leaf to avoid forwarding problems from stale entries. Dedicated border leaf switches make that a clean, contained decision rather than something that affects your compute.
Interface types
Three options for the L3Out interface.
A routed interface is a physical routed port connected directly to an external router. Simplest thing that works.
A subinterface with 802.1Q tagging gives multiple logical Layer 3 connections over one physical link. This is the workhorse where one physical connection carries several VRFs.
An SVI is used where the same VLAN encapsulation must be shared across multiple border leaf switches on the same L3Out, typically over a vPC.
One thing to be firm about: L3Out SVIs are for routing devices. Connecting servers directly to an L3Out SVI without a routing protocol is not something I’d design and not something Cisco recommends. Servers attach to an EPG and a bridge domain. If somebody proposes hanging a server off an L3Out because it’s quicker, that’s a conversation worth having before the build rather than after.
L3Out over vPC
Static routing and dynamic protocol peering over a vPC are both fully supported with no additional design constraints. Up to 32 ports can be configured in the same vPC port channel, being 16 per leaf, which was introduced in ACI 3.2.
This is one of the areas where ACI is less fussy than people expect. Peering over a vPC is normal.
Floating L3Out
Floating L3Out was introduced in ACI 4.2(1) for VMM domains and ACI 5.0(1) for physical domains, and it solves a real problem: what do you do when the external router is a virtual machine that can move?
Without it, you pre-provision the L3Out on every leaf switch where the router might appear, which is either a lot of configuration or a bet on the router not moving. Floating L3Out removes that.
Anchor and non-anchor
The design splits leaf switches into two roles.
An anchor leaf holds a unique primary IP address and establishes the routing protocol adjacency with the external router. Anchor leaf switches are selected explicitly.
A non-anchor leaf extends the external bridge domain from the anchor leaf switches. It has no primary IP address and does not peer.
Non-anchor leaf switches share a floating IP address in the same subnet as the primary addresses. That address is used internally for ARP gleaning, and it cannot be used as a BGP peer address or as a static route next hop. Worth knowing before somebody tries.
With a VMM domain, ACI detects where the virtual router is and deploys the VLAN only where it’s needed. With a physical domain, ACI deploys the VLAN on every interface in the associated AAEP.
That last sentence is the trap. Design the AAEP for a floating L3Out physical domain carefully and dedicate it to that purpose, because ACI will statically deploy the L3Out VLAN on every interface in that AAEP, including all the ones where you didn’t want it.
Design points
Configure at least two anchor leaf switches. As of ACI 6.0(1) the verified scale is 32 non-anchor leaf switches per L3Out.
Use BFD or IP SLA tracking with static or dynamic routing. This matters more than it looks: if all anchor leaf switches fail, a virtual router sitting behind a non-anchor leaf will not detect that its next hop is down, and traffic gets black-holed. The router has no adjacency to lose, because it never had one with the non-anchor leaf.
The traffic pattern is asymmetric by design. Traffic towards the virtual router is forwarded to an anchor leaf first, then on to the non-anchor leaf where the router actually lives. Traffic from the virtual router follows normal endpoint lookup and doesn’t traverse an anchor leaf at all.
At large scale, forcing everything through an anchor leaf is inefficient. BGP next-hop propagate, introduced in APIC 5.0(1), was designed for use with floating SVI to avoid that hairpin. It’s aimed at service provider deployments with hundreds of virtual routers, which is a fair marker for when you should be thinking about it.
Choosing a routing protocol
| Requirement | Protocol |
|---|---|
| General external peering to a campus core or WAN | BGP. Attribute preservation and policy flexibility |
| Inbound route filtering required | BGP or OSPF. EIGRP does not support import route control |
| Peering to spine switches, IPN or remote leaf | OSPFv2. Mandatory |
| Existing estate is EIGRP and cannot change | EIGRP, accepting the filtering limitation |
| Small number of stable prefixes | Static routing with BFD or IP SLA tracking |
BGP is the default answer for external peering. All BGP attributes from the external router are preserved, because MP-BGP carries the routes across the fabric. Both iBGP and eBGP peering are supported, and peer reachability can be via a connected interface, a static route, OSPF, EIGRP or a loopback.
Two BGP points worth designing rather than discovering. First, the fabric AS number appears in the AS_PATH, so design the surrounding BGP network treating the whole fabric as a single AS, and agree that AS number at design time. Changing it later is not a small piece of work. Second, configure BGP Maximum AS Limit per tenant to bound the AS path length you’ll accept.
The common peering pattern is to peer to the neighbour’s loopback, with loopback reachability advertised by OSPF using a passive interface. That survives the loss of an individual link between the border leaf and the external router, which a directly connected peering does not.
OSPF is supported for external peering and is mandatory in several places: external routers connected to spine switches, and remote leaf switches. NSSA or totally stubby areas are recommended so a default route can be received without full area participation. Note that two ACI leaf switches don’t form an OSPF adjacency with each other unless they share the same external SVI.
EIGRP works, with one significant limitation: import route control is not supported. Only BGP and OSPF support inbound route filtering. If inbound filtering is a requirement, EIGRP is not a suitable choice, and that’s a conclusion best reached early.
Static routing is fine for a small number of stable prefixes. Inside the fabric, static route preference is carried in MP-BGP using a cost extended community, and an MP-BGP route from a remote leaf wins over a local static route by default. A static route with a next hop of Null0 is not advertised to other leaf switches. Always pair static routes with BFD or IP SLA tracking, because a failed next hop with no tracking will happily keep attracting traffic.
Convergence
BFD provides fast failure detection. Support on L3Out links arrived in ACI 1.2(2g), and from ACI 3.1(1) it can also run between leaf and spine switches.
The details:
- ACI uses BFD version 1 in asynchronous mode, so both endpoints send hellos.
- BFD is not supported for multihop BGP.
- A global BFD policy exists for IPv4 and IPv6, and a per-tenant policy under Networking, Protocol Policies overrides it.
- Session scale per leaf is in the Verified Scalability Guide, worth checking where you have a lot of peers.
Where BFD earns its place is on L3Out SVIs, because a physical link failure doesn’t necessarily bring down the logical adjacency. On routed interfaces and subinterfaces, physical interface mechanisms already give you fast detection, so BFD adds less.
Related, and easy to miss: SVI auto state. Where an L3Out SVI spans several Layer 2 interfaces, the SVI stays up while any one of them is up. If those interfaces are down but the SVI stays up, other leaf switches still believe the external network is reachable through that border leaf, and they’ll keep sending traffic to it. Auto state withdraws the SVI when no Layer 2 interface is active. Enable it.
My order of preference for convergence work: BFD first, SVI auto state next, and only then consider tuning BGP or OSPF timers. Timers are the thing people reach for and the least reliable of the three.
Forwarding scale profiles
Border leaf switches can be tuned for higher longest prefix match capacity using a forwarding scale profile. This matters where the border leaf carries a large external routing table.
| Profile | IPv4 LPM capacity | Hardware |
|---|---|---|
| Default | 20K | All supported leaf switches |
| High LPM, ACI 3.2 and later | 128K | FX and GX generation |
| High dual stack | 38K IPv4 and 19K IPv6 | EX and FX2 generation |
Configured per leaf under Fabric, Access Policies, Leaf Switches, Policy Groups, Forwarding Scale Profile Policy.
The operational detail that decides when you do this: a change requires a reload of the switch. So apply it during the build, not as a retro-fit. Deciding at design time that you’ll need more than 20,000 external routes costs nothing. Discovering it in production costs you a maintenance window on a device carrying all your external connectivity.
The cheaper answer, where it’s available, is to accept a default route from the external network rather than the full table. It’s the least effort way to avoid an LPM scale problem entirely, and it’s often perfectly adequate.
What I’d take from this
Dedicated border leaf switches in production. Size the forwarding scale profile during design, because changing it means a reload.
BFD on L3Out SVIs, SVI auto state enabled, and tracking on anything static. Timers last.
With floating L3Out, two anchor leaf switches minimum, a dedicated AAEP for the physical domain, and tracking so an anchor failure doesn’t quietly black-hole traffic.
And agree the BGP AS number at design time, treating the fabric as one AS.
None of that is exotic. It’s the same discipline you’d apply to any external boundary. Which is rather the point of this post: the L3Out is the least surprising part of ACI.
The surprising part is what happens when you attach policy to it. The next post covers external EPG classification, route control, and the two or three defaults that will silently drop your traffic.