The last post ended on a claim: the L3Out is the least surprising part of ACI, and the surprises start when you attach policy to it. This post is that claim collected in one place.
There are three defaults in this area that will drop your traffic without telling you. Subnet scope flags, external EPG scoping across the VRF, and the OSPF route tag in a transit design. All three produce the same symptom, which is an adjacency that establishes cleanly, routes that appear in the table, and traffic that goes nowhere.
That symptom is worth recognising on sight, because it sends people looking at the routing when the routing is fine.
External EPG subnet scopes
Subnets defined under an external EPG carry scope flags that control both routing and security behaviour. Two different jobs, on the same object, set with tick boxes that don’t obviously distinguish between them.
| Scope | Effect |
|---|---|
| External Subnets for the External EPG | Marks the subnet as a security boundary. Required for contract enforcement |
| Export Route Control Subnet | Permits the subnet to be advertised out of the fabric through this L3Out |
| Import Route Control Subnet | Controls which routes are accepted inbound. BGP and OSPF only |
| Shared Route Control Subnet | Marks the subnet for leaking into other VRFs for shared services |
| Shared Security Import Subnet | Applies the security policy in a shared service design |
| Aggregate Export or Import | Used with 0.0.0.0/0 to mark all routes as eligible. It does not perform route aggregation |
Setting the wrong flag here is one of the most common causes of an L3Out that establishes an adjacency and passes no traffic. The route control flags govern whether a prefix moves. The security flag governs whether a contract can be applied to it. Tick the export flag and forget the security flag, and the route is advertised perfectly while every packet is dropped by policy.
Two names deserve a note. Aggregate Export does not aggregate anything, despite what the word means everywhere else in networking. It marks all routes as eligible for export. And Shared Security Import Subnet only makes sense in a shared services design, which is post five.
The external EPG scoping problem
This is the one I’d most want somebody to know before their first ACI design.
An external EPG classifies traffic for every L3Out in the same VRF, not only for the L3Out it’s configured under.
Read that twice, because the object sits underneath an L3Out in the tree, which strongly implies it belongs to that L3Out. It doesn’t. Its scope is the VRF.
The practical consequence: you have two L3Outs in one VRF, one to the internet through a firewall and one to the WAN. You give the internet L3Out an external EPG of 0.0.0.0/0, because that’s what you do for an internet path. Traffic destined for the WAN now gets classified into that 0.0.0.0/0 external EPG too, because it matches, and the contracts attached to it apply. Your WAN traffic is now being filtered by your internet security policy, or dropped by it.
The traditional fix is to define specific subnets per external EPG rather than 0.0.0.0/0. That works, and it costs you something: you now need to know in advance which prefixes are reachable through which L3Out, and keep that knowledge current as the external network changes. Which it will, and nobody will tell you.
Dynamic L3Out EPG classification
Dynamic L3Out EPG Classification, introduced in ACI 5.2(4), removes that maintenance burden.
Instead of listing subnets, you define a default import policy of type match prefix and routing policy on each L3Out, with a match prefix list of 0.0.0.0/0 le 32. Prefixes are then classified by the L3Out they were actually learned through.
Prefixes learned through L3Out 1 go into one external EPG. Prefixes learned through L3Out 2 go into another. If a prefix stops being reachable through one path and appears through the other, its classification follows, and therefore so does its security policy.
That last property is the good bit. The security policy tracks reality rather than tracking a list somebody maintained accurately in 2023.
Where more than one L3Out exists in the same VRF, this is what I’d use, and I’d be particularly firm about it where one path is protected by a firewall and the other isn’t. That’s precisely the case where a misclassification means traffic bypassing the firewall, and nothing in the fabric will flag that as unusual.
Policy enforcement direction
The VRF setting Policy Control Enforcement Direction determines where contract filtering for L3Out traffic is applied. It’s under Tenants, Networking, VRFs.
| Direction | Behaviour | When to use |
|---|---|---|
| Ingress | Filtering applied on the compute leaf where the endpoint resides, in both directions | Default. Keeps border leaf policy TCAM free |
| Egress | Filtering for the L3Out to EPG direction applied on the border leaf | Where the external EPG table is large and duplicating it across compute leaf switches costs more |
Ingress is the default and the right answer most of the time, because it keeps external policy off the border leaf, which is also carrying the external routing table. The two big consumers stay separate.
The reason to fix this early rather than leave it: some features work only with one enforcement direction. Service graphs, policy-based redirect and shared L3Out all have constraints here, and they’ve moved between releases. Confirm the requirement against the release notes before you set it, because changing it later affects every L3Out in the VRF, not just the one that prompted the change.
Import, export and bridge domain scopes
Route control is applied at the L3Out through the subnet scopes above. Export route control governs what the fabric advertises outwards. Import route control governs what it accepts inbound, and it’s available for BGP and OSPF only.
Bridge domain subnets carry their own scope flags, which are separate and easier to reason about.
| Scope | Effect |
|---|---|
| Advertised Externally | The subnet is advertised to external routers by the border leaf through an L3Out |
| Shared Between VRF Instances | The subnet is leaked into one or more other VRFs for shared services |
| Private to VRF | Default. Not advertised, not leaked |
Private to VRF being the default is the right call, and it means a newly created bridge domain subnet is invisible outside the fabric until somebody decides otherwise. That’s a good default and it’s also the first thing to check when a new subnet isn’t reachable from outside.
Route summarisation
Summarisation arrived in ACI 1.2(2) for BGP, EIGRP and OSPF. It has three characteristics that differ from a traditional router, and they’re worth stating explicitly in any design document.
Summarisation happens on the border leaf switches only. Summaries are never carried inside the fabric. Internally, the fabric always has the specifics.
It applies to both bridge domain routes and transit routes.
Summary routes are installed pointing to Null0, giving the usual discard behaviour for the summarised range.
To configure it, define the summary address as a subnet under the external EPG, select Export Route Control, and attach the appropriate route summarisation policy. For BGP, the AS-SET option carries path information through the summary, which matters if you’re summarising routes learned from multiple upstream AS numbers and want loop prevention to keep working.
Route profiles
Route profiles are route maps applied to an L3Out. They’re the correct place for anything involving attributes, and the subnet scope flags are the wrong place.
What they’re for:
- Default export and default import policies for an L3Out
- Controlling which transit routes are redistributed between L3Outs
- Filtering specific prefixes using match criteria
- Setting BGP attributes: local preference, MED, community, inbound or outbound
- Permitting redistribution of static routes between L3Outs
That last one is a genuine gotcha. Static routes are not advertised between L3Outs unless a route profile explicitly allows it. If you’ve configured a static route on one L3Out and expected it to appear out of another, this is why it didn’t.
There’s also route target filtering, which discards VPN routes not relevant to any locally instantiated VRF. Auto route target filtering does this automatically and conserves memory on the spine route reflectors. Worth turning on in a large multi-tenant fabric, and largely irrelevant in a small one.
Transit routing
Transit routing lets the fabric advertise routes received on one L3Out out of another, providing connectivity between separate external routing domains through the fabric. The fabric becomes a transit path between Layer 3 domains rather than just an endpoint destination.
Where it comes up:
- Multiple Layer 3 domains such as external pods, mainframes, service nodes or WAN routers peering with ACI and needing to route between each other
- Mainframe connectivity, where LPAR and VIPA host routes are pushed into the fabric for redistribution to WAN interfaces
- Service node transit, where an appliance advertises virtual IP routes that get redistributed to external WAN interfaces
- A border leaf L3Out and a spine handoff used together
Configuration is a matter of specifying which routes imported from one L3Out are announced through another. Three mechanisms:
- Route profiles, using the default export and default import policies
- Export route control subnets, marking specific prefixes for export on the outbound L3Out
- The Aggregate option on a 0.0.0.0/0 entry, marking all transit routes as eligible for export
Not every combination of inbound and outbound routing protocol is supported. Cisco maintains the matrix in the ACI Transit Routing knowledge base article, and it’s worth checking against the specific protocols you’re proposing rather than assuming any pair works.
The route tag trap
This is the third silent failure, and it’s my favourite, in the sense that it’s a well-intentioned safety mechanism doing exactly what it was designed to do at the worst possible moment.
Routes advertised from ACI to external routers through OSPF or EIGRP are tagged with 4294967295 by default. The fabric will not accept inbound routes carrying that tag.
In a normal design that’s correct behaviour. A route that left the fabric and came back has looped, and dropping it is right.
In a transit design, the same route can legitimately leave the fabric and return on a different L3Out. The classic case is a route that goes out to an external firewall, gets inspected, and comes back in on the other side. That’s the design working as intended. The tag then causes the returning route to be dropped, and nothing anywhere reports a loop, a policy denial, or an error. The route is just quietly absent.
The fix is to change the default route tag on the tenant VRF used for transit. It’s a single setting. Finding out that it’s the single setting you need, at three in the morning, is the expensive part.
So: if a design has routes leaving the fabric and returning through a different L3Out, change the route tag on that VRF at build time and test it. Put it in the build document.
What I’d take from this
The pattern across all of this is that ACI’s defaults protect you right up until the moment your design is the exception, and then they fail silently rather than loudly. Private to VRF, the route tag, the whitelist model, all of them are safe defaults. None of them announce themselves when they’re the thing standing in your way.
Concretely:
Avoid 0.0.0.0/0 as an external EPG subnet wherever there’s an alternative. Use dynamic L3Out EPG classification where a VRF has more than one L3Out, especially where the paths have different security postures.
Fix the policy control enforcement direction early, after checking feature support, because it applies to the whole VRF.
Check the route control flags and the security flag separately. They’re different jobs on the same object.
And where routes leave and return through different L3Outs, change the default route tag on the transit VRF, and test it before handover.
The next post covers VRF route leaking, shared services and the handoff options: GOLF, SR-MPLS, and why ESG-based leaking is the one worth learning.