Most explanations of Cisco ACI open with the policy model. Tenants, VRFs, bridge domains, EPGs, contracts. It’s the interesting bit, so I understand the temptation, but I think it’s the wrong place to start.
The thing to understand first is that ACI separates three planes, and the controller is not in the data path. Almost everything surprising about how an ACI fabric behaves when something breaks follows from that one fact. Get it straight at the beginning and the rest of the architecture stops feeling arbitrary.
This is the first of six posts working through ACI as I understand it. The material comes from the Cisco ACI Design Guide, ACI Fundamentals 4.2(x), and the Multi-Pod, Multi-Site and Remote Leaf white papers, plus the notes I’ve built up designing these fabrics. This post covers the physical components, the underlay, and how a packet actually crosses the fabric. The policy model comes next.
The topology
An ACI fabric is a two-tier leaf and spine Clos topology running a VXLAN overlay on an IP underlay. Every leaf switch connects to every spine switch. There are no leaf to leaf links and no spine to spine links.
That constraint isn’t a recommendation, it’s how the forwarding model works. Any two endpoints on different leaf switches are separated by one spine hop, and that never changes as the fabric grows. Bandwidth scales by adding spines, port capacity scales by adding leaf switches, and the path length stays the same either way.
If you’ve built a traditional data centre network, the absence of an aggregation layer is the first thing that feels wrong. It isn’t. The spines do transport and nothing else.
The three planes
Here’s the separation, and it’s worth committing to memory.
| Plane | Function | Where it runs |
|---|---|---|
| Management and policy | Policy definition, configuration, telemetry, image management | APIC cluster |
| Control | Underlay routing (IS-IS), endpoint mapping (COOP), route distribution (MP-BGP) | Leaf and spine switches |
| Data | VXLAN encapsulated forwarding and contract enforcement | Leaf switches, transported by spines |
The payoff is this: if you lose the entire APIC cluster, the fabric keeps forwarding traffic using the policy it already has. Existing endpoints carry on. What stops is configuration change, new endpoint policy programming, and the management interface.
That’s a genuinely different failure mode from a traditional network, where losing a management system is an inconvenience and losing a control plane is an outage. In ACI the controller can be entirely absent and traffic still flows. It also means “the APIC is down” and “the fabric is down” are separate statements, and it’s worth being precise about which one you mean when you’re on a bridge call.
The APIC cluster
The APIC is a clustered controller holding the policy repository, and it renders that policy onto the fabric switches. It does the policy manager, topology manager, observer, boot director and appliance director functions.
Design points I keep coming back to:
- Three to seven active nodes are supported. Cisco recommends three, five or seven. Four is recommended for a two site Multi-Pod fabric with 80 to 200 leaf switches, which is the one exception to the odd-number habit.
- APIC-M appliances support up to 1200 edge ports. Above that you’re on APIC-L.
- Each APIC has two fabric-facing NICs, and they must connect to different leaf switches. Cabling both into the same leaf defeats the point.
- The policy database is sharded and replicated across cluster nodes. Each shard has a replica set with its own leader.
- A standby APIC can be pre-staged and promoted to replace a failed node.
The point that trips people up: adding APIC nodes increases scale and redundancy, not throughput. It’s a controller, not a forwarding element. Size the cluster from the verified scalability figures for your target release, and from the Multi-Pod placement rules if there’s more than one location involved. I’ll come back to placement in post six, because it’s the difference between losing a pod and losing your ability to make changes.
Spines
Spine switches provide the backbone. They forward VXLAN encapsulated traffic between leaf switches, host the COOP endpoint mapping database, and act as MP-BGP route reflectors for the fabric.
Modular Nexus 9500 spines suit high port count or future scale. Fixed Nexus 9300 spines suit smaller fabrics. Spines also terminate the connections to an interpod or intersite network in Multi-Pod and Multi-Site designs.
One hardware constraint matters more than the rest. Multi-Site requires second generation spines, meaning Nexus 9500 cloud scale line cards or equivalent, on the switches connected to the intersite network. First and second generation spines can coexist in the same fabric, but only second generation spines should connect to the external IP network. There’s a reason for that which I’ll get to in post six, and it isn’t a software licensing decision.
Leaf, border leaf, and the third tier
Leaf switches are where everything outside the fabric attaches: servers, storage, hypervisors, firewalls, load balancers, external routers, FEX. All policy enforcement happens on leaf switches. Every contract, every filter, every drop decision.
They’re Nexus 9300 series, and model selection comes down to port speed, port count, buffering, and support for features like analytics, MACsec and IPv6 scale.
Two behaviours are worth calling out. First, leaf switches are the anycast gateway for the bridge domain subnets they serve, so the same gateway address is active on every leaf where that bridge domain is deployed. There’s no HSRP, no active gateway to fail over, no first hop redundancy protocol to tune. Second, a vPC domain is formed between two leaf switches without a dedicated peer link, because peer communication runs across the fabric. If you’re used to building a peer link and a peer keepalive by reflex, that’s one habit to drop.
A border leaf is just a leaf switch carrying an L3Out to an external routing domain. It learns external prefixes, redistributes them into MP-BGP within the infra VRF, and the spines reflect them to the rest of the fabric. A border leaf can be dedicated to external connectivity or can host compute at the same time. Dedicated is preferred at scale, and post three explains why in terms of policy TCAM.
ACI also supports a three-tier topology, where tier-2 leaf switches connect to tier-1 leaf switches rather than to spines. It suits environments needing a lot of low speed access ports. The constraints are specific: maximum 18 ECMP links from a tier-2 leaf to tier-1 leaf switches, tier-2 leaf switches cannot connect to remote leaf switches, and changing a switch between tier-1 and tier-2 requires decommissioning and recommissioning. Combined tier-1 and tier-2 leaf count still counts against the validated leaf limit, which is 400 per pod and 500 per Multi-Pod fabric as of ACI 6.0(1).
The underlay
IS-IS runs automatically between leaf and spine switches to provide TEP reachability. It isn’t exposed for configuration. You don’t design it, you don’t tune it, you don’t troubleshoot it in the usual sense.
Where an interpod or intersite network is used, OSPFv2 or BGP runs between the spines and the external devices. That part you do design, and it’s covered in post six.
Every fabric node gets a tunnel endpoint address from the infra TEP pool. The TEP pool is defined during fabric discovery.
Here’s the part to write down somewhere permanent: the TEP pool cannot be changed without rebuilding the fabric.
Not “cannot be changed easily”. Rebuilt. So it needs planning before the first APIC is initialised, with room for the pods you might add later, not just the ones in front of you today. The specifics:
- The infra TEP pool is a private range and doesn’t need to be routable outside the fabric in a single pod design.
- In Multi-Pod, each pod needs a separate non-overlapping TEP pool, and all pod TEP pools must be routable across the interpod network.
- In Multi-Site, site TEP pools are not exchanged. Only the EVPN Router-ID, Overlay Unicast TEP and Overlay Multicast TEP addresses need to be routable across the intersite network.
- Remote leaf locations need a dedicated remote leaf TEP pool, plus an external TEP pool per pod so APICs, spines and border leaf switches have routable addresses.
My standing recommendation is to configure an external TEP pool on every pod as a matter of course, even where remote leaf is nowhere near the current scope. It costs nothing at build time. Retro-fitting addressing into a production fabric is disruptive in a way that’s difficult to explain to anyone who wasn’t there.
VXLAN and the MTU budget
Traffic entering the fabric is encapsulated in VXLAN at the ingress leaf and decapsulated at the egress leaf. The VXLAN header carries the VNID identifying the bridge domain or VRF, and the class ID identifying the source group for policy enforcement.
That second field is the important one conceptually. The policy identity travels with the packet, which is how the egress leaf knows what the traffic is without needing to look it up.
The numbers:
| Consideration | Value |
|---|---|
| VXLAN encapsulation overhead | 50 bytes, or 54 where the original 802.1Q header is preserved |
| Default fabric access port MTU | 9000 bytes. Jumbo frames don’t need configuring manually |
| Fabric uplink MTU | 9366 bytes from ACI 3.1(2), being 9216 plus 150 bytes of overhead |
| Default control plane MTU between locations | 9000 bytes, tunable under System Settings |
| Minimum interpod or intersite MTU | Must accommodate the control plane MTU. 9150 bytes is the common design value |
And a warning that has cost people real time. MP-BGP adjacencies across an interpod or intersite network can establish successfully even when the MTU is too small, because the adjacency itself uses small packets. The failure shows up later, as an inability to exchange endpoint information, and by then you’re several layers away from the actual cause.
So validate MTU with large packet tests across every path before handover. Not “the interface says 9216”. Actual packets, actual paths, before anyone puts a workload on it.
Endpoint learning and COOP
Leaf switches learn local endpoints from the data plane and report them to the spine COOP database. The spines hold the authoritative mapping of endpoint identity to location.
When an ingress leaf doesn’t know where a destination lives, it forwards to the spine proxy, which looks up COOP and forwards on. That’s the hardware proxy behaviour, and it’s why an ACI fabric doesn’t need to flood unknown unicast the way a traditional Layer 2 network does.
Ingress leaf switches also learn remote endpoints as an optimisation, caching the location of endpoints they’ve talked to so they can forward directly rather than through the proxy. It’s a performance feature, and like most caches it can hold something stale.
Disable remote endpoint learning where first and second generation leaf switches are mixed. It’s also commonly disabled on border leaf switches. Post three covers that in context.
External routes get into the fabric a different way. A border leaf redistributes them into MP-BGP VPNv4 and VPNv6 in the infra VRF, tagged with the originating VRF route target. Spines acting as route reflectors propagate them, and each leaf imports them only into VRFs it actually has instantiated locally. At least one route reflector per pod is required, and two per pod is what I’d deploy.
What I’d take from this
If there’s one thing to keep from this post, it’s the TEP pool. It’s decided once, at the very beginning, by someone who may not realise how permanent the decision is, and it can’t be undone without a rebuild. Plan it properly and configure an external TEP pool while you’re there.
Second thing: the APIC not being in the data path changes what “the fabric is down” means, and it’s worth being precise about that before you’re under pressure.
Third: test the MTU with real packets. An adjacency that comes up is not evidence that the path is correctly sized.
The next post covers the policy model. Tenants, VRFs, bridge domains, EPGs and contracts, and the access policy chain that catches almost everyone the first time.