The conversation about extending ACI across locations usually starts with latency and distance. How far apart are the sites, what’s the round trip time, does it fit inside the supported limit.
Those questions matter, but they’re not the decision. The decision is how big you want your change domain to be. Everything else, including the latency limits, is a consequence of that.
This is the last post in the series, following the previous post on VRF leaking and handoff options. The first covered the fabric, and a couple of things from it come back here: the TEP pool that can’t be changed, and the MTU that has to be tested with real packets. Both cost more in a multi-location design than they do in a single fabric.
The options
| Architecture | APIC clusters | Primary benefit | Introduced |
|---|---|---|---|
| Single fabric | One | Simplest design, single location | ACI 1.0 |
| Multi-Pod | One, distributed across pods | Operational simplicity across locations | ACI 2.0 |
| Multi-Site | One per site, orchestrated by NDO | Fault domain isolation between locations | ACI 3.0 |
| Remote Leaf | Parent fabric cluster | Policy at small sites with no local spine | ACI 3.1 |
| vPod and cloud | Parent or site cluster | Policy in virtual or public cloud environments | ACI 4.0 onwards |
| Stretched fabric | One, stretched | Superseded. Legacy estates only | Legacy |
Stretched fabric splits a single fabric across two locations with leaf switches in each connected to spines in both. It’s superseded, it offers no fault isolation, and the only reason to know about it is to recognise it in an existing estate and plan a migration to Multi-Pod or Multi-Site.
Multi-Pod
Multi-Pod presents multiple leaf and spine pods as a single fabric under a single APIC cluster. Each pod runs its own IS-IS underlay, its own COOP database and its own MP-BGP route reflectors, which contains the control plane blast radius within a pod.
The result is one policy domain, one management point and one change domain covering every pod.
That sentence is both the advantage and the risk, and I’d write it that way in a design document. One change domain means one place to make a change, which is the operational appeal. It also means a configuration error, a faulty upgrade or a control plane defect applies to every pod simultaneously. There’s no boundary. There’s nothing to stop at.
What the interpod network has to provide
| Requirement | Detail |
|---|---|
| Routing | OSPFv2 between spines and IPN devices. VLAN 4 subinterfaces on the point to point links |
| Multicast | PIM Bidir is required to carry BUM traffic between pods |
| DHCP relay | Required so switches and APICs in a remote pod can be discovered |
| MTU | Must support the fabric control plane MTU. 9150 bytes is the common design value |
| TEP routing | Each pod TEP pool must be non-overlapping and routable across the IPN |
| Latency | Maximum 50 msec round trip time between pods, supported from ACI 2.3 |
| QoS | CoS to DSCP mapping must be configured so traffic classes survive the transit |
| Bandwidth | Sized for inter-pod east-west traffic plus APIC cluster replication |
Two of these cause more trouble than the rest.
PIM Bidir is a genuine requirement, not a recommendation, and it needs a rendezvous point with redundancy. A phantom RP design is the usual answer. If the IPN is provided by somebody else, this is the requirement most likely to come back as “we don’t do multicast”, and it’s better to find that out during design than during commissioning.
QoS is the other one. The IPN sits outside the fabric and will not preserve 802.1p markings. ACI marks the outer IP header with DSCP values based on the internal class of service, and a CoS to DSCP mapping has to be configured so the classes survive the crossing. Without it, control plane traffic competes with everything else across a link you don’t control.
APIC placement
The APIC cluster is distributed across pods, and where you put the nodes determines whether the cluster keeps quorum when a pod is lost. That makes it a design decision, not a deployment detail, and it’s one that’s awkward to change later.
The rules:
- Never place three APIC nodes in a single pod. Losing that pod removes the majority of the replica set.
- Three node cluster, two pods: two nodes in pod 1, one in pod 2. Pre-stage a standby APIC in pod 2.
- Three node cluster, three pods: one node in each pod.
- Five node cluster, two pods: a three plus two split leaves only two nodes if the larger pod fails. Pre-stage a standby APIC in the smaller pod so quorum can be recreated.
- Where an even split occurs and the IPN fails, you get a split brain. Some shards stay read-write in each pod. Resolve the partition before making any configuration changes.
I’d treat the standby APIC as mandatory in any Multi-Pod design. It’s the documented mechanism for recovering quorum after a pod loss, it costs very little to pre-stage, and the alternative is discovering during an outage that your options are limited.
What happens when things break
Worth being precise about, because the four failure modes have genuinely different consequences.
Loss of the IPN: pods continue forwarding locally. Inter-pod traffic stops. The APIC cluster partitions as described above.
Loss of a pod: surviving pods continue forwarding. Endpoints in the failed pod are withdrawn from COOP.
Loss of APIC quorum: the fabric continues forwarding with existing policy. Configuration changes are blocked. This is the one that follows from the APIC not being in the data path, and it’s the least bad of the four.
Configuration error: applies fabric wide. There is no boundary between pods for policy changes.
The last one is the reason Multi-Site exists.
Multi-Site
Multi-Site interconnects separate ACI fabrics, each with its own APIC cluster. Each fabric is an independent availability and change domain. The fabrics are joined by an intersite network, and the spines handle the intersite VXLAN encapsulation.
Policy is defined centrally in Nexus Dashboard Orchestrator and pushed to the participating APIC clusters using templates. Per object, you choose whether it’s stretched across sites or kept local. That per-object control is the substantive difference from Multi-Pod, where a bridge domain is available fabric-wide by default.
What the intersite network has to provide
| Requirement | Detail |
|---|---|
| Routing | OSPFv2 between spines and ISN devices, or BGP in later releases |
| Multicast | Not required. BUM is handled by head-end replication on the spines |
| DHCP relay | Not required. Each site bootstraps independently |
| MTU | Must support the control plane MTU. 9150 bytes is the common design value |
| Routable addresses | The EVPN Router-ID, Overlay Unicast TEP and Overlay Multicast TEP of each site |
| Latency | No validated limit between the fabrics themselves |
| Hardware | Second generation spines, EX generation or later, on the ISN facing links |
| Encryption | CloudSec provides line rate encryption of intersite VXLAN traffic where supported |
Compare that with the IPN table and the shape of the difference is clear. No multicast. No DHCP relay. Only three address types need to be routable rather than entire TEP pools. No latency limit between the fabrics.
The ISN asks much less of the network in between, and that’s not an accident. Multi-Pod extends one fabric, so the transit network has to look like part of that fabric. Multi-Site connects two complete fabrics, so the transit network only has to carry traffic between them.
Note the addressing point, though, and connect it to the TEP pool discussion from post one. In Multi-Site the site TEP pools are not exchanged, but the routable addresses in each site must be reachable across the ISN, and where the ISN is a public WAN, those must be public addresses. That’s a decision to make at design time.
Namespace translation
Each site allocates its own VNIDs and class IDs. A bridge domain stretched across two sites carries a different VNID in each fabric, and the same EPG carries a different class ID.
The spines translate between namespaces as traffic crosses the ISN.
Where a contract is defined between EPGs in different sites, NDO creates a shadow EPG in the remote site. The shadow object exists so the contract can be enforced locally without the EPG itself having to be stretched. You’ll see these appear in the APIC without having created them, and they’re expected rather than a fault.
Namespace translation is why Multi-Site requires second generation spines on the ISN facing links. It’s a hardware function, not a software one, which is why no release upgrade will get you round it. That’s the reason I flagged the spine generation constraint back in post one.
BUM handling
Where a bridge domain is stretched and flooding is enabled, one ISN connected spine per bridge domain is elected as designated forwarder. On receiving BUM traffic it replicates a copy to each remote site.
This is head-end replication. It removes the requirement for multicast in the ISN, which is a significant simplification, and it costs replication load on the spine plus bandwidth proportional to the number of sites.
The design conclusion follows directly: disable intersite BUM flooding where the bridge domain doesn’t carry protocols that depend on it. With two sites the cost is modest. With eight, it isn’t.
Nexus Dashboard Orchestrator
NDO runs as a service on Nexus Dashboard and replaces the earlier Multi-Site Orchestrator. It holds the multi-site policy templates, resolves them per site, and pushes configuration to each APIC cluster.
| Constraint | Value |
|---|---|
| Latency between Nexus Dashboard cluster nodes | 50 msec round trip time |
| Latency between a Nexus Dashboard node and an APIC cluster | 500 msec round trip time |
| Latency between ACI fabrics | No validated limit. Constrained by application tolerance |
| Maximum ACI sites in a Multi-Site domain | 14, increased from 12 in NDO release 3.7(1) |
The 500 msec figure is worth noting because it’s often confused with a fabric-to-fabric limit. It isn’t. It’s how far the orchestrator can be from a controller. The fabrics themselves have no validated limit, and what actually constrains you is what your applications tolerate.
Intersite L3Out, introduced in ACI 4.2(1), lets an EPG in one site reach an external network through an L3Out hosted in another site. That supports centralised internet or WAN egress, and gives external connectivity a path to survive the loss of a local L3Out. Contracts are configured between the EPG and the remote external EPG, and the routable TEP addresses of each site must be reachable across the ISN. Intersite L3Out with policy-based redirect is supported in later NDO releases, and that’s a combination to confirm against the release matrix rather than assume.
Remote leaf
Remote leaf extends a pod to a location where deploying spines and an APIC isn’t justified. The remote leaf switches are managed by the APIC cluster in the parent location and behave as normal leaf switches, with local switching continuing if the connection to the parent pod is lost.
| Requirement | Detail |
|---|---|
| Software | ACI 3.1 or later. ACI 4.1(2) or later where combined with Multi-Site |
| Hardware | Second generation spine or later in the main location |
| Latency | Less than 300 msec round trip time to the parent pod |
| Routing | OSPFv2 to the IP network. DHCP relay required for discovery |
| Addressing | Dedicated remote leaf TEP pool, plus an external TEP pool on each pod |
| Topology | Tier-2 leaf switches cannot be connected to remote leaf switches |
| Resilience | Deployed in pairs. Local switching continues if the parent is unreachable |
Branch sites, colocation cages and edge compute are the typical use cases. You get consistent policy without the cost of a full pod.
The addressing requirement is the one that reaches backwards into decisions you’ve already made, which is why I made a point of it in post one. Configure an external TEP pool on every pod as a matter of course, even where remote leaf isn’t in scope. It costs nothing at build time and retro-fitting it is disruptive.
There’s also ACI vPod, which extends the policy model into environments where Cisco hardware can’t be deployed using virtual spine and leaf functions on a hypervisor, and Cloud Network Controller, previously Cloud APIC, which extends policy into AWS, Azure and Google Cloud under NDO orchestration. Both are worth knowing exist. Neither is a substitute for the three main options.
Comparison
| Attribute | Multi-Pod | Multi-Site | Remote Leaf |
|---|---|---|---|
| APIC clusters | One, distributed | One per site | Parent fabric cluster |
| Change domain | Fabric wide | Per site | Parent fabric |
| Availability domain | Fabric wide | Per site | Parent fabric |
| Layer 2 extension | All bridge domains, by default | Selective, per bridge domain | Extended from parent pod |
| Multicast in transit network | PIM Bidir required | Not required | Not required |
| DHCP relay in transit network | Required | Not required | Required |
| Maximum latency | 50 msec RTT | No fabric limit. 500 msec to NDO | 300 msec RTT |
| Namespace | Shared | Per site, translated on spines | Shared with parent |
| Orchestration | APIC | Nexus Dashboard Orchestrator | APIC |
| Local spine required | Yes | Yes | No |
| Typical use | Metro pair, one operations team | Geographically separate DCs, DR | Branch, colo, edge compute |
These aren’t mutually exclusive. The most common large scale pattern is a Multi-Site domain where each site is itself a Multi-Pod fabric, with remote leaf switches hanging off the nearest pod. Local redundancy within a site, isolation between sites, and policy reach out to the small locations.
Combining Multi-Site with remote leaf requires ACI 4.1(2) or later. Confirm the software matrix for whatever combination you’re proposing before committing to it, because these constraints have moved between releases and will move again.
Things that decide the design
Stretched bridge domains
Extending a bridge domain across locations is technically easy and operationally attractive, because workloads move without re-addressing. It also extends the Layer 2 failure domain across locations, which is the thing you were presumably trying to avoid by having two locations.
Stretch only the bridge domains that require it. In Multi-Site that’s a per bridge domain decision and each one should be justified individually. Where Layer 3 reachability is sufficient, keep the bridge domain local and route between sites. That’s the lower risk option and it’s more often sufficient than people expect.
Where a subnet is stretched, ingress traffic from outside the fabric can arrive at either site. Host route advertisement, available from ACI 4.0, resolves this by advertising a host route for each endpoint from the site where that endpoint actually lives. Enable it deliberately, because it increases the number of prefixes you’re presenting to the external network, and confirm the external network can carry them.
Service node placement
Firewall and load balancer placement is one of the harder decisions in a multi-location design, because most stateful devices need traffic symmetry and multi-location designs are good at producing asymmetry.
Active/standby pairs stretched across locations create a dependency on the interconnect and can cause traffic tromboning, where traffic crosses between sites purely to reach the active device and then crosses back.
Independent service node pairs per location, with policy-based redirect steering traffic locally, avoids the tromboning but means managing state per site. That’s usually the better answer, and it’s more work up front.
Where a service graph is stretched in Multi-Site, verify the supported combinations for your intended release. Support here has changed materially between releases, and this is not an area to assume.
Scale
Scale figures move between releases and have to be validated against the Verified Scalability Guide for the specific target release. For sizing conversations only: 400 leaf switches per pod, 500 per Multi-Pod fabric as of ACI 6.0(1), up to 1200 edge ports per APIC-M cluster, 14 sites per Multi-Site domain from NDO 3.7(1), 18 ECMP links from tier-2 to tier-1 leaf, minimum one MP-BGP route reflector per pod with two recommended.
Choosing
I’d rather walk through this in prose than hand over a decision table, because the reasoning matters more than the answer.
One data centre with no immediate second site: single fabric, but plan the TEP pools and addressing as though a second location is coming. That costs nothing now and avoids a rebuild later.
Two data centres in the same metro, one operations team, under 50 msec, no regulatory separation requirement: Multi-Pod. You get one policy domain and one management point, and the operational simplicity is real. You’re accepting a fabric-wide change domain in exchange, and that’s a reasonable trade when one team runs both sites and treats them as a pair.
Two or more geographically separate data centres with a disaster recovery requirement: Multi-Site. Independent change and availability domains, selective Layer 2 extension, and the ability to upgrade one site while the other carries service.
A regulatory or contractual requirement for separation: Multi-Site, because separate APIC clusters give you a defensible administrative boundary. This one is about being able to demonstrate the separation, not just having it.
Here’s the case worth thinking hardest about. Where the requirement is genuine disaster recovery or regulatory separation, Multi-Site is the right answer even where the physical distance would comfortably allow Multi-Pod. Distance permits Multi-Pod. It doesn’t recommend it.
I’ve seen the distance argument used to justify Multi-Pod on sites that were built to be independent of each other, and it’s a category error. If the whole point of the second site is that it survives the first one failing, a fabric-wide change domain undermines the requirement you’re spending the money on.
Large estate needing resilience both within and between locations: Multi-Site where each site is a Multi-Pod fabric.
Small satellite sites, colo cages, edge compute: remote leaf attached to the nearest pod.
Existing stretched fabric: plan a migration to Multi-Pod or Multi-Site.
Where the series lands
Six posts on, the through line is that ACI concentrates its consequential decisions at the beginning. The TEP pool that can’t be changed. The APIC placement that decides whether you keep quorum. The forwarding scale profile that needs a reload. The enforcement direction that applies to a whole VRF. The change domain you draw once and live inside.
None of those are difficult decisions in themselves. They’re all documented, and none of them take long to make. They just have to be made before the first APIC is initialised, by somebody who understands what depends on them, and that’s a scheduling problem as much as a technical one.
Which is the argument for doing the design properly rather than quickly. Not out of thoroughness for its own sake, but because ACI gives you fewer chances than most platforms to change your mind later.