# APIC clustering: failover without the flapping

> Configurable thresholds and cooldown windows keep your Cisco ACI fabric monitored even when controllers go down.

**Operations** · 2026-04-08 · 4 min read · By Beyrak A., Founder & Security/System Engineer at SAMURAI

**Key takeaways**

- Single-controller monitoring goes blind on the first maintenance window; treat APICs as a cluster from day one.
- Failover needs both a threshold and a cooldown: react to real failure, ignore blips, never flap.
- Sync one primary per cluster; polling all three controllers triples load and produces duplicate snapshots.

## The problem with single-controller monitoring

[Cisco ACI fabrics](https://nometa.az/en/cisco-aci-analyzer/) are managed by APIC controllers, typically three per site, forming a cluster. Most monitoring tools connect to one controller and assume it will always be available. When that controller goes down for maintenance, firmware upgrade, or failure, the monitoring tool goes blind. Worse, it may start throwing errors that trigger false alerts, or it may silently stop collecting data without anyone noticing for hours.

SAMURAI treats APIC controllers as clusters from the start. You register the controllers, and the system automatically groups them by site, elects a primary, and handles failover without operator intervention.

## Cluster structure

Clusters are keyed by `ndoID|siteName`. This composite key ensures that multi-site deployments managed by NDO (Nexus Dashboard Orchestrator) group correctly: controllers belonging to the same site under the same NDO instance form one cluster, even if they were registered at different times.

Each cluster tracks its members, the current primary, and a failure counter per member. The primary is the controller that SAMURAI actively syncs from. All other members are standby: they exist in the system, but sync operations skip them and redirect any request for a standby member to the current primary.

## Failover logic

When a sync to the primary controller fails, the failure counter increments. A single failure does not trigger failover. Network blips, brief maintenance windows, and transient API errors are too common to react to immediately. Failover only triggers after `failover_threshold` consecutive failures (default: 3).

When the threshold is reached, SAMURAI promotes the next healthy member to primary and resets the failure counter. But it also starts a `failover_cooldown_seconds` timer (default: 600 seconds). During cooldown, no further failover can occur. This prevents flapping: the scenario where two controllers alternate between primary and standby because both are intermittently reachable.

### Why cooldown matters

Without cooldown, a degraded controller that responds to every other health check would cause rapid primary oscillation. Each failover resets counters, re-authenticates API sessions, and potentially produces duplicate or incomplete sync data. The 600-second cooldown gives the original primary time to either fully recover or fully fail, producing a clean signal for the next failover decision.

## NDO-aware multi-site grouping

In multi-site ACI deployments, Nexus Dashboard Orchestrator manages policies across sites. SAMURAI's cluster grouping is NDO-aware: controllers are grouped by their NDO association and site name, not just by IP range or manual assignment. This means a three-site, nine-controller deployment automatically produces three clusters, each with independent failover, no manual configuration required.

The sync engine skips non-primary members entirely. This is not just an optimization: it prevents duplicate data. All three controllers in a cluster serve the same fabric state. Syncing all three would produce three identical snapshots, waste API capacity, and complicate [change detection](https://nometa.az/en/blog/change-detection-without-audit-logs/). One primary, authoritative sync per cluster is the correct model.

---

Canonical page: https://nometa.az/en/blog/apic-clustering-failover/
Part of SAMURAI, a self-hosted, multi-vendor network monitoring & security platform. Overview: https://nometa.az/llms.txt
Contact: info@nometa.az · Docker Hub: https://hub.docker.com/r/beyrak44/samurai