Most Kafka teams treat replication as synonymous with disaster recovery, when in reality it only covers one piece of the puzzle. Replication moves data, but it doesn't automatically switch clients, preserve security and governance, keep schemas compatible, or guarantee your on-call team can execute a failover under pressure.
Teams that recover in minutes instead of hours have gone beyond replication and addressed six technical areas, and centralized how those areas are managed. This checklist distills those areas into a practical review tool you can bring to your next design review, disaster recovery audit, or game day planning session.
It's the companion to our full guide, Kafka Disaster Recovery Beyond Replication. Use the framework below to frame the work, then run the checklist itself to find your gaps.
Disaster recovery readiness breaks into three phases, each requiring different work. Get all three right and a real failover becomes a calm, sequenced procedure instead of a 3 AM scramble.
- Define RTO/RPO per domain and workload
- Set up cross-cluster replication (MirrorMaker, Cluster Linking)
- Identify Wave 1 applications for priority recovery
- Choose per service: standby (active/passive) or both serving (active/active)
- Map dependencies: apps, topics, schemas, ACLs, certs
- Run chaos tests in staging, then production game days
- Measure actual failover time during drills
- Test the monitoring and decision chain, not just infra
- Verify the runbook is current and executable by on-call
- Pre-stage configs in both regions and check for drift
- Switch clients through a stable endpoint, not per-app
- Validate Wave 1 apps first, then the long tail
- Enforce security and governance during cutover
- Communicate to stakeholders and compliance teams
- Plan failback once stable; reversing the switch needs its own runbook
Before you start
- Replication is one-third of the job. Getting data to a secondary cluster is necessary but nowhere near sufficient; the other two-thirds is security parity, config sync, observability, client switching, and testing.
- A plan you haven't tested is a hypothesis. The only way to know your real recovery time is to measure it in a drill.
- The full 24-item checklist is below. Score yourself against the six areas and focus first where a failure would be unrecoverable.
Six technical areas to verify in your disaster recovery design review. Work through each item and mark where you stand.
Scoring guide
If you can check 20+ items on this list, you're well ahead of most organizations. Fewer than 12 means significant gaps remain. Focus first on the areas where a failure would be unrecoverable: client switching, security parity, and testing.
Security & identity parity
- TLS and certificate strategy. Covers both clusters, including custom CAs and cert renewal processes.
- Authentication parity across regions. mTLS, SASL, or OAuth configured identically; cluster-specific credentials pre-provisioned.
- ACL/RBAC provisioned on secondary. Least privilege maintained; no broader access on failover than on primary.
- Audit logs on secondary. Available, retained, and accessible during and after failover for compliance evidence.
Topic & schema configuration parity
- Topic configs match across clusters. Retention policies, compaction strategies, and quota limits verified in sync.
- Schema Registry replicated. Schemas and compatibility rules match; consumers won't fail on schema mismatches after failover.
- Configuration drift detection. An automated process catches and alerts on divergence between primary and secondary.
- Consumer offset sync configured. Offset translation strategy defined; the reprocessing window is understood and acceptable.
Data protection & compliance continuity
- Encryption parity verified. Field-level and payload-level encryption applied consistently across both clusters.
- Data masking rules travel with the data. Policies enforced regardless of which cluster serves traffic, not tied to co-located infrastructure.
- PII handling consistent across regions. Regulated data treated identically, with no exposure risk during the failover window.
- Region-specific compliance requirements met. The secondary site satisfies data residency, sovereignty, and regulatory obligations for all workloads.
Observability & operational readiness
- Monitoring independent of failed infrastructure. Dashboards, alerts, and on-call routing don't depend on the region or cluster that went down.
- Replication lag tracked continuously. Cross-cluster lag is your effective RPO metric; alerting thresholds are set and tested.
- Secondary cluster sized for catch-up load. At least 1.5x steady-state capacity to absorb the reconnection burst after failover.
- Producer delivery timeouts reviewed.
delivery.timeout.ms exceeds your expected detection + decision window; the 2-minute default is often too short.
Client switching & traffic routing
- Failover from a single control point. One action switches all clients, with no per-app config changes, restarts, or cross-team coordination.
- Client broker discovery method documented. How clients find brokers today, and how that changes during failover, is clearly understood.
- Failover authority defined. Clear ownership of who can trigger a switch, with an approval chain short enough for a 3 AM incident.
- Kafka client versions audited. Clients on 3.8+ support rebootstrap (KIP-899); older clients get stuck and require restarts.
Testing & continuous validation
- Game days include full regional loss. Partial failure tests aren't enough; validate the worst-case scenario your plan is designed for.
- Chaos tests run quarterly at minimum. Simulate broker loss, region loss, and auth failures for mission-critical workloads.
- Named runbook owner with a review cadence. Runbooks are versioned, reviewed on a schedule, and updated after every incident or drill.
- On-call can execute without the author. The person at 3 AM should be able to follow the runbook without the person who wrote it.
No heroics. The sequence should be simple enough that the person on call at 3 AM can execute it without the person who designed it.
- Confirm incident scope
- Broker failure, partial, or full regional outage?
- Choose recovery mode and approve cutover
- Keep the approval chain short
- Route clients via a stable endpoint
- One action, not dozens
- Wave 1 apps first, then the long tail
- Verify end-to-end data flow
- Monitoring, backpressure, comms
- Then plan your failback
Use these indicators to assess where your disaster recovery program stands today, and where the common blind spots are.
✅ Signs you're on track
- Failover time is measured, not estimated. You've run a drill and have a real number to compare against your RTO target.
- Game days run quarterly or more often. Chaos testing is routine, not an annual event, and runbooks are updated afterward.
- Failover is a single operational action. One decision, one action, all clients move; no per-app coordination.
- Security posture survives failover. Encryption, RBAC, masking, and audit logging are enforced identically on both clusters.
- The on-call engineer can execute the runbook. Anyone on rotation can follow it at 3 AM without calling the author.
- Failback has its own tested runbook. Reversing the switch is often harder than the initial cutover; it's been planned and validated separately.
🚫 Common pitfalls to avoid
- Treating replication as the entire DR strategy. Data arrives at the secondary but clients can't connect, authenticate, or find the right schemas.
- Monitoring that depends on what failed. If dashboards and alerts are in the same region as the primary, you're flying blind during failover.
- Failover that requires per-app coordination. If switching 50 services means 50 config changes across 10 teams, your RTO scales with your org chart.
- Producer timeouts shorter than your decision window. The default
delivery.timeout.ms of 2 minutes means producers silently drop data before anyone decides to fail over. - Configuration drift between clusters. Topic settings, schemas, quotas, and ACLs diverge silently over time, so the secondary behaves differently by failover.
- Untested or stale runbooks. If the last drill was a year ago or the author has left, the runbook is a liability, not an asset.
Ready to close the gaps?
Book a 45-minute disaster recovery workshop with our team to review your Kafka estate and build a first-pass wave plan.
Book a DR workshop
