A disaster recovery plan built around the business, not the tools
A financial services client ran critical applications on Kubernetes but had no clear way to recover them if a cluster or site failed. I designed a tiered disaster recovery approach that matched each application's protection to its business importance, with immutable backups, a second recovery location and a documented runbook.
CLIENT
Financial services enterprise
ROLE
DR workshops, design and runbook
EMPLOYER
FABS
TECHNOLOGY
KubernetesVeeam KastenS3-compatible object storage with object lock
[HERO IMAGE: backup policy console or DR architecture, sanitised]
at a glancedesign targets
Tier 1: criticalCustomer-facing and payment services
15 minRPO target1 hourRTO target
Tier 2: importantInternal business applications
4 hoursRPO target4 hoursRTO target
Tier 3: standardReporting, dev and test
24 hoursRPO target24 hoursRTO target
Backups retained immutably for 30 days.
context
The client's applications had moved onto Kubernetes faster than its recovery plans. Some data was backed up, but there was no agreed answer to two basic questions: how much data could the business afford to lose, and how quickly did each application need to come back? Without those answers, every application was being treated the same way, which meant over-spending on systems that didn't need it while under-protecting the ones that did. For a regulated business, it was also a gap auditors would eventually ask about.
targetsRecovery targets had to reflect business priorities and regulatory expectations
scopeApplications needed to be recovered as complete units, including configuration, secrets and data, not just storage volumes
resilienceBackups had to survive a ransomware attack, not just a hardware failure
budgetThe design had to fit the client's budget, so protection levels had to be proportionate
tier 2 · importantRPO 4 h · RTO 4 hinternal business apps
tier 3 · standardRPO 24 h · RTO 24 hreporting, dev and test
backup tool
application-aware, runs in cluster · schedule set per tier
resourcesconfigsecretsvolumes
backup
off-cluster storageS3-compatible object storeobject lock · 30 dayscopies can't be changed or deleted, even by an attacker with cluster access
restore
recovery cluster
second site · standby
restore 1st · tier 1back within 1 h
restore 2nd · tier 2back within 4 h
restore 3rd · tier 3back within 24 h
recovery runbookstep-by-step, in tier orderusable by the client's own teamtested on a regular schedule
targets checked, not assumed
1tiered by impact protection matched to what an outage would cost
2whole applications config, secrets and data backed up together
3ransomware-proof copies immutable and kept off the production cluster
4critical apps first restore in priority order, following the runbook
what i did
01Started with the business, not the tools.Ran workshops with application owners to agree how much data loss and downtime each system could tolerate, then set RPO and RTO targets per tier.
02Classified every application.Grouped workloads into three tiers, so the tightest and most expensive protection went only where it was needed.
03Designed application-aware backups.Backup policies captured each application as a whole: its Kubernetes resources, configuration, secrets and persistent volumes, on a schedule matched to its tier.
04Protected the backups themselves.Copies were stored off-cluster in object storage with immutability enabled, so an attacker who reached the cluster couldn't delete or encrypt the backups.
05Planned recovery to a second location.Defined how applications would be restored onto a separate cluster if the primary one was lost.
06Wrote the runbook.Documented step-by-step recovery procedures, in priority order by tier, so the client's own team could run a recovery under pressure.
07Built in regular testing.Recommended scheduled recovery tests for each tier, so the targets are checked against reality rather than assumed.
why tiering matters
Protecting everything to a 15-minute RPO sounds safe, but it multiplies storage, bandwidth and licensing costs across systems that don't need it. Tiering puts the budget where an outage would actually hurt. This is where my accounting background is useful: DR is ultimately a cost and risk decision, and the business should be able to see what each level of protection costs and what it buys.
deliverables
Agreed RPO and RTO targets for every application tier
Application classification across the Kubernetes estate