Your backups are fine. Nobody has tried a restore.
Backup success rate is the metric everyone puts on a dashboard, and it tells you almost nothing. The number that matters is how long ago somebody last restored one into a working environment.
"Backup successful" tells you data moved from one place to another. It does not tell you the database will mount, the volume is consistent, or that you still hold the key it was encrypted with. Most organisations find out which of those is true during an outage, which is the worst possible time.
The metric everyone tracks is the wrong one
Open any backup console and the number on the dashboard is time since last successful backup. It is the easy number to collect and it answers a question nobody is actually asking.
The question is: if production is gone right now, how long until it is back, and will the data be correct? The only metric that answers that is time since the last successful restore. Almost nobody measures it.
When we run a Well-Architected review, this is the finding that surprises people most. Snapshots are scheduled, retention is set, someone owns the runbook, and no one has ever restored one into a working environment. A backup you have not tested is a hope with a retention policy attached.
Three ways backups fail quietly
Silent corruption
Data can corrupt on the source or on the target without anything raising an alarm. The backup job reports success because it wrote the bytes it was given. Nothing reads them back until you need them.
Configuration drift
A team adds a database, or attaches a new EBS volume, and does not update the backup selection or the IAM role that performs the snapshot. The job keeps running green against the old set. The new, business-critical data is not in it. This is the most common one we see, and it is invisible until the restore comes up short.
The shrinking window
Data grows, the backup takes longer, and eventually it overlaps production hours or hits a timeout. What lands is partial. Partial backups often still report success at the job level, because the job did what it could in the time it had.
What continuous validation looks like
Quarterly manual DR tests are better than nothing and they age badly, because the environment changes weekly and the test happens four times a year. Automate it instead.
- Restore on a schedule into an isolated account or VPC. Not the production account, and not a shared one. AWS Backup restore testing plans will do this for you on EBS, RDS, DynamoDB and EFS, or you can script it.
- Run a real check against what you restored. Mount the volume, start the database, run a query that touches a table your application actually uses, and confirm the row count is in a sane range. A restore that produces a running instance and an empty schema has still failed.
- Record how long it took, then tear the environment down. That elapsed time is your real RTO. It is usually longer than the number in the DR document, sometimes by a factor of three.
- Alert on the age of the last successful restore, not the last successful backup.
Verify integrity as well as recoverability. Checksums on write and a read-back comparison catch the corruption case that a successful mount does not.
The number to write down
Restore one thing. Time it. Write down the number. That single figure changes the conversation with the business more than any DR policy document, because it is measured rather than asserted, and it can be measured again next month to show whether things improved.
If the number is worse than what you have committed to in a contract, you now know that before an auditor or a customer does.
Where the effort actually goes
The scheduling is easy. The hard part is the verification script, because it has to know what "correct" looks like for your application, and that knowledge usually lives in one engineer's head. Budget for that. On the projects we have done it, writing the checks took longer than building the restore automation around them.
How often should a restore be tested?
Monthly for anything with a stated RTO, quarterly for the rest. The frequency matters less than the fact that a failed restore raises an alert instead of sitting undiscovered until an incident.
What does AWS Backup restore testing actually do?
It picks recovery points on a schedule, restores them into a target account or region, optionally runs a validation Lambda against the restored resource, and deletes it afterwards. It supports EBS, RDS, DynamoDB, EFS and several other resource types.
What is a realistic RTO?
The one you measured, not the one in the policy document. A 4TB RDS snapshot restore plus the application checks took us just over two hours on a recent engagement, against a documented RTO of one hour.
Is the validation Lambda necessary?
Yes, and it is the hard part. A restore that completes tells you the storage layer worked. Only a query against the restored data tells you the backup was not silently corrupt.
More notes
What a Well-Architected review actually finds
Two weeks of reading someone else's AWS account, and the same handful of problems keep showing up: single-AZ databases, IAM nobody has audited since 2021, and instances paid for at on-demand rates for three years.
Writing a cloud roadmap people will actually follow
Most cloud roadmaps are a list of technologies with quarters next to them. A useful one names the constraint you are working against, what you will not do, and who owns each piece.
Send us your AWS bill
Thirty minutes with an engineer, no slide deck. We will tell you what we would change first and what it would cost to have us do it.