No Trace in Any Log: How Meta Hunts Silent Data Corruption
A silent data corruption is a hardware error that nothing notices: a faulty chip computes a wrong answer and reports no error, writes no log line, and trips no alert. The bad result flows downstream, spreads across services, and surfaces weeks later as an application bug, with no trail back to the chip that caused it. It is also hard to catch at the source. A chip gets only a few hours of testing at the manufacturer and maybe a couple of days at the company that assembles it into a server, then runs for years, where its flaws worsen with age and heat. Harish Dixit's post describes Meta's answer, drawn from 3+ years in production: since no ordinary signal reveals the failure, you must manufacture the evidence. You ask each machine questions whose answers you already know, and treat a wrong answer as the only symptom you will ever get. Meta does this at two depths: a deep test while a machine is down for maintenance, and a shallow test that runs constantly, right beside the live work. Its central finding is that a fleet needs both.
Plant a fault that computes wrong answers under one data pattern, and watch every health check stay green while corrupted results ship downstream. Arm Fleetscanner and catch it at the next maintenance window, months of exposure later. Arm Ripple and catch it in days. Then plant the rare-mode fault that only the deep test can see, and learn why the fleet needs both hands.
Problem
Start with what a silent data corruption is, because the definition is the whole problem. It is a hardware fault that makes a chip compute an incorrect result. Its causes are physical: the way data flows through particular circuits, temperature swings, and the chip's age, among other factors. It leaves no record or trace in any log. It hides inside a running program and spreads across several services, so the damage shows up in systems far from the faulty chip. That can mean data loss and application bugs that take months to track down, long after the bad result has spread through the software. With no way to catch it early, an application can be exposed for months.
The way chips are made explains why the problem lands in the data center. A chip's testing before deployment is brief:
- a few hours at the manufacturer,
- at best a couple of days at the company that assembles it into a server,
- and only spot-checks after that.
Then the chip enters a data center of enormous scale, where testing is genuinely hard. Every test takes time away from real work. Testing a machine that is already offline still costs time to wind down and back up around each run. Testing a machine that is still live risks leaving behind settings that hurt its performance afterward. Across a whole data center, both the differences between machines and the total time add up. Meta frames the two challenges as a pair. For testing on live machines, share the machine politely, with as little effect on the real work as possible. For testing on offline machines, use the downtime a machine already has, without disturbing the maintenance that created it.
What makes this failure gray, rather than just rare, is the dead end it creates for observation. There is no log line to alert on, no error counter to graph, and no health check that a lying chip fails. Detection cannot be watched into existence. It has to be manufactured.
Solution
Meta's answer is to actively question the hardware at two depths, and its conclusion is that both matter equally: deploy both in any large data center.
The deep test (Meta calls its tool Fleetscanner) rides on the maintenance events machines already go through: restarts, software and firmware updates, wipe-and-reinstalls, setup, and repairs. Because the machine is already offline, the test can run for minutes and be as invasive as it needs to be, which a live machine could never tolerate. Its weakness is its timing, and Meta is honest about it. Given how often machines actually go into maintenance, a given machine gets tested only about once every 180 days (important machines can be set to test more often). A fault that appears just after a test can run for months before the next one. The price is honest too: most maintenance now takes longer by the length of a test, so how long and how often to test is weighed carefully against how much it actually catches. The running total: more than 68 million tests and roughly 4 billion machine-seconds of testing (all the test time added up across the data center), every bit of it harvested from downtime that already existed.
The shallow test (Meta's tool is Ripple) exists because the deep test simply cannot catch three kinds of fault:
- faults that appear between maintenance windows, when the deep test is not running,
- faults that only show up for particular data patterns the deep test never happened to try,
- and faults that only trigger when the chip switches between operating modes.
The shallow test runs right alongside the live work, feeding the chip inputs whose correct answers are known, on a set schedule per service, with fine control over how often it runs, for how long, and on which hardware. Each run is about 1,000 times shorter than a deep test: a few hundred milliseconds. Its constantly changing test values and deliberately forced mode-switches catch faults that only appear after thousands of repetitions. The cost to the real work is genuine but measured as negligible next to ordinary day-to-day fleet management. Meta shared its early results with chip vendors, who built similar checks into their own tools. The scale: about 2.5 billion separate tests and seeds per month, and nearly 100 million seconds of testing run beside live work.
The coverage numbers are the argument for running both. The shallow test reaches 70% of the corruptions in about 15 days; the deep test takes roughly 6 months to reach that same 70%. But 23% of faulty chips are found only by the deep test, and another 7% only by the shallow test once it has run long enough to pile up repetitions. Fast-and-shallow finds most faults quickly; slow-and-deep finds the ones nothing else ever will. The split is expected to change with the type of fault, so Meta keeps revisiting it to tune what each test checks, how often, and for how long. Meta treats the whole thing as permanent infrastructure: applications are too sensitive to corruption, and the exposure windows too costly, for detection to be anything but an always-on priority.
Tradeoffs
- Health signals answer the wrong question. Whether a machine is alive, responsive, cool, or throwing errors: every ordinary signal a data center collects assumes a failure will announce itself, and the whole point of silent corruption is that it does not. No record, no trace, no log line. When a failure is invisible to the system's own signals, adding more monitoring of the same kind catches nothing new. The only question worth asking, is this machine telling the truth, can be answered only by asking questions whose answers you already know.
- Factory testing lasts hours; a chip runs for years. A few hours at the manufacturer, a couple of days at the assembler, spot-checks after that, against a chip whose flaws worsen with age and heat over years of service. The gap between how much a chip is tested when it is made and how long its flaws take to surface is built in. That is why detection had to become a job the data center does continuously, rather than a box the supplier ticks once. It is also why Meta feeding results back to chip vendors closed a loop the industry did not have before.
- The deep test rides on downtime you already paid for. Hanging minutes-long, invasive tests onto restarts, updates, and repairs turns existing maintenance into detection at almost no extra cost. The catch is twofold: every maintenance event now runs a little longer by one test, and the timing is at the mercy of how often machines happen to cycle. About once every 180 days per machine means a fault can own a machine for half a year between deep looks. The deep test is cheap precisely because it does not control its own timing.
- The constant test has to interfere as little as possible with the real work it runs beside. Because it lives on the same machines as live workloads, everything about it is kept small and controllable: runs of a few hundred milliseconds, schedules per service, on/off switches per workload, hardware controls, and a footprint measured as negligible. Keeping it that quiet is what buys the coverage the deep test cannot reach. That means faults that arrive between maintenance windows, that only show up for specific data patterns, or that trigger on a mode switch, plus the faults that surface only under relentless repetition with fresh test values.
- Neither depth replaces the other, and the numbers show it. The shallow test's 15 days to 70% coverage, against the deep test's 6 months, makes shallow-and-constant the fast layer. The deep test's 23% of faults found by nothing else makes deep-and-rare the thorough layer. The shallow test's own 7% found by nothing else closes the gap. A data center running only one of the two is choosing between slow detection and permanent blind spots. Meta's recommendation is explicitly both, and it watches the split over time as the mix of fault types changes.
- Manufactured evidence is the only evidence there will ever be. Feeding in inputs with known answers and comparing the results turns an invisible failure into an ordinary detection problem. The permanent price is running billions of tests a month against machines that are almost always fine. That overhead (4 billion deep-test machine-seconds over the program's life, and about 100 million seconds a month beside live work) is what living with 'no trace in any log' actually costs. You pay for constant certainty about the innocent majority in order to catch the silent few.
Patterns in this article
- Known-Answer Testing
When a failure leaves no signal to observe, you detect it by asking questions whose answers you already know: give the chip a known input, check for the known output, and treat any mismatch as the detection. This article shows the pattern used at two depths, and the point is that you need both. A shallow, constant version runs beside live work (milliseconds, always on) to find most faults fast; a deep, occasional version runs during maintenance (minutes, invasive) to find the ones nothing else catches. Meta's coverage numbers are the proof that neither depth can stand in for the other.
- Fault Isolation
Both testing approaches are built to stay out of the way of the work they run beside. Each has fine controls over how often, how long, and on which hardware a test runs, so that testing a whole data center billions of times over does not disturb the very workloads it is meant to protect. The usual use of fault isolation is to contain the damage a failure causes; here it is turned inward, to contain the cost of the detection itself.
Also solving this
Other systems in behindscale's Gray failure defeats automatic detection class: