No Trace in Any Log: How Meta Hunts Silent Data Corruption
Silent data corruptions are hardware errors that go undetected by the larger system: defective silicon — aggravated by datapath dependencies, temperature variance, and age — computes wrong answers that leave no record or trace in any system log, stay undetected inside workloads, and propagate across services until they surface as application-level mysteries far from the defective part. A CPU gets a few hours of vendor testing, a couple of days at the integrator, then enters a production fleet where at-scale testing is genuinely hard. Harish Dixit's post describes Meta's answer across 3+ years of production experience: two complementary regimes of active interrogation. Fleetscanner piggybacks minutes-long, intrusive tests onto every maintenance event — reboots, kernel and firmware upgrades, reimages, repairs — reaching each machine about once every 180 days, 68 million tests and roughly 4 billion fleet-seconds so far. Ripple co-locates with live workloads, injecting bit patterns with known expected answers in bursts of hundreds of milliseconds — 1,000× shorter than opportunistic runtimes — about 2.5 billion unique test seeds per month, always on, tunable per service. Neither suffices alone: ripple reaches 70% coverage in ~15 days where opportunistic testing needs ~6 months, yet 23% of faulty CPUs are ever found only by the deep tests. Meta's recommendation is both, everywhere.
Plant a defect that computes wrong answers under one data pattern and watch every health check stay green while corrupted results ship downstream. Arm fleetscanner and catch it at the next maintenance window — months of exposure later; arm ripple and catch it in days; then plant the rare-mode defect only the deep test can see, and learn why the fleet needs both hands.
Problem
The post's definitional paragraph is the problem statement: silent data corruptions impact computational integrity at scale; their sources include datapath dependencies, temperature variance, and age among other silicon factors; they do not leave any record or trace in system logs; they stay undetected within workloads and propagate across several services — so the consequences appear in systems far removed from the defective part, as data loss and application-level problems that can take months to debug once the software-level residue of a corruption has spread. Applications can be exposed for months when no accelerated detection exists.
The manufacturing pipeline explains why the fleet inherits the problem. Within a typical cycle, a server or CPU is tested at the vendor for a few hours, at the integrator for a couple of days in the best case, and on a sampling basis — historically, burn-in was the bulk of a CPU's testing life. Then the part enters a production fleet of enormous scale, where implementing testing becomes really challenging: every test iteration takes time from production workloads; out-of-production testing pays ramp-down and ramp-up costs around every run; in-production testing risks leaving residual configurations that hurt workload performance. Scaled across a fleet, both the variation and the total time become significant. The two challenges are stated as duals — for in-production testing, smart colocation with minimal effects; for out-of-production testing, using production downtimes effectively without disturbing the maintenance work that created them.
What makes the failure mode gray rather than merely rare is the epistemic dead end: there is no log line to alert on, no error counter to graph, no health check that a lying CPU fails. Detection cannot be observed into existence. It has to be manufactured.
Solution
Meta's answer is active interrogation at two depths, and the post's conclusion is that both are equally important — deploy both in any large-scale fleet.
Opportunistic testing, facilitated by an internal tool called Fleetscanner, piggybacks on the maintenance events machines already experience: system reboots, kernel upgrades, firmware upgrades, device reimages, host provisioning, namespace reprovisioning, and repairs. Because the machine is already out of production, tests can run for minutes and be intrusive in ways live machines can't tolerate. The cadence is the mechanism's honest limit: across fleet population and upgrade rhythms, a machine passes through opportunistic testing on average once every 180 days (with fine-grained controls to test sensitive tiers more often) — so a corruption can arrive and operate inside the window between two tests. The cost is equally honest: most maintenance activities now take longer by the duration of a test, so test time and frequency are evaluated closely against detection efficacy. The accumulated ledger: more than 68 million tests, roughly 4 billion machine-seconds of testing, opportunistically harvested from downtime that already existed.
Ripple testing exists because opportunistic testing structurally cannot catch three things: corruptions that manifest between testing windows, corruptions dependent on data patterns the benchmark never exercised, and corruptions triggered by switches between operating modes. Ripple runs silent-error detection in conjunction with the workload itself — shadow testing beside production, injecting bit patterns with expected results on a predetermined per-service cadence, with granular controls over test interval, duration, and device configuration. Its runtimes are 1,000× shorter than opportunistic tests — hundreds of milliseconds — scheduled around workload behavior and switchable per workload; its unique seeds and deliberately manufactured transition events catch defects that only manifest across thousands of iterations. The footprint tax is real but observed to be negligible next to routine management activity, and the early results were shared with CPU vendors, who built similar mechanisms into their own tools. The scale: approximately 2.5 billion unique tests and seeds per month, nearly 100 million seconds of colocated testing.
The comparison table is the argument for running both. Ripple reaches 70% of SDC coverage in about 15 days; opportunistic testing takes roughly 6 months to reach the same point. But 23% of faulty CPUs are uniquely detectable only by the deep opportunistic tests, and a further 7% only by repeated ripple instances accumulating iterations. Fast-and-shallow finds most defects quickly; slow-and-deep finds the defects nothing else ever will; the coverage split is expected to vary by defect type and is itself continuously revisited to tune test vectors, cadences, and runtimes. The closing stance treats the problem as permanent infrastructure: applications are too sensitive to corruption, and exposure windows too costly, for detection to be anything but an always-on fleet priority.
Tradeoffs
- Health signals answer the wrong question. Liveness, responsiveness, thermals, error counters — every passive signal the fleet collects presumes failures announce themselves, and silent corruption's defining property is that it doesn't: no record, no trace, no log. The class lesson at its purest: when a failure mode is invisible to the system's own signals, adding more observability of the same kind adds zero detection — the question 'is this machine telling the truth?' can only be answered by asking questions whose answers you already know.
- Vendor testing is hours; fleet life is years. A few hours at the vendor, days at the integrator, sampling thereafter — against silicon whose defects are accelerated by age and temperature across years of service. The gap between manufacturing test coverage and lifetime defect emergence is structural, which is why detection had to become a fleet capability rather than a procurement checkbox — and why Meta feeding results back to CPU vendors closed a loop the industry didn't have.
- Deep testing rides on downtime you already bought. Piggybacking minutes-long intrusive tests onto reboots, upgrades, and repairs converts existing maintenance into detection nearly free — except that every maintenance event now runs longer by a test, and the cadence is hostage to upgrade rhythms: once per ~180 days on average means a defect can own a machine for half a year between deep looks. Opportunism is cheap precisely because it doesn't control its own schedule.
- Constant testing must be a polite guest. Ripple lives beside production workloads, so everything about it is negotiated: hundreds-of-millisecond runtimes, per-service cadences, per-workload switches, device-configuration controls, and a footprint tax kept negligible. The reward for the politeness is what opportunism can't buy — coverage of between-window arrivals, data-pattern-dependent defects, and mode-switch triggers, plus the thousands-of-iterations defects that only surface under relentless repetition with fresh seeds.
- Neither depth substitutes for the other, and the numbers prove it. Ripple's 15 days to 70% coverage versus opportunistic's 6 months makes shallow-and-constant the speed layer; opportunistic's 23% unique detections make deep-and-rare the completeness layer; ripple's own 7% unique (iteration-accumulated) closes the split. A fleet running only one regime chooses between slow detection and permanent blind spots — Meta's recommendation is explicitly both, and the split itself is monitored as defect types evolve.
- Manufactured evidence is the only evidence there will ever be. Injecting known-answer bit patterns and comparing results converts an epistemically invisible failure into an ordinary detection problem — at the permanent price of running billions of tests monthly against machines that are almost always innocent. The overhead ledger (4 billion opportunistic seconds lifetime; ~100 million colocated seconds monthly) is what 'no trace in any log' costs to live with: you pay for certainty about the overwhelming majority to catch the silent few.
Patterns in this article
- Known-Answer Testing
Minted from the post's core epistemology, under the name cryptographic validation already uses: when a failure mode leaves no passive signal, inject questions with predetermined answers — bit patterns with expected results, targeted micro-benchmarks — and treat disagreement as the detection. The definition carries the post's two-depth deployment shape as intrinsic: shallow-and-constant beside live workloads (milliseconds, always-on, polite) for speed, deep-and-rare during maintenance windows (minutes, intrusive, opportunistic) for completeness — with Meta's coverage table as the evidence that neither depth substitutes for the other.
- Fault Isolation
The class's shared response shape appears here as test-blast-radius containment: both regimes carry granular per-tier, per-service, per-workload controls over interval, duration, and device configuration precisely so that testing a fleet at billions of iterations does not disturb the workloads it protects — smart colocation with minimal effects on one side, use of production downtimes without disrupting the maintenance work that created them on the other. The gray-failure classmates (Slack cellular, Cloudflare byzantine) apply the same discipline to the failure's consequences; Meta applies it upstream, to the detection's footprint.
Also solving this
Other systems in behindscale's Gray failure defeats automatic detection class: