Pattern · seen in 1 breakdown across 1 company

Known-Answer Testing

observability

Definition

When a failure mode produces no passive signal — no log line, no error counter, no failed health check — because the failing component doesn't know it's failing, detection must be manufactured: inject inputs whose correct outputs are predetermined, execute them on the component under suspicion, and treat any disagreement between produced and expected answers as the detection event. The component's honesty is tested directly rather than inferred from its self-reported health. Cryptographic module validation named the technique (known-answer tests); fleet-scale silicon screening, storage scrubbing, end-to-end checksum patrols, and canary queries against serving systems are the same move at different layers.

The pattern's deployment shape is two-tiered, and the tiers are complements rather than alternatives. Shallow-and-constant: sub-second known-answer probes co-located with live workloads, always on, scheduled and sized to be a polite guest — buying fast time-to-detection and coverage of defects that only manifest under production-like conditions, mode transitions, or accumulated iterations. Deep-and-rare: minutes-long intrusive test batteries run when the component is already out of service (maintenance windows, upgrades, repairs) — buying coverage of defects the quick probes structurally cannot reach. Evidence from fleet practice shows each tier detecting faults the other never finds; running only one chooses between slow detection and permanent blind spots. Boundary against health checking: a health check asks 'are you up?'; known-answer testing asks 'are you right?' — a component can pass every liveness probe while failing every known-answer probe. Boundary against Independent Observability: that pattern moves the WATCHER out of the failure domain so passive signals survive; this pattern applies when no passive signal exists to preserve, and evidence must be actively created.

When it applies

01Failure modes that corrupt outputs without raising errors: silicon defects, bit rot, memory corruption, miscompiled or misdeployed logic returning plausible-but-wrong results
02Components whose self-reported health is structurally uninformative about correctness — the thing being verified is the truthfulness of computation or storage itself
03Fleets large enough that rare per-unit defect rates become certainties, and exposure windows (time between defect onset and detection) carry real data-loss or propagation cost

Tradeoffs

Almost every test interrogates an innocent component: the steady-state cost is billions of probes against healthy hardware to catch the silent few, and that overhead ledger must be defended continuously
Coverage is only as good as the question bank: defects triggered by data patterns, modes, or iteration counts the probes never exercise remain invisible — the test-vector library needs the same curation as the fleet
In-production probes are guests with obligations: co-location footprint, residual-configuration risk, and per-workload politeness controls all become permanent operational surface

The same move, 1 ways

Every row is a production system that bet on this pattern — the note says how, in that system's own terms.

Meta
Engineering at Meta
2022
When a failure leaves no signal to observe, you detect it by asking questions whose answers you already know: give the chip a known input, check for the known output, and treat any mismatch as the detection. This article shows the pattern used at two depths, and the point is that you need both. A shallow, constant version runs beside live work (milliseconds, always on) to find most faults fast; a deep, occasional version runs during maintenance (minutes, invasive) to find the ones nothing else catches. Meta's coverage numbers are the proof that neither depth can stand in for the other. Read the breakdown →

Problems this pattern answers

The walls where its breakdowns live — each opens the cross-company comparison.

Gray failure defeats automatic detectionMETA →