When a failed check becomes useful evidence

A laboratory robot places a sample vial in a rack, reports successful completion, and withdraws its gripper. An independent camera then records the vial centre 8 mm from its required position, beyond the permitted 5 mm tolerance. The immediate question is not merely whether the robot failed, but whether the discrepancy originates in its movement, its measurement system, its success criterion, or the procedure used to check it.

This lesson follows that laboratory scenario through two illustrative engineering examples. All numerical values are stipulated teaching data, not reported experimental findings. The central objective is to turn an unsuccessful verification result into a defensible account of what remains uncertain and what must be learnt.

The connection to robotic metacognition is substantive: a robot must evaluate the adequacy of its own decisions, not merely execute them. METATOOL investigates computational models of synthetic awareness grounded in human metacognition and examines them through robotic experiments. This research establishes a relevant context for distinguishing successful action from justified confidence in that action. METATOOL project overview

Distinguishing the result from the judgement

A robot may complete its programmed movement while failing to satisfy the placement requirement. Conversely, a faulty measurement procedure may reject an acceptable placement. Verification concerns conformity with specified requirements; validation concerns whether the resulting system fulfils its intended purpose in the relevant operational setting. These questions are complementary, but their evidential targets differ. NASA verification planning guidance

For this lesson, verification failure means that a verification activity does not establish the required conformity. That description deliberately leaves the cause open. A discrepancy is the recorded mismatch requiring investigation, whereas a learning need is a precisely identified limitation in the knowledge, model, procedure, or judgement needed to resolve it. NASA’s verification guidance likewise requires investigation of discrepancies, including possible problems in verification methods, criteria, and environments. NASA verification results guidance

Dimension Requirement not satisfied Verification not trustworthy
Illustrative observation An independently established placement error is 8 mm against a 5 mm limit. The camera reports 8 mm, but its coordinate transformation is unverified.
Warranted judgement The observed placement does not conform. Conformity remains unresolved.
Immediate learning need Establish why placement accuracy is inadequate. Establish whether the measurement supports a valid decision.
Premature response to avoid Declaring the controller defective without isolating the cause. Retraining the controller before checking the measuring system.

The table separates two questions that are easily conflated: “Did the product satisfy the requirement?” and “Can this verification result support that judgement?” In the vial scenario, the same reported error can lead to different investigations depending on the measurement evidence. A failed check identifies an evidential problem before it identifies a defective component.

Establishing whether the failure is genuine

The first analytical task is to preserve the relationship between the requirement, the configuration tested, and the evidence collected. NASA’s systems engineering guidance identifies acceptance criteria, test configuration, measuring equipment, calibration information, and recorded results as important elements of verification documentation. These records make the result interpretable rather than merely repeatable. A value of 8 mm has limited diagnostic meaning if the required reference frame, software version, or measurement accuracy is unknown. NASA Systems Engineering Handbook

For the illustrative vial test, the following sequence establishes the status of the evidence before any corrective change occurs. Each stage consumes a specific record and produces a narrower, more defensible judgement. This is an application of the handbook’s documentation principles, not an additional universal verification standard.

  1. Recover the criterion. Input: requirement P-01 specifies a horizontal centre-position error no greater than 5 mm after release. Output: an explicit pass–fail rule.
  2. Identify the configuration. Input: controller version 1.4, rack identifier R-03, and the recorded camera calibration. Output: the configuration associated with the disputed result.
  3. Examine measurement adequacy. Input: camera reference checks and their documented limitations. Output: a judgement on whether the reported 8 mm error is sufficiently trustworthy for this decision.
  4. Classify the evidence. Input: the requirement, configuration, and measurement assessment. Output: a supported nonconformity, a supported conforming result, or an unresolved result.

The important discipline is to resist converting uncertainty into a convenient binary answer. In this example, a missing calibration record does not establish that the vial is correctly placed; it prevents the camera result from settling the matter. Equally, an independently confirmed 8 mm error should not be dismissed merely because another instrument reports success. NASA explicitly recognises that discrepancies can originate in either the product or the verification arrangements. NASA product realisation guidance

Explaining the discrepancy without guessing

Once the evidence is credible, the investigation shifts from establishing what happened to identifying what could explain it. NASA’s verification-results guidance calls for comparing actual and expected results, determining causes, evaluating proposed changes, and planning renewed verification where necessary. The implication is methodological restraint: a corrective action requires an evidential connection to the discrepancy. “The placement failed” is an observation; “the controller caused the failure” remains a hypothesis until supported. NASA verification results guidance

In the vial scenario, several mechanisms can produce the same final displacement. A consistently displaced target estimate suggests a different investigation from a vial that moves only when the gripper opens. The most useful additional observation is therefore one that distinguishes competing explanations, rather than one that merely reproduces the final error. The following table expresses that diagnostic logic for the stipulated case.

Diagnostic dimension Target-estimation error Movement error Release-related displacement
Proposed mechanism The robot aims at the wrong location. The robot does not reach the commanded location. The vial moves during release.
Discriminating evidence Compare the estimated rack centre with an independent reference. Compare commanded and observed gripper positions before release. Compare vial position immediately before and after release.
Illustrative signature Target estimate displaced by 6 mm. Gripper remains 7 mm from the commanded position. Error increases from 2 mm to 8 mm during release.
Corresponding learning need Improve the target-location model. Characterise positioning behaviour. Characterise the release interaction.

The table does not imply that only one mechanism can operate at a time. For example, a small target-estimation error and a release disturbance could jointly exceed the tolerance. Its purpose is to prevent a plausible explanation from becoming an unquestioned conclusion before the relevant intermediate observations exist.

A common pitfall is to change several parameters simultaneously and interpret subsequent success as confirmation of the preferred explanation. In this scenario, changing camera calibration, approach direction, and release timing together would leave their individual contributions unresolved. A more informative investigation preserves an explicit relationship between each proposed mechanism and the evidence capable of challenging it.

Turning uncertainty into a learning specification

“Improve the robot” does not identify what must be learnt, who or what must learn it, or how improvement will be recognised. A usable learning specification names the uncertainty, the intended knowledge change, the evidence required, and the boundary of the resulting claim. Here, learning may concern the robot’s model, the engineer’s causal understanding, or the adequacy of the verification procedure. It need not entail machine-learning retraining.

Robotic metacognition introduces a further distinction between making a decision and judging its reliability. A METATOOL-associated research paper proposes confidence as a second-order judgement within robotic decision-making and emphasises embodied action monitoring. This supports attention to the robot’s assessment of its decisions, rather than treating its internal success signal as decisive evidence. The proposal concerns computational self-monitoring; it does not establish that a robot possesses human-like introspection. Meera and colleagues, Robot Metacognition

For the vial scenario, the following specification translates an unresolved release mechanism into a bounded learning objective. The numerical evidence threshold is illustrative and applies only to this teaching case. It is not a general reliability or safety certification criterion.

Specification element Illustrative definition
Uncertainty Whether release introduces a direction-dependent displacement.
Learning target Characterise displacement during release for the specified vial and rack.
Required evidence Independently measured positions immediately before and after release.
Demonstration criterion Ten predefined placements each satisfy the 5 mm final-position limit.
Claim boundary Applies only to the tested vial, rack, approach directions, and configuration.

A passing result would support conformity within that bounded demonstration, not universal competence. NASA’s verification guidance similarly emphasises recording limitations, configuration differences, corrective actions, and the evidence needed for closure. The broader lesson is that further learning becomes accountable when its endpoint is an observable evidential condition rather than an aspiration. NASA verification results guidance

Example: locating the error at release

Locate the change before naming the cause

In the first illustrative case, the laboratory team establishes that its independent measurement is adequate for the stipulated decision. The recorded vial error is 2 mm immediately before release and 8 mm immediately afterwards. The observations locate the additional displacement within the release interval, but do not yet distinguish gripper contact from vial settling. The learning need therefore concerns the release interaction, not an unspecified deficiency in the entire robot.

  1. Establish the starting position. Input: the pre-release record shows a 2 mm error. Output: evidence that the vial is within tolerance at that instant.
  2. Locate the change. Input: synchronised pre-release and post-release records show 2 mm and 8 mm respectively. Output: a 6 mm increase in error magnitude during release.
  3. Bound the causal claim. Input: contact and settling remain unresolved. Output: a release-interval hypothesis rather than a confirmed gripper fault.
  4. Define closure evidence. Input: the revised release configuration and the predefined ten-placement demonstration. Output: a documented acceptance decision for the stated conditions.

Vial centre-position error before release is 2 mm, within the 5 mm limit; afterwards it is 8 mm, outside the limit. The error magnitude increases by 6 mm during release, but the cause remains unresolved.

Suppose the revised configuration subsequently produces ten measured errors between 1 mm and 4 mm. The immediate benefit is a bounded demonstration that satisfies the stipulated placement criterion, accompanied by a more precise account of the original discrepancy. The limitation is equally important: those results do not establish performance for different vial geometries or rack arrangements. In the laboratory workflow, the revised configuration and its supporting evidence remain linked to the discrepancy record rather than replacing the original failure history.

Example: correcting confidence rather than motion

Continuing with the same robot, a second illustrative review examines its internal success predictions for another vial type. Before each of ten placements, the robot assigns a success probability of 0.95; independent verification records only six conforming placements. The observed success proportion is therefore 0.60 for this small batch. That difference warrants investigation, although ten observations do not establish the robot’s long-run calibration.

  1. Recover the predictions. Input: ten recorded probabilities of 0.95, issued before the outcomes are known. Output: an auditable prediction record.
  2. Match predictions to outcomes. Input: six conforming and four nonconforming placements. Output: a batch success proportion of 0.60.
  3. Identify the learning gap. Input: the prediction–outcome discrepancy and the vial-type record. Output: a question about whether confidence remains informative for this condition.
  4. Separate the acceptance decisions. Input: physical placement evidence and confidence evidence. Output: distinct judgements about task performance and self-evaluation.

The distinction changes the interpretation of improvement. Lowering the robot’s confidence without correcting placement would not make the four failed placements conforming; improving placement without examining the confidence mechanism would leave its self-evaluation unverified. Confidence-informed robotic decision-making provides the research rationale for examining both, while the particular batch comparison here remains illustrative. Meera and colleagues, Robot Metacognition

The human reviewer’s learning is also explicit in this case. “The robot reports success” becomes a claim to examine rather than a substitute for independent evidence. Within the laboratory’s review process, the useful outcome is a documented distinction between what the robot achieves, what it predicts, and what the available observations justify.

Web Sources

From discrepancy to justified learning

  • Establish the evidence before attributing the failure: distinguish product nonconformity from an inadequate verification result.

  • Connect explanations to discriminating observations: retain the distinction between an observed discrepancy and its hypothesised cause.

  • Specify further learning precisely: identify the uncertainty, required evidence, acceptance criterion, and claim boundary.

  • Evaluate self-assessment separately: successful action and justified confidence require distinct judgements.

The central discipline is to make robotic performance explainable without claiming more than the evidence permits. A failed verification result becomes useful when it yields a narrower uncertainty and a defensible learning objective. This connects robotics, scientific investigation, and metacognitive judgement within one repeatable approach to accountable improvement.

Last modified: Saturday, 31 October 2026, 2:33 PM