Rare AI errors need a different review from average performance

An application may answer routine questions accurately while occasionally exposing restricted information or taking an unauthorised action. Those failures should not be diluted by a large number of easy successes.

Define critical failure categories

Identify outcomes that require separate attention because of their consequence. Appropriate risk owners should determine those categories for the actual workflow.

NIST’s Generative AI Profile addresses risks specific to generative AI. It is a risk-management reference, not a certification that a chosen deployment is safe.

An illustrative internal assistant could be useful at summarisation while being unsuitable if it reveals another team’s confidential documents. The access failure requires its own investigation.

Construct targeted tests as well as ordinary samples

Use authorised scenarios that probe the relevant boundaries. Keep these tests distinct from a representative performance sample so readers understand what each result means.

A small challenge set can reveal a vulnerability. Passing it does not prove the absence of all rare failures. Report the scope and limits rather than claim comprehensive safety.

Investigate the system around the model, including permissions and tool execution. Some consequences can be constrained outside the generated answer.

Connect findings to release decisions

Define which failures block release, require restricted scope or trigger additional review. A high aggregate score should not automatically override an unresolved critical issue.

Create a release report section for critical failures that remains visible even when the overall score improves. The reviewer should see whether each issue is resolved, constrained by scope or still open. This prevents a summary chart from silently turning a consequential exception into a tolerable average error.

GAT’s enterprise security-review guide asks suppliers to produce relevant assurance. Rare-error evaluation adds a practical layer by showing how the organisation treats the failures that matter most, even when they occur infrequently.