How to stop AI hallucination in document analysis

A language model can be instructed to quote its sources, avoid guessing and say when it is uncertain. Those instructions are useful. They are not controls. The same model that generates an unsupported answer can generate a plausible quotation, an incorrect page reference or a confident explanation of why the answer follows.

Document analysis needs a harder boundary. An extracted value should not reach the user unless the system can produce the underlying words and prove that those words occur in the source. This converts citation from presentation into a release condition. The model may still make mistakes internally, but unsupported values cannot pass through the gate.

For credit documents, that distinction matters. A reported leverage ratio, basket amount, maturity date or voting threshold may enter a model, committee paper or restructuring analysis. “The model was told to be careful” is not an adequate basis for relying on it.

What is quote-gating?

Quote-gating requires every extracted value to travel with a verbatim span from the source document. A deterministic verifier then checks that the submitted span actually appears in the authorised document text. The value is released only if the check succeeds.

The minimum extraction record might contain:

{
  "field": "required_lenders_threshold",
  "value": "more than 50% of aggregate commitments",
  "source_span": "Required Lenders means ... more than 50% of the aggregate amount of the Commitments ...",
  "document_id": "credit-agreement",
  "page": 27
}

The page number is useful for navigation, but it is not the evidence. Page references can be generated incorrectly. The source span is the evidence, and the verifier must locate it in the text derived from the identified document.

A basic implementation follows five steps:

1. Define the field and the documents within scope.
2. Ask the model to return a structured value and a verbatim supporting span.
3. Search the authorised source text for that span.
4. assign a result state based on the search and extraction process.
5. Expose the value only if its state is verified.

The gate should operate at field level. A response containing ten extracted terms is not verified merely because one supporting quotation matches. Each value needs its own evidence.

Why does verification work better than asking the model to be careful?

A prompt is evaluated inside the same probabilistic process that produces the answer. It can reduce undesirable behaviour, but it cannot enforce an invariant. When the model encounters a long definition chain, poor OCR or an unfamiliar drafting structure, it may still complete the expected pattern instead of reporting that the evidence is missing.

Post-hoc verification separates generation from admission. The model proposes an extraction. Another component decides whether the proposal satisfies a test that does not depend on confidence, fluency or apparent legal sophistication.

The critical invariant is simple:

No extracted value may surface without a source span that the system can locate in the permitted source material.

This blocks a broad class of hallucination. A fabricated threshold accompanied by fabricated wording fails because the wording is not in the document. A real-looking quotation copied from another transaction fails because it is not in the authorised document. A number inferred from market convention fails unless the document itself supports it.

The gate does not require the model to understand why it failed. It only requires the system to withhold unsupported output.

Why should the verifier be a program rather than another model?

Where the verification question is mechanical, the verifier should be mechanical. Whether a string occurs in a document is a search problem, not a legal-reasoning problem.

Using another model to decide whether a quotation “looks supported” recreates the original weakness. The second model can overlook substitutions, accept paraphrases or rationalise a mismatch. Agreement between two models is not proof that the text exists.

A program can check:

• whether the cited document is within scope;
• whether the source span is non-empty;
• whether it appears in the extracted document text;
• whether the reported page or location contains that span;
• whether the span meets minimum context requirements;
• whether multiple reported fields improperly rely on an unrelated fragment.

Exact matching is the cleanest rule, but source preparation complicates it. PDF extraction may alter line breaks, ligatures, hyphenation or whitespace. OCR may confuse characters. The system can maintain both the original text and a deterministically normalised representation. It may then compare normalised strings while preserving the original span for display.

Normalisation rules must be narrow and auditable. Collapsing repeated whitespace is different from permitting semantic paraphrase. If the verifier accepts wording merely because it means roughly the same thing, quote-gating has become model-based entailment rather than source verification.

What are the three possible outcomes?

Every requested field should terminate in one of three states: verified, absent or unverifiable.

Verified means the evidence exists in the source. It does not automatically mean the legal interpretation is correct. A model might quote a ratio from the wrong covenant, overlook a proviso or confuse a grower basket with a fixed basket. Quote-gating establishes provenance. Additional rules must test interpretation.

Absent means more than “nothing was returned.” It requires a completed, bounded search. The system must know which documents, schedules, amendments and page ranges were included, and what counted as a qualifying result.

Unverifiable covers incomplete or defective analysis. Examples include unreadable pages, failed OCR, a missing exhibit, a quotation that does not match, interrupted retrieval or a model response without an adequate source span. It is the correct state whenever the system cannot support either a value or a finding of absence.

Why must absent be a first-class result?

A system that reports only what it found cannot distinguish absence from failure to look. An empty result could mean the provision is not present. It could also mean the relevant amendment was omitted, the retrieval query failed or the model stopped before reaching the applicable section.

That ambiguity is dangerous in covenant work. Consider a request for an unrestricted-subsidiary designation capacity. Returning no answer does not establish that the capacity is absent. The mechanism may sit in a definition, an investment basket, a general exception or an amendment. The system needs a declared search scope and a completed procedure before it can report absence.

Absence should therefore be represented as data, not encoded as a blank cell, null value or application error. A useful record includes:

{
  "field": "unrestricted_subsidiary_designation",
  "status": "absent",
  "documents_searched": ["credit-agreement", "amendment-no-1"],
  "search_completed": true
}

This also improves review. A professional can challenge the scope behind an absent result. Without that record, there is no way to tell whether the system reached a conclusion or merely produced nothing.

How should the system decide that a field is absent?

Presence and absence require different proof. Presence can be supported by one qualifying span. Absence requires coverage.

The system should define, before extraction:

• the document set that governs the question;
• the field’s acceptable drafting variants;
• relevant defined terms and cross-references;
• the units or forms the value may take;
• the conditions that make the search complete.

For a maturity date, the search may need the operative agreement plus amendments affecting maturity. For a restricted payments basket, it may need the covenant, incorporated definitions, provisos and applicable schedules. The required coverage follows the legal structure of the field.

Deterministic search can help establish coverage, but keyword failure alone rarely proves absence. Drafting varies. A model may identify candidate passages and follow cross-references, while the program records which sources were processed. The final absent state should be permitted only when all required search tasks completed without a qualifying verified span.

If a required document is missing, the result is unverifiable. If a candidate provision is found but its text cannot be checked against the source, the result is unverifiable. Absence is a conclusion, not a fallback.

What does quote-gating fail to solve?

Quote-gating prevents unsupported extraction. It does not eliminate every analytical error.

A verified quotation may still be:

• taken from an inapplicable subsection;
• superseded by an amendment;
• limited by an exception outside the quoted span;
• dependent on a defined term elsewhere;
• descriptive of capacity that is unavailable under current facts;
• correctly quoted but incorrectly normalised into a value.

These failures require further controls. The system can require surrounding context, resolve defined terms, collect linked provisos and validate numerical transformations. High-impact fields may require rule-based checks or human review.

The evidence span must also be long enough to support the proposition. A quotation containing only “50%” proves that those characters exist, not that they represent a lender voting threshold. The extraction schema should require the operative language, relevant subject and necessary qualification.

Quote-gating is therefore a foundation, not a complete legal-analysis methodology. Its value is precise: it makes unsupported answers structurally ineligible for release.

What should an implementation log?

The audit trail should explain both successful and unsuccessful results. At minimum, retain the field requested, document version, extracted value, submitted span, matched location, verification method, result state and failure reason.

Version control matters. A span found in an original credit agreement does not establish that the same term remained operative after an amendment. The verifier must search the identified source set, not an undifferentiated collection of transaction documents.

Logs should also preserve the distinction between model generation and system verification. “The model cited page 42” is not the same as “the verifier located the submitted span on page 42 of the authorised document.” Reviewers need to know which statement they are examining.

The practical standard is straightforward. Show the value when its evidence passes. Report absent only after the defined search completes. Otherwise return unverifiable. That design does more than encourage careful answers: it prevents a value without documentary support from becoming an answer at all.