Why AI agents fail on long documents

A long document does not usually defeat an agent because the underlying legal analysis is too sophisticated. More often, the relevant language never reaches the model. The attachment was not parsed. Search targeted the wrong field. A result limit removed the last few matches. An answer was clipped between a tool and the agent. The final response then looks careless even though the decisive failure happened upstream.

Delivery is necessary but not sufficient. Material can reach the model and still go unused: as the amount of competing text grows, the chance that the decisive passage is the one actually relied on falls. So the ladder has a third rung between plumbing and reasoning — the passage arrived, but it arrived buried among enough near-misses that selection failed. The fix there is not a better prompt or a larger model; it is putting less, and more relevant, material in front of it.

This distinction matters for credit work. A question such as “identify every exception to the debt covenant” is not satisfied by locating a representative passage. It requires enumeration across definitions, provisos, incorporated baskets and cross-references. If the harness returns only the first page of matches, the model cannot reason about provisions it never sees. Changing the model may alter the prose while leaving the omission intact.

The correct default is therefore: assume a harness problem until the evidence points to a reasoning problem. Diagnose the path from source document to final answer in order.

What is the diagnostic ladder?

Use the following ladder whenever an agent gives an incomplete, unsupported or apparently inattentive answer.

Do not begin at step six. First prove that the model received the complete evidence needed to answer the question.

Can the agent actually reach the document?

Start with access, not interpretation. Confirm that the agent can identify the document, open it and retrieve known language from it.

A file appearing in a user interface does not establish that the agent can read it. Upload processing may have failed. A connector may expose metadata but not file contents. Permissions may differ between the user session and the document tool. A scanned amendment may be stored as images without usable OCR.

Test access with a known-answer probe. Choose a distinctive phrase visible in the document and ask the system to return the surrounding text with a page or document reference. Then test material from the beginning, middle and end. This catches partial ingestion and page-range failures that a single successful search would miss.

For a credit agreement, include annexes and schedules in the test. Defined terms, pricing grids, exceptions and subsidiary designations may sit outside the main covenant section. “Document loaded” is not enough if the ingestion boundary excluded those components.

Was the text extracted faithfully?

An agent can reach a file yet receive a damaged representation of it. PDF extraction commonly disrupts multi-column text, headers, tables and footnotes. OCR can confuse clause labels, monetary symbols or negative language. Page breaks can separate a qualification from the provision it limits.

These defects are especially consequential in credit documents. The difference between “and” and “or” may change how conditions operate. A missing “not” reverses a restriction. A debt-capacity table flattened into running text can detach a ratio from the basket it qualifies.

Compare the extracted text against the source for several structurally difficult pages. Inspect dense definitions, nested clauses, tables and signature-adjacent schedules. Also verify document completeness: page count, amendment coverage and whether exhibits were processed as separate files. If extraction is defective, retrieval and reasoning tests performed on top of it will produce misleading conclusions.

Did retrieval return the right evidence?

Once access and extraction are established, inspect retrieval. The issue is not merely whether search returned something relevant. It is whether it returned all material required by the question.

A semantic search for “debt baskets” might retrieve the main permitted debt covenant while missing a definition that expands acquired debt, a general basket in another clause, or an incremental facility provision using different language. Keyword search has the opposite weakness: it may find every occurrence of “Debt” while burying operative exceptions among definitions and cross-references.

Enumeration questions require a retrieval strategy designed for coverage. Useful approaches include:

• Search for the governing covenant and each defined term it incorporates.
• Follow explicit cross-references rather than treating chunks as self-contained.
• Search amendments alongside the original agreement.
• Separate discovery from synthesis: collect candidate provisions first, then classify them.
• Preserve document identity, page location and clause hierarchy with every passage.

Review the actual query arguments and returned passages. If the agent searched only for the user’s wording, the retrieval layer may have failed to translate the question into the document’s vocabulary.

Was the result set cut off?

Caps are among the most damaging long-document failure modes because they can look exactly like model carelessness.

Some caps are silent. A search tool returns ten passages without saying that thirty matched. The agent reasonably treats the ten as the complete result set and produces an incomplete list.

The more deceptive case is an announced cap. The tool may say “showing first 20 results.” That disclosure sounds responsible, but the substantive failure remains. Anything that sorts late is absent. If the omitted provision is the final exception to a restricted payments covenant, the answer will still be wrong.

An announcement is not a continuation mechanism. The agent needs the total number of matches and a way to request the next page. It must also recognise when the task requires continuation. A single-result question may not require exhaustive paging. “Identify all,” “compare every,” and “list each exception” plainly do.

Check for limits at every boundary:

1. Source-to-parser limits, including page or file-size restrictions.
2. Parser-to-index limits, including dropped sections or chunk-count ceilings.
3. Search limits, including default result counts and ranking cut-offs.
4. Tool-response limits, including character or token clipping.
5. Model-context limits, including evidence removed before generation.
6. Final-answer limits, including output caps that cut off the completed analysis.

Raising a cap can produce a larger quality improvement than changing models because it changes the evidence available for reasoning. A stronger model cannot recover a covenant exception that the harness discarded.

What should replace silent or announced caps?

Use progressive disclosure. Return a bounded result page, but make incompleteness machine-readable and actionable.

A suitable tool response should include:

• The results on the current page.
• The total number of matching results, where available.
• The number returned and the applied limit.
• A cursor or page token for the next request.
• An explicit indication of whether more results remain.
• Stable identifiers so results can be deduplicated across pages.

The agent can then decide whether to continue based on the question. For exhaustive work, it should page until completion or state a documented stopping condition. For targeted work, it can stop after obtaining sufficient direct evidence.

Progressive disclosure also protects the context window. The system need not inject an entire agreement at once. It can expose the document in bounded units while preserving the agent’s ability to reach every relevant unit.

Why is observability the highest-leverage investment?

Without tool-level observability, all upstream failures collapse into the same symptom: a bad answer. Teams then change prompts or models because those are the only visible components.

A useful trace records the tool selected, its arguments, the document searched, filters applied, result totals, returned passages, pagination state and truncation flags. It should also show what evidence was ultimately placed in the model’s context. This creates a chain of custody from source language to conclusion.

For a missed basket, the trace should let a reviewer answer three questions quickly:

• Did the agent search for the provision?
• Did the tool find and return it?
• Did the model receive it before answering?

If the first answer is no, improve planning or query construction. If the second is no, fix retrieval, parsing or access. If the third is no, fix caps and context assembly. Only when all three are yes should model reasoning become the primary focus.

When is the model actually the problem?

A model-level diagnosis is justified when the harness supplied complete, legible and relevant evidence, yet the model misapplied it. Examples include treating disjunctive conditions as cumulative, overlooking a proviso present in the supplied passage, or presenting an inference as express document language.

At that point, test prompt structure, decomposition and model choice. Ask the agent to distinguish quoted terms from analysis, resolve definitions before applying covenants and state which evidence supports each conclusion. But preserve the trace. Otherwise, an apparent reasoning improvement may simply reflect a different retrieval path.

The practical sequence is simple: verify access, extraction, retrieval, caps and evidence delivery; then evaluate reasoning. That order turns “the agent missed it” from a vague complaint into a fixable engineering diagnosis.