Context Window (Legal AI)

The maximum amount of text a legal AI model can process in a single interaction — directly determining how much of a contract, brief, or document the AI can analyze at once without losing context or resorting to document chunking.

Context window size is one of the most practically consequential technical specifications for legal AI — yet it is one of the most commonly overlooked factors in tool evaluation. The context window determines a fundamental limitation: how much of a legal document the AI can actually see and reason about at one time.

Legal documents are long. A sophisticated M&A purchase agreement may run 200-400 pages. A complex software license with exhibits and schedules may total 150 pages. An eDiscovery document set for a major litigation may encompass millions of pages. Context window size determines whether a legal AI tool can analyze these documents comprehensively in a single pass or must resort to strategies that introduce accuracy limitations.

The stakes of context window limitations in legal analysis are not trivial. Contracts are complex webs of cross-references. A defined term established in Section 1 may appear 80 times across a 200-page agreement, each use carrying the precise meaning of the definition. A payment obligation in Section 8 may be conditioned on a concept defined in Section 3.2 and limited by a provision in Schedule C. An AI analyzing a chunk of Section 8 without the full context of Section 3.2 and Schedule C will produce an analysis that misunderstands the obligation.

Context window size has expanded dramatically since 2022 — from models with 4,000-token windows to models supporting 128,000 tokens or more, with some research models supporting windows of 1 million tokens or beyond. Understanding where a vendor's tools fall in this range, and what strategies they use when documents exceed the window, is essential for accurately assessing their capability on the complex, long-form legal documents that characterize sophisticated legal practice.

How It Works

Tokens — the unit of context measurement:

AI models do not count characters or words; they count tokens. A token is approximately 0.75 words in English, but varies by word length and character complexity. Common legal words tend to token-compress efficiently; specialized legal terminology and citation formats may be more token-dense.

Practical token-to-page conversions (approximate): - 1,000 tokens ≈ 750 words ≈ 2-3 pages of standard legal text - 32,000 tokens ≈ 24,000 words ≈ 60-80 pages - 100,000 tokens ≈ 75,000 words ≈ 200-250 pages - 128,000 tokens ≈ 96,000 words ≈ 250-320 pages - 200,000 tokens ≈ 150,000 words ≈ 400-500 pages

These conversions are approximate and depend on document formatting, table density, and the specific tokenizer the model uses.

What occupies the context window:

The full context window must accommodate not just the document being analyzed, but all the other content the AI needs:

• System prompt: The vendor's instructions to the AI about how to behave, what role to play, and what tasks to perform. These can consume 500-2,000 tokens.
• User query: The lawyer's specific question or instruction. A detailed prompt with examples might consume 200-500 tokens.
• Conversation history: In interactive tools where the AI maintains context across a multi-turn conversation, prior exchanges accumulate in the context window, consuming space that could otherwise hold document text.
• Retrieval content: In RAG systems, retrieved legal documents or excerpts are injected into the context window alongside the document being analyzed. A research query might inject 5,000-20,000 tokens of retrieved case law into the window.
• The document itself: Only the remaining context window capacity is available for the actual legal document to be analyzed.

This means the effective document length a tool can process is often significantly less than the nominal context window size.

Context window strategies when documents exceed the limit:

When a legal document exceeds the available context window, vendors apply one of several strategies:

Chunking with overlap: The document is split into segments of appropriate token length, with overlapping portions at segment boundaries to reduce loss of cross-boundary context. Each chunk is analyzed separately, and the outputs are aggregated. The limitation is that cross-chunk relationships — a defined term in Chunk 1 used in Chunk 5 — may not be correctly resolved.

Hierarchical summarization: Long documents are first summarized to fit within the context window, and then the summary is analyzed. This loses detail from the original document — exactly the detail that may matter for legal accuracy.

Map-reduce approaches: A specific question is posed to each chunk independently ("does this section contain a change of control provision?"), and the chunked results are combined ("yes, Section 12 contains a change of control provision"). This works for factual extraction but poorly for analysis requiring understanding of how provisions interact.

Selective context injection: The system identifies the most relevant portions of the document for the specific query and injects only those portions, discarding the rest. This can work well for targeted queries but will miss context that is relevant but not obviously related to the query.

Context window in leading legal AI platforms:

Harvey AI is built on GPT-4 and GPT-4o, which support context windows of 128,000 tokens — sufficient for most single contracts but potentially limiting for very long agreements with extensive schedules. Harvey uses chunking strategies for documents that exceed effective context limits. CoCounsel processes documents with context management strategies designed for legal document types, including handling of long contracts and document sets used in due diligence. Spellbook operates within Microsoft Word, processing the currently visible document or selected text — which creates practical context limitations based on what portion of a document the user is working on.

The rapid evolution of context windows:

Context window sizes have expanded rapidly and continue to do so. In 2023, a 16,000-token context window was considered large for legal AI applications. By 2025, models with 128,000-token windows are commonplace, and frontier models support windows of 200,000 tokens or more. This expansion has reduced the practical significance of context window limitations for most standard legal documents — though very long agreements (major credit facilities, complex M&A transaction documents) still challenge even large context windows.

Key Considerations for Law Firms

Measure the context your actual documents require: Before evaluating legal AI tools on context window size, measure the token length of your typical documents — specifically the longest and most complex agreements in your practice. This gives you a concrete requirement to evaluate vendor context window claims against.

Understand the vendor's chunking strategy: For documents that may exceed the context window, ask vendors to describe their chunking or context management strategy. Request examples of how their tool handles a specific cross-reference that spans two chunks. The quality of the chunking strategy significantly affects accuracy on long, complex documents.

Include long documents in your evaluation test set: When benchmarking candidate tools, include your longest and most complex typical documents in the test set. Accuracy on a 20-page NDA is not a reliable predictor of accuracy on a 200-page credit agreement. Test on documents representative of your most challenging use cases.

Monitor context utilization in interactive workflows: In conversational AI interfaces where conversation history accumulates in the context window, very long conversations may eventually crowd out document context. This can cause the AI to lose track of document content as the conversation extends. Be aware of this limitation in extended interactive analysis sessions.

Consider context requirements for multi-document analysis: Some legal tasks require analyzing multiple documents simultaneously — comparing a definitive agreement against an LOI, reviewing a contract against a playbook, analyzing transaction documents as an integrated set. Multi-document analysis requires larger context windows than single-document analysis; verify vendor capabilities for multi-document scenarios.

Limitations and Risks

Context window size does not equal comprehension: A model with a 128,000-token context window that receives a 100,000-token contract may still fail to correctly reason about relationships between provisions that are far apart in the document. Attention mechanisms — the way models weight different parts of the context when generating outputs — can produce "lost in the middle" effects where content in the middle of a very long context receives less attention than content near the beginning or end.

Effective context is smaller than nominal context: The nominal context window size must be divided among system prompts, conversation history, retrieval content, and the document itself. The effective document context available in a 128,000-token window may be 80,000-100,000 tokens after these other uses are accounted for.

Chunking introduces cross-reference failures: Any document chunking strategy creates potential for the AI to miss or misinterpret cross-referenced provisions. This is the most significant practical risk for complex, heavily cross-referenced legal agreements.

Rapid evolution creates information asymmetry: Context window sizes are increasing rapidly. Vendor marketing materials may quote current model context windows that differ from the version actually in production for enterprise customers. Verify the context window of the specific model version available to your deployment.