AI users are familiar with session transcripts. When a model runs in a chat interface or an agent runs in a harness, the session transcript displays a running stream of actions, including reasoning (i.e. thinking), commands, tool calls, and other processes. The screenshot below shows the transcript of a Claude.ai chat session in which Sonnet 5 at max effort ran a modified version1 of the build-litigation-case-timeline task from Harvey’s Legal Agent Benchmark (LAB), an open-source repository of synthetic legal tasks.2 As detailed in a previous Benchmarking Claude for Legal article, that task supplies a fifteen-document record for a fictional breach of contract dispute and asks the model to build a litigation case timeline with strategic annotations for summary judgment preparation.

When run in the Claude.ai web interface, Sonnet-5_Max ran 40 commands, viewed 7 files, and edited 5 files. The session transcript provides a running narration of the actions Sonnet-5_Max executed, beginning with “Scrutinizing litigation documents,” “Examining document extraction methods,” and “Sequencing documents chronologically.”

When Sonnet-5_Max runs the same task in Claude Code, a session transcript is displayed by the terminal. For example, the left half of the image below shows Sonnet-5_Max beginning the build-litigation-case-timeline task by “exploring the documents directory to see what’s available for review.”

Apart from displaying the session transcript, Claude Code also records a trajectory for each session. The above image compares the session transcript (left) and trajectory (right) for the same Sonnet-5_Max session, with annotations indicating corresponding entries. For example, both include the “I’ll start by exploring the documents directory to see what’s available for review” action (annotated in blue). Then, where the transcript mentions running “1 shell command,” the trajectory identifies that command as “fin ./documents -type f 2>/dev/null | sort” (annotated in red). The transcript summarizes what the model did, while the trajectory records the specific actions it took.

Session trajectories may be decomposed into behavioral characteristics. For example, a behavioral profile of Sonnet-5_Max on the build-litigation-case-timeline task identifies characteristic P1 from the model’s response to being unable to parse .docx files directly: it checked what conversion tools were available and converted the documents to markdown with pandoc.

Considered as a whole, a session’s cumulative behavioral characteristics define its behavioral profile.

Here, the Sonnet-5_Max trajectory decomposes into eleven characteristics, which are summarized in the table and compiled in the executive profile above. Individual session profiles are then evaluated as part of a broader behavioral analysis.

Methods: Tiered Behavioral Analysis

This article analyzes the behavior of three Anthropic models (Fable-5, Opus-5, and Sonnet-5) on the twelve Harvey LAB tasks set forth in a previous Twelve-Task Claude-for-Legal Benchmark.3 The tasks span six practice areas, two per area, and serve here as a native baseline for the Claude for Legal (CfL) skill comparison reserved for a subsequent article.

Each model ran each task once at each of three effort levels (High, Xhigh, and Max), producing 108 sessions (12 tasks × 3 models × 3 efforts). The analysis of those sessions proceeded through three tiers. The first tier (C1) constructed 108 session behavioral profiles by analyzing the session trajectory of each model/task/effort combination. The second tier (C2) constructed 36 model-on-task behavioral profiles by comparing session behaviors for each model/task combination across High, Xhigh, and Max. The third tier (C3) constructed 12 cross-model task profiles by comparing model behaviors for each task across Fable-5, Opus-5, and Sonnet-5.

The first tier constructed session behavioral profiles from observable behavioral characteristics in each sessions trajectory. The Sonnet-5_Max build-litigation-case-timeline profile shown above illustrates the first tier. It was constructed from eleven observable characteristics, including the adaptive recovery from a document-access failure characteristic (P1).

The second analysis tier constructed model-on-task behavioral profiles from each model’s three session profiles on a task at High, Xhigh, and Max efforts. Below, the Sonnet-5 profile on the build-litigation-case-timeline task illustrates this analysis. Its behavioral overview identifies the behaviors common to the three Sonnet-5 sessions and how those behaviors differed by session.

A behavior observed in all three sessions was recorded as a recurring behavior of the model on that task. For example, all three Sonnet-5 sessions changed their source extraction method after failing to access the source documents directly.

Conversely, where a behavior varied by session, the variation was recorded as a behavioral difference. Although every Sonnet-5 session changed how it accessed the source materials, the adopted extraction and review methods differed.

Here, the session-level variation in extraction and review methods is detailed in the manifestations of the C2-G01 characteristic. Sonnet-5_High adapted with composite extraction of the source materials and segmented follow-up reads. Sonnet-5_Max converted the source documents to a readable format, then tracked its traversal of the converted documents. Meanwhile, Sonnet-5_Xhigh adopted a hybrid conversion or extraction behavior with sequential review. Each session’s behavioral profile supplies further detail. For example, Sonnet-5_Max’s adaptive recovery from a document-access failure characteristic (P1) provides that the session “convert[ed] all DOCX files to Markdown.”

The third analysis tier constructed task-level behavioral profiles from each task’s three model-on-task profiles (Fable-5, Opus-5, and Sonnet-5). Below, the build-litigation-case-timeline task profile illustrates this analysis. Its behavioral overview identifies the behaviors common to the three models and how those behaviors differed by model.

As at the second tier, a behavior observed in all three models was recorded as a recurring behavior on that task. For example, all three models adapted their access methods to the source documents of the build-litigation-case-timeline task.

Model-level behavioral differences were likewise recorded by task. For example, the extraction and review methods adopted by each model differed.

Here, model- and effort-level manifestations are recorded in both characteristic descriptions (C3-G01 and C3-H01).

Each behavior in the analysis rests on one or more session trajectories. Task-level behaviors are derived from three model-level profiles, model-level behaviors from three session-level profiles, and session-level behaviors from session trajectories. The results below present task-level profiles for the twelve evaluated tasks.

Results: Recurring Behaviors for the Twelve Evaluated Tasks

The results below present, for each of the twelve tasks, the behaviors that recurred across Fable-5, Opus-5, and Sonnet-5.4 The behavioral differences recorded for each task, and deeper task-specific analysis, will follow in subsequent articles.

Analyze EU AI Act (AI governance)

Recurring behaviors

C3-G01: Each model family prepared the nominal supplied corpus, issued retrievals covering its visible sources, and explicitly continued the token-capped legal-summary read.

C3-G02: Each model family materialized the requested memo through recorded file operations and received creation, non-error, or mechanical verification metadata before delivery.

C3-G03: Across all observed efforts, each model family completed the visible task in one registered parent stream without a visible subagent dispatch-return relationship.

Triage Vendor AI Contracts for Compliance with EU AI Liability Directive (AI governance)

Recurring behaviors

C3-G01: Across the observed sessions, each model family established readable working representations of the office-format inputs before substantive source review.

C3-G02: Across the observed sessions, each model family attempted pre-draft acquisition across the framework, portfolio and incident context, and the supplied agreements.

C3-G03: Across the observed sessions, the parent stream retained task-level coordination and final synthesis and delivery responsibility.

C3-G04: Across the observed sessions, each model family externalized a substantial initial memo through a visible creation operation after source review.

C3-G05: Each model family exhibited an explicit post-create artifact check in at least one observed effort before delivery.

Master Service Agreement, Counterparty Review 03 (commercial)

Recurring behaviors

C3-G01: Registered task execution remained within one parent stream without visible delegation or cross-stream result integration.

C3-G02: Before formal output creation, the workflow prepared readable working materials and accessed principal, contextual, and ancillary sources.

C3-G03: When long-source access was capped or rejected, the workflow issued follow-up requests that continued or repartitioned the affected source.

C3-G04: The redline and memorandum were maintained as separate deliverables and created through serialized formal output operations.

SaaS API Subscription, Counterparty Review 04 (commercial)

Recurring behaviors

C3-G01: The workflow prepared the mixed-format source package and visibly traversed the identified source categories before deliverable drafting.

C3-G02: The workflow treated redline and memo production as successive artifact phases, completing one visible production phase before transitioning to the other.

C3-G03: Final synthesis, deliverable construction, and delivery remained centralized in the registered parent stream.

C3-G04: Before delivery, the workflow performed at least one explicit integrity-oriented check connected to the source package or generated artifacts.

Draft Diligence Summary Memo (corporate)

Recurring behaviors

C3-G01: Across all three model families, the retained record places observed task activity in registered parent streams and contains no accepted direct finding of subagent dispatch or return.

C3-G02: Across all three model families, the workflow inventoried the seven-file mixed-format package, produced locally readable representations, and accessed representations spanning the supplied source categories before memo creation.

C3-G03: Across all three model families, the workflow adjusted local source-handling operations after an intermediate access or output condition and continued toward synthesis.

C3-G04: Across all three model families, the workflow externalized a substantial initial memo after source review and proceeded to delivery.

Review Commercial Contracts Diligence 01 (corporate)

Recurring behaviors

C3-G01: Across all three model families on this task, the workflow created or used tool-readable source representations and visibly accessed planning, correspondence, and contract evidence before memo construction.

C3-G02: Across all three model families on this task, the workflow visibly addressed schedule- or planning-derived assertions or discrepancies through calculation, comparison, or contract-oriented checking before drafting.

C3-G03: Across all three model families on this task, final memo synthesis and file creation remained controlled by the parent workflow after contract-level review.

C3-G04: Across all three model families on this task, the workflow made at least one pre-draft analytical, phase, or progress state observable beyond raw source access.

C3-G05: Across all three model families on this task, the workflow materialized the requested memo as a file and performed at least one post-creation artifact check before delivery.

C3-G06: Across all three model families on this task, at least one visible obstruction or anticipated tool constraint was met by changing the operational tactic and continuing the workflow.

Identify Issues in IP Assignment Agreement (intellectual property)

Recurring behaviors

C3-G01: Across all three model families and retained efforts, the visible workflow inventoried and normalized the enumerated filesystem corpus and obtained returned access records covering every enumerated source before memorandum construction.

C3-G02: Across all three model families and retained efforts, every registered task operation remained in one parent stream without a visible dispatch-return dependency.

C3-G03: Across all three model families and retained efforts, completed visible corpus review preceded a post-review synthesis or construction phase that led to creation of the requested memorandum.

C3-G04: Across all three model families, delivery occurred without a visible post-creation substantive comparison of the completed memorandum against the source corpus.

Compare Asserted Patent Claims to Accused Product (intellectual property)

Recurring behavior

C3-G01: Across all three model families and released efforts, the visible workflow accessed each of the six named source files before the initial artifact Write.

C3-G02: Across all three model families and released efforts, visible task execution remained within the sole registered parent stream without delegation, dispatch, or subagent return.

C3-G03: Across all three model families and released efforts, the requested artifact was first materialized through one visible whole-file Write after access to the six named sources.

C3-G04: Across all three model families, Xhigh’s initial artifact Write was smaller than both High’s and Max’s in recorded characters and recorded lines.

Draft Deposition Outline (litigation)

Recurring behaviors

C3-G01: Across the observed effort sessions, each model family staged broad mixed-format evidence acquisition before the recorded artifact-writing operation and used format-responsive access paths.

C3-G02: Across all three model families and effort sessions, registered task execution remained in one parent stream without visible subagent dispatch or return linkage.

C3-G03: Each model family explicitly created the requested file artifact through recorded mutation operations before delivery.

C3-G04: Across all three model families, post-create assurance was visibly effort-conditioned: every family exposed a structural check in at least one effort, but no family exposed one uniform assurance implementation across all efforts.

C3-G05: Each model family used visible process text to mark a transition from evidence work toward drafting or another ensuing workflow phase.

Build Litigation Case Timeline (litigation)

Recurring behaviors

C3-G01: The workflow targeted the evident mixed-format source corpus through format-adapted, text-oriented access before artifact production.

C3-G02: When a visible source-access or execution complication occurred, the workflow changed method or resumed and continued toward artifact production.

C3-G03: The workflow moved from source review into artifact production at the requested destination and then issued a delivery-shaped assistant response.

C3-G04: Each model family has an observed effort session in which artifact creation was followed by a recorded output-confirmation step before delivery.

C3-G05: All registered task-directed execution remained in one parent stream without a visible dispatch-return dependency.

Analyze CPRA Compliance Gaps Against Current Privacy Program (privacy)

Recurring behavior

C3-G01: Before drafting, each model family prepared a broad working set from the heterogeneous local corpus and continued retrieval when binary incompatibility, truncation, or oversized output impeded direct access.

C3-G02: Across all three model families, observable task execution remained in the sole registered parent stream without a visible subagent dispatch or return.

C3-G03: In every model-family profile, visible corpus review preceded creation of the memo artifact.

C3-G04: Within every model family, Max exposed a more extended persisted-artifact lifecycle than High and Xhigh, through incremental assembly, an added post-write control phase, or both.

Privacy Addendum, Counterparty Review 04 (privacy)

Recurring behavior

C3-G01: Across all three model families, the observed workflows prepared and consulted multiple source categories and continued token-capped reads before drafting.

C3-G02: Across all three model families, major workflow phases were made externally visible during execution before terminal delivery.

C3-G03: Across all three model families, the first persisted redline preceded persistence of the companion memo.

C3-G04: Across all three model families, registered task work remained in a sole parent stream in the observed High and Xhigh sessions.

C3-G05: Each model family exhibited explicit terminal-stage mechanical or structural checking of both requested deliverables in at least one observed effort session.

The twelve tables above present the behaviors that recurred across all three models (Fable-5, Opus-5, and Sonnet-5) on each of the evaluated tasks. The details of these recurring behaviors, along with the behavioral differences recorded in the task, model, and session profiles, will be presented in subsequent analyses and published to GitHub.

Conclusion: The Future of Behavioral Analysis

This article introduces an analysis of legal AI behavior. Rather than analyzing legal AI results, behavioral analysis assesses how those results were produced. Characterizing model, effort, and session behaviors on a particular task may identify the best model and effort for that task. Additionally, skills can be analyzed for their influence on model behavior and tailored to improve it. Finally, combined with results, behavioral analysis may improve legal AI assessment and benchmarking. The recurring behaviors reported here are the baseline. Questions of optimization, skills, and integrated grading will be taken up in future articles.

This analysis is for educational and informational purposes only. It is not legal, financial, or other professional advice and does not create an attorney-client or any other professional relationship. The analysis is based solely on publicly available information. Any views expressed are solely those of the author in the author’s individual capacity.

[July 27, 2026] While preparing for retrieval augmented generation (RAG) of Harvey’s new M&A Due Diligence tasks, a document ingestion pipeline returned a regulatory license that contained no license:

That is the entire tennessee-wholesale-distributor-license-tn-wd-729048.docx document of the pharma-pipeline-acquisition task. A separate ingestion run on a different task surfaced the same thing:

That is the entire sec-closure-letter-workplace-conduct-disclosure-investigation.docx document of the gaming-strategic-acquisition task. In total, 40 such artifacts were identified across nine of Harvey’s M&A Due Diligence tasks. Those familiar with vibe coding (or even just AI chatting) will recognize these as AI refusals, generated when a model is asked to do something that its programming prohibits.

On July 17, Harvey extended its Legal Agent Benchmark (LAB) to include new diligence tasks meant to represent the virtual data rooms (VDRs) common to M&A due diligence.1 The stated goal of these new tasks is to create “realistic, high-scale agent environments to evaluate agents’ ability to perform end-to-end legal work and support open model training and agent research.” While traditional LAB tasks contain about a dozen documents and perhaps 100,000 tokens, the new diligence tasks run to thousands of documents and as much as 80 million tokens.

Further, Harvey describes these larger diligence tasks as RL environments that will “support open model training and agent research,” including by providing “[e]nvironments to train agents for diligence” that “allow [them] to train diligence agents that can take an entire dataroom, client communication and deal context and product [sic] a diligence memo” by “scal[ing] RL + post-training to handle this scale and complexity.”

Large data sets benefit from techniques like retrieval augmented generation (RAG) because those sets exceed the context windows of frontier models. RAG can resolve such issues by providing AI agents with the ability to search, filter, identify, and collect the most pertinent context from a larger set. Harvey’s announcement prompted “excite[ment] to test this larger dataset by giving frontier models a queryable RAG agent”:

However, the AI refusal artifacts indicate that LAB’s new diligence tasks may not be ready for RAG and that AI-generated content may have been published without verification.

First, AI refusals themselves are less remarkable than their presence in the preeminent benchmark for the accuracy of legal AI analysis. The publication of AI refusals without verification is consistent with the kind of autonomous AI generation (i.e., generation without an attorney in the loop) that is irking courts and leading to fines and suspensions. Indeed, an attorney who files 40 AI refusal artifacts in a brief or diligence report is likely to face sanctions. Even more concerning, such artifacts appear throughout tasks designed to train agents for M&A diligence through reinforcement learning (RL), for which accuracy and comprehensiveness seem paramount. If attorneys are sanctioned for relying on LAB-trained agents, will Harvey take responsibility?

Second, LAB’s realism should be tested rather than assumed. Apart from AI artifacts (e.g., refusals), the diligence sets are unusually uniform and exceptionally easy to parse. They are limited to five top-level file types (.docx, .xlsx, .pptx, .eml, .txt), with no PDFs, no comments (other than .pptx speaker notes), no hyperlinks, no MIME attachments, no images, no macros, and no scanned or OCRed documents. Consistent with generation by an AI model, LAB documents appear to have similar structure, voice, tone, and syntax. Attorneys may wish to ask whether that resembles the legal data sets (e.g., data rooms) that they’re familiar with before trusting LAB to evaluate models or train agents.

This article is part of a larger series describing the author’s experience “validat[ing] and improv[ing] the benchmark” pursuant to Harvey’s invitation in the announcement of LAB.2 A first article attempted to reproduce Harvey’s initial LAB results and instead found them highly dependent on methodological choices. A second article sought to isolate the scoring variance associated with grading judge model selection, found that this variance rendered all-pass scoring unreliable, and advocated for per-criterion scoring. A third article identified hallucinated scoring criteria while running skill-matched Claude-for-Legal assessments on LAB tasks. RAG ingestion of LAB’s diligence tasks identified the AI refusals described here.

In view of repeated shortcomings, LAB’s standing as the preeminent legal AI benchmark may be worth reconsidering. Identifying these AI refusals required nothing more than a parser or a basic grep. A benchmark that skipped such nominal diligence should not own the conversation about legal AI, where diligence, accuracy, and verification matter.

Methods: The Case for Retrieval Augmented Generation (RAG)

Large data sets are common in several legal practice areas, including M&A diligence and litigation discovery. RAG is a well-established technique for mitigating context-window limits and context bloat by providing AI agents (e.g., frontier AI models in an agent harness) with the ability to query external indexes and retrieve the most pertinent chunks of context for a specific analysis. This is not a comprehensive review of RAG, which encompasses numerous approaches: VectorRAG, GraphRAG, hybrid search, re-ranking, iterative agentic retrieval, and structure-aware chunking. The recently published “Retrieval-Augmented Generation for Natural Language Processing: A Survey” provides a good overview of RAG:

Shangyu Wu et al., “Retrieval-Augmented Generation for Natural Language Processing: A Survey,” Artificial Intelligence Review (June 1, 2026).

For LAB’s diligence tasks, the attempted implementation was a structure-aware vector RAG pipeline. First, source documents (.docx, .xlsx, .pptx, and .eml, with .txt pass through) were parsed into candidate structural units using a custom ingestion pipeline. Qwen3.5-9B then adjudicated those units by determining whether they should remain separate, be merged, or be subdivided into retrieval chunks. The RAG pipeline was interrupted here due to Qwen3.5-9B’s identification of AI refusal artifacts while adjudicating chunk boundaries.

Had the RAG pipeline not been interrupted, the resulting chunks would have been embedded into vector indexes using Qwen3-Embedding-8B, then stored in a SQLite-backed Chroma database. Retrieval would have run through a query tool built on Qwen3-Embedding-8B and Qwen3-Reranker-8B, allowing frontier models (e.g., Opus, Fable, GPT-5.6, Grok 4.5) operating in an agent harness (e.g., Claude Code, Codex, Grok Build) to agentically query the RAG system for the most pertinent context.3

RAG was abandoned once the AI refusal artifacts indicated LAB’s diligence tasks were not ready for analysis. As detailed below, at least 40 refusal artifacts sit in plain document body text, readily identifiable by any parser (including a basic grep) and visible to anyone who opened the files or extracted their contents. Apparently, no one did.

Results: At Least 40 AI Refusal Artifacts

Once the tennessee-wholesale-distributor-license-tn-wd-729048.docx of the pharma-pipeline-acquisition task and the sec-closure-letter-workplace-conduct-disclosure-investigation.docx of the gaming-strategic-acquisition task were identified, the full diligence folder of the LAB repository was scanned. That scan identified the following AI refusals.

The aerospace-vertical-integration task includes at least 9 AI refusal artifacts:

prairie-aerostructures-holdings-inc-delaware-certificate-of-good-stand.docx

prairie-aerostructures-uk-ltd-companies-house-certificate-of-good-stan.docx

prairie-aerostructures-holdings-inc-kansas-foreign-corporation-status-.docx

prairie-aerostructures-uk-ltd-companies-house-current-filing-status-ce.docx

780-max-fuselage-traveler-msn-7110.docx

780-max-recovery-traveler-msn-7190.docx

as9100d-recertification-audit-report-april-2022-prairie-opco-wichita-t.docx

qp-0704-odar-02-authorized-inspection-delegation-and-stamp-control-rev.docx

wichita-building-4-plug-door-verification-record-780-qa-2406-122.docx

The cybersecurity-tuck-in task includes at least 1 AI refusal:

ipr2024-00313-exhibit-1001-u-s-patent-no-11-502-847.docx

The enterprise-software-diversification task includes at least 1 AI refusal:

scdf-fire-certificate-cecil-court-18th-floor-novavault-singapore-offic.docx

The gaming-strategic-acquisition task includes at least 8 AI refusal artifacts:

sec-closure-letter-workplace-conduct-disclosure-investigation.docx

confetti-mobile-ltd-texas-foreign-registration-and-franchise-status-ce.docx

meridian-ireland-ltd-certificate-of-good-standing.docx

meridian-studios-llc-delaware-certificate-of-good-standing.docx

palisade-interactive-inc-delaware-certificate-of-good-standing.docx

palisade-japan-kk-certificate-of-registered-matters.docx

accc-informal-merger-review-closure-letter-northwind-palisade.docx

cma-provisional-findings-issues-letter-original-transaction.docx

The grocery-horizontal-merger task includes at least 4 AI refusal artifacts:

cottonwood-companies-inc-california-combined-corporation-franchise-tax.docx

cottonwood-specialty-rx-llc-delaware-certificate-of-good-standing-july.docx

cottonwood-companies-inc-delaware-certificate-of-good-standing-july-20.docx

dea-registration-certificate-and-renewal-file-cottonwood-pharmacy-az-3.docx

The media-recap task includes at least 3 AI refusal artifacts:

2100-riverside-parkway-atlanta-production-center-annual-fire-operation.docx

501-north-state-wmdn-tv-chicago-studios-occupancy-and-fire-safety-cert.docx

5555-melrose-studios-culver-city-annual-fire-clearance-2024.docx

The pharma-pipeline-acquisition task includes at least 5 AI refusal artifacts:

tennessee-wholesale-distributor-license-tn-wd-729048.docx

bothell-fei-3004512890-fda-establishment-registration-2022-confirmatio.docx

kb-6091-ind-154288-may-proceed-letter.docx

kb-6814-ind-160455-may-proceed-letter.docx

companies-house-certificate-of-good-standing-kestrel-biotherapeutics-u.docx

The restaurant-pe-buyout task includes at least 7 AI refusal artifacts:

norrell-stanton-bank-statement-and-reconciliation-bsi-operating-accoun.docx

norrell-stanton-bank-statement-boardwalk-gift-card-settlement-account-.docx

norrell-stanton-bank-statement-boardwalk-global-marketing-coop-ad-fund-010.docx

norrell-stanton-bank-statement-bsc-boardwalknext-fee-collections-accou-024.docx

norrell-stanton-bank-statement-fwh-franchise-royalty-collections-accou-022.docx

norrell-stanton-bank-statement-fwh-ifpc-rebate-receipts-account-octobe.docx

thames-mercantile-bank-statement-boardwalk-uk-operating-account-octobe.docx

The sports-media-consolidation task includes at least 2 AI refusal artifacts:

kentucky-sales-and-use-tax-return-and-eft-confirmation-q1-2022.docx

bloodline-live-chicago-2023-09-12-local-event-permit-packet.docx

This analysis is for educational and informational purposes only. It is not legal, financial, or other professional advice and does not create an attorney-client or any other professional relationship. The analysis is based solely on publicly available information. Any views expressed are solely those of the author in the author’s individual capacity.

[July 14, 2026] Last week, a benchmark of twelve Legal Agent Benchmark (LAB) tasks was published to evaluate twelve matched Claude for Legal skills.1 As part of this benchmark, 26 model/effort combinations were run against each LAB task to establish baseline performance of native models. These 26 combinations include Sonnet 5 (High-Max), Opus 4.7 (Medium-Max), Opus 4.8 (Medium-Max), Fable 5 (High-Max), GPT-5.5 (High-Extra High), GPT-5.6 Luna (Medium-Extra High), GPT-5.6 Terra (High-Max), GPT-5.6 Sol (High-Max), and Grok 4.5 (High).

Cumulatively, scoring rubrics for these twelve LAB tasks comprise 738 criteria.2 The vast majority of these criteria were passed by most or all of the evaluated AI models, whose average rubric coverage ranged from 80.5% to 93.9%. However, five criteria (0.7%) were failed by every single model/effort combination. As explained below, these criteria are wholly or partially unsupported by the documentary record of their corresponding task. Because these five fabricated criteria cannot be derived from task documents, they cannot be passed without hallucination and were correctly failed by all 26 model/effort combinations.

The fabricated criteria span three tasks: Build Litigation Case Timeline, Compare Asserted Patent Claims Against Accused Product, and Review Commercial Contracts Diligence Scenario 01. Specifically, criteria C-027 and C-028 of the Build Litigation Case Timeline task, criterion C-030 of the Compare Asserted Patent Claims Against Accused Product task, and criteria C-014 and C-015 of the Review Commercial Contracts Diligence Scenario 01 task.

While the fabrication of five of 738 rubric criteria (0.7%) may not appear consequential, the contamination of three of twelve benchmarked tasks means that approximately 25% of LAB tasks cannot be passed by any accurate (non-hallucinating) AI model. Under Harvey’s all-pass scoring methodology, for example, this imposes a hard 75% ceiling on all-pass performance that no model can exceed without hallucinating.3 Furthermore, the presence of fabricated scoring criteria demonstrates that the LAB itself and corresponding scores are at least partially hallucinated.

The principal purpose of this article is to encourage greater scrutiny of AI benchmarks and, hopefully, to provoke improvement of LAB, but it’s also worth asking whether LAB should be used to assess the accuracy of AI models for legal analysis. How can a benchmark with fabricated criteria assess the accuracy of AI models against hallucinations?

Rubric criteria C-027 and C-028 of the Build Litigation Case Timeline LAB Task are fabricated because the task’s fifteen-document record does not support a January 6, 2025 Chakrabarti deposition or a January 13, 2025 Buckley deposition

The Build Litigation Case Timeline task provides a fifteen-document record for the fictional Harborview v. Greenleaf breach of contract dispute.4

(Accessed on July 13, 2026).5

When evaluated on the Build Litigation Case Timeline task, AI models are tasked to “Review the attached documents and build a detailed litigation case timeline with strategic annotations for summary judgment preparation” and to output a litigation-case-timeline.docx deliverable based on the fifteen-document record.

(Accessed on July 13, 2026).

Each evaluated AI model’s generated litigation-case-timeline.docx output is graded against 66 criteria in the Build Litigation Case Timeline task.json file.6

Five of the Build Litigation Case Timeline task’s graded criteria require an evaluated AI model to identify a deposition by deponent and date. Three of these criteria (C-023, C-024, C-025) require identification of fact-witness depositions consistent with the Build Litigation Case Timeline task’s fifteen-document record: Derek Holcomb on October 18, 2024, Lisa Fong on November 5, 2024, and Randy Beckett on November 22, 2024.7 These criteria were passed by 22-26 of the evaluated AI models,8 demonstrating that the evaluated AI models are proficient at identifying and extracting depositions from a factual record.

Meanwhile, criteria C-027 and C-028 require the AI generated litigation-case-timeline.docx timelines to identify the depositions of expert witnesses Dr. Priya Chakrabarti on January 6, 2025 and Dr. Aaron Buckley on January 13, 2025.

(Accessed on July 13, 2026).

All 26 evaluated models correctly failed criteria C-027 and C-028 because the task’s fifteen-document factual record does not support a January 6, 2025 Chakrabarti deposition or a January 13, 2025 Buckley deposition. An accurate chronology cannot satisfy these criteria because nothing in the task’s fifteen-document record supports the depositions they require. Indeed, the only January 2025 date present in the task’s record is the scheduling order’s January 15, 2025, the deadline for completing all fact and expert depositions.

For confirmation, the Build Litigation Case Timeline LAB task is currently available at the following links.

• The task is located here: https://github.com/harveyai/harvey-labs/tree/main/tasks/litigation-dispute-resolution/build-litigation-case-timeline;

• Documents comprising the file record are located here: https://github.com/harveyai/harvey-labs/tree/main/tasks/litigation-dispute-resolution/build-litigation-case-timeline/documents;

• The scoring rubric (task.json) is located here: https://github.com/harveyai/harvey-labs/blob/main/tasks/litigation-dispute-resolution/build-litigation-case-timeline/task.json.

Consistent with this article’s purposes of encouraging benchmark scrutiny and provoking LAB’s improvement, it is anticipated that the identified fabrications will be corrected. However, to ensure that fabrications are not merely papered over without addressing deeper issues, Harvey’s LAB has also been forked into a July 13, 2026 snapshot that provides the Build Litigation Case Timeline task, documents, and rubric at the following links.9

All 26 evaluated AI models correctly failed the C-027 and C-028 criteria, which are unsupported fabrications that were likely hallucinated during the LAB’s development.

Rubric criterion C-030 of the Compare Asserted Patent Claims Against Accused Product LAB Task is fabricated because the task’s six-document record does not support a Lindström EP 3,102,887 A1 prior art reference

The Compare Asserted Patent Claims Against Accused Product task provides a six-document record for a fictional patent infringement dispute in which Luminos asserts U.S. Patent No. 10,847,233, covering multi-path signal phase correction, against Meridian’s VectorStream 9000. The record comprises the ’233 patent, prosecution history excerpts, Luminos’s infringement contentions, the VectorStream 9000’s engineering specification and product brief, and an internal Meridian email.

(Accessed on July 13, 2026).10

When evaluated on the Compare Asserted Patent Claims Against Accused Product task, AI models are tasked to “Compare the asserted claims against the VectorStream 9000’s actual implementation, correct any mischaracterizations in the infringement contentions, and prepare a non-infringement analysis with litigation risk assessment” and to output a claim-comparison-and-noninfringement-analysis.docx deliverable based on the six-document record.

(Accessed on July 13, 2026).

Each evaluated AI model’s generated claim-comparison-and-noninfringement-analysis.docx output is then graded against 54 criteria in the Compare Asserted Patent Claims Against Accused Product task.json.11

Criterion C-030 of the Compare Asserted Patent Claims Against Accused Product task’s rubric requires the AI generated claim-comparison-and-noninfringement-analysis.docx outputs to reference a Lindström European patent application as prior art invalidating Claim 12. Specifically, the rubric provides: “PASS if the output references the Lindström European patent application (EP 3,102,887 A1) as relevant prior art for invalidating Claim 12, noting that Lindström describes an RLS-based multipath phase correction system that would fall within Claim 12’s broader claim scope. FAIL if Lindström is not mentioned in the context of Claim 12 invalidity.”

(Accessed on July 13, 2026).

All 26 evaluated models correctly failed criterion C-030 because nothing in the task’s six-document record supports the existence of Lindström prior art. The task’s six-document record identifies precisely four prior art references: Kobayashi (U.S. Patent No. 9,312,445), El-Amin (U.S. Pub. No. 2016/0087744 A1), Chen (U.S. Patent No. 8,750,391), and Petrov (U.S. Patent No. 9,048,922). There is no reference named Lindström, no application numbered EP 3,102,887, and no European patent document of any kind. No accurate noninfringement analysis could satisfy criterion C-030 without hallucinating an unsupported prior art reference.

For confirmation, the Compare Asserted Patent Claims Against Accused Product LAB task is currently available at the following links.12

• The task is located here: https://github.com/harveyai/harvey-labs/tree/main/tasks/intellectual-property/compare-asserted-patent-claims-against-accused-product;

• Documents comprising the file record are located here: https://github.com/harveyai/harvey-labs/tree/main/tasks/intellectual-property/compare-asserted-patent-claims-against-accused-product/documents ;

• The scoring rubric (task.json) is located here: https://github.com/harveyai/harvey-labs/blob/main/tasks/intellectual-property/compare-asserted-patent-claims-against-accused-product/task.json.

All 26 evaluated AI models correctly failed criterion C-030, which is an unsupported fabrication that was likely hallucinated during the LAB’s development.

Rubric criteria C-014 and C-015 of the Review Commercial Contracts Diligence (Scenario 01) task are partially fabricated because the required 54% customer-dependency figure and 9-12 month replacement timeline are neither stated in nor derivable from the task's eleven-document record

Scenario 01 of the Review Commercial Contracts Diligence task provides an eleven-document record for a fictional acquisition in which Pinnacle conducts commercial-contracts diligence on SaaS target CloudMesh Solutions. The record comprises five CloudMesh customer agreements, CloudMesh’s IaaS agreement with Stratos Cloud, the Lumen Analytics technology partnership agreement and its accompanying escrow agreement, a customer contract schedule workbook, a NovaCast renewal email, and Pinnacle’s diligence request list.

(Accessed on July 13, 2026).13

When evaluated on Scenario 01 of the Review Commercial Contracts Diligence task, AI models are tasked to “Review the attached CloudMesh commercial contracts against the contract schedule and diligence request list; produce a full diligence memo” and to output a commercial-contracts-diligence-memo.docx deliverable based on the eleven-document record.

(Accessed on July 13, 2026).

Each generated commercial-contracts-diligence-memo.docx deliverable is graded against 59 rubric criteria in the Review Commercial Contracts Diligence task.json file.14

Unlike the fabricated depositions and prior art references discussed above, CloudMesh’s dependency on Lumen is a real issue presented by Scenario 01 of the Review Commercial Contracts Diligence task that a competent diligence memo would flag. However, criteria C-014 and C-015 require specific quantifications of this dependency that are not supported by the eleven-document task record. Criterion C-014 requires the memo to note “that approximately 54% of CloudMesh’s customer base (approximately 116 of 214 customers) actively uses the Lumen-powered MeshInsights feature,” and fails the memo “if the scope of customer dependency on Lumen/MeshInsights is not quantified.” Meanwhile, criterion C-015 requires the memo to note “that replacing Lumen’s technology would take approximately 9-12 months,” and fails the memo “if the replacement timeline is not stated.”

(Accessed on July 13, 2026).

Neither the 54% (~116 of 214 customers) nor the 9-12 month timeline is stated in the task’s eleven-document record or derivable from it.

With respect to C-014, the record supports only the criterion’s 214-customer denominator. A dependency quantification is derivable only in revenue terms: the Lumen agreement’s fee and revenue-share provisions imply that roughly 55% of CloudMesh’s FY2024 subscription revenue derives from relevant customers. But no document states how many customers use the feature, and equating 55% of revenue with 54% of customers (~116 of 214) is exactly the unsupported leap an accurate AI model should not make. All 26 models failed C-014, which cannot be passed without asserting a customer count the record does not contain.

With respect to C-015, no document estimates how long replacing Lumen’s technology would take. The record supports only contractual runway: 60 days’ notice plus a 180-day wind-down guarantees ~8 months from Lumen’s notice, and at most ~11 months from closing. The rubric’s “approximately 9-12 months” matches neither endpoint and measures time required instead of time available. A memo accurately reporting the record’s timeline correctly fails C-015 as written.

For confirmation, the Review Commercial Contracts Diligence Scenario 01 LAB task is currently available at the following links.15

• The task is located here: https://github.com/harveyai/harvey-labs/tree/main/tasks/corporate-ma/review-commercial-contracts-diligence/scenario-01;

• Documents comprising the file record are located here: https://github.com/harveyai/harvey-labs/tree/main/tasks/corporate-ma/review-commercial-contracts-diligence/scenario-01/documents;

• The scoring rubric (task.json) is located here: https://github.com/harveyai/harvey-labs/blob/main/tasks/corporate-ma/review-commercial-contracts-diligence/scenario-01/task.json.

All 26 evaluated models correctly failed criteria C-014 and C-015. Accurate diligence memos could not satisfy these criteria because nothing in the task’s eleven-document record supports the specific 54% of customers or 9-12 month timeline required to pass them.

This analysis is for educational and informational purposes only. It is not legal, financial, or other professional advice and does not create an attorney-client or any other professional relationship. The analysis is based solely on publicly available information. Any views expressed are solely those of the author in the author’s individual capacity.

5

GitHub history of Build Litigation Timeline task documents. The deposition-summary-holcomb.docx document was edited on May 22, 2026. Otherwise, the task record has not been modified since the LAB’s original May 6, 2026 commit.

(Accessed on July 13, 2026).

6

GitHub history of Build Litigation Timeline task.json file shows that it has not been modified since the LAB’s original May 6, 2026 commit.

(Accessed on July 13, 2026).

7

Criteria C-023, C-024, and C-025 of the Build Litigation Case Timeline task:

(Accessed on July 13, 2026).

10

GitHub history of Compare Asserted Patent Claims Against Accused Product task documents. The vectorstream-9000-engineering-spec document was edited on May 22, 2026. Otherwise, the task’s record has not been modified since the LAB’s original May 6, 2026 commit.

(Accessed on July 13, 2026).

11

GitHub history of Compare Asserted Patent Claims Against Accused Product task.json file shows that it has not been modified since the LAB’s original May 6, 2026 commit.

(Accessed on July 13, 2026).

13

The Review Commercial Contracts Diligence (Scenario 01) task record has not been modified since the LAB’s original May 6, 2026 commit.

(Accessed on July 13, 2026).

14

GitHub history of Review Commercial Contracts Diligence (Scenario 01) task.json file shows that it has not been modified since the LAB’s original May 6, 2026 commit.

(Accessed on July 13, 2026).

Claude for Legal is Anthropic’s open-source toolkit of legal skills that plug into a harness (Cowork or Claude Code) running a native Claude model to provide structured protocols that guide the model through specified legal analyses.2 Claude for Legal works by generating one or more profiles (company profile, practice profile, matter profile, etc.) that provide organizational, practice, and/or matter-specific context for legal analyses invoked by any particular skill. A previous Benchmarking Claude for Legal article tested the chronology skill of Claude for Legal’s litigation plugin with eleven model/effort combinations against a single build litigation case timeline task from Harvey’s LAB. That article found the chronology scores scaled with model capability and effort from 40% coverage of the task’s rubric with Haiku 4.5 Low to 86% coverage with Opus 4.7 Medium, then plateaued at higher efforts. No model/effort combination achieved 100% coverage and three rubric criteria (C-027, C-028, and C-045) were not captured by any of the eleven combinations. This first benchmark left open whether breadth of coverage is a good proxy for legal-analysis quality or whether the chronology results could generalize to other Claude for Legal skills.

The Legal Agent Benchmark is an open-source repository of synthetic legal tasks published by Harvey.3 Tasks are associated with various legal practice areas (e.g., litigation dispute resolution) and defined by a prompt, a scoring rubric, and documents comprising a factual record. Each task is designed to simulate a legal analysis performed by attorneys within the associated legal practice area. Harvey published the first iteration of LAB on May 6, 2026, published initial LAB results on May 26, 2026, then expanded LAB to include in-house contracting tasks on June 12, 2026. A previous Investigating the Legal Agent Benchmark (LAB): Part 1 article attempted to reproduce Harvey’s initial LAB results and instead found that they depended on undisclosed methodological choices, including task selection, grader model selection, and scoring methodology. A subsequent Investigating the LAB: Part 2, Judge Variation and the Case for Per-Criterion Scoring article determined that grader selection alone could shift an evaluated model’s per-criterion score by up to 11 percentage points, and that under all-pass grading roughly 40% of competitive evaluations fell within the resulting margin of judge disagreement. That article advocated for per-criterion scoring, which is adopted by this updated Claude-for-Legal benchmark.

A Twelve-Skill, Task-Matched Claude-for-Legal Benchmark

Claude for Legal skills are divided into twelve practice areas, nine of which are fields of law with substantive legal skills. When mapped to the closest corresponding LAB tasks, six of those nine yielded strong skill-to-task matches: AI governance, commercial, corporate, intellectual property, litigation, and privacy.4 The benchmark comprises the two strongest skill-task matches from each of these six practice areas.5^,6

Each skill-task pairing reflects the Claude for Legal skill best exercised by a corresponding LAB task. The benchmark runs each skill on its paired task and scores the resulting work product against that task’s published rubric, criterion by criterion. It has been run against the four current-generation Claude models that can execute Claude for Legal — Sonnet 5, Opus 4.7, Opus 4.8, and Fable 5 — each at high, extra high, and max effort.

Conclusion

This benchmark is defined independently of the models it evaluates and of how their outputs are graded. Current-generation Anthropic models Sonnet 5, Opus 4.7, Opus 4.8, and Fable 5 have been evaluated, and results will be presented for each skill-task pairing. Recently released Grok 4.5, Meta Spark 1.1, and OpenAI's GPT-5.6 Sol, Terra, and Luna are being evaluated as comparative baselines; none is an Anthropic model, so none can run Claude for Legal skills.

This analysis is for educational and informational purposes only. It is not legal, financial, or other professional advice and does not create an attorney-client or any other professional relationship. The analysis is based solely on publicly available information. Any views expressed are solely those of the author in the author’s individual capacity.

To isolate the effects of judging, this article holds constant the previous article’s tasks, evaluated models, and generated outputs. Those are six LAB tasks selected to represent the three practice areas defined by the initial LAB results, with each task run by six evaluated AI models to generate 36 outputs. Those 36 outputs were scored against the appropriate LAB rubric by six judge models: GPT-5.4, GPT-5.5, Haiku 4.5, Sonnet 4.6, Opus 4.7, and Opus 4.8. Scoring was performed at extra high (xhigh) effort by Opus and GPT judges and at high effort by Sonnet and Haiku judges (for which extra high effort is not an option). Because the same outputs, rubrics, and evaluated models were scored by each judge, differences in scoring isolate the act of judging itself.

Based on the following results, LAB scores do not represent model performance, they represent a judge’s perspective of model performance according to a rubric. Judge variation affects model scoring. Under all-pass scoring, this variation appears to overshadow about 40% of the all-pass assessments of evaluated models on LAB tasks. That renders all-pass scoring unreliable because it measures the judge nearly as much as the evaluated model. This article therefore abandons efforts to reproduce Harvey’s initial LAB results because, in light of observed judge variation, they cannot be reproduced. Instead, this article concludes that per-criterion scoring is the superior LAB evaluation methodology.

Results: Judge Behavior

The six task rubrics comprised 375 criteria (about 63 per task). Each judge made the same 2,250 score determinations by applying these 375 criteria to the six evaluated models. Because each judge scored the same generated outputs against the same criteria, differences in 2,250 determinations are attributable to judging.

Overall pass rates clustered in a narrow band

Across 2,250 score determinations, overall pass rates ranged from 80.8% for GPT-5.4 to 86.3% for Opus 4.8.

This distribution reveals permissiveness gradients across iterative generations of the same model, and for Anthropic by model tier. Within the GPT model family, overall pass rate increased 0.7% from 5.4 (80.8%) to 5.5 (81.5%). Within Opus, they increased 0.9% from 4.7 (85.4%) to 4.8 (86.3%). Meanwhile, overall pass rates increased 5% across Anthropic model tiers from the throughput-focused Haiku 4.5 (81.3%) to the flagship Opus 4.8 (86.3%). This may indicate that more advanced reasoning results in higher pass rates, but does not necessarily mean that more permissive judges are more accurate.

Per-criterion rankings were consistent, no significant biases observed

Judges did not demonstrate significant bias by evaluated model and per-criterion scores were stable.

None of the models demonstrated significant bias or self-judging preference. Generations of the Opus and GPT model families demonstrate consistent scoring with earlier generations judging 0-1% lower than newer generations. The only notable variation is Opus’s relatively higher GPT-5.5 pass rate, but this doesn’t appear to indicate concerning bias. Haiku is remarkably consistent with GPT models despite the previously described variation in individual score determinations. Consistent with broad cross-model alignment, Sonnet split the difference.

Agreement was high within families, lower across them

Generations of the Opus and GPT model families showed high scoring alignment. For example, Opus 4.7 and 4.8 agreed on 98.7% of scoring determinations, while GPT-5.4 and 5.5 agreed 96.5% of the time. Sonnet achieved the broadest cross-model alignment, averaging 96.05% consensus with Opus models, 94.25% consensus with GPT models, and 94% consensus with Haiku. Meanwhile, cross-model agreement between Opus, GPT, and Haiku falls to 92% (92.2-92.7% when Opus/GPT generations are averaged).

Although 92% alignment might appear good, the converse 7.4-7.8% disagreement is more consequential where overall pass rates are compressed into a relatively narrow 5.5% band (80.8-86.3%). In this context, alignments of less than 94.5% indicate that variance is not isolated to marginal pass determinations. Rather, there is cross-model discord in which higher pass rate judges fail criteria that stricter judges pass and vice versa.

Where judges split, the splits were structured

Across 2,250 score determinations, the judges universally agreed that an evaluated model passed a task criterion for 1,694 determinations (75.3%), universally agreed on failure for 260 determinations (11.6%), and disagreed in 296 split determinations (13.2%).

Split determinations generally followed the judges’ overall permissiveness and alignment patterns. They were structured, not random.

Opus judges formed a permissive bloc, in which one or both Opus judges voted pass in 85.1% of splits. GPT judges formed a strict bloc, in which one or both GPT judges voted fail in 68.9% of splits. Sonnet rarely dissented. Meanwhile, Haiku-4.5 was the most discordant judge, appearing as a lone dissenter in 63 splits.

Discussion: Judge Variation Makes Per-Criterion Scoring More Reliable Than All-pass Scoring

Judge behavior has implications for the most appropriate scoring methodology. For example, Opus, GPT, and Haiku disagreed with each other on an average of 4.72 criteria per evaluated model per task. Sonnet disagreed with GPT on 3.58 criteria per task and with Opus on 2.47. Even GPT-5.4 and GPT-5.5 disagreed on 2.19 criteria per task. That variation affects model scores. In this analysis, for example, GPT-5.5 was graded 11% higher by Opus 4.8 (91.5%) than by Haiku 4.5 (80.5%).3 Under per-criterion scoring, that variation remains visible because it can be traced to specific disagreements over specific criteria. Here, overall rankings remained stable across all six judges despite shifts of up to 11% in per-criterion scores.

All-pass scoring does the opposite. It magnifies judge variation by turning criterion-level disagreements into complete task-level failures. Had all-pass scoring been used for this analysis, model ranks would have been unstable and scores unreliable: some judges would have passed no models on any task, while others would have passed different models on different tasks.

This instability becomes more pronounced when near-passes are considered. Excluding Gemini 3.1 Pro, the analysis produced 180 judge–model–task evaluations under the all-pass rule: 5 evaluated models × 6 LAB tasks × 6 judges. Of those, 70 failed by fewer than 4.72 criteria, the observed average criterion-level disagreement among Opus, GPT, and Haiku. Thus, for competitive models, 38.9% of evaluations scored under the all-pass rule fell within the empirical margin of judge variation. This article uses per-criterion scoring because model-task evaluations should not pass or fail based on judge variation.

According to Harvey’s release of LAB, the case for all-pass scoring is grounded in how legal work is used.

The evaluation methodology for LAB puts it even more bluntly:

This article doesn’t dispute Harvey’s reasoning that incomplete legal analysis presents material risks or the LAB’s objective of measuring complete legal analysis. Rather, it presents empirical results showing that the all-pass score does not accomplish that objective. All-pass scoring works against that objective because it lets methodological variance obscure the assessment of the evaluated model. For example, in this analysis, the all-pass scores of different judges deviated on multiple vectors: not only in how many tasks were passed by each evaluated model, but which models passed which tasks. Ultimately, the all-pass score appears to measure judging methodology as much as it measures the evaluated model's performance.

Conclusion: Judge Variation Favors Per-Criterion Scoring

Per-criterion scoring is superior due to the variation inherent in judge selection and other methodological choices. All-pass scoring does not reliably measure the quality or completeness of legal analysis. It compresses each task into a single pass-or-fail verdict that magnifies and obscures methodological variation. This manifests as unstable and unreliable results that could not be reproduced. Meanwhile, per-criterion scoring makes methodological variance visible and interrogable with higher precision and more reliable results. All-pass scoring should be abandoned in favor of per-criterion scoring for LAB analysis.

This analysis is for educational and informational purposes only. It is not legal, financial, or other professional advice and does not create an attorney-client or any other professional relationship. The analysis is based solely on publicly available information. Any views expressed are solely those of the author in the author’s individual capacity.

A previous article used the “build-litigation-case-timeline” task from the “litigation-dispute-resolution” practice area to test the chronology skill of the open-source (Apache 2.0 licensed) Claude-for-Legal toolkit across various Anthropic models and effort levels. This article turns to LAB itself, attempting to reproduce several aspects of the initial results Harvey published last week. As it developed, this article’s study diverged from the initial LAB results — and that divergence, rather than a clean reproduction, became its most useful result. Follow-up articles will examine how task-specific and practice-area-specific results were influenced by the structure and methodology of LAB.

Methods: Building a Proxy to Reproduce the Initial LAB Results

Harvey’s release anticipated that lawyers and researchers in the legal-AI community would join the conversation around LAB by running benchmark analyses, testing tasks, and reporting results under comparable conditions. The initial LAB results provide a natural baseline for that kind of comparison. This article attempts to participate in the LAB conversation by performing a study designed to reproduce aspects of the initial LAB results. Where it reproduces those results, it provides a baseline for more detailed analysis of them; where it diverges, it provides a means to investigate how the structure and methodology of LAB affect the scoring of evaluated models.

Ten AI models are evaluated by the initial LAB results on a set of legal tasks grouped into three practice areas: regulated and emerging company work; corporate transactions and funds; and privacy, tax, and private client work. Four models are scored within those practice areas: Opus 4.7, Sonnet 4.6, GPT-5.5, and Gemini 3.5 Flash.

[Image copied from Initial Results on Legal Agent Benchmark]

Because the initial LAB results do not disclose which specific tasks populate each practice-area grouping, this article cannot run the same evaluation exactly. Instead, it constructs a proxy by running six evaluated AI models against six representative LAB tasks. The evaluated models included Opus 4.7, Sonnet 4.6, GPT-5.5, and Gemini 3.5 Flash, along with Gemini 3.1 Pro and the newly released Opus 4.8. Two tasks were chosen to represent each of the three groupings for a total of six tasks.

For corporate transactions and funds, the representative tasks are “draft diligence summary memo” and “review commercial contracts diligence.” For regulated and emerging company work, the representative tasks are “triage vendor ai contracts for compliance with eu ai liability directive” and “analyze eu ai act high.” For privacy, tax, and private client work, the representative tasks are “analyze cpra compliance gaps against current privacy program” and “analyze counterparty markup of data processing agreement.”

The evaluated models’ generated outputs were initially judged by five models. Sonnet 4.6 and GPT-5.4 did not provide reliable scoring,3 leaving three reportable judges: GPT-5.5, Opus 4.7, and Opus 4.8. As discussed below, those scores support comparison across tasks, practice-area groupings, and judge models.

Results: How the Proxy Analysis Scored AI Models

When scores from all three judges (Opus 4.7, Opus 4.8, and GPT-5.5) are averaged across all six tasks, Opus 4.8 and Opus 4.7 achieve the best per-criterion coverage of approximately 95%. In practical terms, the Opus models rarely missed more than a couple of rubric criteria on any task.

Sonnet 4.6 and GPT-5.5 formed a second tier, trailing the Opus models by roughly four to seven percent. Both met approximately nine of every ten rubric criteria, though this varied by task. Sonnet 4.6, for example, achieved the highest average “draft diligence summary memo” score (75/76) while placing second to last on the “triage vendor ai contracts for compliance with eu ai liability directive” task.

Gemini 3.5 Flash formed a third tier at roughly 80% per-criterion coverage, with especially task-dependent performance.4 It tied for the second-highest average “analyze counterparty markup of data processing agreement” score (57/59) but otherwise ranked fourth or fifth on the evaluated tasks.

Gemini 3.1 Pro ranked last on every task, at approximately 54% average per-criterion coverage and 24.6% on its weakest task. Based on this analysis, Gemini 3.1 Pro does not appear competitive with the other evaluated models on the selected LAB tasks.

These per-criterion results differ from Harvey’s reported baseline for LAB in ways the next section examines. Task-specific and practice-area-specific results will follow in subsequent articles; in the meantime, model outputs and judged rubric scores are progressively being uploaded to GitHub.

Discussion: Asking Why the Proxy’s Results Diverged from the Initial LAB Results

Despite attempting to reproduce aspects of the initial LAB results, the proxy produced different ranking and score patterns. That makes it less useful as a direct reproduction of the reported LAB baseline, but more useful as a way to inspect how LAB results may be shaped by task selection, scoring methodology, judge model selection, and aggregation under the all-pass standard.

The most visible example is the model ranking. Excluding models not reported by practice area, the initial LAB results and the proxy ranked the evaluated models in the same order for only one practice-area grouping: corporate transactions and funds, where both ranked them (1) Opus 4.7, (2) Sonnet 4.6, (3) Gemini 3.5 Flash, and (4) GPT-5.5.

The other two groupings diverged. The initial LAB results show different models outperforming one another across the three practice areas, whereas the proxy’s rankings were more consistent across them. Individual tasks still showed model-specific variation, but that variation tended to narrow with aggregation, even with only two tasks per grouping.

The most immediate explanation for the proxy’s deviation is that models were ranked by all-pass scoring in the initial LAB results, whereas the proxy ranked them by per-criterion coverage. These metrics answer different questions, so part of the divergence may be definitional rather than substantive. The two scores should correlate, since higher per-criterion coverage generally produces higher all-pass scores. However, all-pass scoring compresses per-criterion scoring into binary pass/fail results, which can obscure analytically important distinctions. The Opus models illustrate this point by achieving roughly 95% per-criterion coverage while failing to achieve all-pass on most tasks.

Beyond different scoring methodologies, the proxy’s divergence from the initial LAB results may be explained by task selection, judge model selection, and run variance:

• Task selection may matter substantially. The initial LAB results report practice-area scores in 2.5-percentage-point increments, suggesting each grouping may include 40 tasks. If so, those groupings are necessarily more selective than the tasks in practice-area folders. The corporate-ma folder, for example, includes 136 tasks; even if the corporate transactions and funds grouping drew only from that folder, fewer than one in three of its tasks would be included. Tasks selected to represent the corporate transactions practice area may not have been part of the grouping reflected in the initial LAB results, and the same is true for the tasks chosen to represent the other two groupings.

• Judge model selection can materially affect results. Across all six evaluated models, the three reportable judges marked criteria as passing at different rates: Opus 4.8 at 86.3%, Opus 4.7 at 85.4%, and GPT-5.5 at 81.5%. These are judge pass rates and should not be confused with the per-criterion coverage scores reported above. All three judges agreed on 92.4% of criteria, and the two Opus judges agreed on 98.7%. That leaves 7.6% of binary pass/fail judgments on which at least one judge disagreed, most often GPT-5.5. Furthermore, abandoned runs of Sonnet 4.6 and GPT-5.4 suggest that unreliable judge models can have outsized effects. These partially completed runs produced lower interim pass rates of roughly 67.5% for Sonnet 4.6 and roughly 61.3% for GPT-5.4, which would have weighted scores toward failure if included.

• Run variance may matter at the margins. This study ran each model on each task once. Frontier models are not deterministic, so repeated runs may produce different outputs and different scores. Inter-run variance could contribute to task-level variation, but it is unlikely to explain all of the recurring divergence from the initial LAB results across multiple tasks and practice-area groupings.

No single one of these variables explains the divergence. Together, they show that assessment of evaluated models is highly dependent on methodological choices: which tasks are selected, how they are grouped, how they are scored, and which model does the judging. Because these choices appear to materially affect the assessment of evaluated models, they warrant further investigation.

Conclusion: A Benchmark, Not a Leaderboard

Harvey invited lawyers to help test LAB. This study set out to accept that invitation by reproducing aspects of the initial LAB results through approximation of as much of Harvey's methodology as could be derived from those results. The initial LAB results were not reproduced. Instead, the proxy’s divergence shows why LAB should be studied as a benchmark, not merely consulted as a leaderboard. Because task selection, and judge model selection can change which model ranks highest and whether an output passes at all, the methodology behind a LAB score deserves as much scrutiny as the score itself. Follow-up articles will undertake that investigation in task-specific and practice-area-specific context.

This analysis is for educational and informational purposes only. It is not legal, financial, or other professional advice and does not create an attorney-client or any other professional relationship. The analysis is based solely on publicly available information. Any views expressed are solely those of the author in the author’s individual capacity.

Yet published evidence of what the toolkit produces when implemented remains limited. What kind of analysis Claude-for-Legal actually produces is an empirical question, one that this article seeks to answer.

This article benchmarks Claude-for-Legal’s chronology skill across eleven distinct combinations of Anthropic models (Haiku, Sonnet, Opus) at different effort levels (Low through Max). The chronology skill, one of several skills in the litigation plugin, extracts events from a case file and builds a timeline. The skill is benchmarked against Harvey Labs’ build-litigation-case-timeline task, which provides a synthetic 15-document breach of contract case file (Harborview v. Greenleaf).

Background: What Claude-for-Legal Actually Is

Claude-for-Legal is not a standalone application. It is a directory of skills and agents that loads into a harness (Claude Code, Claude Cowork) running a Claude model. The model, harness, and Claude-for-Legal layers each do a different job. The model (Haiku, Sonnet, or Opus) supplies general reasoning capability and may be configured to run at one of several levels of effort. Claude Code and Claude Cowork are harnesses that provide the model with an environment to load files, write files, run shell commands, and hold sessions together across turns. Claude-for-Legal sits on top of the harness as a toolkit of twelve practice-area plugins that each provide a set of skills for specific legal tasks.2 The plugins also include MCP connectors that link to platforms like legal-research databases and document management systems, providing skills with context the harness alone cannot reach.

This isn’t a novel architecture, AI skills typically perform abstract tasks by running structured protocols inside a harness. What differentiates Claude-for-Legal are profiles that provide company (or firm), practice, and case specific context when skills are performed. Generally, these profiles are generated from templates through a cold-start-interview, then nested within Claude-for-Legal’s config directory. They include a company profile, practice profiles, portfolio registers, matter profiles, and event logs.

The company profile provides organization-level context and is read by every plugin, while additional nested profiles provide practice-area specific and matter specific context. When invoked, skills draw context from pertinent parts of the profile stack to generate results that are tailored to an attorney’s company, practice, and the relevant matter.

Yet plugin architecture cannot answer what kind of analysis Claude-for-Legal provides. That question requires testing a realistic legal task.

Methods: Testing Claude-for-Legal’s Chronology Skill

One of Claude-for-Legal’s most testable skills is “chronology,” a litigation skill that extracts events from a case file, identifies the user’s theory of the case and tags each event’s significance to this theory, flags potential information gaps (i.e., missing documents), and constructs a timeline.3 The build-litigation-case-timeline benchmarking task, provided by Harvey Labs4 under an open-source (MIT) license, presents a good test case for the chronology skill because it provides a synthetic case file and scoring rubric for chronological extraction. Indeed, this task asks AI models to develop a “Chronological Event Summary” from the 15-document case file of a fictional Harborview v. Greenleaf breach of contract dispute. Equally important, the Harvey Labs task comes with a 66-criterion rubric for scoring the extracted chronologies.5

Consistent with the “Breach of Contract and Fraud Defense” focus of Harvey Labs task, the Claude-for-Legal chronology skill was tested from the perspective of a defense-focused firm. Here, Greenleaf’s counsel. This perspective was reflected during Claude-for-Legal’s cold-start-interview to generate appropriate firm and practice profiles to simulate the chronology skill on Harvey Labs’ build-litigation-case-timeline task.6

Following the cold-start-interview, the chronology skill was run against the Harborview v. Greenleaf case file under eleven model and effort combinations: Haiku Low, Medium, and High; Sonnet Low, Medium, and High; Opus Low, Medium, High, Xhigh, and Max.7 Each run produced a summary and full chronology report. Example summaries from Haiku High, Sonnet Low, Sonnet High, and Opus Medium are shown below.

The eleven chronology simulations took 2–16 minutes to complete, consumed 85,000–249,000 tokens, and extracted 23–73 events into resulting chronology reports.

[Select figure for interactive chart]

These resulting chronologies were scored against the 66 criteria of Harvey Labs’ rubric. Scoring was performed independently by two LLM judges: Opus 4.7 and GPT-5.5.8 The two-judge setup was chosen to reduce single-model scoring bias and to surface ambiguous criteria. Judge agreement was tracked per criterion and used as a reliability signal in the analysis.

Results: How Well Each Model Captured the Rubric Criteria

Judged scores across the eleven simulations ranged from 26.5 to 57 (40.2% to 86.4%) coverage of the 66 criteria, with the runs clustering into two distinct sets. The lower cluster (Haiku at every effort level, plus Sonnet and Opus at Low effort) had shorter run times, used less context, and covered 40.2% to 62.1% of the rubric criteria. The higher cluster (Sonnet at Medium effort and above, plus Opus at Medium effort and above) had longer run times, used more context, and covered 76.5% to 86.4% of the rubric criteria. Top coverage was tied between Opus at Medium effort and Opus at Max effort, both at 57 criteria (86.4% coverage).

[Interactive chart]

Coverage scores measure whether each model simulation satisfied the rubric's criteria. Whether coverage breadth approximates analysis quality is an open question.

The overall coverage of each model and effort combination can be broken down across the 66 rubric criteria. The eleven runs demonstrate distinct patterns across pre-litigation events, litigation timeline events, and issue identification, as illustrated in the coverage heat map below. Each row represents one of the 66 rubric criteria; each column, one model/effort run; cells are filled where the run satisfied the criterion, hashed where the two judges disagreed.

[Interactive chart]

Patterns emerge by model and by criterion type. Opus models (except at Low effort) cover nearly all pre-litigation and litigation events (C-001 through C-028), show sporadic issue spotting (C-032 through C-053), and produce serviceable claim and damages assessments (C-054 through C-058 and C-060 through C-062). Sonnet models nearly match Opus on pre-litigation and litigation events but show slightly sparser issue identification (see, e.g., C-044, C-046, C-048, C-051). Haiku models are significantly sparser: their coverage concentrates on pre-litigation events (C-001 through C-013), thins rapidly for litigation timeline events (C-014 through C-028), and is minimal for issue identification or claim and damages assessment. Across all models, coverage is better for event identification than for issue identification and claim and damages assessment. Consistent with the more subjective nature of issue identification and claim and damages assessment, Opus 4.7 and GPT-5.5 more commonly disagreed on whether these criteria were satisfied.

Conclusion: What Benchmarks Can and Cannot Say

Claude-for-Legal’s chronology skill, run against Harvey Labs’s build-litigation-case-timeline task, produces chronologies that vary substantially in rubric coverage across model and effort combinations. Coverage tended to scale with model capability and, within more capable models (Sonnet, Opus), plateaued around Medium effort. These coverage results report what each run did against a specific rubric on a specific synthetic case. They do not report whether the resulting chronologies are good legal work product for any particular use.

Those evaluating Claude-for-Legal for legal analysis work will want to ask several questions this benchmark does not answer:

• Do the coverage patterns observed here translate to other legal analysis tasks, or are they specific to extracting events into a chronology or timeline?

• Which categories of legal analysis is AI well-suited to perform (factual event extraction, legal event extraction, issue spotting, claim assessment, damages assessment), and which require human work?

• Is coverage breadth a good approximation for analysis quality, or should a different test be developed?

What the benchmark offers is comparison material to start from: the model and effort choices, the comparable outputs on a known case file, and the cost-per-run profile. The legal-quality assessment is the reader’s to make.

This analysis is for educational and informational purposes only. It is not legal, financial, or other professional advice and does not create an attorney-client or any other professional relationship. The analysis is based solely on publicly available information. Any views expressed are solely those of the author in the author’s individual capacity.