This benchmark evaluates coding agents on five complementary forms of long-horizon work:
- DeepSWE subset (20 tasks): repository-level software-engineering tasks selected from DeepSWE v1.1, with balanced language coverage and difficulty stratification.
- Rewrite subset (5 tasks): four open-source command-line tools rewritten from Rust to Python, plus one single-page HTML reference rebuilt as a full-stack TanStack Start application.
- Design subset (2 tasks): open-ended visual and interactive HTML deliverables. These tasks are executed and archived, but are intentionally excluded from the automated scoring harness.
- Repository-style MCP subset (20 tasks): self-contained implementation tasks derived from a pinned MCPMark filesystem snapshot and exposed through real MCP JSON-RPC over stdio.
- Stateful MCP workflow subset (10 tasks): deterministic multi-service workflows that require schema discovery, dependency-safe tool calls, and verified final service state.
The resulting inventory contains 57 tasks in total. Of these, 55 have programmatic harnesses and 2 are design-mode tasks without a harness. The July engineering matrix covers the original 25 harness-scored tasks; the August MCP workflow pilot covers a separate 10-task matrix. Repository-style MCP tasks do not yet have a canonical result publication. These subsets measure different capabilities and must be reported separately. A single aggregate score is not the primary result because binary repository repair, behavioral compatibility, full-stack reconstruction, visual design, repository work through MCP, and cross-service state transitions are not commensurate measurements.
This document describes the task-selection criteria, data normalization rules, evaluation boundaries, known anomalies, and limitations. The task inventory, executable contracts, selection logic, and published result artifacts are maintained in this repository; the current test-set evidence record applies the methodology to the July 2026 engineering artifacts and the August 2026 MCP workflow pilot.
The primary unit of comparison is the complete configured agent system. The four-configuration matrix evaluates outcome and resource measurements jointly; it does not estimate an isolated runtime-component effect.1
| Analysis question | Primary estimand | Required controls or qualifications |
|---|---|---|
| Configuration outcome | DeepSWE pass proportion; rewrite task-macro and assertion-micro rates | Report numerator, denominator, task revision, replicate count, model, effort, build, and retry policy |
| Resource allocation | Observed tokens, rounds, estimated cost, and command records | Keep token components separate; command records are runtime-specific units |
| Additional reasoning effort | Codex High-minus-Medium configuration contrast | Build and effort differ simultaneously; the contrast is not an effort-only effect |
| Round and command associations | Q1-to-Q3 difference in fitted success probability | Descriptive binomial models; task difficulty and stopping behavior remain uncontrolled |
| Submitted production-code volume | Within-task association between code additions and run-level harness success ratio across all 25 harness tasks | Equal run weights; task fixed effects; task-clustered uncertainty; configuration-adjusted and subset sensitivity models; missing source remains missing |
| MCP workflow outcome | Completed workflows / valid runs, plus task- and agent-level pass proportions | Keep the MCP pilot separate from the engineering matrix; report model, effort, contract revision, replicates, requests, usage, and failed check IDs |
Tura Balanced operationalizes the verification-reinvestment configuration and Tura Direct operationalizes the token-and-round-reduction configuration. Their labels identify configured policies; they do not encode a success criterion. Every comparison reports harness outcome, observed model tokens, model rounds, estimated cost when usage is available, and the relevant uncertainty or identification limit.
This matrix is not a feature-level ablation. Runtime architecture, tool orchestration, context policy, reasoning effort, instructions, and verification behavior vary jointly. A component-level causal estimand requires a crossed design that holds the remaining factors constant.
The suite follows six principles.
- Behavior before implementation shape. Where an automated verifier is available, success is based on observable behavior rather than matching a reference patch or reproducing internal symbol names. This follows the behavioral-verifier rationale described by DeepSWE and the broader repository-level evaluation setup established by SWE-bench.2 3
- Coverage before convenience. DeepSWE sampling is stratified by programming language and estimated difficulty rather than drawn only from the easiest or most common tasks.
- Pinned, auditable inputs. Rewrite tasks identify the source repository, commit, tag, target language, and stable harness items. MCPMark source is commit-pinned, and each workflow scenario retains its vendor-contract source and revision date. Run artifacts retain task, agent, model, and runtime metadata.
- No invented evidence. Missing logs, assertion text, token fields, or scores remain missing. They are not reconstructed from model summaries or inferred from nearby runs.
- Separate objective and subjective evaluation. Deterministic or programmatic checks belong in the harness. Design quality remains outside the harness until a validated human-review or multimodal-evaluation protocol is defined.
- Strategy before feature attribution. Interpret each agent configuration as a complete budget-and-verification policy. Do not relabel a system-level result as evidence that one architectural component caused it.
These choices are also consistent with reproducible benchmark practice: the experimental design, software versions, parameters, and result metadata should remain tied together rather than being reported as disconnected tables.4
| Subset | Tasks | Primary capability | Evaluation mode | Publication boundary |
|---|---|---|---|---|
| DeepSWE v1.1 subset | 20 | Repository exploration, implementation, debugging, and verification | Official program-based verifier; binary task outcome | July 25-task engineering matrix |
| Rewrite subset | 5 | Behavioral compatibility, source porting, and full-stack reconstruction | Task-specific multi-item harness | July 25-task engineering matrix |
| Design subset | 2 | Visual communication, research, interaction, and artifact quality | Artifact capture and separate review | Separate, non-harness review |
| Repository-style MCP subset | 20 | Repository implementation through an MCP filesystem surface | Task-specific deterministic verifier | No canonical result publication yet |
| Stateful MCP workflow subset | 10 | Cross-service orchestration and final-state correctness | Five deterministic protocol, order, and state checks | August 10-task pilot; reported separately |
| Total | 57 | Mixed long-horizon agent work | Mixed | No single aggregate |
The suite is a curated capability sample, not a random sample of all software-engineering work. Results therefore support comparison on this fixed suite; they do not directly estimate performance on all repositories, languages, or development tasks.
The 20 tasks under tasks/mcp are derived from the filesystem scenarios in MCPMark commit cd45b7f57923b9b3985467f5139927575f83141c. Each task packages its own fixture, task contract, stdio MCP server, adapter declarations, runner, and deterministic verifier. They test repository implementation through MCP rather than live external services. Because no canonical result cohort is currently published for this subset, these tasks belong to the benchmark inventory but not to the current evidence tables.
The 10 tasks under tasks/mcp_workflow model multi-step work across vendor-aligned service contracts. They use real MCP JSON-RPC lifecycle and tool-result envelopes against deterministic, run-scoped mock state. The mocks never contact a live user account. Every scenario embeds the selected contract source, revision date, tool schemas, initial state, required calls, dependency graph, and expected final state. A tool is labeled official-mcp only when its exercised surface comes from a published provider MCP contract; otherwise it is labeled vendor-api-adapter.
Each workflow run receives five checks: initialization, tools/list discovery, completion of required operations, dependency-safe call order, and independent verification of final state and generated artifacts. Rejected calls remain in the trace and do not fail a later corrected workflow. Scoring is entirely programmatic; no LLM judge or human rating contributes to the result.
The August pilot manifest at results/mcp/report-mcp-workflow-gpt56-sol-low-20260809/manifest.json contains 90 runs: ten tasks, three configurations, and three replicates, all using GPT-5.6 SOL at Low reasoning. It is an exploratory pilot and is not pooled with the July engineering matrix. The detailed task and adapter contract is documented in MCP benchmark tasks and workflow harness.
DeepSWE v1.1 contains 113 original tasks across 91 active open-source repositories and five languages: Go, Python, TypeScript, Rust, and JavaScript.5 2 The official repository publishes the underlying task definitions.6 Each task provides an instruction, a pinned environment, resource limits, and a purpose-written verifier in the Harbor task format.7
The subset-selection artifact records these official inputs:
- task metadata:
https://deepswe.datacurve.ai/artifacts/v1.1/tasks.json; - trial records:
https://deepswe.datacurve.ai/artifacts/v1.1/trials.json; - official task count at selection time: 113;
- eligible official scored trials at selection time: 18,396;
- selection schema:
tura.benchmark.deep-swe-selection.v1.
Only official trials satisfying all three conditions are used to estimate task difficulty:
source == "deep-swe"
eval_scope == "full"
included_in_score == true
For task (t), the official completion rate is:
official_completion_rate(t)
= number of eligible official trials with passed == true
/ number of eligible official trials for t
In the repository and official artifacts this quantity is named official_pass_rate. This document uses completion rate and pass rate synonymously only for that field. It is a historical empirical rate over the official model/trial pool, not an intrinsic property of the task.
The subset contains exactly four tasks from each official language:
| Language | Selected tasks |
|---|---|
| Go | 4 |
| Python | 4 |
| TypeScript | 4 |
| Rust | 4 |
| JavaScript | 4 |
This equal allocation prevents the larger language pools from dominating the suite. It is a deliberate macro-balancing choice, not a reflection of language prevalence in production software.
The intended difficulty profile is four levels per language, anchored where possible around 80%, 60%, 40%, and 20% official completion rates. Higher historical completion implies an easier task; lower completion implies a harder task.
The original inventory was produced with rank bands because some language pools are too small or do not contain tasks near every target rate:
- rank all eligible tasks within each language by descending official pass rate;
- divide that language-specific ranking into four approximately equal bands;
- label the bands
easy,medium-easy,medium-hard, andhard; - select the highest-pass-rate task in each band;
- break equal-rate ties deterministically by task ID.
This produced four tasks per language and five tasks per difficulty band. The 20/40/60/80 values are therefore difficulty anchors, not guaranteed bins. The selected 20 task IDs are now pinned in deep_swe/canonical_tasks.json. Current official trial data may refresh the recorded rates and ranks, but it must never change task membership. This prevents later updates to the online trials.json artifact from silently changing the comparison cohort.
The selected rates demonstrate the resulting approximation:
| Language | Selected official completion rates, hard to easy |
|---|---|
| Go | 44%, 59%, 70%, 80% |
| Python | 36%, 51%, 60%, 87% |
| TypeScript | 26%, 36%, 69%, 91% |
| Rust | 13%, 44%, 59%, 61% |
| JavaScript | 25%, 30%, 66%, 73% |
Rates in this summary are rounded to the nearest percentage point for readability. Selection and auditing use the unrounded values.
| Language | Difficulty band | Official pass rate | Task | Requested behavior |
|---|---|---|---|---|
| Go | Easy | 79.88% | actionlint-action-pinning-lint |
Add action-pinning linting for actions and reusable workflows |
| Go | Medium-easy | 70.12% | abs-stepped-slices |
Add stepped slices for arrays and strings |
| Go | Medium-hard | 59.15% | yaegi-go-embed-directives |
Add go:embed directive support for interpreted packages |
| Go | Hard | 44.38% | dasel-html-document-format |
Add HTML document-format handling to Dasel |
| Python | Easy | 87.20% | narwhals-rolling-window-suite |
Add rolling minimum, maximum, median, and quantile methods |
| Python | Medium-easy | 59.88% | numba-stencil-boundary-modes |
Add boundary modes to @stencil |
| Python | Medium-hard | 50.61% | bandit-incremental-cache-control |
Add incremental cache controls to Bandit |
| Python | Hard | 35.58% | langchain-request-coalescing |
Add request coalescing to Runnable |
| TypeScript | Easy | 91.46% | happy-dom-abort-pending-body-reads |
Abort pending body reads on shutdown |
| TypeScript | Medium-easy | 69.14% | dynamodb-toolbox-conditional-attribute-requirements |
Add conditional required attributes to schemas |
| TypeScript | Medium-hard | 35.63% | awilix-async-container-initialization |
Add dependency-aware asynchronous container initialization |
| TypeScript | Hard | 25.77% | quill-shared-toolbar-focus |
Reuse one toolbar across multiple Quill editors |
| Rust | Easy | 60.98% | wasmi-trap-coredumps |
Add trap coredump generation to wasmi |
| Rust | Medium-easy | 59.26% | fd-deterministic-multi-key-sorting |
Add deterministic multi-key sorting to fd |
| Rust | Medium-hard | 44.03% | boa-hierarchical-evaluation-cancellation |
Add hierarchical evaluation cancellation to Boa |
| Rust | Hard | 12.80% | pest-character-class-coalescing |
Coalesce qualifying choices into character classes |
| JavaScript | Easy | 73.17% | yjs-map-conflict-detection |
Add deterministic map-conflict detection to Y.Map writes |
| JavaScript | Medium-easy | 65.64% | testem-per-launcher-reports |
Partition reports by launcher and expand report templates |
| JavaScript | Medium-hard | 29.81% | csstree-shorthand-expansion-compression |
Add shorthand expansion and compression to the lexer |
| JavaScript | Hard | 24.54% | katex-multicolumn-array-spans |
Add \multicolumn column spans to array-like environments |
Each selected task had between 159 and 164 eligible official trials in the captured v1.1 data. The selection artifact recorded zero official error trials for these 20 tasks after applying the eligibility filter.
Each run starts from the task's pinned base commit and isolated environment. The agent receives the task instruction and edits the workspace. The official task verifier then evaluates the resulting repository state. Pier provides the upstream workspace-and-trace execution model for Harbor tasks, while the local benchmark repository normalizes agent runs and verifier artifacts into its own contracts.8 A valid verifier report with reward 1 is a pass; a valid report with reward 0 is a task failure.
The managed DeepSWE checkout defaults to upstream commit a40d7298b18999c2d9b0ded7d6928e3ee26b5524. The July published harness.json and harness-report.json files nevertheless identify the upstream grader location with the tag v1.1, not a verifier commit SHA and container-image digest. Those artifacts therefore preserve the recorded patch and verdict but do not independently prove bit-for-bit verifier identity. Future canonical cohorts must record the resolved grader commit and verifier image digest in each run contract.
The DeepSWE grader and its hidden fixtures execute from the upstream corpus rather than a complete vendored copy in this repository. A reader can inspect the published patch, normalized report, and reward, but cannot re-derive every reward from this repository alone when the upstream test fixture is not public. This is a reproduction limit, not a reason to recode a valid verifier failure as infrastructure-invalid. The distinction and the other audit boundaries below respond to benchmark issue #1.
For DeepSWE, every Tura configuration uses the Bash tool surface and launches as
tura exec bash --json. This setting is mandatory rather than an optional CLI
preference: disabling Bash can severely reduce Tura's effectiveness on
repository-level investigation, editing, and verification. The runner forces
the setting and rejects a Tura invocation that does not archive the expected
argument prefix; such a run is configuration-invalid and cannot be pooled with
the published Tura DeepSWE results.
Infrastructure outcomes are not task failures. A non-zero verifier process exit, missing report, malformed reward, unavailable image, workspace-preparation failure, timeout outside the task contract, or artifact-write failure is labeled invalid/infrastructure failure and excluded from the pass-rate denominator until rerun or explicitly reported as missing. Treating infrastructure failures as zero would confound agent capability with benchmark availability.
The rewrite subset is designed to test whether an agent can recover and reproduce behavior from an existing artifact or codebase rather than implement a narrowly localized issue. The repository and result category is named rewrite; “rebuild” describes the work performed inside these tasks, not a separate benchmark subset. A task is included when it has:
- a legally accessible and inspectable source or reference artifact;
- a pinned source commit/tag or benchmark-owned reference snapshot;
- a concrete target technology;
- a runnable, task-specific harness with stable score-item IDs;
- enough behavioral breadth to require exploration, implementation, and testing rather than a one-file patch;
- no dependency on private credentials or proprietary services for core scoring.
The four CLI tasks use differential or reference-equivalence checks: the target implementation is exercised with representative commands and compared with the pinned reference behavior. The HTML task combines structural, browser, backend, database, test, and maintainability checks. Harness item counts describe the number of stable assertions, not five directly comparable percentage scales.
| Task | Source and pin | Target | Harness items | Scope |
|---|---|---|---|---|
eza |
eza, Rust, tag v0.23.3, commit 05d20d11c488b2ad3f0d63ac0b529281cc1c16ef |
Python CLI | 52 | Rebuild directory listing, long view, tree traversal, sorting, hidden-file behavior, and related option/error semantics; icons and colors are disabled to keep output comparable. |
nushell |
Nushell, Rust, tag 0.106.1, commit 682d593d3f53e5337dceedf98c9603a698af6a64 |
Python CLI | 48 | Reproduce the selected nu -c workflow: expressions, tables, JSON, CSV, strings, mathematics, and filesystem snippets. This is a compatibility subset, not a full Nushell reimplementation. |
xsv |
xsv, Rust, tag 0.13.0, commit 2b4cbaa0eecf7b507a612632fe00289b1b358c15 |
Python CLI | 55 | Rebuild CSV behavior for headers, count, select, slice, search, sort, table, format, statistics, and frequency operations, including relevant argument and output semantics. |
zip-password-finder |
zip-password-finder, Rust, tag v0.11.1, commit 7c1a4c93841220fc740ed81d3b97784e450fc6a6 |
Python CLI | 18 | Rebuild the single-command interface, argument validation, dictionary search, and brute-force ZIP password search behavior. |
prompt-gallery-tanstack-fullstack-rebuild |
Benchmark-owned makeup.html, snapshot tag report-20260708-20260709 |
TypeScript / TanStack Start | 63 | Convert a single-page prompt-marketplace reference into a functioning full-stack application. Checks cover TanStack Start structure, visual fidelity, storefront/detail/cart/checkout/filter/favorite/creator/admin flows, server operations, local database schema and seed data, computed analytics, runnable tests, browser robustness, and code quality. |
Each task reports passed assertions and total valid assertions from its own harness. Recommended reporting is:
task_score = passed valid harness items / total valid harness items
For replicated runs, pool valid harness items within each task before computing that task's score. Report both the numerator and denominator. The task-level macro average gives each of the five tasks equal weight:
rewrite_macro_average = mean(task_score for the five rewrite tasks)
The published README also reports an assertion-weighted micro rate from the canonical manifest:
rewrite_micro_rate = sum(passed valid harness items) / sum(total valid harness items)
Keep the macro and micro rates labeled and adjacent. The micro rate gives the 63-item HTML rebuild 3.5 times the weight of the 18-item ZIP task merely because its harness is more granular; it is useful as an auditable count of all checks, but it is not a task-balanced score. Never average run percentages directly.
The harness does not require source-level similarity. Alternative implementations are acceptable when they satisfy the declared behavior. Conversely, compilation or visual resemblance alone is insufficient when behavioral checks fail.
The July 2026 rewrite publication contains five tasks, four configurations, and
two replicates per configuration: 40 canonical runs. The 30-run Tura Balanced,
Tura Direct, and Codex Medium source is
report-20260710-gpt56-sol.
The 10-run Codex High source is
report-20260714-codex-cli-0.144.1-gpt56-sol-high.
Per-run prompts, normalized rounds, aggregate usage, retained workspaces, and
harness reports remain under those report directories.
The published rewrite tasks do not include a benchmark-owned known-good target implementation executed through each final harness. The four CLI harnesses compare selected behavior with pinned source programs, and the HTML harness checks a benchmark-owned reference specification, but neither path is a substitute for a recorded reference target passing every assertion. Until such reference runs are published, harness satisfiability is supported by individual checks and source behavior rather than demonstrated by one complete known-good build.
The design tasks have stable prompts, run metadata, and required output paths, but no harness.json. They are excluded from automated score aggregation because their central outcomes—visual hierarchy, information design, editorial quality, interaction clarity, and responsible use of sources—cannot currently be reduced to the same deterministic pass/fail contract used by the engineering tasks.
Simple existence checks such as “index.html was created” are useful integrity checks but are not evidence of design quality. Until a separate rubric is validated, these tasks should be reported as completed artifact / invalid artifact / not run, followed by blinded human review or clearly labeled qualitative analysis. They must not silently receive a zero or a perfect score in the 25-task harness result.
| Task | Required deliverable | Core requirements | Evaluation boundary |
|---|---|---|---|
east-asian-squid-recipes-slides |
A navigable English HTML presentation at ./index.html |
Fifteen illustrated slides covering ten distinct squid cooking methods from East Asian countries or regions; each method needs cultural attribution, ingredient quantities, preparation and cooking steps, timing, a recipe-source link, and a working YouTube cooking-video link; all assets remain in the workspace. | Review completeness, factual sourcing, editorial structure, image relevance, readability, navigation, and link validity. No automated harness score. |
paris-summer-temperature-3d |
A responsive English interactive 3D HTML experience at ./index.html |
Show the evolution of Paris summer temperature from 1986 through 2026 with a clear time axis, spatial depth, animation, and controls for yearly values, trends, and notable heat events; keep assets local and distinguish observed historical values from provisional or projected 2026 values. | Review data provenance, historical/provisional labeling, legibility, interaction stability, 3D communication value, responsiveness, and accessibility. No automated harness score. |
Every task is keyed by a stable task ID. Repository tasks additionally retain the repository URL and base commit. Rewrite tasks retain their source tag/commit and target runtime. MCP tasks retain the scenario or source snapshot revision and adapter-contract provenance. When an external verifier is used, canonical identity also requires the resolved verifier commit and executable image digest; a floating tag is descriptive provenance, not an immutable identity. Results from different task or verifier revisions must not be merged under one ID without a revision field or migration record.
A run record should include at least:
- benchmark and task version;
- task ID and subset;
- agent/runtime ID;
- model identifier and reasoning/effort setting;
- replicate number;
- start/end state and bounded timeout;
- source commit or reference snapshot;
- harness version and report path, plus resolved verifier commit and image digest when applicable;
- observable token/usage fields without imputation;
- infrastructure status and retry lineage.
Agentic runs are stochastic. Replicates are independent observations, not backup files to be cherry-picked. Retry only a documented environment failure or provider failure that invalidates the attempt. A normal task timeout, agent non-zero exit, agent-reported error, or valid verifier failure is experimental behavior and must remain in the dataset without retry.
- Raw layer: untouched provider events, stdout/stderr, workspace state, and verifier output.
- Normalized layer: schema-validated rounds, tool calls, usage, task reports, and harness reports.
- Published layer: compact manifests and result tables linked back to normalized and raw evidence.
Normalization may rename or structure fields, but it must not invent commands, tool results, token counts, assertions, or scores. Cumulative usage updates must be deduplicated before summation; otherwise repeated provider snapshots inflate cost and token totals.
Codex Medium used a locally instrumented Codex build to retain round commands,
timing, and provenance. Its normalized round contracts omit per-round input and
output components, while run-level aggregate usage remains retained. Codex
High used the unmodified official Codex CLI 0.144.1 release. Keep the two
configurations separate in every table and fit. The build boundary is a
confounder; do not attribute a High-versus-Medium difference solely to
reasoning effort.
Harness success means that the declared checks passed. It does not imply that every unrelated file or external object remained unchanged. The July engineering reports and August MCP workflow schema retain final workspaces, diffs or state traces where available, but they do not publish a universal agent-handoff manifest followed by an automated off-task-state diff for every task family. Reviewers may inspect retained artifacts, but manual inspectability is not the same as a scored guard.
Future canonical harnesses should record the task-visible workspace and relevant service-state manifest at handoff, declare allowed mutation paths or objects, and fail or separately flag undeclared changes. This guard must be scoped carefully: build products, caches, logs, and task-authorized generated files should not become accidental failures.
Use explicit states rather than coercing all anomalies to zero:
| Condition | Treatment |
|---|---|
| Valid harness reward or assertion result | Include in task score |
| Agent completed; valid verifier returns failure | Count as task failure |
| Agent process or task reaches its declared task timeout and verifier can still run | Preserve timeout status and score only from the valid verifier |
| Agent process times out or exits non-zero | Retain as experimental behavior; do not retry |
| Provider or environment failure invalidates the attempt | Mark invalid and retry with lineage retained |
| Verifier crashes, report is absent/malformed, or source cannot be prepared | Mark invalid/infrastructure failure; do not count as task failure |
| Token or cost field unavailable | Keep null/missing; do not estimate |
| Assertion text absent in an archived report | Keep evidence text empty; retain stable assertion ID if known |
| Duplicate cumulative provider-usage event | Deduplicate using the cumulative state before aggregation |
| Design artifact missing or entry path wrong | Mark invalid artifact; do not manufacture a design score |
| External link unavailable during design review | Record link-check time and failure separately from artifact rendering |
Configuration-level result tables use all 280 published harness-scored runs.
Cross-run relationship figures use a 278-run population after excluding exactly
two Tura Balanced observations above 90 rounds: 113 rounds for
quill-shared-toolbar-focus and 242 rounds for
dynamodb-toolbox-conditional-attribute-requirements. The observations remain
in raw contracts and configuration-level aggregates. Record their identities,
values, and exclusion reason in assets/model-run-statistics/excluded-runs.csv.
Do not apply additional visual trimming or replace missing values with zero.
The threshold changes the estimand from the full empirical population to the declared non-long-tail relationship population. Every statistical figure and caption must state the 278-run denominator and the two-run exclusion. The submitted-code analysis additionally states its 272-run observed-code population, 206-run pooled identifying population, task-cluster counts, and six missing-source records.
Report the five subsets separately:
- DeepSWE: passes / valid task runs and pass rate, with replicate-level results retained;
- Rewrite: assertion score per task, the five-task macro average, and the separately labeled assertion-weighted micro rate;
- Design: artifact validity and separate rubric dimensions or qualitative findings, explicitly labeled non-harness.
- Repository-style MCP: passes / valid task runs when a canonical cohort exists; do not infer a result from task availability alone.
- Stateful MCP workflow: completed workflows / valid runs, with the five check IDs, task- and agent-level rates, requests, token components, estimated cost, and failed-run evidence retained.
For every strategy comparison, report these outcome metrics beside observed total model tokens, model rounds, and computed cost when the provider usage record supports it. Also retain task-level distributions and severe long tails; aggregate savings alone can hide expensive failures. Verification activity may be summarized from traceable test, build, lint, browser, link, source, or rerun evidence, but raw command counts must not be treated as equal atomic work units across runtimes with different batching granularity.
For comparisons between agents, use the same task revision, model where the agent comparison requires it, effort setting, timeout policy, network policy, and replicate count. Publish the run matrix before interpreting differences.
For every regression analysis, state the analysis population, response,
predictor transformation, weighting or trial denominator, adjustment variables,
estimand, interval construction, missing-data treatment, and exclusion rule.
Report coefficients only with their units or transformations. Report fitted
probability differences in percentage points and odds ratios as exp(β).
Round and command models use separate binomial logistic regressions by
configuration: logit(P(success_i)) = α + β log(1 + x_i). Their estimand is the
Q1-to-Q3 change in fitted success probability within that configuration's
observed predictor range. Harness check count supplies the binomial trial
denominator; it is not a semantic-difficulty weight.
Token volume uses pooled quadratic OLS on the natural round axis. Effective
billed rate is cost × 1,000,000 / total tokens and uses pooled log-linear OLS.
These pooled coefficients combine within- and between-configuration variation.
Command counts are not compared as equal atomic-work units across runtimes.
The submitted-code model uses one equal-weight fractional-logit observation per
run, task fixed effects, and CR1 covariance clustered by task. Its primary
estimand is the common within-task association between standardized
log(1 + additions) and expected harness ratio. Report pooled, DeepSWE-only,
rewrite-only, and task-plus-configuration specifications together. The
association remains vulnerable to attempt-scope, architecture, stopping-rule,
and semantic-coverage confounding and is not a causal effect of writing more
lines.
Configuration differences are system-level contrasts. A component-level causal claim requires a crossed design. Codex High versus Medium is jointly confounded by build and reasoning effort.
If an overall engineering score is required for the July matrix, use a task-level macro average over the 25 harness-scored tasks so that each task contributes equally after its own harness has produced a task score. Label the formula and keep the subset scores adjacent. Do not include the two design tasks unless a separate, predeclared scoring protocol exists. Do not append the MCP pilot to that score: its task shape, reasoning setting, configuration matrix, and five-check workflow contract define a different estimand.
Always show counts with percentages. Regression figures report 95% intervals and identify their covariance estimator, clustering unit, and reference distribution. Round and command intervals are model-based and do not correct for task dependence. Submitted-code intervals use CR1 covariance clustered by task but still rely on only 19 pooled, 15 DeepSWE, or 4 rewrite identifying clusters. Twenty DeepSWE tasks and five rewrite tasks do not support precise population generalization beyond the curated subset. The official DeepSWE site likewise reports uncertainty and cautions against overinterpreting small qualitative frequencies.5 2
The official pass rate depends on the models, agent harness, effort settings, and trial mix present in the v1.1 official records. A task labeled hard may be easy for a later model, and a low rate can partly reflect verifier or environment friction. Difficulty labels should be regenerated or versioned when the official trial pool changes.
Go and Python offered 34 eligible tasks each and TypeScript 35, but Rust and JavaScript offered only five each in the captured selection. Four strata over five candidates cannot closely match four fixed completion-rate targets. Equal language representation is preserved at the cost of a less uniform difficulty profile.
Selecting the first item in each rank band is deterministic but sensitive to small rate changes near a band boundary. It also tends to select the easier edge of every band. A future revision could predeclare nearest-target matching with uniqueness constraints, but changing the algorithm would define a new subset version and should not retroactively alter existing results.
One harness item can represent a narrow argument check or a broad browser flow. Assertion counts are therefore not units of semantic difficulty. This is why task-level macro aggregation is preferred over pooling all assertions.
CLI output can vary with operating system, locale, filesystem ordering, path separators, terminal capabilities, timestamps, permissions, and archive libraries. Fixtures should disable irrelevant color/icon output, pin locale and dependency versions, normalize only declared nondeterministic fields, and preserve exit code, stdout, and stderr semantics.
Repositories, package registries, videos, recipe pages, and climate-data endpoints can change or disappear. Source commits and local task assets must be pinned where licensing permits. External-link checks should record their date; link rot is not automatically an agent failure if the artifact used a valid source at run time.
Program-based verifiers approximate a specification; they are not the specification itself. They can miss valid alternative behaviors or permit incomplete implementations. DeepSWE's authors explicitly motivate behavioral verification and also identify verifier design as an area for continued improvement.2 Harness changes require versioning and re-evaluation of comparability.
Human design ratings can vary with reviewer background, display, browser, cultural familiarity, and aesthetic preference. Any future design comparison should use multiple blinded reviewers, a predeclared rubric, calibrated examples, and inter-rater agreement. Automated visual checks may detect clipping or missing assets, but should not be presented as a complete measure of quality.
The benchmark measures performance under specific prompts, tools, timeouts, environments, and verifiers. It does not fully measure maintainability, security, product judgment, long-term operation, collaboration, or whether a patch would be accepted by upstream maintainers.
DeepSWE covers five languages but excludes major ecosystems such as Java and C++. Its official corpus is concentrated in TypeScript, Go, and Python, and is drawn from established open-source repositories; DeepSWE's authors note these same coverage limits.2 Equal-language sampling further differs from real-world language prevalence.
The rewrite subset is small and intentionally heterogeneous. All four CLI ports begin with Rust sources and target Python, so the result should not be generalized to arbitrary language pairs. The HTML task tests one framework and one product shape.
The MCP workflow subset uses deterministic mocks and ten authored scenarios. It measures protocol use and orchestration against the declared contract, not authentication, rate limits, permission drift, latency, partial outages, undocumented provider behavior, or safety constraints in live accounts. Passing a mock workflow is not evidence of complete vendor MCP conformance.
The DeepSWE subset is stratified, not random. It overrepresents Rust and JavaScript relative to their available task pools and chooses deterministic band-edge examples. The rebuild and design tasks were purposefully selected for breadth and evaluability. Reported performance is conditional on this curation.
DeepSWE reduces direct benchmark leakage by using original tasks rather than fixes copied from existing public commits.2 This lowers but does not eliminate contamination: models may have seen the underlying repositories, libraries, task descriptions after publication, or similar implementations. Research on code-generation benchmarks finds that both surface and semantic overlap with training corpora can materially inflate measured performance.9
The four rebuild sources are public and may be present in model training data. They should be interpreted as behavioral reconstruction tasks, not contamination-free tests of novel algorithm discovery.
Model APIs, agent implementations, package registries, benchmark artifacts, and source repositories evolve. Every publication should state the benchmark revision, selection timestamp, model identifier, agent version, configuration, and execution period. Results from different revisions are not directly comparable without a compatibility audit.
Twenty DeepSWE tasks and five rewrite tasks provide limited power. Outcomes within a repository, language, or agent runtime may be correlated, so treating every harness assertion as an independent sample understates uncertainty. Replicates reduce stochastic noise but do not create new independent tasks.
Long-horizon performance is sensitive to token budget, reasoning effort, tool-call limits, wall-clock timeout, network access, and service tier. More resources may improve completion rate while increasing cost. Capability and efficiency should therefore be reported together, not collapsed without an explicit utility function.
The current matrix does not isolate compact-context behavior, command batching, operation-manual text, or reasoning effort. Cross-task-group differences in model rounds and recorded command output define descriptive associations only; they do not estimate an individual mechanism's causal effect. A controlled ablation must hold the build, task set, model, effort, timeout, service tier, network policy, and retry policy constant.
Tura-AI develops the Tura runtime, owns this benchmark repository, defines the Tura Balanced and Tura Direct configurations, and publishes comparisons against Codex. This is a direct conflict of interest. Public prompts, contracts, traces, workspaces, usage records, and verifier reports make the claims auditable, but they do not provide independent task authorship, execution, or replication. Readers should weight the results accordingly, and independent reproduction should be reported separately from project-run evidence.
Before publishing or comparing a run:
- freeze the benchmark revision and DeepSWE selection artifact;
- verify that the selection contains 20 unique DeepSWE tasks, four per language and five per difficulty band;
- record the official task/trial artifact URLs and retrieval time;
- validate all task declarations and harness schemas;
- pin source commits, dependency lockfiles, container images, locale, and runtime versions;
- record the resolved grader commit and verifier image digest rather than only a tag;
- publish the agent/model/effort matrix, replicate count, timeout, concurrency, and network policy;
- preserve raw events, normalized rounds, repository diffs, verifier output, and retry lineage;
- publish a start-state manifest and an automated off-task-state comparison where the task contract permits one;
- distinguish valid task failures from infrastructure-invalid runs;
- run a known-good reference target through each rewrite harness and retain the result;
- report DeepSWE, rewrite, design, repository MCP, and workflow MCP results separately;
- include counts and denominators with every rate;
- identify the published, relationship-model, and observed-code populations;
- publish every regression formula, estimand, adjustment set, and interval assumption;
- keep design tasks outside harness aggregation;
- document every exclusion, rerun, harness revision, and manual judgment.
- disclose benchmark ownership, configuration authorship, and other material conflicts of interest.
Additional implementation evidence is available in the [Tura Benchmark repository]10, DeepSWE selection implementation, task definitions and harnesses, runtime schemas, and published result manifests.
Footnotes
-
Tura AI, “Tura,” agent architecture, tool orchestration, context-management design, and public strategy-level benchmark framing, https://github.com/Tura-AI/tura (accessed 2026-07-13). ↩
-
Datacurve AI, “DeepSWE: Measuring frontier coding agents on original, long-horizon engineering tasks,” methodology, analysis, and limitations, https://deepswe.datacurve.ai/blog/deepswe (accessed 2026-07-12). ↩ ↩2 ↩3 ↩4 ↩5 ↩6
-
Carlos E. Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik Narasimhan, “SWE-bench: Can Language Models Resolve Real-World GitHub Issues?”, ICLR 2024, arXiv:2310.06770, https://arxiv.org/abs/2310.06770. ↩
-
Stephanie C. Mangul, Lana S. Martin, Brian L. Hill, Angela Ka-Mei Lam, Margaret G. Distler, Eleazar Eskin, and Jonathan Flint, “Reproducible and replicable comparisons using SummarizedBenchmark,” Bioinformatics 35(8), 2019, https://doi.org/10.1093/bioinformatics/bty627. ↩
-
Datacurve AI, “DeepSWE,” official benchmark website and v1.1 leaderboard, https://deepswe.datacurve.ai/ (accessed 2026-07-12). ↩ ↩2
-
Datacurve AI, “deep-swe,” task definitions and benchmark source repository, https://github.com/datacurve-ai/deep-swe (accessed 2026-07-12). ↩
-
Harbor Framework, “Task Structure,” task metadata, instructions, environment, verifier, solution, and network-policy format, https://www.harborframework.com/docs/tasks (accessed 2026-07-12). ↩
-
Allen Institute for AI, “Pier: Workspace manager for coding agents,” https://github.com/allenai/pier (accessed 2026-07-12). ↩
-
Yiming Yang, Wenjin Yao, Yujia Zhang, Patricio P. B. Gusmao, and others, “Quantifying Contamination in Evaluating Code Generation Capabilities of Language Models,” Proceedings of ACL 2024, https://aclanthology.org/2024.acl-long.761/. ↩
-
Tura AI, “Tura Benchmark,” methodology, task definitions, canonical manifests, and published evidence, https://github.com/Tura-AI/benchmark (accessed 2026-07-13). ↩