Public sources
S01 / checked-original
From AI-Assisted to AI-Native: Building a Frontier Development Team
Clare Liguori (Senior Principal Engineer, AWS) / 2026-08-28 (published)
Supports: AWS-reported pilot: 50 normal-distribution teams on existing (non-greenfield) systems; metric = deployment velocity to production; half the teams achieved less than 3x; the higher group had a median of 4.5x and some more than 10x; ~90% of those teams used Kiro among other internal tools; speaker's reading is that it was about how they worked, not the tools. Separate headline at 1:29 ('a median of 4.5x ... sometimes more than 10x') is less qualified than the subgroup account at 6:50-7:04.
Limit: Observational, speaker-reported company account; not independent or causal evidence. Video upload date from YouTube page metadata (2026-08-28, AI Engineer World's Fair 2026 per organizer page). Read the organizer's timestamped transcript (https://ai.engineer/talks/pqlWNihgdjI-youtube-ca313d4e0a12a2daf850), which has automatic-transcription errors (speaker name rendered 'Claire La Gory', Kiro as 'Kuro'); the organizer page also carries AI-written summaries that were NOT used as evidence. Organizer chapters: 0:12, 2:11, 5:41, 8:10, 11:10, 13:33, 15:02, 18:32 (earlier notes' 13:58/15:09 are approximate).
https://youtu.be/pqlWNihgdjIChecked 2026-10-05
S02 / checked-original
Effective context engineering for AI agents
Anthropic Applied AI team (Prithvi Rajasekaran, Ethan Dixon, Carly Ryan, Jeremy Hadfield) / 2025-09-29 (published)
Supports: Context is a finite resource with diminishing marginal returns; aim for the smallest set of high-signal tokens; just-in-time retrieval, compaction, structured note-taking and sub-agents for long-horizon work.
Limit: Team writing: never attribute to Dario Amodei or any executive. Also contains a contrary line (see Q-C05): agentic design will trend toward 'progressively less human curation' and 'do the simplest thing that works'. Does not establish any context asset as a commercial moat.
https://www.anthropic.com/engineering/effective-context-engineering-for-ai-agentsChecked 2026-10-05
S03 / checked-original
Demystifying evals for AI agents
Anthropic (Mikaela Grace, Jeremy Hadfield, Rodrigo Olivares, Jiri De Jonghe) / 2026-01-09 (published)
Supports: Defines outcome as the final environment state (not what the transcript says); evaluating an agent means evaluating harness and model together; combines code-based, model-based and human graders; model-based graders require calibration with human graders.
Limit: Offline evals are one source of confidence, not proof of production behaviour. Earlier notes said date unverified; page shows 'Published Jan 09, 2026'.
https://www.anthropic.com/engineering/demystifying-evals-for-ai-agentsChecked 2026-10-05
S04 / checked-original
How we built our multi-agent research system
Anthropic (Jeremy Hadfield, Barry Zhang, Kenneth Lien, Florian Scholz, Jeremy Fox, Daniel Ford) / 2025-06-13 (published)
Supports: In Anthropic's data agents use about 4x more tokens than chat and multi-agent systems about 15x; multi-agent systems need tasks valuable enough to pay for the extra performance.
Limit: Ratios are specific to Anthropic's 2025 research workload. The 90.2% internal-eval improvement is not a general productivity figure. Supplementary for the talk (not a primary slide).
https://www.anthropic.com/engineering/multi-agent-research-systemChecked 2026-10-05
S05 / indexed-only
Harness engineering: leveraging Codex in an agent-first world
Ryan Lopopolo (Member of the Technical Staff, OpenAI) / 2026-02-11 (indexed)
Supports: OpenAI team built and shipped an internal beta product with 0 lines of manually-written code; its stated lesson is that the engineering job becomes designing environments, specifying intent and building feedback loops ('Humans steer. Agents execute.').
Limit: Direct fetch was blocked (HTTP 403 via curl; simplified fetch returned only the title). Body text was read only as a search-engine copy of the original page (Glean/Brave), including the authorship line and the quoted sentence; the 2026-02-11 date comes from the earlier indexed metadata and was NOT independently re-verified. Internal agent-first experiment, not enterprise policy; do not equate lines of code or PR volume with customer value.
https://openai.com/index/harness-engineering/Checked 2026-10-05
S06 / checked-original
Software Factories And The Agentic Moment
Justin McCarthy (co-founder, CTO, StrongDM AI) / 2026-02-06 (published)
Supports: Non-interactive development driven by specs and scenarios; scenarios are end-to-end user stories stored outside the codebase (holdout-like) and validated by an LLM; 'satisfaction' replaces boolean test success; Digital Twin Universe = behavioural clones of Okta, Jira, Slack, Google Docs/Drive/Sheets for high-volume validation. Rules: code must not be written by humans; code must not be reviewed by humans.
Limit: Company-authored account; page byline reads 'February 6th, 2026' (earlier note said date unknown). Experimental team constraint by a security-company AI team; not proof of general production safety. The '$1,000 on tokens today per human engineer' line is a provocation, not a benchmark.
https://factory.strongdm.ai/Checked 2026-10-05
S07 / checked-original
How StrongDM's AI team build serious software without even looking at the code
Simon Willison / 2026-02-07 (published)
Supports: Eyewitness account (he visited the three-person team in October 2025) of StrongDM's software factory, holdout scenarios and Digital Twin Universe; he calls 'Code must not be reviewed by humans' the most interesting rule and asks how it could be sensible given inhuman LLM mistakes.
Limit: Commentary on a demo, not independent validation of reliability or economics. Willison himself questions the $1,000/day figure and says he hopes lower-spend patterns exist.
https://simonwillison.net/2026/Feb/7/software-factory/Checked 2026-10-05
S08 / checked-original
Writing about Agentic Engineering Patterns
Simon Willison / 2026-02-23 (published)
Supports: Defines agentic engineering as professional engineers using coding agents that can generate and execute code to amplify existing expertise; keeps 'vibe coding' for coding without paying attention to the code.
Limit: Definition and practitioner perspective, not a controlled result. Supplementary vocabulary slide only.
https://simonwillison.net/2026/Feb/23/agentic-engineering-patterns/Checked 2026-10-05
S09 / checked-original
Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity
METR / 2025-07-10 (published)
Supports: RCT, 16 experienced open-source developers, 246 real issues: with AI allowed tasks took 19% longer; developers expected a 24% speedup and afterwards still believed AI had sped them up by 20%.
Limit: Must be paired with S10. The page now carries an update stating the historical results no longer reflect current impact. Note METR's 2026 update restates the 2025 result as '20% slowdown' in prose and '19% longer' with CI +2% to +39%; use '19% longer' from the original.
https://metr.org/blog/2025-07-10-early-2025-ai-experienced-os-dev-study/Checked 2026-10-05
S10 / checked-original
METR: February 2026 uplift update (late-2025 AI tools)
METR / 2026-02-24 (published)
Supports: Raw 2026 estimates point to speedup (original developers who returned: -18% completion time, CI -38% to +9%; newly recruited: -4%, CI -15% to +9%), but METR calls the data an unreliable signal because of selection effects (developers declining to work without AI, lower pay $50/hr vs $150/hr, task-type shifts) and says it is only very weak evidence for the size of the increase while believing developers are likely more sped up now.
Limit: Do not present as a clean 2026 speedup number. Task-level productivity, not customer value or reliability. METR's blog index (read 2026-10-05) listed no later uplift post, but the index may be incomplete and this is not an exhaustive search.
https://metr.org/blog/2026-02-24-uplift-update/Checked 2026-10-05
S11 / checked-original-partial
State of AI-assisted Software Development 2025
DORA (Google Cloud) / 2025 (year-only)
Supports: Landing page: AI's primary role is as an amplifier of an organization's existing strengths and weaknesses; greatest returns come from the underlying organizational system, not the tools themselves.
Limit: Only the landing page was read; month of release not shown there; full report and any stability statistics NOT reviewed. Supplementary; do not cite numbers.
https://dora.dev/research/2025/dora-report/Checked 2026-10-05
S12 / checked-original
Clouded Judgement 12.12.25 - Long Live Systems of Record
Jamin Ball (Clouded Judgement; disclaimer references Altimeter Capital) / 2025-12-12 (published)
Supports: Agents don't make systems of record irrelevant: cross-system agents need explicit canonical definitions (which ARR, which table wins), semantic contracts and controlled write boundaries.
Limit: Investor opinion, not empirical proof of incumbent durability.
https://cloudedjudgement.substack.com/p/clouded-judgement-121225-long-liveChecked 2026-10-05
S13 / checked-original
AI's trillion-dollar opportunity: Context graphs
Jaya Gupta and Ashu Garg (Foundation Capital) / 2025-12-22 (published)
Supports: Responds to S12: systems of record remain but the missing layer is decision traces (exceptions, overrides, precedents, cross-system context); they argue agent startups in the execution path have a structural advantage in capturing them and call the accumulated structure a context graph (explicitly not model chain-of-thought).
Limit: CORRECTION: byline is 'Posted Dec 22, 2025'; the earlier note's 'September 29, 2026' page metadata was not found in this pass. Investor thesis with commercial interest; 'incumbents can't capture this' is an argument, not an established fact.
https://foundationcapital.com/ideas/context-graphs-ais-trillion-dollar-opportunityChecked 2026-10-05
S14 / checked-original
Strands Agents with Clare Liguori (InfoQ podcast)
Clare Liguori interviewed by Thomas Betts / 2026-07-20 (published)
Supports: Liguori (technical lead on the open-source Strands Agents SDK) reports her own test: simple four-sentence instructions achieved an 82.5% pass rate and beat explicit workflows, while steering hooks reached 100% in her case study; advises evaluating by tool calls where possible; discusses procedural memory (20:25).
Limit: Date from page structured metadata (2026-07-20). Accuracy numbers are one case study, not universal. Hooks are runtime guardrails she still pairs with evals; she also notes model instructions 'can't be completely relied upon'. Supplementary (second Liguori source) unless the contrary quote Q-C04 is used.
https://www.infoq.com/podcasts/strands-agents/Checked 2026-10-05
S15 / checked-original
Writing code is cheap now (Agentic Engineering Patterns)
Simon Willison / 2026-02-23 (announced)
Supports: Delivering new code is nearly free but delivering good code (works, known to work, right problem, error handling, simple, tested, documented, maintainable) is still significantly more expensive.
Limit: Chapter was announced 2026-02-23 (S08); the live chapter has no visible revision date, so the current revision date is unknown. Practitioner argument, not an economic model.
https://simonwillison.net/guides/agentic-engineering-patterns/code-is-cheap/Checked 2026-10-05
S16 / checked-original
LLM Wiki (gist: llm-wiki.md)
Andrej Karpathy / 2026-04-04 (created)
Supports: Pattern in which an LLM incrementally builds and maintains a persistent, interlinked markdown wiki between raw sources and queries ('the wiki is a persistent, compounding artifact'), rather than re-deriving answers by RAG each time.
Limit: Created April 4, 2026 16:25 per the gist page (1 revision at check time); earlier note said date unknown. It is an idea file, not evidence of accuracy; generated summaries can go stale or be wrong; enterprise authorization is a separate problem.
https://gist.github.com/karpathy/442a6bf555914893e9891c11519de94fChecked 2026-10-05
S17 / checked-original
DeepSWE leaderboard (v1.1) and methodology
Datacurve (v1.1 post by Wenqi Huang, Peter Jiang) / 2026-09-22 (data-updated)
Supports: Fixed-harness (mini-swe-agent) benchmark of 113 long-horizon tasks across 91 repositories and 5 languages; v1.1 (released 2026-06-15 per changelog; post dated 2026-06-14) grades only the committed patch in a clean isolated container, with structured per-test reports, and drops wall-clock time as unreliable. Pass@1 and average cost per task are plotted per model+reasoning-effort configuration.
Limit: Not DeepSWE-Preview (Agentica/Together AI). The page's own limitations section: one bash tool and shared prompt for every model, so it does not reflect native harnesses (Codex CLI, Claude Code, Cursor, Gemini CLI); open-source >=500-star repos only. Costs depend on provider price changes (see changelog 2026-07-30, 08-14, 08-20, 08-21) and the raw JSON artifact disagrees with the rendered page for 19 configurations (see benchmark.caveats). See benchmark object.
https://deepswe.datacurve.ai/Checked 2026-10-05
S18 / checked-original
Harness design for long-running application development
Prithvi Rajasekaran (Anthropic Labs team) / 2026-03-24 (published)
Supports: A planner/generator/evaluator harness materially beat a solo run, but the first harness was 'bulky, slow, and expensive'; the author argues each harness component encodes an assumption about what the model can't do and should be stress-tested and removed as models improve; an out-of-the-box LLM evaluator is lenient toward LLM output and a poor QA agent until tuned, and still missed bugs.
Limit: Single-author Labs experiment, not Anthropic corporate position. Both aligned (separate evaluator helps but is not an oracle) and contrary (harness complexity should shrink) for Tom's thesis.
https://www.anthropic.com/engineering/harness-design-long-running-appsChecked 2026-10-05
S19 / checked-original
Not all AI-assisted programming is vibe coding (but vibe coding rocks)
Simon Willison / 2025-03-19 (published)
Supports: His golden rule for production-quality AI-assisted programming: don't commit code you couldn't explain to someone else; reviewing, testing and being able to explain LLM code is software development, not vibe coding.
Limit: A 2025 position, dated, and not his current stance on all workflows: in 2026 he treats StrongDM's no-human-review approach as an open question worth exploring (S07, S22). The rule is about being able to explain what code does, not specifically line-by-line review.
https://simonwillison.net/2025/Mar/19/vibe-coding/Checked 2026-10-05
S20 / checked-original
Comprehension Debt - the hidden cost of AI generated code
Addy Osmani / 2026-03-14 (published)
Supports: Argues that excessive reliance on AI code review/generation creates a growing gap between code that exists and code any human understands; tests and detailed specs are necessary but not sufficient substitutes for review.
Limit: Author's page bio (read 2026-10-05) lists him as a Member of Technical Staff at Anthropic working on Claude Code; he also says he relies heavily on tests with coding agents ('I love tests') and does not argue against agents. O'Reilly Radar repost (HTTP 403 to fetch) not used. Cites an Anthropic RCT (52 engineers, 17% lower comprehension quiz score) that was NOT independently checked.
https://addyosmani.com/blog/comprehension-debt/Checked 2026-10-05
S21 / checked-original
Fragments: February 13 (Martin Fowler)
Martin Fowler (Thoughtworks) / 2026-02-13 (published)
Supports: Fowler notes an increasing feeling that programmers will become supervisors of agents and says they will still be accountable for generated code, while calling the effect of context-switching an open question; the page also relays Margaret-Anne Storey's cognitive-debt post.
Limit: Date inferred from the fragment URL/title (year 2026 from URL); a short blog fragment, not a study. Related fragment 2026-04-02 (https://martinfowler.com/fragments/2026-04-02.html) endorses verification-first reorganisation and wants ways for humans to comprehend larger ranges of tests.
https://martinfowler.com/fragments/2026-02-13.htmlChecked 2026-10-05
S22 / checked-original
2026 in LLMs (so far)
Simon Willison / 2026-09-27 (published)
Supports: Annotated keynote notes: calls StrongDM 'six months ahead', says the central question is how to be confident of quality without reading the code and what agents can do to verify their work; says 'code must not be reviewed by humans' was a huge topic of the year and many sessions at the event were about code review.
Limit: Keynote notes, not a controlled study. Establishes that Willison's 2026 stance is exploratory rather than the 2025 golden rule alone.
https://simonwillison.net/2026/Sep/27/2026-in-llms-so-far/Checked 2026-10-05
S23 / checked-original
Note on 24th September 2026
Simon Willison / 2026-09-24 (published)
Supports: Two-sentence note: working with coding agents has convinced him they make software engineering even harder, requiring extraordinary discipline and knowledge.
Limit: Short note; supportive of 'engineering makes it trustworthy', and a counterweight to 'agents make engineering easy'.
https://simonwillison.net/2026/Sep/24/harder/Checked 2026-10-05
S24 / checked-original
The Empty Promise of Data Moats
Martin Casado and Peter Lauten (Andreessen Horowitz) / 2019-05-09 (published)
Supports: Argues data scale effects and data network effects are often weaker moats than assumed: the value of incremental data falls while its cost rises; defensibility is not inherent to data itself, though proprietary data can be defensible in some situations (scarce or reticent sources, compliance, custodianship).
Limit: 2019, pre-LLM and about enterprise startups' training-data moats, not about live enterprise context for agents. Page last modified 2024-04-25 per metadata. Tom already concedes raw accumulation isn't sufficient, so this challenges only the stronger moat claim. Use clearly dated.
https://a16z.com/the-empty-promise-of-data-moats/Checked 2026-10-05
S25 / checked-original-partial
Get started with Adobe Workfront Planning
Adobe Experience League / 2026-04-01 (last-updated)
Supports: Workfront Planning connects operational records and other systems across planning, execution, delivery and measurement. The introduction uses campaign, audience and program-performance questions; record views include timelines and calendars.
Limit: Introduction, availability and setup sections read. Planning access depends on the purchased package; highlighted preview features are not generally available. Jira for marketers is Tom's simplified analogy, not Adobe's quotation or a product equivalence.
https://experienceleague.adobe.com/en/docs/workfront/using/adobe-workfront-planning/adobe-workfront-planning-general-information/planning-overviewChecked 2026-10-06
S26 / checked-original-partial
Unified review and approval overview
Adobe Experience League / publication date unverified (unknown)
Supports: The documentation describes sequential and parallel approval paths, teams and individual participants, deadlines and notifications, and asset review through a Frame.io viewer. It explicitly distinguishes Frame.io comments from the Workfront project.
Limit: FAQ body read; publication/update date not verified. Features and limits vary by package and release. Do not imply all assets, comments or marketing-result data physically reside in Workfront.
https://experienceleague.adobe.com/en/docs/workfront/using/review-and-approve-work/document-approvals-overviewChecked 2026-10-06
S27 / checked-original
Jira as the system of record for OpenAI Symphony
Atlassian / publication date unverified (unknown)
Supports: Atlassian describes pairing Symphony agent orchestration with Jira and Teamwork Graph context. Intent, status, session history, review and follow-up remain attached to a work item; engineers retain the decision about what should ship.
Limit: Vendor account of an integration, not evidence of an acquisition attempt, a guaranteed moat or measured productivity. Original body read; byline and publication date not verified.
https://www.atlassian.com/blog/ai-at-work/jira-system-of-record-openai-symphonyChecked 2026-10-06
S28 / checked-original
NHTSA ODI opening resume: PE25012
NHTSA Office of Defects Investigation / 2025-10-07 (investigation-opened)
Supports: The original investigation document describes FSD (Supervised) and FSD (Beta) as Level 2 partial automation requiring an attentive supervising driver who remains responsible for driving and traffic-law compliance.
Limit: All three PDF pages read through local text extraction. This is an investigation-opening record, not its final findings or a comparative crash/fatality-rate study. It does not establish Tesla superiority to humans, a current software-version result or suitability for unsupervised driving.
https://static.nhtsa.gov/odi/inv/2025/INOA-PE25012-19171.pdfChecked 2026-10-06
S29 / checked-original
Full Self-Driving (Supervised) Vehicle Safety Report
Tesla / publication date unverified (live-report; rolling twelve months, exact endpoints not stated in the view read)
Supports: The original live North America / All Road Classes chart reports 7,141,797,524.951 FSD-supervised miles and 1,255 major collisions (5,690,675.319 miles per collision), versus 36,081,121,994.416 manually driven Tesla miles with active safety and 17,380 major collisions (2,076,013.924 miles per collision). Their ratio is about 2.74. Major means airbag or other non-reversible restraint deployment. FSD engagement within five seconds before a collision counts as FSD; no fault is assigned.
Limit: Read the original chart and methodology in CMUX after programmatic fetches returned 403. These are manufacturer-reported observational data, not randomized or independently audited rates. Driver supervision remains required; exposure, driver, road, conditions, hardware and software mix may differ. Tesla does not provide an observed injury/fatality-rate comparison here: projected lives/injuries avoided extrapolate collision reductions. The national baseline uses estimates and proxies; the deck uses the manual-active-safety Tesla comparator instead.
https://www.tesla.com/fsd/safetyChecked 2026-10-06
S30 / checked-original
Reuters: senators seek scrutiny of Tesla's FSD safety statistics
Reuters, republished by Claims Journal / 2026-06-17 (publisher-url-date)
Supports: Reuters reports that researchers challenged Tesla's national crash comparison on severity criteria and vehicle age, and that two senators requested NHTSA scrutiny. NHTSA said it had received the letter and was reviewing it.
Limit: Read the Reuters syndication on Claims Journal, not the inaccessible original May investigation. This is reporting and a request for scrutiny, not a final regulator finding or an independent reanalysis of the chart data. Do not assume every criticism describes the current report's stated CISS methodology unchanged.
https://www.claimsjournal.com/news/national/2026/06/17/338276.htmChecked 2026-10-06
S31 / checked-original
NHTSA ODI opening resume: EA26002
NHTSA Office of Defects Investigation / 2026-03-18 (investigation-opened)
Supports: The two-page original engineering-analysis opening record lists nine crashes, including one fatality, in the reduced-visibility investigation of FSD Beta and FSD Supervised. NHTSA identifies concerns about degradation detection and timely driver warnings, possible under-reporting of subject crashes, and six additional potentially related incidents to analyze.
Limit: All two pages read through PDF text extraction. Nine is the investigation's identified subset, not the total FSD fleet crash count; the six additional incidents are not confirmed members of that subset. This is an opening record, not a completed adjudication, matched rate study or result for every current software version. Its counts cannot be divided by the Tesla report's unrelated mileage denominator.
https://static.nhtsa.gov/odi/inv/2026/INOA-EA26002-10023.pdfChecked 2026-10-06
S32 / checked-original
LLM Leaderboard: models
Artificial Analysis / publication date unverified (live-leaderboard)
Supports: Read the original rendered leaderboard and captured its first 25 current configurations in descending Intelligence Index order, retaining the displayed order for ties. The table explicitly provides Intelligence Index, creator and Cost per Task in USD. Selected displayed scores range from 46 to 58 and costs from $0.13 to $7.67; efforts are separate configurations of the same exact model version.
Limit: Snapshot of the first 25 rows, not 25 distinct models or every model tied at the cutoff. Scores and costs preserve the table's displayed precision. API pricing, cache assumptions, model configuration and the benchmark suite affect results. The chart does not rank enterprise outcomes, interpolate unmeasured efforts, show speed or establish statistically significant differences.
https://artificialanalysis.ai/leaderboards/modelsChecked 2026-10-06
S33 / checked-original
Benchmarking methodology: Cost per Task definition
Artificial Analysis / publication date unverified (live-methodology)
Supports: Cost per Task is the benchmark-weighted mean USD cost of one Intelligence Index task. It combines input, cached and output tokens consumed with their prices, divides by task count within each benchmark, and uses the Index's benchmark weights. Prices use first-party API rates or provider medians; caching varies by provider.
Limit: Original page read in CMUX, including definitions and the formula. This is not blended price per million tokens, cost per successful task, or total enterprise delivery cost. Reasoning/output length can increase per-task cost at identical token prices. Reading date is not publication date.
https://artificialanalysis.ai/methodologyChecked 2026-10-06
S34 / checked-original-partial
Artificial Analysis Intelligence Index v4.3.2 methodology
Artificial Analysis / publication date unverified (live-methodology)
Supports: The original introduction identifies v4.3.2, ten evaluations and category weights: Agents 30%, Coding 20%, Scientific Reasoning 20%, General 30%. The suite is primarily text-based and English-language. The page describes API token counts, typical cache-hit measurements and testing parameters.
Limit: Introduction, suite table and general testing parameters read; detailed implementations were not fully audited. Index scores are not pass percentages or a universal capability measure. The publisher's aggregate uncertainty statement is not a per-point confidence interval or a test of pairwise equality; the chart does not manufacture error bars.
https://artificialanalysis.ai/methodology/intelligence-benchmarkingChecked 2026-10-06
Benchmark limitations
Not DeepSWE-Preview (Agentica/Together AI). This is Datacurve's benchmark.
Numbers were parsed read-only from the live page's embedded leaderboard data and cross-checked against the rendered table text (page header 'updated September 22, 2026'); the six rows match the displayed table (74% +/- 3% $4.43 30k 29; 74% +/- 1% $2.36 143k 166; 74% +/- 4% $11.84 118k 99; 69% +/- 3% $3.99 80k 124; 59% +/- 2% $13.22 135k 120; 54% +/- 4% $26.40 214k 268). Step counts here are unrounded means; the page rounds them (e.g. 28.8 shown as 29, 268.5 as 268).
passRate = passed/attempted scored rollouts (Pass@1, mean over 4 whole-benchmark runs); uncertainty = published 95% run-to-run half-width (1.96*std(runs)/sqrt(4 runs)) as a fraction (0.0287 = 2.87 percentage points). Overlapping intervals (the top three here) do not establish a rank difference. With only 4 runs the interval is itself rough.
costUsd = mean cost per scored attempt in Datacurve's own accounting (cost_basis states the provider rate card assumed, not reproduced here). It is NOT per-token pricing, NOT speed, and NOT a comparison of native harnesses: every model runs in mini-swe-agent with a single bash tool and a shared prompt, which Datacurve says may hold models below their native-harness ceiling. Wall-clock time is explicitly not reported in v1.1.
Rows are model+reasoning-effort configurations, not model-only scores. Effort setting changes both pass rate and cost materially (e.g. claude-opus-5 high/xhigh/max are separate rows; gpt-6-astra low-to-max span roughly 67%-74%). Do not compare 'max' on one vendor to 'xhigh' on another as like-for-like compute.
DISCREPANCY: the raw artifact https://deepswe.datacurve.ai/artifacts/v1.1/leaderboard-live.json (HTTP 200, same generated_at 2026-09-22T06:27:15Z, last-modified Sep 30 2026) differs from the page-embedded data/rendered table in avg cost for 19 configurations, all matching changelog repricing entries (GPT-5.6 Sol/Terra/Luna Jul 30 and Aug 20, DeepSeek v4 Aug 21, Gemini 3.6 Flash Aug 14, plus glm-5.3-flash). Examples: gpt-5.6-sol[max] $6.46 on page vs $8.39 in JSON; gpt-5.6-luna[max] $0.61 vs $3.03; deepseek-v4-pro[max] $1.67 vs $0.24; glm-5.3-flash[max] $0.24 vs $0.48. Pass rates are identical. Not resolved which is intended; the page/changelog-consistent values are what a viewer sees. Hence none of those rows are used here and earlier-note rows gpt-5.6-sol[max] and gpt-5.6-luna[max] were dropped. Recheck the live chart just before presenting.
Versus the earlier clean-room note: its claude-opus-4.8[max] and claude-sonnet-5[max] values and its gpt-6-astra, gemini-3.8-flash rows are confirmed unchanged, but it omitted claude-opus-5[max] (73.6%, $11.84, now one of the top rows) and its gpt-5.6-sol[max] ($6.46) and gpt-5.6-luna[max] ($0.61) costs are subject to the discrepancy above. Opus 4.8 [max] and Sonnet 5 [max] have 429 and 442 scored attempts (fewer than 452), so they rest on slightly fewer attempts.
claude-fable-5 rows have incomplete trials (page note: 73 of 2,260 trials did not complete because access was suspended by a US government directive); not used in the six rows.
Datacurve's 'contamination free' (tasks written from scratch) and its audit of SWE-bench Pro verifier error rates are the authors' own claims and were not independently verified. The sample is open-source repos with at least 500 GitHub stars in 5 languages; results may not generalize to proprietary enterprise codebases.
The chart does not measure customer value, maintainability, review quality, safety certification or cost per accepted outcome; it supports only 'a similar pass rate can come with very different cost per attempted task and token/step usage'. Cost per accepted outcome would divide by pass rate, which is derived rather than published.