Enterprise AI Architecture

Public edition / Tom Valletta / Yerevan, October 6, 2026. Personal perspective, not an Adobe statement or roadmap. Examples are illustrative; research is attributed to public sources.

Core: 45 minutes. Optional depth: 15 minutes. Q&A/networking: final 30 minutes. Budgets aren't measured timings.

Open the presentation / Include optional depth / Source register

01 / The customer's job

Enterprise AI
Architecture

Lessons from large-scale systems

01 / 36

Enterprise AI Architecture

Budget: 1.0 minutes.

This is my personal engineering perspective, not an Adobe statement, roadmap or disclosure of internal results. The examples are illustrative; the research is attributed to public sources. We're not claiming we've solved this.

  1. OPEN
    Building AI products and using AI to build software raise many of the same questions. These are engineering lessons, not a product announcement.
  2. SET EXPECTATIONS
    Some of you are further down the path. Some are earlier. Some are on a completely different path, and that's okay.
  3. THE THREAD
    What does the customer want? What does the agent need to know? How do we trust what it does?

NEXT SLIDE: Start with an illustrative campaign demo and the job it might miss.

Evidence: Personal perspective / not an employer statement

01 / The customer's job

Meet customers
where they're at.

The demo: AI does the whole job

The idea

The creative

The campaign

Illustrative customer needs

Creative control

Keep the part of the job they care about.

Less management

Dates, follow-up, delivery and budgets.

Their context

Previous campaigns, data and decisions.

Their team

Marketing isn't a single-person job.

02 / 36

Meet customers where they're at.

Budget: 2.5 minutes.

Imagine a compelling demo around "Help me generate a campaign." It shows the power of AI. But a marketer might want help managing the work while keeping creative control. This is an illustrative scenario, not a report of customer interviews or internal demos.

  1. THE DEMO
    Imagine a demo that generates a campaign from beginning to end, as if AI should do the entire job.
  2. BUILD 1: THE POSSIBLE NEED
    The marketer might want help with management, dates and follow-up, without giving up the creative part.
  3. WHAT IT CAN MISS
    Previous campaigns, data and team coordination. Marketing doesn't suddenly become a single-person job.
  4. LAND
    The impressive demo isn't necessarily the useful product. Find the actual job before automating it.

Builds: advance with Right/Space when ready. In Flat mode, the graphic is already complete.

NEXT SLIDE: This isn't just a product lesson. It's also a lesson about what we tell our agents.

Evidence: Illustrative example / not customer research

01 / The customer's job

Not just what.
Why.

01

What

What do you want built?

02

Why

What job are you trying to get done?

03

Decisions

What can the agent infer from that?

Doing what I asked isn't the same as doing what I meant.

03 / 36

Not just what. Why.

Budget: 1.5 minutes.

AI will do a lot more for you than you think it can. But it needs to understand what you're trying to accomplish, not just what you want built.

  1. WHAT
    I can give an agent a complex job and send it on its way. More capability doesn't fix a misunderstanding of the job.
  2. BUILD 1: WHY
    Why do we want this? Who is it for? What job are we trying to accomplish? That's setting the intent.
  3. BUILD 2: DECISIONS
    When the model and harness understand why, they have clues for the decisions we didn't spell out. That's different from handholding every step.
  4. LAND
    The same tools can support very different workflows. We'll come back to the public AWS report when we talk about engineering.

Builds: advance with Right/Space when ready. In Flat mode, the graphic is already complete.

NEXT SLIDE: That intent belongs in the project, not just in the prompt I happen to type today.

Evidence: Working thesis

02 / The situation

It knows projects.
It doesn't know your project.

The code and README aren't the whole story.

04 / 36

It knows projects. It doesn't know your project.

Budget: 1.5 minutes.

I have all this context in my head. My agent doesn't, unless I give it a way to get it. That's why it can make a decision that seems dumb to me.

  1. THE GAP
    The harness is the system around the model: tools, instructions and feedback. For architecture, delivery and scale, the code and README don't tell it everything.
  2. PROJECT INTENT
    What is this project trying to accomplish? Who are the customers? Why would they use ours instead of a competitor?
  3. MY PRACTICE
    You might use SpecKit or Superpowers. I use Compound Engineering for project context. The principle isn't which plugin you use. Capture why, not just what.
  4. LAND
    It can't infer the context from thin air. Give it the clues you already have.

NEXT SLIDE: Consider the same model and harness working without those connections.

Evidence: Architecture principle

S02

02 / The situation

Your agent shouldn't
be on an island.

Code + repo Wiki + decisions CI + deploy Slack + people Customers + intent Memory Your agent

Same model. Same harness. Different context.

05 / 36

Your agent shouldn't be on an island.

Budget: 1.5 minutes.

Imagine two agents with the same harness and model. One has authorized project context; the other is operating on an island. This is a conceptual comparison, not an evaluation of a colleague or an internal system.

  1. THE ISLAND
    There's the agent, sitting under a palm tree. Same model, same harness, but no idea what's happening off the island.
  2. BUILD 1: ENGINEERING
    Connect the repo and pipeline within the user's permissions. It should know what green means and which deployment lane is authorized.
  3. BUILD 2: PEOPLE + DECISIONS
    Connect the Wiki and team channels. What did we decide? Who do we work with? These are sources to evaluate, not instructions to obey.
  4. BUILD 3: PURPOSE + MEMORY
    It needs the customers, the purpose and the history. A web agent with these connections could be just as capable. Connected versus isolated is the point.

Builds: advance with Right/Space when ready. In Flat mode, the graphic is already complete.

NEXT SLIDE: Here's a short instruction that makes all of those connections matter.

Evidence: Capability comparison / not browser versus terminal

02 / The situation

"Create an MR and keep the
pipeline green through QA."

01

Create MR

WaitingOpened
02

CI + review

WaitingGreen
03

Merge

WaitingMerged
04

Deploy to QA

WaitingIn QA
05

Verify QA

WaitingGreen

Required checks and approvals still apply. Red means stop and fix, not skip.

06 / 36

"Create an MR and keep the pipeline green through QA."

Budget: 1.5 minutes.

In this example, the agent understands the workflow. It isn't five unrelated prompts. It's one bounded job in an environment it knows, with checks and authorization at every consequential step.

  1. CREATE MR
    MR means merge request, as in GitLab. In this example the agent knows the repo, branch conventions and intent.
  2. BUILD 1: CI + REVIEW
    Watch the pipeline. Follow up on failures and review feedback. Keep it green; don't just report that an MR exists.
  3. BUILD 2: MERGE
    If the checks and required approvals are satisfied, and I've authorized it, merge it.
  4. BUILD 3: DEPLOY
    Then follow the deployment to QA. It knows which lane I mean.
  5. BUILD 4: VERIFY
    QA also has to stay green. If something fails, stop and surface it. Knowing the workflow doesn't grant extra authority.

Builds: advance with Right/Space when ready. In Flat mode, the graphic is already complete.

NEXT SLIDE: I don't want to explain that environment from scratch every time. It should learn along the way.

Evidence: Illustrative GitLab workflow / not a live demo

02 / The situation

It should learn
along the way.

Related / compounding memory
the wiki is a persistent, compounding artifact.
Andrej KarpathyLLM Wiki author / 2026-04-04 / S16
01

Work

Use the agent

02

Capture

Decisions + corrections

03

Correct

Review + update

04

Recall

When it's relevant

Bring it back when it's relevant

Memory should compound. It should also be correctable.

07 / 36

It should learn along the way.

Budget: 1.5 minutes.

The wiki is a persistent, compounding artifact. I think that's critical. I want hooks into the work, not another document I have to maintain separately.

  1. THE QUOTE
    Karpathy's line is the principle: the useful understanding shouldn't disappear when the chat ends.
  2. BUILD 1: CAPTURE
    Capture the decisions, the things it got wrong that I corrected, and the important work I'm focused on.
  3. BUILD 2: CORRECT
    Automatically captured doesn't mean automatically true. Keep the source, date and scope. Let me correct or delete it; recheck permissions.
  4. BUILD 3: RECALL
    Bring it back when it's relevant: customers, tools, people, why we're doing the work. It doesn't matter which memory system I use if it works and stays current.

Builds: advance with Right/Space when ready. In Flat mode, the graphic is already complete.

NEXT SLIDE: That doesn't mean putting everything we've ever learned into one huge prompt.

Evidence: Related perspective / memory requires governance

S16 / S02

02 / The situation

Not one huge prompt.
A way to get the right context.

01IntentWhat job are we trying to get done?
02MeaningOur words, glossary and working rules
03Live statePeople, work, dates and dependencies
04MemoryDecisions and things we've corrected
05AuthorityWhat's risky? What's allowed?
08 / 36

Not one huge prompt. A way to get the right context.

Budget: 1.5 minutes.

I shouldn't have to supply all the context in a huge prompt. The agent needs a way to pull the right context from the project, the systems and memory.

  1. BREAK IT DOWN
    The job, the meanings of our words, the systems we interact with, the decisions we've made.
  2. GET IT WHEN NEEDED
    Curate the tools and make sure they work in your environment. More context isn't automatically better context.
  3. THE BOUNDARY
    It should know what's risky and what's not. But permissions and approvals are enforced by the system, not by asking the model to remember them.
  4. LAND
    Context makes a capable model more useful. It doesn't make a wrong answer impossible.

NEXT SLIDE: The same context principle applies to a marketer using an agent connected through MCP.

Evidence: Conceptual decomposition

S02 / S12

02 / The situation

They bring the agent.
The provider supplies context.

Customer brings

Agent + harness

  • Their model
  • Their tools and skills
  • Their memory and instructions
Provider supplies

Work tools + context

  • Who they are and their role
  • What work they interact with
  • What those objects mean

A conceptual integration boundary, not a product announcement.

09 / 36

They bring the agent. The provider supplies context.

Budget: 1.5 minutes.

In this architecture, a customer brings an agent, model, tools and skills. A work-management provider exposes authorized tools and context. This is a general design pattern, not a statement about a deployed or planned Adobe service.

  1. THE HANDOFF
    MCP can expose a provider's tools to an agent. The context that matters to an engineer matters just as much to someone managing marketing work.
  2. THE REQUEST
    "Help me make sure the right people are assigned to this work." Which people? Which work? What does this person normally do?
  3. THE PERSPECTIVE
    Are they assigning work, running reports, getting the executive view, or creating an ad? The role changes which data is useful.
  4. LAND
    Tools expose actions. Authorized context helps the agent understand the work. Its harness still decides what to use.

NEXT SLIDE: If you don't know Workfront, let me describe the work we're talking about.

Evidence: Conceptual system boundary / not a current product contract

02 / The situation

Workfront:
Jira for marketers.

01

Campaigns

02

People + assets

03

Approvals

04

Calendars

05

Results

Work records here. Assets and results can live in connected systems.

10 / 36

Workfront: Jira for marketers.

Budget: 1.0 minutes.

Workfront is a public example of work-management software. Think of Jira for marketers: a useful analogy, not a claim that they're the same product. This description uses public Adobe documentation, not internal product information.

  1. THE ANALOGY
    Jira tracks engineering work. Workfront tracks work around marketing campaigns: who's doing it, what needs approval, and when it's going out.
  2. THE CONNECTIONS
    The assets, creative tools and results can be in other systems. The useful part is connecting them to the people and workflow.
  3. LAND
    There's a lot of detail here about what customers did and how they did it. That matters for what they should do next.

NEXT SLIDE: Now the general business question: if software is cheap to build, what remains valuable?

Evidence: Simplified analogy / capabilities vary by package and integration

S25 / S26

02 / The situation

When software is cheap,
what remains valuable?

Context + trust

HistoryWhat we did
MeaningHow + why
OutcomesWhat worked
Capability

Frontier
models

General reasoning

Customer value

The next
decision

Informed by their work

Useful context + general capability = a better next decision.

11 / 36

When software is cheap, what remains valuable?

Budget: 2.0 minutes.

A moat is a business's defense against competition. Public writers debate whether agents strengthen or weaken systems of record. My personal hypothesis is that useful, authorized work context can matter. This isn't an Adobe strategy or a guarantee that incumbents win.

  1. THE CASTLE
    Why pay for a work-management product? It isn't just because nobody else can build a project-management screen.
  2. BUILD 1: THE MOAT
    What did the customer do before? How and why did they do it? What worked? Add the approvals, permissions and team coordination. That's useful work context, not just a pile of data.
  3. BUILD 2: THE NEXT DECISION
    The frontier models bring general capability. Systems of record and their connections bring the work context. Put those together so the next decision, and the one after that, can benefit from what happened before.
  4. LAND
    Customers own their data. Providers have to earn usefulness and trust. That's a general value hypothesis, not a claim about a company's roadmap.

Builds: advance with Right/Space when ready. In Flat mode, the graphic is already complete.

NEXT SLIDE: That can accelerate a lot of work. But the processes around that work weren't built for this speed.

Evidence: Strategic hypothesis / usefulness and trust have to be earned

S12 / S27

02 / The situationOptional depth

A useful memory has an owner.

01PreferenceUser-owned; easy to correct
02DecisionSource, participants and constraints
03ProcedureVersioned; replayed before reuse
04AuthorityRechecked at read and action time
12 / 36

A useful memory has an owner.

Optional depth / Budget: 1.5 minutes.

Ask who owns a budget fact versus a working preference. A remembered trajectory can be useful; it can also perpetuate an old error. Explain provenance, permission revocation and deletion. These are proposed rules, not a claim that every current agent implements them.

NEXT SLIDE: Memory can be large and still unhelpful.

Evidence: Optional depth / proposed operating rules

S16 / S14

02 / The situationOptional depth

More context can make
the system worse.

Context, therefore, must be treated as a finite resource with diminishing marginal returns.
Anthropic Applied AI teamAnthropic Applied AI team / 2025-09-29 / S02

Retrieve the smallest useful, current, authorized set.

13 / 36

More context can make the system worse.

Optional depth / Budget: 2.0 minutes.

Talk about conflicting facts, stale instructions and attention limits. Large context windows don't eliminate relevance decisions. Test context contribution separately from the model and harness; more tokens are not automatically more value.

NEXT SLIDE: The strategic question is where the valuable memory should live.

Evidence: Related caution / attention is a finite resource

S02

02 / The situationOptional depth

Who owns the memory?

The user? The employer? The product? The agent platform?

14 / 36

Who owns the memory?

Optional depth / Budget: 1.5 minutes.

Use this as a short discussion question, not an answer we already know. Portability, tenant boundaries and the right to correct matter. A moat built by trapping customer data is different from a moat built by making it useful.

NEXT SLIDE: Bring the audience back to the next constraint: faster work running into processes built around humans.

Evidence: Optional depth / open strategic question

S12 / S13

03 / The production system

We accelerate right
into a brick wall.

Create Drafts Human validation Delivered AI work Review + test Same capacity

Faster work. The same human validation.

15 / 36

We accelerate right into a brick wall.

Budget: 1.5 minutes.

Our processes were built around humans doing the work and making the decisions. AI can accelerate parts of that work. Then it runs straight into the validation that still waits for a human.

  1. THE OLD PROCESS
    This isn't just software. Marketing approvals and a lot of other work were designed around human speed.
  2. BUILD 1: MORE WORK
    We build more, faster. But more drafts aren't more delivered value. They can just make the queue bigger.
  3. BUILD 2: THE BRICK WALL
    We don't trust the result yet, so someone has to look at it and fix the last few things. That doesn't scale at the same rate.
  4. LAND
    The next unlock isn't just faster generation. It's verification that can keep up.

Builds: advance with Right/Space when ready. In Flat mode, the graphic is already complete.

NEXT SLIDE: Consider a review workflow where automation can remove a manual handoff.

Evidence: Conceptual queue / no measured throughput or universal speedup

S01

03 / The production system

My agent reviewing
your agent.

Implementation

Build agent

Change + intent

Manual dispatch

Person starts
review

Automatic trigger

Workflow
starts review

Challenge

Review agent

Findings + evidence

Required human approval still applies.

Automate the handoff. Keep the checks and accountability.

16 / 36

My agent reviewing your agent.

Budget: 1.5 minutes.

Imagine one person using an agent to build a change and another person starting an agent to review it. Automating the dispatch removes a manual step. This is a proposed workflow, not a report about an employer's review process.

  1. THE RELAY
    My agent builds it. You start your agent to review it. In the end, it's my agent reviewing your agent, or the other way around.
  2. BUILD 1: AUTOMATE THE HANDOFF
    Why are we waiting for a person to start the review? Trigger it from the workflow, with the intent, context and actual change attached.
  3. THE REVIEW STILL HAS TO WORK
    More review capacity doesn't prove better review. Challenge the change, reproduce findings, and check behavior. Another model can share the same blind spots.
  4. LAND
    Automating dispatch doesn't replace a required human approval. That policy still applies. It does remove a manual step we don't need to keep doing.

Builds: advance with Right/Space when ready. In Flat mode, the graphic is already complete.

NEXT SLIDE: The bigger job is building the checks that let us trust the whole workflow.

Evidence: Illustrative review pattern / not proof of correctness

S03 / S18

03 / The production system

The next unlock:
verification that scales.

01

Intent

Job + constraints

02

Review

Challenge the change

03

Exercise

End-to-end tests + evals

04

Outcome

Check the actual final state

Trust the system's evidence. Not the agent's self-report.

17 / 36

The next unlock: verification that scales.

Budget: 1.5 minutes.

I think agents can take on entire, bounded jobs. We don't get there by trusting harder. We need tools that check the work from beginning to end. That's an engineering challenge, not a claim of completed production assurance.

  1. INTENT
    The right context matters here too. What is the job? What does correct mean? Which constraints cannot change?
  2. BUILD 1: REVIEW
    Challenge the implementation. Don't just ask another agent whether it looks good.
  3. BUILD 2: EXERCISE
    Run it beginning to end in a realistic, isolated environment. Protect the checks from being rewritten to make the agent pass.
  4. BUILD 3: OUTCOME
    "I've updated the plan" isn't evidence. Are the right records changed, the permissions respected, and the constraints preserved? Calibrate subjective graders with people.
  5. LAND
    That can let us stop watching every step of a job that has earned that freedom. It doesn't mean automating every human role or ignoring release authority.

Builds: advance with Right/Space when ready. In Flat mode, the graphic is already complete.

NEXT SLIDE: And we have to judge that evidence against the right baseline.

Evidence: Proposed assurance architecture / checks must be meaningful and independent

S03 / S06

03 / The production system

The same standard.
Not a demand for perfection.

Tesla FSD + human supervision

Illustration, not a product demonstration.

North America / rolling 12 months

Miles per major collision

FSD (Supervised)

5.69M

Manual + active safety

2.08M

2.7×as many miles between
recorded major collisions

Reported association. Not proof of fewer deaths.

Measure against a baseline. Then ask what the comparison actually proves.

18 / 36

The same standard. Not a demand for perfection.

Budget: 1.5 minutes.

Humans aren't perfect either. Driving makes that intuitive, but the numbers have to earn the claim. Tesla publishes a useful collision comparison. It doesn't establish that we've solved driving, or that autonomous software is safe.

  1. THE QUESTION
    We know people make mistakes. Does this system do the job better, and how would we know?
  2. BUILD 1: SUPERVISED
    This is FSD with a person still responsible for supervising. The car is an illustration, not a product demonstration or an incident reconstruction.
  3. BUILD 2: COMPARE
    Tesla's own North American fleet: 5.69 million miles per recorded major collision with supervised FSD, versus 2.08 million for manual driving with active safety. About 2.7 times as many miles between those collisions. Major means a restraint deployment, not a fatality.
  4. THE LIMIT
    These are manufacturer-reported observations, not a randomized comparison. Road and driver mix can differ. I am not claiming Tesla has a lower fatality rate. Its projected lives saved aren't observed fatality reductions.
  5. SERIOUS FAILURES
    NHTSA's reduced-visibility investigation lists nine crashes, including one fatality. That's a subset, not a fleet rate. A better average doesn't excuse a critical failure or earn unsupervised operation.
  6. LAND
    That's the lesson for us: same work, same conditions, same consequences. Define the baseline, measure the failures, and earn more autonomy for bounded jobs. This driving data doesn't prove anything about our software.

Builds: advance with Right/Space when ready. In Flat mode, the graphic is already complete.

NEXT SLIDE: That baseline also tells us what happens when delivery volume increases.

Evidence: Tesla-reported / all road classes / checked Oct 6 / observational, not a fatality rate

S29 / S30 / S31

03 / The production system

If delivery accelerates,
confidence has to scale.

Baseline100 deployments x 5%
5expected failed changes
10x volume1,000 deployments x 5%
50expected failed changes
Stronger assurance1,000 deployments x 0.5%
5expected failed changes

Comparable changes assumed. The failure probability can change too.

19 / 36

If delivery accelerates, confidence has to scale.

Budget: 1.5 minutes.

Expected failed changes equal deployment volume times change-failure probability under comparable conditions. 100 at 5% gives 5; 1,000 at 5% gives 50; 1,000 at 0.5% gives 5. AI may change both volume and failure probability. DORA describes AI as an amplifier of the surrounding system. Don't claim an empirical law of linear AI outages.

NEXT SLIDE: To scale that confidence, we have to measure the whole system.

Evidence: Conditional illustration / failed changes are not necessarily outages

S11

03 / The production system

Choose a system,
not just a model.

01ModelGeneral reasoning and generation
02Agent harnessTools, context, state, budgets, recovery
03Eval harnessTasks, traces, grading, repeatability
04EnvironmentAuthority, integrations, operational limits
20 / 36

Choose a system, not just a model.

Budget: 1.5 minutes.

Define the agent harness and evaluation harness separately. They are not interchangeable. A native coding agent and a common benchmark harness are different systems even with the same model. We will return to the engineering environment in the final act.

NEXT SLIDE: A benchmark can still teach us something if we respect that boundary.

Evidence: Terminology / distinct responsibilities

S03 / S18

03 / The production system

Capability and cost
don't move together.

Benchmark pass rate 0%25%50%75%100% $1$10$100 Average USD / attempted task (log scale) gpt-6-astra [xhigh]: 74.12%, +/-2.87 percentage points, $4.43 per attempted task gpt-6-astra gemini-3.8-flash [high]: 73.83%, +/-1.42 percentage points, $2.36 per attempted task gemini-3.8-flash claude-opus-5 [max]: 73.65%, +/-3.87 percentage points, $11.84 per attempted task claude-opus-5 glm-5.3 [max]: 68.96%, +/-3.02 percentage points, $3.99 per attempted task glm-5.3 claude-opus-4.8 [max]: 58.97%, +/-1.76 percentage points, $13.22 per attempted task claude-opus-4.8 claude-sonnet-5 [max]: 53.85%, +/-4.24 percentage points, $26.40 per attempted task claude-sonnet-5
Selected configurations
Model / effortPassUSD
gpt-6-astra [xhigh]74%$4.43
gemini-3.8-flash [high]74%$2.36
claude-opus-5 [max]74%$11.84
glm-5.3 [max]69%$3.99
claude-opus-4.8 [max]59%$13.22
claude-sonnet-5 [max]54%$26.40

DeepSWE v1.1 / 113 tasks
Checked 2026-10-05
Bars: 95% run-to-run intervals; rates rounded.

21 / 36

Capability and cost don't move together.

Budget: 2.0 minutes.

Read the axes before discussing models. X is average USD per attempted task; Y is benchmark pass rate. The common harness holds one part of the system constant. Selected points do not establish a universal best model or a complete Pareto frontier. Quote the checked version and uncertainty caveats. No speed inference from output tokens or steps.

NEXT SLIDE: Another benchmark makes the cost of reasoning effort visible.

Evidence: Selected verified rows / conflicting-price rows excluded / not native-agent performance

S17

03 / The production system

More thinking has a price.

Artificial Analysis Intelligence Index 4045505560 $0.1$0.2$0.5$1$2$5$10 Cost per task (USD, log scale) Opus 5.5 Sonnet 5.5 Fable 5.1 GPT-6 Astra Gemini 4 Argon GPT-6.1 Sol Muse Spark 1.3 Grok 4.7 MiMo-V2.6-Pro Claude Opus 5.5 (max with fallback) / Anthropic: Index 58, $5.98 per Index taskClaude Sonnet 5.5 (max with fallback) / Anthropic: Index 56, $7.67 per Index taskClaude Opus 5.5 (xhigh with fallback) / Anthropic: Index 56, $3.46 per Index taskClaude Opus 5.5 (high with fallback) / Anthropic: Index 54, $1.82 per Index taskClaude Fable 5.1 (max with fallback) / Anthropic: Index 53, $7.63 per Index taskClaude Fable 5.1 (xhigh with fallback) / Anthropic: Index 53, $5.98 per Index taskGPT-6 Astra (max) / OpenAI: Index 53, $3.26 per Index taskGemini 4 Argon (high) / Google: Index 53, $1.99 per Index taskGPT-6 Astra (xhigh) / OpenAI: Index 52, $2.31 per Index taskClaude Sonnet 5.5 (xhigh with fallback) / Anthropic: Index 52, $2.75 per Index taskGPT-6.1 Sol (max) / OpenAI: Index 52, $0.72 per Index taskClaude Opus 5.5 (medium with fallback) / Anthropic: Index 51, $1.34 per Index taskClaude Fable 5.1 (high with fallback) / Anthropic: Index 51, $3.91 per Index taskGPT-6.1 Sol (xhigh) / OpenAI: Index 51, $0.39 per Index taskGPT-6 Astra (high) / OpenAI: Index 51, $1.73 per Index taskGPT-6.1 Sol (high) / OpenAI: Index 50, $0.32 per Index taskGPT-6 Astra (medium) / OpenAI: Index 50, $1.54 per Index taskClaude Fable 5.1 (medium with fallback) / Anthropic: Index 49, $2.98 per Index taskMuse Spark 1.3 (max) / Meta: Index 48, $1.60 per Index taskGPT-6.1 Sol (medium) / OpenAI: Index 48, $0.21 per Index taskClaude Fable 5.1 (low with fallback) / Anthropic: Index 47, $2.37 per Index taskClaude Sonnet 5.5 (high with fallback) / Anthropic: Index 47, $1.12 per Index taskGrok 4.7 (xhigh) / SpaceXAI: Index 46, $3.74 per Index taskGrok 4.7 (high) / SpaceXAI: Index 46, $2.73 per Index taskMiMo-V2.6-Pro / Xiaomi: Index 46, $0.13 per Index task Model lab AnthropicOpenAIGoogleMetaSpaceXAIXiaomi

Top 25 current settings / Intelligence Index v4.3.2 / score axis zoomed to 40-60

22 / 36

More thinking has a price.

Budget: 2.0 minutes.

Here's the top 25 currently listed settings from Artificial Analysis. Each dot is a model configuration, not a whole lab. Changing effort moves the same model to a different place on the capability/cost trade-off.

  1. READ THE AXES
    Up is the Artificial Analysis Intelligence Index. Right is more dollars per benchmark task. Cost is logarithmic; the score axis zooms to 40-60. The score isn't a pass percentage.
  2. FOLLOW ONE MODEL
    Follow Opus 5.5 from medium to high, xhigh and max. Those dots are connected because they're the same model version. We don't connect Opus to Sonnet or Fable just because they're all Anthropic.
  3. THE TRADE-OFF
    More effort can buy a higher score, but it can also buy very little movement. Compare that against the cost. Colors and shapes identify the model's lab; lines join observed effort settings, not an interpolated performance curve.
  4. THE LIMIT
    These are the first 25 configurations in the current table, including its order for ties. Names retain the publisher's fallback settings. Cost is a weighted mean of token-use costs across Index tasks, including caching. Different tasks from DeepSWE, so don't compare those dollars as if the workload were the same. It isn't our cost per accepted outcome, and this doesn't select a universal best model.

NEXT SLIDE: In a product, even benchmark cost isn't the metric we ultimately pay for.

Evidence: Displayed leaderboard values / checked Oct 6 / benchmark tasks, not enterprise delivery

S32 / S33 / S34

03 / The production system

The cheaper call can be
the more expensive outcome.

Option A / 100 attempted tasks
$1050 accepted
$0.20per accepted outcome
Option B / 100 attempted tasks
$1590 accepted
$0.17per accepted outcome

All attempts + verification cost / accepted outcomes

23 / 36

The cheaper call can be the more expensive outcome.

Budget: 2.0 minutes.

Over 100 tasks: A costs $10 for 50 accepted outcomes; B costs $15 for 90. B has the cheaper accepted outcome. Actual accounting includes retries, repeated context, tools, compute, verification and rework. Development costs are different from recurring shipped-feature costs. Don't manufacture a current per-token price table.

NEXT SLIDE: Now apply these same principles to the engineering agent.

Evidence: Illustration, not provider pricing / equal task mix assumed

03 / The production systemOptional depth

A test result isn't
a deployment guarantee.

Controlled tests

Possible gains

  • Accuracy
  • Calls and tokens
  • Elapsed time
Deployment setting

What changes?

  • Instructions?
  • Retrieval behavior?
  • Task and role mix?
24 / 36

A test result isn't a deployment guarantee.

Optional depth / Budget: 1.5 minutes.

A controlled evaluation might improve accuracy, calls, tokens or time without predicting performance in a different deployment. These are hypothetical metrics, not internal results. Compare representative tasks and starting states across harnesses, context on and off; repeat trials and validate the grader. Keep holdouts protected. A context effect isn't the same as a harness capability score.

NEXT SLIDE: The grader is part of the measurement system.

Evidence: Optional depth / hypothetical transfer risk; no internal results

S03

03 / The production systemOptional depth

An eval needs real teeth.

01StateDid the right records change?
02ConstraintsDid protected values stay protected?
03JudgmentAre subjective graders calibrated?
04RegressionCan it still do what worked?
05EvidenceIs the trace enough to explain failure?
25 / 36

An eval needs real teeth.

Optional depth / Budget: 2.0 minutes.

Use one project-update example. A language-model grader can be useful but is not objective by naming it an eval. Repeated trials and final state reveal different failure types. Turn observed failures into cases without leaking the holdout to the implementation agent.

NEXT SLIDE: Offline tests also need a production boundary.

Evidence: Optional depth / outcome verification

S03

03 / The production systemOptional depth

What evidence may
we learn from?

Customer consent / tenant isolation / retention / authorized feedback

26 / 36

What evidence may we learn from?

Optional depth / Budget: 1.5 minutes.

Trace collection is not automatically authorized. Discuss consent, scope, retention and how to turn corrections into validated shared behavior without copying private customer data. These are design responsibilities, not an asserted current service implementation.

NEXT SLIDE: Return to the engineering system we control.

Evidence: Optional depth / open governance question

04 / The engineering system

The same problem.
Now it's how we build.

From shipping AI to using AI to ship.

27 / 36

The same problem. Now it's how we build.

Budget: 1.0 minutes.

We've covered assurance for customer agents and used an illustrative code-review workflow to make the bottleneck concrete. Now apply the same principles to engineering workflows, using public research and thought experiments. The outcomes and environments differ; intent, evidence and boundaries still matter.

NEXT SLIDE: AWS saw large gains when teams changed the way they worked.

Evidence: Explicit transition / customer product to engineering workflow

04 / The engineering system

The gain was in
the way they worked.

Aligned / ways of working
frontier engineering is about intentionally changing the way that you work.
Clare LiguoriSenior Principal Engineer, AWS / 2026-08-28 / S01

About half reported under 3x. The detailed subgroup account qualifies the headline.

50 teams

Ordinary teams in the pilot

4.5x

Reported median in the higher-gain group

>10x

Some reported deployment-velocity gains

ContextIntentChecksDeliver Job

Conceptual job flow, not pilot throughput.

28 / 36

The gain was in the way they worked.

Budget: 2.5 minutes.

Clare Liguori's August 28, 2026 talk: 50 teams; about half under 3x; the higher-gain group median 4.5x, some above 10x. About 90% used Kiro alongside other tools. The detailed account qualifies an earlier headline. Themes: feed agents, make intent explicit, accept initial slowdown, fast local test feedback. Decision speed becomes the bottleneck. This is not a controlled universal productivity multiplier.

  1. THE REPORT
    Fifty teams. About half reported less than three times the deployment velocity. The higher-gain group had a median of 4.5 times, with some above ten. Not every team got ten times faster.
  2. THE QUOTE
    The important part is changing the way we work, not just swapping the tool.
  3. BUILD 1: THE WHOLE JOB
    Give the job its context, intent and feedback. Let the agent carry it through the workflow instead of handholding every step. The moving package is a metaphor, not a measurement of the pilot.
  4. LAND
    Those gains were reported by teams changing how they worked. They aren't a universal multiplier or a claim about inference speed.

Builds: advance with Right/Space when ready. In Flat mode, the graphic is already complete.

NEXT SLIDE: That points toward a factory, but there is a real disagreement about confidence.

Evidence: Speaker-reported observational pilot / deployment velocity, not inference speed

S01

04 / The engineering system

No line-by-line review
is not no verification.

Factory position
Code must not be reviewed by humans
Justin McCarthyCo-founder and CTO, StrongDM / 2026-02-06 / S06
Counterposition: human understanding
Making code cheap to generate doesn’t make understanding cheap to skip.
Addy OsmaniEngineering leader / Comprehension Debt author / 2026-03-14 / S20
Change Evidence checkpoint Release owner Illustration, not live pass or approval status.

My direction: protected scenarios + service simulations + accountable release decisions.

29 / 36

No line-by-line review is not no verification.

Budget: 2.5 minutes.

StrongDM advocates no human code writing or review, with external scenarios and simulated services. Addy Osmani's March 2026 counterposition is that tests and specs cannot fully replace comprehension. Don't brush it aside: protected scenarios, realistic service simulations and human accountability are additional layers, not proof that every missing requirement has been captured. My personal stance is to explore bounded dark-factory workflows, not assert an employer's release policy.

  1. TWO POSITIONS
    StrongDM says humans don't have to write or review the code. Addy says tests don't make understanding optional. That's a real disagreement, not a quote I can just wave away.
  2. BUILD 1: THE CHECKPOINT
    The change reaches a checkpoint, not an automatic shipping chute. Protected scenarios, realistic services and an accountable release decision. Those checks need real teeth.
  3. LAND
    I'm advocating bounded exploration of this direction. This isn't an employer policy, and the animation doesn't prove the checks work.

Builds: advance with Right/Space when ready. In Flat mode, the graphic is already complete.

NEXT SLIDE: Here's a thought experiment for the confidence problem.

Evidence: Competing positions / independent checks don't make understanding optional

S06 / S20

04 / The engineering system

What if you couldn't
read the implementation?

The implementation

No comfy code review.

Outcome

Did it do the job?

Constraints

What must never change?

Failure paths

What happens when it breaks?

Observability

Can we explain the result?

In a sandbox: how would you prove the behavior is right?

30 / 36

What if you couldn't read the implementation?

Budget: 2.5 minutes.

Try a language you don't know in a sandbox. What evidence would replace a comfortable code review? Behavioral contracts, protected scenarios, properties, fault injection and observable outcomes. Language expertise still matters for runtime, concurrency, security and incident response. Ignorance is not a safety mechanism.

  1. THE BOX
    You can't read the implementation. Give that a second. What would you need before you trusted it?
  2. BUILD 1: THE JOB
    Start with the actual outcome and the constraints. Not whether the agent sounds confident, and not whether the code looks familiar.
  3. BUILD 2: BREAK IT
    Exercise failures and make the behavior observable. The probes are questions to answer, not live tests that have passed.
  4. LAND
    This is a sandbox thought experiment. Not knowing the language isn't a safety mechanism. The point is to build evidence that scales beyond a comfortable code review.

Builds: advance with Right/Space when ready. In Flat mode, the graphic is already complete.

NEXT SLIDE: A workflow upgrade can create that same confidence gap.

Evidence: Thought experiment / not a policy to remove language expertise

04 / The engineering systemOptional depth

Fast by which measure?

AWS pilot

Deployment velocity

  • Observational teams
  • Changed ways of working
  • Large reported subgroup gains
METR update

Task completion time

  • Historical slowdown is dated
  • Newer gains likely increased
  • Current magnitude is unreliable
31 / 36

Fast by which measure?

Optional depth / Budget: 1.5 minutes.

Don't set up AWS versus METR as a winner-take-all contradiction. METR studied experienced repository contributors on selected tasks; AWS reports deployment velocity for teams changing their work. The February 2026 update warns about selection effects and concurrent-agent timing. Both caution us to define the metric and setting.

NEXT SLIDE: External verification is one approach to scaling confidence.

Evidence: Optional depth / different populations, dates and measures

S01 / S09 / S10

04 / The engineering systemOptional depth

Move confidence outside
the implementation.

01ContractBehavior and constraints before code
02ScenariosHeld out from the implementation agent
03EnvironmentRealistic services and isolated tests
04OperationsSafe rollout, observed health, rollback
05AccountabilityA person owns the release decision
32 / 36

Move confidence outside the implementation.

Optional depth / Budget: 2.0 minutes.

Discuss StrongDM's external scenarios and digital-twin environment as a case, not a mandatory platform design. Holdouts need meaningful coverage and protection. A factory can automate code production without automating away responsibility.

NEXT SLIDE: And we still need to decide where autonomy is appropriate.

Evidence: Optional depth / patterns, not universal proof

S06 / S07 / S21

04 / The engineering systemOptional depth

Which work has earned
that much autonomy?

Reversible actions first. Bigger authority needs stronger evidence.

33 / 36

Which work has earned that much autonomy?

Optional depth / Budget: 1.5 minutes.

Use the reversibility and blast-radius lens. A bounded branch change is different from a production schema migration. Ask what evidence, approval and recovery each action needs. Don't imply every enterprise should ship unreviewed code tomorrow.

NEXT SLIDE: A more capable model doesn't automatically preserve the old confidence.

Evidence: Optional depth / open architectural question

04 / The engineering system

Better model.
Where are the brakes?

Capability Confidence Changed model + harness Regression risk Challenge+ evidence Illustration, not a measured gain.

A workflow upgrade needs renewed assurance, not just a better model.

34 / 36

Better model. Where are the brakes?

Budget: 2.5 minutes.

Imagine upgrading both the model and harness. Previous assurance doesn't automatically carry over. Repeat representative behavioral checks, review failure paths and validate any new review practice. Changing several variables doesn't isolate a cause. Agreement between agents isn't proof, and adding a reviewer doesn't establish a measured failure-rate improvement.

  1. THE PIT STOP
    Imagine a new model and a changed harness. More engine. Have we renewed the assurance around it?
  2. BUILD 1: THE CHANGE
    Several variables changed. Recheck the workflow and its failure paths rather than assuming capability means reliability.
  3. BUILD 2: THE BRAKES
    A different-model adversarial reviewer can challenge the change. Reproduce its findings and check behavior; a second opinion alone isn't evidence of fewer failures.
  4. LAND
    Agreement between agents isn't proof. The brakes are meaningful evidence and an accountable release decision, not a reassuring second opinion.

Builds: advance with Right/Space when ready. In Flat mode, the graphic is already complete.

NEXT SLIDE: That's how we build toward autonomy that has earned its authority.

Evidence: Hypothetical upgrade / not an incident report or measured result

04 / The engineering system

Autonomy has to
earn its authority.

Models

Make more possible.

Context

Makes it useful.

Engineering

Makes it trustworthy.

Bounded authority

An earned permission.
Not a blank check.

Independent evidence. A person owns the release.

35 / 36

Autonomy has to earn its authority.

Budget: 1.5 minutes.

I think we should build toward agents doing real engineering work without a person reading every line. We don't get there by trusting harder. We get there by engineering intent, context, evidence, boundaries and feedback that scale. The same principles apply to customers and to the agents building our software.

  1. BRING IT TOGETHER
    Models make more possible. Context makes it useful. Engineering makes it trustworthy.
  2. BUILD 1: EARNED AUTHORITY
    The permission at the end is bounded. It follows meaningful evidence and a responsible release decision. This isn't a live approval or a blank check.
  3. LAND
    I want more autonomy. We have to build the system that earns it, not just decide we're going to trust harder.

Builds: advance with Right/Space when ready. In Flat mode, the graphic is already complete.

NEXT SLIDE: These are the questions I'm still working on.

Evidence: Tom's position / more autonomy, independent evidence, human accountability

04 / The engineering system

What are we still
figuring out?

01

Who owns the useful memory?

02

How does context survive a different harness?

03

What evidence is enough to ship?

04

Which work should become a dark factory?

Questions + conversation / final 30 minutes

36 / 36

What are we still figuring out?

Budget: 0.5 minutes.

Leave the final 30 minutes for questions and networking. Use the chapter map or overview to return to evidence. Don't pretend the open questions are already resolved.

  1. OPEN THE ROOM
    These are the questions I'm still figuring out. Pick one of these, or bring one of your own.
  2. LEAVE SPACE
    Use the chapter map to return to a story or source. We have the final thirty minutes for questions and conversation.

NEXT SLIDE: Discussion.

Evidence: Open questions

Public sources

S01 / checked-original

From AI-Assisted to AI-Native: Building a Frontier Development Team

Clare Liguori (Senior Principal Engineer, AWS) / 2026-08-28 (published)

Supports: AWS-reported pilot: 50 normal-distribution teams on existing (non-greenfield) systems; metric = deployment velocity to production; half the teams achieved less than 3x; the higher group had a median of 4.5x and some more than 10x; ~90% of those teams used Kiro among other internal tools; speaker's reading is that it was about how they worked, not the tools. Separate headline at 1:29 ('a median of 4.5x ... sometimes more than 10x') is less qualified than the subgroup account at 6:50-7:04.

Limit: Observational, speaker-reported company account; not independent or causal evidence. Video upload date from YouTube page metadata (2026-08-28, AI Engineer World's Fair 2026 per organizer page). Read the organizer's timestamped transcript (https://ai.engineer/talks/pqlWNihgdjI-youtube-ca313d4e0a12a2daf850), which has automatic-transcription errors (speaker name rendered 'Claire La Gory', Kiro as 'Kuro'); the organizer page also carries AI-written summaries that were NOT used as evidence. Organizer chapters: 0:12, 2:11, 5:41, 8:10, 11:10, 13:33, 15:02, 18:32 (earlier notes' 13:58/15:09 are approximate).

https://youtu.be/pqlWNihgdjI

Checked 2026-10-05

S02 / checked-original

Effective context engineering for AI agents

Anthropic Applied AI team (Prithvi Rajasekaran, Ethan Dixon, Carly Ryan, Jeremy Hadfield) / 2025-09-29 (published)

Supports: Context is a finite resource with diminishing marginal returns; aim for the smallest set of high-signal tokens; just-in-time retrieval, compaction, structured note-taking and sub-agents for long-horizon work.

Limit: Team writing: never attribute to Dario Amodei or any executive. Also contains a contrary line (see Q-C05): agentic design will trend toward 'progressively less human curation' and 'do the simplest thing that works'. Does not establish any context asset as a commercial moat.

https://www.anthropic.com/engineering/effective-context-engineering-for-ai-agents

Checked 2026-10-05

S03 / checked-original

Demystifying evals for AI agents

Anthropic (Mikaela Grace, Jeremy Hadfield, Rodrigo Olivares, Jiri De Jonghe) / 2026-01-09 (published)

Supports: Defines outcome as the final environment state (not what the transcript says); evaluating an agent means evaluating harness and model together; combines code-based, model-based and human graders; model-based graders require calibration with human graders.

Limit: Offline evals are one source of confidence, not proof of production behaviour. Earlier notes said date unverified; page shows 'Published Jan 09, 2026'.

https://www.anthropic.com/engineering/demystifying-evals-for-ai-agents

Checked 2026-10-05

S04 / checked-original

How we built our multi-agent research system

Anthropic (Jeremy Hadfield, Barry Zhang, Kenneth Lien, Florian Scholz, Jeremy Fox, Daniel Ford) / 2025-06-13 (published)

Supports: In Anthropic's data agents use about 4x more tokens than chat and multi-agent systems about 15x; multi-agent systems need tasks valuable enough to pay for the extra performance.

Limit: Ratios are specific to Anthropic's 2025 research workload. The 90.2% internal-eval improvement is not a general productivity figure. Supplementary for the talk (not a primary slide).

https://www.anthropic.com/engineering/multi-agent-research-system

Checked 2026-10-05

S05 / indexed-only

Harness engineering: leveraging Codex in an agent-first world

Ryan Lopopolo (Member of the Technical Staff, OpenAI) / 2026-02-11 (indexed)

Supports: OpenAI team built and shipped an internal beta product with 0 lines of manually-written code; its stated lesson is that the engineering job becomes designing environments, specifying intent and building feedback loops ('Humans steer. Agents execute.').

Limit: Direct fetch was blocked (HTTP 403 via curl; simplified fetch returned only the title). Body text was read only as a search-engine copy of the original page (Glean/Brave), including the authorship line and the quoted sentence; the 2026-02-11 date comes from the earlier indexed metadata and was NOT independently re-verified. Internal agent-first experiment, not enterprise policy; do not equate lines of code or PR volume with customer value.

https://openai.com/index/harness-engineering/

Checked 2026-10-05

S06 / checked-original

Software Factories And The Agentic Moment

Justin McCarthy (co-founder, CTO, StrongDM AI) / 2026-02-06 (published)

Supports: Non-interactive development driven by specs and scenarios; scenarios are end-to-end user stories stored outside the codebase (holdout-like) and validated by an LLM; 'satisfaction' replaces boolean test success; Digital Twin Universe = behavioural clones of Okta, Jira, Slack, Google Docs/Drive/Sheets for high-volume validation. Rules: code must not be written by humans; code must not be reviewed by humans.

Limit: Company-authored account; page byline reads 'February 6th, 2026' (earlier note said date unknown). Experimental team constraint by a security-company AI team; not proof of general production safety. The '$1,000 on tokens today per human engineer' line is a provocation, not a benchmark.

https://factory.strongdm.ai/

Checked 2026-10-05

S07 / checked-original

How StrongDM's AI team build serious software without even looking at the code

Simon Willison / 2026-02-07 (published)

Supports: Eyewitness account (he visited the three-person team in October 2025) of StrongDM's software factory, holdout scenarios and Digital Twin Universe; he calls 'Code must not be reviewed by humans' the most interesting rule and asks how it could be sensible given inhuman LLM mistakes.

Limit: Commentary on a demo, not independent validation of reliability or economics. Willison himself questions the $1,000/day figure and says he hopes lower-spend patterns exist.

https://simonwillison.net/2026/Feb/7/software-factory/

Checked 2026-10-05

S08 / checked-original

Writing about Agentic Engineering Patterns

Simon Willison / 2026-02-23 (published)

Supports: Defines agentic engineering as professional engineers using coding agents that can generate and execute code to amplify existing expertise; keeps 'vibe coding' for coding without paying attention to the code.

Limit: Definition and practitioner perspective, not a controlled result. Supplementary vocabulary slide only.

https://simonwillison.net/2026/Feb/23/agentic-engineering-patterns/

Checked 2026-10-05

S09 / checked-original

Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity

METR / 2025-07-10 (published)

Supports: RCT, 16 experienced open-source developers, 246 real issues: with AI allowed tasks took 19% longer; developers expected a 24% speedup and afterwards still believed AI had sped them up by 20%.

Limit: Must be paired with S10. The page now carries an update stating the historical results no longer reflect current impact. Note METR's 2026 update restates the 2025 result as '20% slowdown' in prose and '19% longer' with CI +2% to +39%; use '19% longer' from the original.

https://metr.org/blog/2025-07-10-early-2025-ai-experienced-os-dev-study/

Checked 2026-10-05

S10 / checked-original

METR: February 2026 uplift update (late-2025 AI tools)

METR / 2026-02-24 (published)

Supports: Raw 2026 estimates point to speedup (original developers who returned: -18% completion time, CI -38% to +9%; newly recruited: -4%, CI -15% to +9%), but METR calls the data an unreliable signal because of selection effects (developers declining to work without AI, lower pay $50/hr vs $150/hr, task-type shifts) and says it is only very weak evidence for the size of the increase while believing developers are likely more sped up now.

Limit: Do not present as a clean 2026 speedup number. Task-level productivity, not customer value or reliability. METR's blog index (read 2026-10-05) listed no later uplift post, but the index may be incomplete and this is not an exhaustive search.

https://metr.org/blog/2026-02-24-uplift-update/

Checked 2026-10-05

S11 / checked-original-partial

State of AI-assisted Software Development 2025

DORA (Google Cloud) / 2025 (year-only)

Supports: Landing page: AI's primary role is as an amplifier of an organization's existing strengths and weaknesses; greatest returns come from the underlying organizational system, not the tools themselves.

Limit: Only the landing page was read; month of release not shown there; full report and any stability statistics NOT reviewed. Supplementary; do not cite numbers.

https://dora.dev/research/2025/dora-report/

Checked 2026-10-05

S12 / checked-original

Clouded Judgement 12.12.25 - Long Live Systems of Record

Jamin Ball (Clouded Judgement; disclaimer references Altimeter Capital) / 2025-12-12 (published)

Supports: Agents don't make systems of record irrelevant: cross-system agents need explicit canonical definitions (which ARR, which table wins), semantic contracts and controlled write boundaries.

Limit: Investor opinion, not empirical proof of incumbent durability.

https://cloudedjudgement.substack.com/p/clouded-judgement-121225-long-live

Checked 2026-10-05

S13 / checked-original

AI's trillion-dollar opportunity: Context graphs

Jaya Gupta and Ashu Garg (Foundation Capital) / 2025-12-22 (published)

Supports: Responds to S12: systems of record remain but the missing layer is decision traces (exceptions, overrides, precedents, cross-system context); they argue agent startups in the execution path have a structural advantage in capturing them and call the accumulated structure a context graph (explicitly not model chain-of-thought).

Limit: CORRECTION: byline is 'Posted Dec 22, 2025'; the earlier note's 'September 29, 2026' page metadata was not found in this pass. Investor thesis with commercial interest; 'incumbents can't capture this' is an argument, not an established fact.

https://foundationcapital.com/ideas/context-graphs-ais-trillion-dollar-opportunity

Checked 2026-10-05

S14 / checked-original

Strands Agents with Clare Liguori (InfoQ podcast)

Clare Liguori interviewed by Thomas Betts / 2026-07-20 (published)

Supports: Liguori (technical lead on the open-source Strands Agents SDK) reports her own test: simple four-sentence instructions achieved an 82.5% pass rate and beat explicit workflows, while steering hooks reached 100% in her case study; advises evaluating by tool calls where possible; discusses procedural memory (20:25).

Limit: Date from page structured metadata (2026-07-20). Accuracy numbers are one case study, not universal. Hooks are runtime guardrails she still pairs with evals; she also notes model instructions 'can't be completely relied upon'. Supplementary (second Liguori source) unless the contrary quote Q-C04 is used.

https://www.infoq.com/podcasts/strands-agents/

Checked 2026-10-05

S15 / checked-original

Writing code is cheap now (Agentic Engineering Patterns)

Simon Willison / 2026-02-23 (announced)

Supports: Delivering new code is nearly free but delivering good code (works, known to work, right problem, error handling, simple, tested, documented, maintainable) is still significantly more expensive.

Limit: Chapter was announced 2026-02-23 (S08); the live chapter has no visible revision date, so the current revision date is unknown. Practitioner argument, not an economic model.

https://simonwillison.net/guides/agentic-engineering-patterns/code-is-cheap/

Checked 2026-10-05

S16 / checked-original

LLM Wiki (gist: llm-wiki.md)

Andrej Karpathy / 2026-04-04 (created)

Supports: Pattern in which an LLM incrementally builds and maintains a persistent, interlinked markdown wiki between raw sources and queries ('the wiki is a persistent, compounding artifact'), rather than re-deriving answers by RAG each time.

Limit: Created April 4, 2026 16:25 per the gist page (1 revision at check time); earlier note said date unknown. It is an idea file, not evidence of accuracy; generated summaries can go stale or be wrong; enterprise authorization is a separate problem.

https://gist.github.com/karpathy/442a6bf555914893e9891c11519de94f

Checked 2026-10-05

S17 / checked-original

DeepSWE leaderboard (v1.1) and methodology

Datacurve (v1.1 post by Wenqi Huang, Peter Jiang) / 2026-09-22 (data-updated)

Supports: Fixed-harness (mini-swe-agent) benchmark of 113 long-horizon tasks across 91 repositories and 5 languages; v1.1 (released 2026-06-15 per changelog; post dated 2026-06-14) grades only the committed patch in a clean isolated container, with structured per-test reports, and drops wall-clock time as unreliable. Pass@1 and average cost per task are plotted per model+reasoning-effort configuration.

Limit: Not DeepSWE-Preview (Agentica/Together AI). The page's own limitations section: one bash tool and shared prompt for every model, so it does not reflect native harnesses (Codex CLI, Claude Code, Cursor, Gemini CLI); open-source >=500-star repos only. Costs depend on provider price changes (see changelog 2026-07-30, 08-14, 08-20, 08-21) and the raw JSON artifact disagrees with the rendered page for 19 configurations (see benchmark.caveats). See benchmark object.

https://deepswe.datacurve.ai/

Checked 2026-10-05

S18 / checked-original

Harness design for long-running application development

Prithvi Rajasekaran (Anthropic Labs team) / 2026-03-24 (published)

Supports: A planner/generator/evaluator harness materially beat a solo run, but the first harness was 'bulky, slow, and expensive'; the author argues each harness component encodes an assumption about what the model can't do and should be stress-tested and removed as models improve; an out-of-the-box LLM evaluator is lenient toward LLM output and a poor QA agent until tuned, and still missed bugs.

Limit: Single-author Labs experiment, not Anthropic corporate position. Both aligned (separate evaluator helps but is not an oracle) and contrary (harness complexity should shrink) for Tom's thesis.

https://www.anthropic.com/engineering/harness-design-long-running-apps

Checked 2026-10-05

S19 / checked-original

Not all AI-assisted programming is vibe coding (but vibe coding rocks)

Simon Willison / 2025-03-19 (published)

Supports: His golden rule for production-quality AI-assisted programming: don't commit code you couldn't explain to someone else; reviewing, testing and being able to explain LLM code is software development, not vibe coding.

Limit: A 2025 position, dated, and not his current stance on all workflows: in 2026 he treats StrongDM's no-human-review approach as an open question worth exploring (S07, S22). The rule is about being able to explain what code does, not specifically line-by-line review.

https://simonwillison.net/2025/Mar/19/vibe-coding/

Checked 2026-10-05

S20 / checked-original

Comprehension Debt - the hidden cost of AI generated code

Addy Osmani / 2026-03-14 (published)

Supports: Argues that excessive reliance on AI code review/generation creates a growing gap between code that exists and code any human understands; tests and detailed specs are necessary but not sufficient substitutes for review.

Limit: Author's page bio (read 2026-10-05) lists him as a Member of Technical Staff at Anthropic working on Claude Code; he also says he relies heavily on tests with coding agents ('I love tests') and does not argue against agents. O'Reilly Radar repost (HTTP 403 to fetch) not used. Cites an Anthropic RCT (52 engineers, 17% lower comprehension quiz score) that was NOT independently checked.

https://addyosmani.com/blog/comprehension-debt/

Checked 2026-10-05

S21 / checked-original

Fragments: February 13 (Martin Fowler)

Martin Fowler (Thoughtworks) / 2026-02-13 (published)

Supports: Fowler notes an increasing feeling that programmers will become supervisors of agents and says they will still be accountable for generated code, while calling the effect of context-switching an open question; the page also relays Margaret-Anne Storey's cognitive-debt post.

Limit: Date inferred from the fragment URL/title (year 2026 from URL); a short blog fragment, not a study. Related fragment 2026-04-02 (https://martinfowler.com/fragments/2026-04-02.html) endorses verification-first reorganisation and wants ways for humans to comprehend larger ranges of tests.

https://martinfowler.com/fragments/2026-02-13.html

Checked 2026-10-05

S22 / checked-original

2026 in LLMs (so far)

Simon Willison / 2026-09-27 (published)

Supports: Annotated keynote notes: calls StrongDM 'six months ahead', says the central question is how to be confident of quality without reading the code and what agents can do to verify their work; says 'code must not be reviewed by humans' was a huge topic of the year and many sessions at the event were about code review.

Limit: Keynote notes, not a controlled study. Establishes that Willison's 2026 stance is exploratory rather than the 2025 golden rule alone.

https://simonwillison.net/2026/Sep/27/2026-in-llms-so-far/

Checked 2026-10-05

S23 / checked-original

Note on 24th September 2026

Simon Willison / 2026-09-24 (published)

Supports: Two-sentence note: working with coding agents has convinced him they make software engineering even harder, requiring extraordinary discipline and knowledge.

Limit: Short note; supportive of 'engineering makes it trustworthy', and a counterweight to 'agents make engineering easy'.

https://simonwillison.net/2026/Sep/24/harder/

Checked 2026-10-05

S24 / checked-original

The Empty Promise of Data Moats

Martin Casado and Peter Lauten (Andreessen Horowitz) / 2019-05-09 (published)

Supports: Argues data scale effects and data network effects are often weaker moats than assumed: the value of incremental data falls while its cost rises; defensibility is not inherent to data itself, though proprietary data can be defensible in some situations (scarce or reticent sources, compliance, custodianship).

Limit: 2019, pre-LLM and about enterprise startups' training-data moats, not about live enterprise context for agents. Page last modified 2024-04-25 per metadata. Tom already concedes raw accumulation isn't sufficient, so this challenges only the stronger moat claim. Use clearly dated.

https://a16z.com/the-empty-promise-of-data-moats/

Checked 2026-10-05

S25 / checked-original-partial

Get started with Adobe Workfront Planning

Adobe Experience League / 2026-04-01 (last-updated)

Supports: Workfront Planning connects operational records and other systems across planning, execution, delivery and measurement. The introduction uses campaign, audience and program-performance questions; record views include timelines and calendars.

Limit: Introduction, availability and setup sections read. Planning access depends on the purchased package; highlighted preview features are not generally available. Jira for marketers is Tom's simplified analogy, not Adobe's quotation or a product equivalence.

https://experienceleague.adobe.com/en/docs/workfront/using/adobe-workfront-planning/adobe-workfront-planning-general-information/planning-overview

Checked 2026-10-06

S26 / checked-original-partial

Unified review and approval overview

Adobe Experience League / publication date unverified (unknown)

Supports: The documentation describes sequential and parallel approval paths, teams and individual participants, deadlines and notifications, and asset review through a Frame.io viewer. It explicitly distinguishes Frame.io comments from the Workfront project.

Limit: FAQ body read; publication/update date not verified. Features and limits vary by package and release. Do not imply all assets, comments or marketing-result data physically reside in Workfront.

https://experienceleague.adobe.com/en/docs/workfront/using/review-and-approve-work/document-approvals-overview

Checked 2026-10-06

S27 / checked-original

Jira as the system of record for OpenAI Symphony

Atlassian / publication date unverified (unknown)

Supports: Atlassian describes pairing Symphony agent orchestration with Jira and Teamwork Graph context. Intent, status, session history, review and follow-up remain attached to a work item; engineers retain the decision about what should ship.

Limit: Vendor account of an integration, not evidence of an acquisition attempt, a guaranteed moat or measured productivity. Original body read; byline and publication date not verified.

https://www.atlassian.com/blog/ai-at-work/jira-system-of-record-openai-symphony

Checked 2026-10-06

S28 / checked-original

NHTSA ODI opening resume: PE25012

NHTSA Office of Defects Investigation / 2025-10-07 (investigation-opened)

Supports: The original investigation document describes FSD (Supervised) and FSD (Beta) as Level 2 partial automation requiring an attentive supervising driver who remains responsible for driving and traffic-law compliance.

Limit: All three PDF pages read through local text extraction. This is an investigation-opening record, not its final findings or a comparative crash/fatality-rate study. It does not establish Tesla superiority to humans, a current software-version result or suitability for unsupervised driving.

https://static.nhtsa.gov/odi/inv/2025/INOA-PE25012-19171.pdf

Checked 2026-10-06

S29 / checked-original

Full Self-Driving (Supervised) Vehicle Safety Report

Tesla / publication date unverified (live-report; rolling twelve months, exact endpoints not stated in the view read)

Supports: The original live North America / All Road Classes chart reports 7,141,797,524.951 FSD-supervised miles and 1,255 major collisions (5,690,675.319 miles per collision), versus 36,081,121,994.416 manually driven Tesla miles with active safety and 17,380 major collisions (2,076,013.924 miles per collision). Their ratio is about 2.74. Major means airbag or other non-reversible restraint deployment. FSD engagement within five seconds before a collision counts as FSD; no fault is assigned.

Limit: Read the original chart and methodology in CMUX after programmatic fetches returned 403. These are manufacturer-reported observational data, not randomized or independently audited rates. Driver supervision remains required; exposure, driver, road, conditions, hardware and software mix may differ. Tesla does not provide an observed injury/fatality-rate comparison here: projected lives/injuries avoided extrapolate collision reductions. The national baseline uses estimates and proxies; the deck uses the manual-active-safety Tesla comparator instead.

https://www.tesla.com/fsd/safety

Checked 2026-10-06

S30 / checked-original

Reuters: senators seek scrutiny of Tesla's FSD safety statistics

Reuters, republished by Claims Journal / 2026-06-17 (publisher-url-date)

Supports: Reuters reports that researchers challenged Tesla's national crash comparison on severity criteria and vehicle age, and that two senators requested NHTSA scrutiny. NHTSA said it had received the letter and was reviewing it.

Limit: Read the Reuters syndication on Claims Journal, not the inaccessible original May investigation. This is reporting and a request for scrutiny, not a final regulator finding or an independent reanalysis of the chart data. Do not assume every criticism describes the current report's stated CISS methodology unchanged.

https://www.claimsjournal.com/news/national/2026/06/17/338276.htm

Checked 2026-10-06

S31 / checked-original

NHTSA ODI opening resume: EA26002

NHTSA Office of Defects Investigation / 2026-03-18 (investigation-opened)

Supports: The two-page original engineering-analysis opening record lists nine crashes, including one fatality, in the reduced-visibility investigation of FSD Beta and FSD Supervised. NHTSA identifies concerns about degradation detection and timely driver warnings, possible under-reporting of subject crashes, and six additional potentially related incidents to analyze.

Limit: All two pages read through PDF text extraction. Nine is the investigation's identified subset, not the total FSD fleet crash count; the six additional incidents are not confirmed members of that subset. This is an opening record, not a completed adjudication, matched rate study or result for every current software version. Its counts cannot be divided by the Tesla report's unrelated mileage denominator.

https://static.nhtsa.gov/odi/inv/2026/INOA-EA26002-10023.pdf

Checked 2026-10-06

S32 / checked-original

LLM Leaderboard: models

Artificial Analysis / publication date unverified (live-leaderboard)

Supports: Read the original rendered leaderboard and captured its first 25 current configurations in descending Intelligence Index order, retaining the displayed order for ties. The table explicitly provides Intelligence Index, creator and Cost per Task in USD. Selected displayed scores range from 46 to 58 and costs from $0.13 to $7.67; efforts are separate configurations of the same exact model version.

Limit: Snapshot of the first 25 rows, not 25 distinct models or every model tied at the cutoff. Scores and costs preserve the table's displayed precision. API pricing, cache assumptions, model configuration and the benchmark suite affect results. The chart does not rank enterprise outcomes, interpolate unmeasured efforts, show speed or establish statistically significant differences.

https://artificialanalysis.ai/leaderboards/models

Checked 2026-10-06

S33 / checked-original

Benchmarking methodology: Cost per Task definition

Artificial Analysis / publication date unverified (live-methodology)

Supports: Cost per Task is the benchmark-weighted mean USD cost of one Intelligence Index task. It combines input, cached and output tokens consumed with their prices, divides by task count within each benchmark, and uses the Index's benchmark weights. Prices use first-party API rates or provider medians; caching varies by provider.

Limit: Original page read in CMUX, including definitions and the formula. This is not blended price per million tokens, cost per successful task, or total enterprise delivery cost. Reasoning/output length can increase per-task cost at identical token prices. Reading date is not publication date.

https://artificialanalysis.ai/methodology

Checked 2026-10-06

S34 / checked-original-partial

Artificial Analysis Intelligence Index v4.3.2 methodology

Artificial Analysis / publication date unverified (live-methodology)

Supports: The original introduction identifies v4.3.2, ten evaluations and category weights: Agents 30%, Coding 20%, Scientific Reasoning 20%, General 30%. The suite is primarily text-based and English-language. The page describes API token counts, typical cache-hit measurements and testing parameters.

Limit: Introduction, suite table and general testing parameters read; detailed implementations were not fully audited. Index scores are not pass percentages or a universal capability measure. The publisher's aggregate uncertainty statement is not a per-point confidence interval or a test of pairwise equality; the chart does not manufacture error bars.

https://artificialanalysis.ai/methodology/intelligence-benchmarking

Checked 2026-10-06

Benchmark limitations

Not DeepSWE-Preview (Agentica/Together AI). This is Datacurve's benchmark.

Numbers were parsed read-only from the live page's embedded leaderboard data and cross-checked against the rendered table text (page header 'updated September 22, 2026'); the six rows match the displayed table (74% +/- 3% $4.43 30k 29; 74% +/- 1% $2.36 143k 166; 74% +/- 4% $11.84 118k 99; 69% +/- 3% $3.99 80k 124; 59% +/- 2% $13.22 135k 120; 54% +/- 4% $26.40 214k 268). Step counts here are unrounded means; the page rounds them (e.g. 28.8 shown as 29, 268.5 as 268).

passRate = passed/attempted scored rollouts (Pass@1, mean over 4 whole-benchmark runs); uncertainty = published 95% run-to-run half-width (1.96*std(runs)/sqrt(4 runs)) as a fraction (0.0287 = 2.87 percentage points). Overlapping intervals (the top three here) do not establish a rank difference. With only 4 runs the interval is itself rough.

costUsd = mean cost per scored attempt in Datacurve's own accounting (cost_basis states the provider rate card assumed, not reproduced here). It is NOT per-token pricing, NOT speed, and NOT a comparison of native harnesses: every model runs in mini-swe-agent with a single bash tool and a shared prompt, which Datacurve says may hold models below their native-harness ceiling. Wall-clock time is explicitly not reported in v1.1.

Rows are model+reasoning-effort configurations, not model-only scores. Effort setting changes both pass rate and cost materially (e.g. claude-opus-5 high/xhigh/max are separate rows; gpt-6-astra low-to-max span roughly 67%-74%). Do not compare 'max' on one vendor to 'xhigh' on another as like-for-like compute.

DISCREPANCY: the raw artifact https://deepswe.datacurve.ai/artifacts/v1.1/leaderboard-live.json (HTTP 200, same generated_at 2026-09-22T06:27:15Z, last-modified Sep 30 2026) differs from the page-embedded data/rendered table in avg cost for 19 configurations, all matching changelog repricing entries (GPT-5.6 Sol/Terra/Luna Jul 30 and Aug 20, DeepSeek v4 Aug 21, Gemini 3.6 Flash Aug 14, plus glm-5.3-flash). Examples: gpt-5.6-sol[max] $6.46 on page vs $8.39 in JSON; gpt-5.6-luna[max] $0.61 vs $3.03; deepseek-v4-pro[max] $1.67 vs $0.24; glm-5.3-flash[max] $0.24 vs $0.48. Pass rates are identical. Not resolved which is intended; the page/changelog-consistent values are what a viewer sees. Hence none of those rows are used here and earlier-note rows gpt-5.6-sol[max] and gpt-5.6-luna[max] were dropped. Recheck the live chart just before presenting.

Versus the earlier clean-room note: its claude-opus-4.8[max] and claude-sonnet-5[max] values and its gpt-6-astra, gemini-3.8-flash rows are confirmed unchanged, but it omitted claude-opus-5[max] (73.6%, $11.84, now one of the top rows) and its gpt-5.6-sol[max] ($6.46) and gpt-5.6-luna[max] ($0.61) costs are subject to the discrepancy above. Opus 4.8 [max] and Sonnet 5 [max] have 429 and 442 scored attempts (fewer than 452), so they rest on slightly fewer attempts.

claude-fable-5 rows have incomplete trials (page note: 73 of 2,260 trials did not complete because access was suspended by a US government directive); not used in the six rows.

Datacurve's 'contamination free' (tasks written from scratch) and its audit of SWE-bench Pro verifier error rates are the authors' own claims and were not independently verified. The sample is open-source repos with at least 500 GitHub stars in 5 languages; results may not generalize to proprietary enterprise codebases.

The chart does not measure customer value, maintainability, review quality, safety certification or cost per accepted outcome; it supports only 'a similar pass rate can come with very different cost per attempted task and token/step usage'. Cost per accepted outcome would divide by pass rate, which is derived rather than published.