2.1s tool interval · 47s outside tools
range 41.4s–60.3s ·3 runs · 12/12 checksAgent Experience evidence
Inspect the evidence behind the customer findings.
Loading results
Inspect the evidence behind the customer findings.
Inspect the evidence behind the customer findings.
3 of 3 comparisons differ
Claude Code · selected / trials · each row runs 0 to 100%
51/51Task result checks3 tasks · 9 runs
24/24Cause identified
24/24Next step identifiedthe same 24 answers as the cause assessment
CSV contains normalized run and evidence rows across the measured AX faces. JSON preserves each face's reviewed results, conditions, limitations, and evidence hashes. Neither export contains credentials or raw transcripts.
Northwind task completion, repeated-run reliability, and recovery are primary. Other endpoints provide comparison context only.
Agents completed wallet inspection, address-history backfill, and the 50-request block task against Northwind. Every Northwind run in the endpoint comparison passed its task checks.
9/9 Northwind runs completed; 51/51 Northwind result checks passed.The endpoint comparison separates total agent task time from time spent inside tools. Other endpoints are included for reference.
All 9 Northwind runs completed and passed 51/51 result checks. Most elapsed time occurred outside tool execution, so this measurement does not point to a Northwind API-latency fix.
2.1s tool interval · 47s outside tools
range 41.4s–60.3s ·3 runs · 12/12 checks2.5s tool interval · 63.2s outside tools
range 50.9s–92.1s ·3 runs · 15/15 checks19.1s tool interval · 60.2s outside tools
range 76.6s–83.6s ·3 runs · 24/24 checksBurst result: all three Northwind runs completed 50/50 requests; no rate-limit response was observed in this matrix. This does not claim rate limits never occur.
The other endpoints are context only. Their ranges overlap in most cells, so no speed ranking is claimed. The only large reproduced reference difference was Competitor H address-history backfill: 252.7s total and 43.7s in tool intervals on average, with 11–18 RPC calls.
| Task | Endpoint | Total | Tool interval | Checks |
|---|---|---|---|---|
| Inspect wallet holdings | Northwind | 49.1s (41.4s–60.3s) | 2.1s | 12/12 |
| Inspect wallet holdings | Competitor A | 53.1s (50.4s–58.1s) | 3.5s | 12/12 |
| Inspect wallet holdings | Competitor C | 48.8s (43.9s–52.4s) | 2.7s | 12/12 |
| Inspect wallet holdings | Competitor G | 50.7s (49s–53.8s) | 3.3s | 12/12 |
| Inspect wallet holdings | Competitor H | 49s (43s–53.2s) | 2.9s | 12/12 |
| Backfill address history | Northwind | 65.6s (50.9s–92.1s) | 2.5s | 15/15 |
| Backfill address history | Competitor A | 100.5s (59.3s–130s) | 4.2s | 15/15 |
| Backfill address history | Competitor C | 68.9s (46.8s–103.4s) | 3.3s | 15/15 |
| Backfill address history | Competitor G | 91s (85.3s–102.2s) | 4.3s | 15/15 |
| Backfill address history | Competitor H | 252.7s (235.7s–280.2s) | 43.7s | 15/15 |
| Fetch 50 blocks in parallel | Northwind | 79.3s (76.6s–83.6s) | 19.1s | 24/24 |
| Fetch 50 blocks in parallel | Competitor A | 67.7s (64.8s–69.9s) | 12.8s | 24/24 |
| Fetch 50 blocks in parallel | Competitor C | 72.5s (66.8s–79.8s) | 13.2s | 24/24 |
| Fetch 50 blocks in parallel | Competitor G | 86.5s (82.8s–93.3s) | 15.5s | 24/24 |
| Fetch 50 blocks in parallel | Competitor H | 77.5s (71.1s–84s) | 13.2s | 24/24 |
This earlier exploratory comparison is retained for qualitative interaction evidence. Its duration values are superseded by the three-repeat B2 baseline above and must not be read as a before/after trend.How to read the earlier verdicts
| Developer task | Northwind | Competitor A | Competitor G | Competitor C | Competitor H |
|---|---|---|---|---|---|
| Fetch a wallet's holdings (balance + all token accounts) | ✓8/8 checks · 252s | ✓8/8 checks · 146s | ✓8/8 checks · 275s | ✓8/8 checks · 173s | ⚠8/8 checks · 240s |
| Backfill an address's transaction history with pagination | ✓10/10 checks · 156s | ✓10/10 checks · 197s | ✓10/10 checks · 144s | ✓10/10 checks · 182s | ✕7/9 checks · 217s |
| Burst-fetch 50 recent blocks and classify endpoint behavior | ✓16/16 checks · 208s | ✓16/16 checks · 90s | ✓16/16 checks · 142s | ✓16/16 checks · 121s | ✓16/16 checks · 273s |
No endpoint pair separated statistically at this sample size; the reproducible differences below are qualitative findings, not rankings.
Each measured Northwind task produced the same result on both attempts. No Northwind reliability fix is indicated by this test.
Northwind was stable in every measured task. One non-Northwind endpoint produced different outcomes across identical address-history attempts; this does not indicate a Northwind fix.
A second agent scaffold (Codex CLI) repeated the core tasks and reached the same endpoint verdicts — 10/10 task-endpoint verdicts matched. These comparison results are context for Northwind, not the subject of this page.
We sent the same request to Northwind twice: getBalance with an invalid API key. Both attempts returned the same 401 Unauthorized response. In the separate recovery test, agents identified invalid credentials and chose the correct next step.
| Endpoint | HTTP | Response | Agent read |
|---|---|---|---|
| Northwind | 401 | Unauthorized | not JSON-RPC · names the cause |
| Endpoint | HTTP | Response | Agent read |
|---|---|---|---|
| Competitor A | 401 | {"error":"UNAUTHORIZED","instance":"<redacted-instance>","request-id":""} | not JSON-RPC · names the cause |
| Competitor G | 404 | <html> <head><title>404 Not Found</title></head> <body> <center><h1>404 Not Found</h1></center> <hr><center>nginx</center> </body> </html> | not JSON-RPC · cause unclear |
| Competitor C | 403 · 403 | Your IP or provider is blocked from this endpoint | JSON-RPC body · cause unclear |
| Competitor H | 403 · 4 | Your token is invalid or expired | JSON-RPC body · names the cause |
Whether agents found the documentation and agent signup path, and whether the published signup terms matched the live flow.
For the affected agent, placing the Northwind docs in the workspace was not enough. Adding a short docs link to the instruction file it reads automatically at startup caused it to select the Northwind-native API.
View the agent discovery fix in AX Findings →Target API: address history API
Putting the docs in the workspace alone produced 0/6. Linking them from AGENTS.md produced 0/6 because this Claude Code version reads CLAUDE.md at startup, not AGENTS.md.
This panel measures one thing: which API an agent selects when asked to write address-history backfill code — Northwind-exclusive address history API versus standard-blockchain fallbacks — under controlled documentation-delivery conditions. It is not a measurement of Northwind's overall agent experience, of other Northwind products, or of other agents' behavior.
Measured 2026-08-18 · Claude Code 2.1.227 (auto-loads CLAUDE.md; web search off (sandboxed, no network tools used)) · Codex CLI 0.147.0 (auto-loads AGENTS.md; web search off (workspace-write sandbox)) · Gemini CLI 0.55.1 (auto-loads GEMINI.md; web search off — google_web_search and web_fetch excluded via workspace settings)
primary (equal-n cells)
Outcome measured: Target API selected address history API
context (in-weight ceiling; unequal exploratory n)
Codex CLI selected the target in every recorded run regardless of condition, so a docs-delivery effect cannot be measured on it.
Outcome measured: Target API selected address history API
secondary (equal-n key cells; earlier search-on runs relabeled to the network condition)
Outcome measured: Target API selected address history API
address history APISelected without docs0/695% interval 0–39%Transaction Submission API.northwind-rpc.com/fast submitTransactionSelected without docs6/695% interval 61–100%Streaming API Subscribe (confirmed)Selected without docs6/695% interval 61–100%The docs-delivery cells above exist for the task where unaided knowledge failed; tasks the agent already knows unaided leave no gap for documentation to close.
Runtime validation: 6 of 7 recorded executions of generated scripts exited cleanly against the live endpoint (read-only; logs stored key-redacted); a separate time-capped pagination run advanced 59 pages before its limit — and a fallback script also completed without error, so the wrong-API outcome does not surface as a runtime failure.
The affected agent fell back to the human dashboard until its startup instruction file included a short link to the Northwind signup documentation.
View the agent-native signup fix in AX Findings →Signup path: northwindAccount (MCP) / Northwind signup (CLI)
This panel measures one thing: which signup path an agent selects when asked to write code that obtains a Northwind API key programmatically — the agent-native path (the northwindAccount MCP tool or the Northwind signup CLI) versus falling back to the human web-dashboard flow. Scripts must stop before any payment and are never executed, so this is a path-selection measurement, not a live signup completion measurement, and not a claim about Northwind's overall agent experience.
Measured 2026-08-18 · Claude Code 2.1.227 (auto-loads CLAUDE.md; web search off (sandboxed, no network tools used)) · Codex CLI 0.147.0 (auto-loads AGENTS.md; web search off (workspace-write sandbox)) · Gemini CLI 0.55.1 (auto-loads GEMINI.md; web search off — google_web_search and web_fetch excluded via workspace settings)
primary (equal-n cells)
Outcome measured: Signup path selected northwindAccount (MCP) / Northwind signup (CLI)
context (knows the CLI signup path in-weight; exploratory n)
Codex CLI selected the target in every recorded run regardless of condition, so a docs-delivery effect cannot be measured on it.
Outcome measured: Signup path selected northwindAccount (MCP) / Northwind signup (CLI)
context (baseline only; exploratory n)
Outcome measured: Signup path selected northwindAccount (MCP) / Northwind signup (CLI)
northwindAccount (MCP) / Northwind signup (CLI)Selected without docs0/695% interval 0–39%The docs-delivery cells above exist for the task where unaided knowledge failed; tasks the agent already knows unaided leave no gap for documentation to close.
Static verification only: generated signup scripts are never executed, because completing the flow would create accounts and move funds — verdicts come from code inspection, and each script must stop before any payment step by design.
The official documentation shows 1 USD; the current CLI and backend payment intent request 10 USD. Northwind needs to confirm which price is intended.
View the Agent-plan price fix in AX Findings →Documentation and payment requests were compared on 2026-08-18. The source records below have sensitive details removed and have been verified against their hashes.
Test runs are grouped by where they stopped. Recorded error excerpts are also checked for request identifiers. This exploratory analysis does not measure real-world failure rates.
67 Northwind test runs grouped by outcome. Product issues observed in the test are separated from stops required by the test procedure and cases with insufficient evidence. These counts do not estimate real-world frequency.
Agents understood both measured Northwind errors and chose the correct next step every time. Keep the existing error body and Retry-After behavior unchanged. A request ID is evaluated separately as an untested traceability proposal.
What this measures: an agent is shown one failed JSON-RPC call and the error response that came back, and must name the cause and the right next step. The six responses are real observed replies from five providers, shown to the agents with provider names removed (some fields and envelopes are reconstructed from captured excerpts, noted per cell). Scores are checked by code against a fixed answer key. This is a comparison of error responses, not of providers overall — the two Northwind cells are the only Northwind surfaces measured — and because the responses differ in several ways at once, it shows what happened, not which single property caused it.
Method: Each agent saw only the request and the error response, with provider names removed and no web access; answers use a fixed vocabulary and are scored by code against a preregistered answer key. All 72 runs completed and passed automated validity checks.
Cause checks whether the agent named the true cause; Right next step checks whether it chose the correct recovery action. Non-primary cells are labeled exploratory.
Measured 2026-08-19 · 72 of 72 runs valid · Claude Code 2.1.227 (claude-fable-5; default configuration with web access disabled) · Codex CLI 0.147.0 (gpt-5.6-native-token; default configuration; no web access) · Gemini CLI 0.55.1 (gemini-3.6-flash; default configuration with web access disabled)
Retained private live-source capture SHA-256: 9684ff63bb26c5338315c4fd35e4a0d98869ff371660cf6cdd8535fc8f9bb604
Retest gate: After vendor confirmation or a fix, repeat the same CLI flow with a new unpaid intent, verify the displayed amount across all five surfaces, then separately authorize payment before testing resume and first API call.
Exploratory regrouping of designed test conditions, not real-world incidence rates. Several stops were deliberately induced by withholding documentation or by the evaluation protocol; the bars must not be read as field prevalence.
The 13 Northwind-side cases are already covered by the findings above: 6 native-API selection losses, 6 native-signup selection losses, and 1 signup-price mismatch. The protocol, docs-not-retrieved, and unresolved buckets do not support an additional Northwind fix from this evidence.
Labeled 2026-08-19 · exploratory relabel pending second review. Existing Northwind tests were grouped into mutually exclusive outcomes. They cover documentation delivery, signup-path selection, signup pricing, and Northwind's endpoint-comparison results. Tests against other providers are excluded. This is not an overall agent-experience assessment, a ranking, an average across studies, or a combined score. Test-environment errors and stops required by the test procedure are kept separate from Northwind issues. Signup scripts were designed to stop before payment and were never executed. The analysis does not determine Northwind's intended price.
Agents correctly diagnosed invalid credentials from the response, but neither repeated 401 included a request-specific ID an agent could give to support.
View the request-ID proposal in AX Findings →