See what happened after an agent chose a provider and attempted the task.
See what happened after an agent chose a provider and attempted the task.
2findings
2 to change
Agent discoveryNeeds attention
0/6 → 6/6selected without / with the startup link
Agent-plan signupNeeds attention
1 ≠ 10 USDdocs and the payment request
3findings
3 with no issue
ExecutionNo issue found
51/51result checks passed
Workflow timeNo issue found
49.1-79.3 smean wall time per task
ReliabilityNo issue found
3/3task cells repeated the same result
2findings
1 to verify · 1 with no issue
RecoveryNo issue found
24/24runs got the cause and next step right
Support traceabilityImprovement opportunity
No request IDin the stored 401 response
Task completion, repeats, and error recovery, measured; the support process itself was not tested.
Some agents missed Northwind-native APIs and signup paths when the documentation was present but not linked from the file they load automatically.
Add a short Northwind section to AGENTS.md, CLAUDE.md, and GEMINI.md, which coding agents read when they start.
Northwind's official docs list the Agent plan at 1 USD, but northwindctl x.y.z and the live backend payment request asked for 10 USD.
Confirm the intended price, then align the documentation, CLI, and amount requested during signup.
In two repeated invalid-API-key calls, Northwind returned 401 Unauthorized without a request or correlation ID. Agents correctly diagnosed invalid credentials, but had no request-specific identifier for support.
Add a non-empty request or correlation ID without changing the existing 401 error meaning. Treat this as an untested traceability proposal.
Agents completed every measured Northwind task cleanly.
No Northwind speed issue appeared in these tasks: wallet inspection averaged 49.1s total (2.1s in tools), history backfill 65.6s (2.5s), and the 50-request burst 79.3s (19.1s).
Every measured Northwind task produced the same outcome on both attempts.
Agents correctly diagnosed each measured Northwind error and chose the right recovery step.
