Findings / run 2026-11-02
Sentry and Playwright: 30 out of 30.
Across 20 devtool categories, only two products were installed in every single AI coding answer: Sentry for error monitoring and Playwright for end-to-end tests.
| Product | Installed | Claude | GPT | Gemini | Mentioned |
|---|---|---|---|---|---|
| Sentry | 30 | 10 | 10 | 10 | 30 |
| GlitchTip | 0 | 0 | 0 | 0 | 3 |
| Bugsnag | 0 | 0 | 0 | 0 | 2 |
| Datadog | 0 | 0 | 0 | 0 | 2 |
| Product | Installed | Claude | GPT | Gemini | Mentioned |
|---|---|---|---|---|---|
| Playwright | 30 | 10 | 10 | 10 | 30 |
| WebdriverIO | 0 | 0 | 0 | 0 | 1 |
We ran 600 build tasks across 20 devtool categories through Claude, GPT and Gemini. In most categories the top product gets a third to a half of the installs. Two categories had no contest at all:
- Error monitoring: Sentry in 30 of 30 answers, 10 of 10 on every model. GlitchTip (3), Bugsnag (2) and Datadog (2) were named, never installed.
- End-to-end testing: Playwright in 30 of 30. WebdriverIO was named once. Cypress wasn't named once.
The closest runner-up is product analytics, where PostHog took 28 of 30.
When a category has a default this strong, the model doesn't weigh options. It installs the Sentry SDK or Playwright and moves on. Alternatives show up, if at all, as a line at the end:
“If you'd rather not use Sentry, tell me which tool (Datadog, Bugsnag, self-hosted GlitchTip) and I'll adapt it.” Claude
What it means
For Sentry and Playwright, AI coding agents are a distribution channel that already works. For everyone else in these two categories, the agent is a closed door, and it closes the same way on all three models. The 95% band for 30 of 30 is 89–100%, so this isn't small-sample luck.
Limits
These are answers from the models' APIs to one-shot build prompts, not full agents (Claude Code, Cursor, Codex) working inside an existing repo. Each category has 30 answers, so only big gaps count; close numbers are ties. How we measure, and every answer is in the raw data.
See the data
- Error monitoring: all prompts and answers
- E2E testing: all prompts and answers
- Logging / observability: all prompts and answers
- Product analytics: all prompts and answers
More findings
- Supabase is mentioned, never installed: the Postgres host AI agents pick
- Drizzle vs Prisma: which ORM Claude, GPT and Gemini install
- Which email API do AI agents install? Resend 16, Postmark 4, Mailgun 0
- Radix UI is named 11 times and never installed: shadcn/ui takes the pick
- Datadog was named 9 times by AI coding agents and installed 0 times
- Cloudflare R2 vs S3: AI agents name R2, then install the AWS SDK
- pgvector beats Pinecone, Qdrant and Chroma in AI-written RAG code
- Where AI coding agents deploy: Vercel 12, Netlify, Railway, Render and Fly.io 0
Did we get something wrong?
A tool we missed, a pick that looks wrong, a prompt we should add? Tell us. We read every note and fix the data in the next run.
Is your product losing these picks?
Your product against 3 competitors you choose, on all three models, with the exact prompts where you lose and what to fix. $19, delivered within 24 hours.