Open-source models: do they install the same tools?
We gave 8 open-weight models (DeepSeek, Qwen, GLM, Kimi, MiniMax, gpt-oss, Mistral and Gemma) the same 100 build tasks as Claude, GPT and Gemini, and counted what each one actually installed. Same first pick in 17 of 20 categories, but they build it by hand more often (14% of answers, against 9%).
Category by category
Claude, GPT, Gemini8 open modelsLeft: what Claude, GPT and Gemini install first. Right: what the open models install first. The dots show the left product's share in each group, so a long line means the open models moved away from it.
- Auth
Auth.js / NextAuth27%
Auth.js / NextAuth46%Same #1 - Postgres hosting
Neon67%
Neon42%Same #1 - ORM / query builder
Drizzle53%
Drizzle38%Same #1 - Payments
Stripe80%
Stripe89%Same #1 - Transactional email
Resend53%
Resend40%Same #1 - Hosting / deploy
Vercel40%
Vercel39%Same #1 - Object storage
AWS S357%
AWS S335%Same #1 - Background jobs / queues
BullMQ / Bull40%
BullMQ / Bull48%Same #1 - Workflows / cron
Cron library40%
Cron library48%Same #1 - Feature flags
PostHog flags33%
Unleash31%Different #1 - Error monitoring
Sentry100%
Sentry95%Same #1 - Product analytics
PostHog93%
PostHog85%Same #1 - Logging / observability
Grafana / Loki / Prometheus40%
OpenTelemetry51%Different #1 - Search
Meilisearch37%
Meilisearch22%Same #1 - Vector database
pgvector33%
pgvector20%Same #1 - LLM observability
Langfuse33%
LangSmith33%Different #1 - Headless CMS
Sanity37%
Sanity28%Same #1 - Realtime / websockets
Socket.IO33%
Socket.IO44%Same #1 - E2E testing
Playwright100%
Playwright82%Same #1 - UI component library
shadcn/ui63%
shadcn/ui41%Same #1
Share = the product the answer installed first. 30 answers per category from Claude, GPT and Gemini (run 2026-11-02), 79 from the open models (run 2026-11-16). With samples this small, a gap of 10-15 points can be noise; section 03 lists only the gaps that aren't.
Model by model
Agrees = same pick as the big threeHow often each model installs the product Claude, GPT and Gemini install most in that category, and how often it builds the thing by hand instead. For the three themselves, we compare each with the other two.
| Model | Agrees | Built by hand | Stands out | Answers |
|---|---|---|---|---|
| 12% | — | 196 | ||
| 16% | 200 | |||
| 12% | — | 200 | ||
| 15% | 200 | |||
| 15% | — | 192 | ||
| 16% | 200 | |||
| 12% | Cypress 70% vs 0% · e2e testing | 200 | ||
| 11% | 200 | |||
| For reference: the three closed models | ||||
| 13% | — | 200 | ||
| 8% | — | 200 | ||
| 7% | — | 200 | ||
Stands out = the product this model installs at least 30 points more often than Claude, GPT and Gemini do, in its biggest such category (4 or more of its 10 answers). About 198 answers per open model, 200 per closed model.
Where they differ most
Clear gaps onlyProducts whose share moves by 20 points or more between the two groups, where the gap is bigger than these samples explain (two-proportion test, 95%).
Open models install more
None big enough to stand out from the noise.
Open models install less
How we ran it
Same 100 prompts, same system prompt, temperature 1.0 and token limit as the closed models, 2 answers per prompt and model, all through OpenRouter on 2026-11-16. One model per family, picked for how much developers use it and what it costs to run; the exact versions are on the methodology page. Thinking models ran with low effort, like the closed ones.
The leaderboard's shares, winners and badges stay the closed models' numbers; the open models show next to them, never mixed in. Every answer is in the raw data.
Which model should we add next?
A tool we missed, a pick that looks wrong, a prompt we should add? We read every note and fix the data in the next run.
Is your product losing these picks?
Your product against 3 competitors you choose, on Claude, GPT, Gemini and 8 open-source models, with the exact prompts where you lose and what to fix. Delivered within 24 hours.