Leaderboard / Open modelsOpen run 2026-11-16 · 8 open models

Open-source models: do they install the same tools?

We gave 8 open-weight models (DeepSeek, Qwen, GLM, Kimi, MiniMax, gpt-oss, Mistral and Gemma) the same 100 build tasks as Claude, GPT and Gemini, and counted what each one actually installed. Same first pick in 17 of 20 categories, but they build it by hand more often (14% of answers, against 9%).

8open models
1,588answers
17 of 20same first pick
14% vs 9%built by hand
01

Category by category

Claude, GPT, Gemini8 open models

Left: what Claude, GPT and Gemini install first. Right: what the open models install first. The dots show the left product's share in each group, so a long line means the open models moved away from it.

  1. AuthAuth.js / NextAuth27%Auth.js / NextAuth46%Same #1
  2. Postgres hostingNeon67%Neon42%Same #1
  3. ORM / query builderDrizzle53%Drizzle38%Same #1
  4. PaymentsStripe80%Stripe89%Same #1
  5. Transactional emailResend53%Resend40%Same #1
  6. Hosting / deployVercel40%Vercel39%Same #1
  7. Object storageAWS S357%AWS S335%Same #1
  8. Background jobs / queuesBullMQ / Bull40%BullMQ / Bull48%Same #1
  9. Workflows / cronCron library40%Cron library48%Same #1
  10. Feature flagsPostHog flags33%Unleash31%Different #1
  11. Error monitoringSentry100%Sentry95%Same #1
  12. Product analyticsPostHog93%PostHog85%Same #1
  13. Logging / observabilityGrafana / Loki / Prometheus40%OpenTelemetry51%Different #1
  14. SearchMeilisearch37%Meilisearch22%Same #1
  15. Vector databasepgvector33%pgvector20%Same #1
  16. LLM observabilityLangfuse33%LangSmith33%Different #1
  17. Headless CMSSanity37%Sanity28%Same #1
  18. Realtime / websocketsSocket.IO33%Socket.IO44%Same #1
  19. E2E testingPlaywright100%Playwright82%Same #1
  20. UI component libraryshadcn/ui63%shadcn/ui41%Same #1

Share = the product the answer installed first. 30 answers per category from Claude, GPT and Gemini (run 2026-11-02), 79 from the open models (run 2026-11-16). With samples this small, a gap of 10-15 points can be noise; section 03 lists only the gaps that aren't.

02

Model by model

Agrees = same pick as the big three

How often each model installs the product Claude, GPT and Gemini install most in that category, and how often it builds the thing by hand instead. For the three themselves, we compare each with the other two.

ModelAgreesBuilt by handStands outAnswers
DeepSeek V4.1 FlashDeepSeek12%—196
Qwen3 Coder NextAlibaba16%OpenTelemetry 80% vs 33% · logging200
GLM 5.3Z.ai12%—200
Kimi K2 (0905)Moonshot AI15%OpenTelemetry 80% vs 33% · logging200
MiniMax M3.1 FlashMiniMax15%—192
gpt-oss-120bOpenAI16%SMTP library 80% vs 23% · transactional email200
Devstral 2Mistral AI12%Cypress 70% vs 0% · e2e testing200
Gemma 4 31BGoogle11%LangSmith 70% vs 30% · llm observability200
For reference: the three closed models
Claude Sonnet 5.5Anthropic13%—200
GPT-5.6 TerraOpenAI8%—200
Gemini 3.1 ProGoogle7%—200

Stands out = the product this model installs at least 30 points more often than Claude, GPT and Gemini do, in its biggest such category (4 or more of its 10 answers). About 198 answers per open model, 200 per closed model.

03

Where they differ most

Clear gaps only

Products whose share moves by 20 points or more between the two groups, where the gap is bigger than these samples explain (two-proportion test, 95%).

Open models install more

None big enough to stand out from the noise.

04

How we ran it

Same 100 prompts, same system prompt, temperature 1.0 and token limit as the closed models, 2 answers per prompt and model, all through OpenRouter on 2026-11-16. One model per family, picked for how much developers use it and what it costs to run; the exact versions are on the methodology page. Thinking models ran with low effort, like the closed ones.

The leaderboard's shares, winners and badges stay the closed models' numbers; the open models show next to them, never mixed in. Every answer is in the raw data.

Feedback

Which model should we add next?

A tool we missed, a pick that looks wrong, a prompt we should add? We read every note and fix the data in the next run.

The $19 report

Is your product losing these picks?

Your product against 3 competitors you choose, on Claude, GPT, Gemini and 8 open-source models, with the exact prompts where you lose and what to fix. Delivered within 24 hours.