Methodology · run 2026-11-02

How InstallRate measures what agents install

Developers now ask an AI to “build me X”, and the AI chooses the auth library, the database and the email API. We measure those choices.

The prompts

100 build tasks across 20 categories, 5 per category. They read like real requests (“Add sign-up and login to my Next.js app”), use different stacks, and never name a product. Each one asks for the install commands, so the choice shows up as a package name and not just a mention. Every prompt is listed on its category page.

The models

ModelExact version
Claudeanthropic/claude-sonnet-5.5
GPTopenai/gpt-5.6-terra
Geminigoogle/gemini-3.1-pro-preview

Each prompt runs 2 times per model through the public API, with the same short system prompt (“You are a coding agent working in the user's project. Make the technical choices yourself…”), temperature 1.0, and up to 2000 output tokens.

Reading the pick

We read what the answer actually installs, in this order: install commands (npm i, pip install, npx … init), then package.json or requirements files, then CLI commands (vercel, fly, wrangler), then imports. The first product of the category found that way is the pick. If an answer only names a product in prose, a small model labels it. If it chooses nothing, it counts as “None”. We hand-check a random 60 answers per run and publish only when at least 95% agree.

Limits

Raw data

Every answer, with the prompt, model version and our extracted pick: 2026-11-02.jsonl.

Categories