# LLM benchmarks: real agents, real work > 24 models run against 5 real Cotera agents — 120 graded runs, scored on the work they actually had to do. Source: https://cotera.co/benchmarks --- Every model runs the same 5 agents with the same tools and the same rubric. Total cost of the matrix: $29.66 across 120 runs. ## Results | Model | Passed | Total cost | | --- | --- | --- | | Mistral Large 3 | 5/5 | $0.267 | | GPT-5.6 Luna | 5/5 | $0.296 | | GPT-5 Mini | 5/5 | $0.376 | | GPT-5.6 Terra | 5/5 | $0.392 | | Nemotron 3 Super 120B | 5/5 | $0.610 | | Gemini 3 Flash | 5/5 | $0.758 | | GPT-5.5 | 5/5 | $1.21 | | Gemini 3.5 Flash | 5/5 | $1.41 | | Claude Sonnet 5 | 5/5 | $1.44 | | Gemini 3.1 Pro | 5/5 | $2.73 | | Claude Sonnet 4.6 | 5/5 | $3.83 | | Claude Opus 4.8 | 5/5 | $4.58 | | GPT-5.6 Sol | 4/5 | $0.697 | | Claude Haiku 4.5 | 4/5 | $0.941 | | DeepSeek V3.1 | 3/5 | $0.113 | | GLM 5.2 | 3/5 | $0.521 | | MiniMax M3 | 3/5 | $1.10 | | GLM 5.1 | 3/5 | $1.19 | | Kimi K2.6 | 3/5 | $1.27 | | Llama 4 Maverick | 2/5 | $0.004 | | Grok 4.1 Fast | 1/5 | $0.089 | | Nemotron 3 Ultra 550B | 1/5 | $2.46 | | Nemotron 3 Nano 30B | 0/5 | $0.411 | | Qwen 3.6 Plus | 0/5 | $2.98 | ## Benchmarks - [Get financials on a company](https://cotera.co/benchmarks/agents/sales-crunchbase-funding-snapshot.md): Find Hightouch on Crunchbase and return their total funding, last round type, and a one-line description. - [Find what customers are recommending on Reddit](https://cotera.co/benchmarks/agents/marketing-reddit-enrichment-tools.md): Read a real r/sales thread on enrichment tools and rank the top 5 by how many people recommended them. - [Find what customers are complaining about](https://cotera.co/benchmarks/agents/cx-google-shopping-complaints.md): Pull Google Shopping reviews for AirPods Pro 2 (USB-C) and return the top 3 recurring complaints, with verbatim quotes. - [Read API docs and write working code](https://cotera.co/benchmarks/agents/coding-stripe-webhook-verifier.md): Read Stripe's official docs and write a real, working webhook-verification function in TypeScript. - [Scrape a competitor's pricing page](https://cotera.co/benchmarks/agents/tool-usage-apollo-pricing-scrape.md): Scrape Apollo.io's pricing page and return every tier (Free, Basic, Professional, Organization) with name, price, and top 3 features. ## Model write-ups - [Mistral Large 3](https://cotera.co/benchmarks/models/mistral-large-3.md): The quiet 5/5. Half the price of GPT-5.5 and didn't whine about JSON once. - [GPT-5 Mini](https://cotera.co/benchmarks/models/azure-gpt-5-mini.md): GPT-5 Mini quietly did the matrix in $0.38 — bring receipts the next time someone says "mini is too dumb." - [GPT-5.5](https://cotera.co/benchmarks/models/azure-gpt-5.5.md): 5/5 at $1.21. Worth it when latency-vs-cost on the long-tail edge cases matters more than the median. - [GPT-5.6 Sol](https://cotera.co/benchmarks/models/azure-gpt-5.6-sol.md): The new flagship went 4/5 — beaten on schema discipline by its own cheaper siblings. - [GPT-5.6 Terra](https://cotera.co/benchmarks/models/azure-gpt-5.6-terra.md): 5/5 at $0.39 — GPT-5.5's scorecard at a third of the bill. - [GPT-5.6 Luna](https://cotera.co/benchmarks/models/azure-gpt-5.6-luna.md): The $1-per-million tier went 5/5 for $0.30 — the second-cheapest perfect score in the matrix. - [Gemini 3 Flash](https://cotera.co/benchmarks/models/gemini-3-flash.md): Cheap on the easy ones, ate $0.34 on AirPods complaints. Flash that occasionally remembers it's Pro. - [Gemini 3.5 Flash](https://cotera.co/benchmarks/models/gemini-3.5-flash.md): Newer Flash, almost double the bill of the model it replaced. We'd still pick the older one. - [Gemini 3.1 Pro](https://cotera.co/benchmarks/models/gemini-3.1-pro-preview.md): Pro-tier Gemini got the answer every time. CX cost $1.48 on a benchmark Mistral nailed for $0.066. - [Claude Haiku 4.5](https://cotera.co/benchmarks/models/claude-haiku-4-5.md): 4/5 with a faceplant on the easiest benchmark — wrote "I'll help you research..." then logged off. - [Claude Sonnet 4.6](https://cotera.co/benchmarks/models/claude-sonnet-4-6.md): 5/5 every time, billed like a senior engineer. Worth it when correctness is non-negotiable; brutal at scale. - [Claude Opus 4.8](https://cotera.co/benchmarks/models/claude-opus-4-8.md): 5/5 at $4.58. The most expensive perfect score, and the only one we actively don't recommend. - [Grok 4.1 Fast](https://cotera.co/benchmarks/models/grok-4.1-fast.md): Calls the tools, reads the responses, and then forgets to write anything down. 1/5. - [DeepSeek V3.1](https://cotera.co/benchmarks/models/deepseek-v3.1.md): Cheapest model in the matrix. Also the most likely to bail at "let me search for…" without ever calling a tool. - [Llama 4 Maverick](https://cotera.co/benchmarks/models/llama-4-maverick.md): Made zero tool calls across all five benchmarks. Answered from training memory or didn't answer at all. - [MiniMax M3](https://cotera.co/benchmarks/models/minimax-m3.md): Thinks in `` blocks. Sometimes forgets to write the answer afterward. - [Kimi K2.6](https://cotera.co/benchmarks/models/fw-kimi-k2p6.md): Confident prose, missing code. Token-repetition loop on Reddit. Read the actual outputs before you trust this one. - [Qwen 3.6 Plus](https://cotera.co/benchmarks/models/fw-qwen3p6-plus.md): 0/5. Burned $1.91 on Crunchbase alone to produce zero parseable output. The worst run in the matrix. - [GLM 5.1](https://cotera.co/benchmarks/models/fw-glm-5p1.md): Smart quotes in Stripe code. Empty final on Reddit. JSON encoding is GLM 5.1's nemesis. - [Claude Sonnet 5](https://cotera.co/benchmarks/models/claude-sonnet-5.md): The other clean 5/5. Never fumbled a final answer, never over-searched, never mangled JSON. - [GLM 5.2](https://cotera.co/benchmarks/models/glm-5.2.md): The cheap open-weights contender — 3/5 for $0.52, tripped up only by JSON hygiene. - [Nemotron 3 Super 120B](https://cotera.co/benchmarks/models/nemotron-3-super-120b.md): The open-weights 5/5 — swept the matrix for $0.61, an order of magnitude cheaper than the frontier models. - [Nemotron 3 Ultra 550B](https://cotera.co/benchmarks/models/nemotron-3-ultra-550b.md): Biggest and priciest Nemotron, worst value — 1/5 at $2.46, most of it burned looping. - [Nemotron 3 Nano 30B](https://cotera.co/benchmarks/models/nemotron-3-nano-30b.md): The small one. 0/5 — cheap, but never assembled a passing answer.