When a team says they are “performance testing” an LLM, I have learned to ask which job they mean.
Sometimes they mean capability. Can the model reason, code, follow instructions, or stay safe enough for the product?
Other times they mean systems. Can the serving stack hit latency and throughput targets without torching the token budget?
Those are different questions. Mix them and a chat Elo chart gets treated like a latency SLA. Or a tokens-per-second graph gets treated like a coding hire.
This post is a plain map of LLM benchmarks that eng leads cite from 2024 through 2026. I stay with bench families and what they measure. I do not reprint rival model scores. Scores move. Definitions last longer.
Two lanes of LLM “performance”
| Lane | Everyday question | Typical metrics |
|---|---|---|
| Capability / quality | How well does the model do the task? | Accuracy %, pass@k, Elo / win rate, success rate, judge score |
| Systems / serving | How well does the stack serve the model? | TTFT, TPOT / ITL, tokens/s, RPS, $/M tokens, goodput under SLA |
Capability benches answer whether the model can do the work. For agent products, that includes the scaffold too.
Systems benches answer a different question. They ask whether your stack can serve that work under concurrency, latency percentiles, and cost.
So a production pick usually needs at least one gate from each lane.
That split also answers a question people type into search. What are the main types of LLM benchmarks?
First, capability suites: knowledge, reasoning, math, coding, instruction following, chat preference, long context, agents, safety, and multimodal. Next, systems suites: latency, throughput, cost, and accuracy-under-load. Domain suites then sit on top when the product is vertical.
Capability lane: what quality benches measure
Knowledge and hard reasoning
MMLU and MMLU-Pro are the broad multi-task quiz families. Classic MMLU is largely maxed out at the frontier.
MMLU-Pro is harder and leans more on reasoning. It also drops a lot of trivial noise. Still, both grade exam-style knowledge, not your live product task.
When you need expert science rather than broad trivia, look at GPQA (especially the Diamond subset).
Also look at Humanity's Last Exam (HLE).
They grade hard expert questions that still have headroom. Small sets mean scores swing. Because of that, tool settings must match when you compare cards.
Math
GSM8K and MATH are the classic math sets. Frontier models have largely maxed out GSM8K.
Year-stamped AIME-style tests show up on 2025 and 2026 model cards. Only compare the same year, grading rules, and tool policy. They are problem sources, not one fixed harness.
Coding and software engineering
This is the cluster eng leads ask about most.
- HumanEval: write Python from docstrings. The metric is pass@k. It is small and largely maxed out at the frontier.
- LiveCodeBench: contest problems that refresh often, so train-on-test is harder. Always cite the version or date window.
- SWE-bench (including Verified, and harder Pro variants): fix real GitHub issues with tests. Resolve rate is highly scaffold-dependent. Agent harness, tools, and retries dominate. Compare only under a stated setup.
If your product is “fix bugs in our monorepo,” SWE-bench-style families are closer than HumanEval. If your product is “complete a function in a notebook,” the opposite is true.
Instruction following and chat preference
IFEval checks clear constraints (length, keywords, format). It grades obedience to instructions, not deep reasoning or helpfulness.
Preference families measure which reply humans (or a judge model) prefer:
- LMSYS Chatbot Arena / lmarena: blind human pairwise battles with an Elo-style rank. It is crowdsourced. Style and length bias exist. It is not a task accuracy measure.
- MT-Bench, AlpacaEval 2.0 (prefer the length-controlled variant), and Arena-Hard: curated or mined prompts. They use LLM-as-judge scores or win rates.
Arena is a useful helpfulness signal. However, it is a poor stand-in for a coding resolve rate or a latency SLO.
Long context, agents, safety, multimodal
Long context families ask whether the advertised window is usable:
- Needle-in-a-haystack variants (see Kamradt's harness): find a planted fact at varying depths. Easy NIAH can look “solved” while multi-hop work still fails.
- RULER: a synthetic suite beyond plain NIAH (multi-needle, tracing, aggregation).
- LongBench / LongBench-v2: more real long-context tasks. v1 and v2 are not interchangeable.
Agents and tool use:
- GAIA: real-world assistant questions that need tools and multi-step work.
- AgentBench: agent skill across environments.
- BFCL (Berkeley Function-Calling Leaderboard): correctness of function and tool calling. Match the leaderboard version.
Terminal, computer use, and workplace agents
Public agent benches are no longer just GAIA-style assistant questions. Eng leads should split at least four jobs:
- Terminal agents that finish work in a shell (Terminal-Bench, plus Terminal-Bench-Science for research workflows).
- Computer-use agents that drive a real desktop (OSWorld / OSWorld 2.0).
- Knowledge-work deliverables (GDPval and the independent GDPval-AA board).
- Cross-app business execution (AutomationBench).
Harder coding peers such as SWE-bench Pro sit beside classic SWE-bench. Browser and tool peers also matter for research browsing or multi-app tool use. For example: BrowseComp, WebArena / WebVoyager-class, τ-bench, and Toolathlon.
Vendor IDE harnesses such as CursorBench show the same coding-agent test. Still, they are not fully public suites. Say so when you cite them.
Always freeze the scaffold, effort and cost notes, task-pack date, and whether live safeguards were on.
Safety and truthfulness:
- HarmBench: automated red-teaming and refusal strength.
- TruthfulQA: tendency to copy human falsehoods. It is narrow. It is not a general RAG hallucination bench.
Multimodal:
- MMMU and MMBench: college-level and broad multimodal skill. Vision encoder and prompt format matter.
Domain suites (HealthBench, FinanceBench, FinBen, and peers) exist when the product is vertical. Name the family. Then read the official paper or leaderboard for the version you cite. Do not treat a vertical suite as a clinical or financial certificate.
Systems lane: latency, throughput, and cost
This is what many platform teams mean by LLM performance testing.
Public harness families include:
- MLPerf Inference LLM workloads: standard Offline / Server / Interactive-style scenarios. They report tokens/s and TTFT / TPOT-style limits at stated percentiles, plus required accuracy floors. Vendor submissions are co-designed stacks, not your prompt mix.
- Ray LLMPerf: client-side load and correctness against LLM APIs. The repo has been read-only since Dec 2025. Teams still cite it often.
- NVIDIA GenAI-Perf / AIPerf: TTFT, inter-token latency, and token / request throughput against NIM or OpenAI-compatible endpoints.
- Open serving docs such as vLLM's: throughput and latency under batching and related features. This is not one fixed bench. Publish ISL/OSL, concurrency, hardware, and precision with every number.
Definitions to keep straight:
- TTFT: time to first token (prompt / prefill heavy).
- TPOT / ITL: time per output token / inter-token latency (decode heavy).
- Tokens/s: aggregate throughput.
- RPS / QPS: requests per second at a concurrency.
- Goodput: throughput that still meets latency and quality goals.
- $/M tokens: price card times measured usage. Not a quality score.
Synthetic ISL/OSL loads can look great. However, real multi-turn RAG or agent traffic with tool waits often looks different. So pair a systems run with at least one quality gate or an online eval.
A practical decision table
Use this when someone pastes a leaderboard screenshot into the architecture channel.
Match the job to a bench family
| Question you need answered | Criterion | Example bench family | What “good” means for you |
|---|---|---|---|
| Can it handle broad knowledge tasks? | Multitask accuracy | MMLU-Pro | Accuracy on your subject mix, not only the public average |
| Does it survive hard expert questions? | Expert reasoning | GPQA Diamond, HLE | Stable accuracy under a fixed tool policy |
| Can it do contest-style math? | Math reasoning | MATH, year-stamped AIME evals | Same year, same grading, same tools |
| Can it write and fix real code? | Coding / SWE | LiveCodeBench, SWE-bench Verified | Pass or resolve rate under your scaffold |
| Does it obey format constraints? | Verifiable instructions | IFEval | Strict accuracy on the constraints you care about |
| Do humans prefer its chat replies? | Preference | Chatbot Arena, Arena-Hard | Elo or win rate on prompts like your users |
| Does the long window actually work? | Long context | RULER, LongBench-v2 | Task accuracy at the lengths you ship |
| Can it call tools correctly? | Tool use / agents | BFCL, GAIA | Success under your tool schema and env |
| Can an agent finish hard work in a real shell? | Terminal / CLI agents | Terminal-Bench (cite version); Terminal-Bench-Science for research workflows | Resolve rate under your agent harness and budget |
| Can it operate a desktop like a human user? | Computer use | OSWorld 2.0 (cite task-pack date) | Strict and/or partial success at a step budget you can afford |
| Can it produce professional work artifacts? | Knowledge-work quality | GDPval gold / GDPval-AA (cite board version) | Preference or Elo under the same grading protocol |
| Can it complete cross-app business workflows? | Business automation | AutomationBench | Strict final-state pass rate (and cost per task) |
| Is the IDE coding agent actually helpful on messy multi-file work? | Agentic IDE coding | SWE-bench Pro (public) or CursorBench (vendor harness; label it) | Resolve / correctness under the harness you will ship |
| Will it refuse harmful asks robustly? | Safety | HarmBench | Refusal robustness without wrecking useful asks |
| Will the API feel fast under load? | Serving latency | MLPerf Server-style, GenAI-Perf / AIPerf | TTFT and TPOT inside your SLA percentiles |
| Can we afford the traffic? | Cost under load | Serving harness + price card | $/M tokens at the quality and concurrency you need |
Read the row that matches your job
Pick the row that matches the job you are hiring the model or agent to do. Then treat that family as a gate, not as a global ranking.
Why a single leaderboard is not enough
A single public ranking folds different jobs into one number.
Chatbot Arena does not tell you SWE-bench resolve rate. SWE-bench does not tell you TTFT at p95. MLPerf does not tell you whether the model follows your JSON schema.
LiveCodeBench without a date window is not comparable to last quarter's screenshot. Agent numbers without the scaffold are marketing.
Contamination, prompt sensitivity, judge bias, length gaming, and scaffold dependence are normal caveats. They are not footnotes.
Before you trust a chart, match the scaffold, the date window, and the metric definition. Also match the SLA percentile you will actually enforce.
Then match effort / cost, safeguard policy, and task-pack date. Do not stop at model name and suite name.
What platform teams should measure before they pick
Before you choose a model or agent stack, write the choice as criteria, not as a vendor name.
- Pick one capability gate tied to the job (coding resolve, tool calling, long-context QA, preference, or a domain suite).
- Pick one systems gate tied to the experience (TTFT and TPOT at target concurrency, plus $/M tokens at that quality).
- Freeze the scaffold, prompt template, tool policy, ISL/OSL, and hardware notes so the next run is comparable.
- Add a third gate only when a hire/buy decision needs another criterion (safety, multimodal, or a vertical suite).
That is enough to stop most category errors. Then expand the scorecard when the product choice is large enough to pay for the extra runs.
Bluelupin close
At Bluelupin we treat LLM benchmarks as a map, not a trophy case.
Start with one capability gate and one systems gate that map to real tools. For the coding job you ship, that might be LiveCodeBench or SWE-bench Verified.
For the stack, run a GenAI-Perf / AIPerf or vLLM-style load test. Check TTFT, TPOT, and cost at the concurrency you expect.
Expand the set when a hire or buy decision needs another criterion. Measure the job, then load-test the stack.