The same model, 0/5 to 5/5 — what we measured across 71 providers

We built ModelIndex.ai because comparing model prices turned out to be the easy part.

The harder question was whether the same open-weight model actually behaves the same way across providers.

Often, it doesn't.

A single checkpoint may be offered by dozens of hosts, with different quantizations, inference engines, chat templates, sampler defaults, and infrastructure. Those implementation details are usually invisible from the model listing, so we started measuring the endpoints themselves.

We now probe 223 model-provider endpoints across 39 open-weight models served by multiple providers. The broader index covers 321 models, 71 providers, and roughly 1,100 endpoints.

Here are the results that surprised us most.

1. Provider choice can change whether basic capabilities work at all

We run five small capability checks against each endpoint:

JSON-schema conformance
strict instruction following
tool calling from a supplied specification
long-context retrieval
benign-task compliance

They are graded programmatically. There is no LLM judge. A failed probe gets one automatic retry.

Of the 39 models currently measured across multiple providers, 30 receive different capability scores depending on the provider serving them.

Some differences are large.

Qwen3.5-27B passes all five checks on Alibaba and AtlasCloud. On SiliconFlow and DeepInfra it fails JSON-schema conformance and long-context retrieval, scoring 3/5. On Phala it passes one of five.

Qwen3.6-35B-A3B scores 5/5 on AtlasCloud and 1–2/5 on five other hosts. Two of those endpoints fail the tool-calling check.

DeepSeek V4 Pro scores 5/5 on DigitalOcean and Ionstream and 0/5 on CoreWeave.

These are endpoints advertised as serving the same model or checkpoint. We cannot assume that every provider is using bit-identical weights, quantization, templates, or serving configuration, and that distinction matters. What we can observe is that selecting the same model listing does not guarantee the same behavior.

We publish each individual probe result by provider in the SMOKE column on the model pages rather than collapsing everything into a single model-level score.

2. Our original throughput test was wrong for some serving stacks

The first version of our latency and throughput probe used one sentence repeated roughly 360 times to create a long input.

That looked harmless. It wasn't.

Highly repetitive text is unusually favorable to speculative decoding and related serving optimizations. When we replaced that filler with varied deterministic text and reran every shared model-provider pair, the median result barely changed: about +1%.

The outliers changed dramatically.

Cloudflare / mistral-small-3.1: 43 → 1.2 tok/s (−97%)
Novita / GLM-4.5-Air: 51 → 4.2 tok/s (−92%)
DeepInfra / gpt-oss-120b: 222 → 54 tok/s (−76%)

The original test was overstating throughput on some of the stacks that benefited most from compressible input.

So we retired those numbers and versioned the methodology.

One frontier model also refused the repetitive-input prompt entirely because its safety system treated the input as adversarial. That was another useful signal that the test itself was unrealistic.

We found a similar problem in an early capability probe.

Version 0.1 of our strict-instruction test allowed only 20 output tokens. That was enough for ordinary models, but some reasoning models consumed the budget internally before producing visible output.

Of 134 failures, 121 were caused by that test artifact.

We increased the output allowance without changing the grading criterion and reran the screen. The pass count moved from 89/223 to 213/223.

That is why methodology changes are versioned and trigger a re-screen rather than silently replacing previous results.

3. The OpenRouter hop has been smaller than we expected

For some model-provider combinations, we can measure both:

the provider's API directly, and
OpenRouter pinned to that same provider.

We expected routing through OpenRouter to add an obvious time-to-first-token penalty.

So far, that is not what the measurements show.

For gpt-oss-120b on Groq, we measure about 0.63s TTFT direct vs 0.69s through OpenRouter.

On Cerebras, we currently measure about 0.39s direct vs 0.34s routed. Cerebras also sustains roughly 1,600 tok/s in our measurements for that model, the highest throughput we have recorded for it so far.

The sample sizes are still small, so we would not generalize from those numbers to every model or request shape. But they were enough to change our assumption that the routing layer would necessarily impose a meaningful latency cost.

If latency is the reason you are avoiding a router, it is worth measuring the specific path you intend to use.

A few measurement decisions that matter

Not every API response should turn into a benchmark number.

One model, Claude Fable 5, continues to refuse our standard probe input as a policy decision. We show ✕, meaning "probed, but no measurable stream," rather than assigning it a latency or throughput figure.

That is different from —, which means we have never probed that endpoint.

Failed pairs are retried automatically, with the oldest failures prioritized.

Some reasoning models present a different problem: they can consume our standard 200-token output allowance entirely in hidden reasoning. Under the standard profile, those attempts remain failed measurements rather than being converted into an artificial TTFT value. We use a separate reasoning profile with a 4K budget for those models.

We also keep service tiers separate.

Flex, batch, and priority pricing are labeled as such and excluded from standard "best price" comparisons. Mixing them can materially distort comparisons. We previously classified some tiered OpenAI and Google prices incorrectly, fixed the data, added explicit tier labels, and verified the prices against the providers' official pricing pages.

Benchmark quality data is treated differently from our own probes. Those scores come from third parties, and every benchmark entry includes its source, date, and whether it is INDEPENDENT or SELF-REPORTED. When vendor and independent results disagree, we display both rather than averaging them together.

The probing system is also spend-capped at $50/month, and the estimated cost of each probe is recorded in the repository data.

Limitations

Single vantage point. All probes currently run from one region. Your absolute TTFT can be different. The measurements are more useful for relative comparisons than as promises about the latency you will see from your own infrastructure.

Small sample sizes. We recently seeded most of the matrix and now have measurements for 865 of 911 provider rows. Recurring probes run daily, so many trailing-72-hour medians are still based on only a handful of observations. The history becomes more useful as those samples accumulate.

Routed measurements include another network hop. Measurements taken through OpenRouter are marked with ° throughout the site. When both a direct and routed measurement exist, we prefer the direct result.

The capability checks are smoke tests, not intelligence benchmarks. Five inexpensive pass/fail checks can identify an endpoint that appears misconfigured or incapable of basic expected behavior. They cannot tell you which model is generally "smarter." Broader quality claims on the site come from separately sourced benchmark data.

Why we built this

A model name, context window, and token price do not tell you enough about the endpoint you are actually buying.

Two providers can advertise the same model while delivering substantially different capability results, latency, throughput, pricing structure, or request behavior.

That variation is what ModelIndex.ai is intended to make visible.

The site currently tracks 321 models, 71 providers, and roughly 1,100 endpoints, with verified pricing, measured TTFT and throughput, capability smoke tests, sourced quality benchmarks, per-model RSS feeds, and embeddable badges.

The data refreshes daily, and the methodology and probe versions are published alongside it.