Ask how fast a model is and you usually get one number from a benchmark run. Your users feel something else: most answers come back quickly, some are slow, and in a chat the first words matter more than the last. So our API speed page shows all three, measured on our gateway from live calls.
Key takeaways
- Typical answer time is the number to plan with.
- The slow end tells you what one user in twenty waits for.
- First token is what a chat user feels.
- A model is only shown once it has enough real calls.
What we measure
| Number | What it means |
|---|---|
| Typical | The middle call: half are faster, half slower. |
| Slow | 1 call in 20 is slower than this. |
| Fast end | 1 call in 10 is quicker than this. |
| First token | How soon the first words arrive when the answer is streamed. |
| Output speed | How fast the rest of the answer streams after the first words. |
Where the numbers come from
Every call that goes through the withConflux gateway is timed. For the speed page we take live calls from the last 7 or 30 days, grouped by model. We leave out test keys, test accounts and failed calls. First token and output speed come from streamed calls only.
A model needs 100 calls before its answer time is shown, and 20 streamed calls before first token and output speed are shown. Until then the page says so, instead of showing a number we can't stand behind. The numbers are recalculated every hour, and the page shows the date and time they were checked.
We only ever show totals per model. Nothing on the page is per customer, and no customer is named.
Why a range beats a rank
Two models can have the same typical time and very different slow ends. If you run a chat, the slow end is the user who gives up. If you run batch jobs overnight, the typical time is what decides your throughput. That is why each model is shown as a bar from the fast end to the slow end, with a line at the typical call.
Long prompts and long answers take longer on any model. Compare models on the same kind of work, and test your own prompts before you switch.
Speed is half of the choice
The other half is what a month costs. Each compare page puts price, context and speed for two models side by side.
See how fast each model answers, measured on our gateway.