Methodology

The rules

• Same inputs, same client, same region. Every provider gets the identical task set, called from the same harness in the same cloud region. The region is recorded with each run.
• Documented defaults. Each provider runs on its documented default or recommended settings. Any setting we change is listed on the results page.
• Providers see their results first. Five business days before a page goes up, each provider gets its own results and the method. Genuine misconfigurations are fixed and noted in the changelog.
• Dated and priced. Every page shows the run date, the API versions, the prices used and where they came from, and the known limitations.
• Raw results published. Every provider's score on every task, and its output wherever the provider's terms allow it.
• Terms checked before running. We read each provider's terms for limits on publishing benchmark results before running it. A provider whose terms restrict it is left out, and the page says so.
• Cost at list price. Cost is units used times the public list price on the date captured. Where a provider reports what it billed, we record both.
• No paid placement. No provider pays for its ranking.

The harness

The harness calls every provider through the same interface, with the same retries, timeouts, and concurrency. Responses are cached on disk, so a rerun scores the same outputs without paying for them again, and every run records its git commit, date, provider versions, settings, and region.

Disclosure

Kiwi offers access to these providers through one API key. No provider pays for its ranking.