Best GPU clouds for AI inference (2026)
Inference economics differ from training: latency, cold-start, billing granularity, and per-token cost all matter more than peak FLOPS. The right GPU for an 8B model is rarely the right GPU for a 405B model. Here's how the three axes split.
The cost pick, pricing and availability refresh hourly. Our performance and operations picks are curated.
Hot Aisle · 8× MI300X
192 GB HBM3 per GPU fits Llama-3.1-405B / DeepSeek-V3 in a single 8× box with no tensor-parallel overhead. ROCm has caught up for vLLM and SGLang in 2025. Bare-metal, 100% availability coverage in our 30-day window.
Runpod · 1× H100
Pod (long-lived) and serverless (per-second billing, sub-30s cold start) both available. GraphQL API with documented lifecycle hooks. Spot pricing exposed for cost control on bursty workloads. Free egress.
Cheapest in-stock 1× L40S
Vast · Virginia, US
$0.47/hr on demand · 1× L40S
Limited stock when checked
How we choose
We use the 48 GB L40S for small-model inference. Throughput and cost per token depend on the model and serving setup. We compare self-serve on-demand listings for 1× L40S. Each listing needs a stock report of available or limited within the past 6 hours. The lowest hourly price wins. Ties are ordered by provider, then region.
Compare L40S listings →Avoid: Vast
Marketplace listings (200+ configs) are great for batch and dev work, but the underlying machine can vanish without notice. Reliability scores are real-only after 48h coverage for a reason. Don't use Vast for production inference SLAs.
The caveat we wish more pages mentioned
Egress is often the dominant inference cost. A 100 RPS API serving 200 KB responses pushes ~50 TB/month of egress. That's $4,500/mo on AWS or GCP at $0.087-0.12/GB — and $0 on a free-egress neocloud. Often more than the GPU bill itself. Compare on per-month TCO including egress, not on $/hr alone.
Data we don't yet show — and how it might change the call
- Tokens/sec or TFLOPS benchmarks per (GPU, framework, model) — single biggest gap for honest cost-per-token recommendations
- Cold-start time on serverless endpoints (model weights into VRAM)
- Auto-scaling minimum charge / idle time billing
Honesty about gaps beats false confidence. We add data as it becomes structurally available.
Notable absences
- Replicate — Obvious managed-inference recommendation. Different category (PaaS, not IaaS) — they abstract away the GPU choice and bill per-prediction. Runpod serverless is the closest analog in our IaaS catalog.