It depends on what you're serving and how much traffic you're handling. Here's the honest answer: most inference workloads don't need an H100, and you'll save a fortune by starting with RTX 4090s.
The RTX 4090 at $0.18/hr on io.net handles quantized models up to 13B parameters beautifully. It pushes 80-120 tokens/second on a 7B model with vLLM, which covers the majority of production chatbots, coding assistants, and content generation APIs. The H100 at $2.20/hr makes sense when you're serving 70B+ models at scale, need FP8 inference for maximum throughput, or require the Transformer Engine for latency-sensitive applications.
That's a 12x price difference. So unless you specifically need what the H100 offers, the 4090 is almost certainly the right call.
When the RTX 4090 Wins
The 4090 has 24GB VRAM, which is enough for:
- 7B models in FP16 — Llama 3 8B, Mistral 7B, Qwen 2.5 7B fit comfortably
- 13B models in 4-bit quantization — GPTQ or AWQ quantized 13B models run at near-full quality
- Stable Diffusion XL — generates images in 2-3 seconds per batch
- Whisper large-v3 — real-time transcription at 30x speed
- Embedding models — BGE, E5, or sentence-transformers for RAG pipelines
Throughput benchmarks (vLLM, Llama 3 8B, batch size 32):
| Metric | RTX 4090 | H100 SXM |
|---|---|---|
| Tokens/sec | 95 | 380 |
| Time to first token | 45ms | 18ms |
| Cost per 1M tokens | $0.53 | $1.62 |
| Requests/sec (512 tokens) | 5.8 | 23 |
The 4090 delivers tokens at one-third the cost. For most startups and mid-size deployments, you'd need 4x the traffic volume before the H100's raw throughput advantage justifies the price.
When the H100 Makes More Sense
There's a crossover point. The H100 becomes the better economic choice when:
1. You're serving 70B+ models unquantized
The H100's 80GB HBM3 memory fits a 70B model in FP16. The 4090 can't do that — you'd need tensor parallelism across 4 cards, which adds complexity and latency.
2. Traffic exceeds ~200 concurrent users on a single endpoint
At high concurrency, the H100's 4x throughput advantage means you need fewer GPUs total. The math flips around 200+ concurrent users:
- Serving 500 users: 5x RTX 4090 ($0.90/hr) vs 2x H100 ($4.40/hr) — 4090 wins
- Serving 2,000 users: 20x RTX 4090 ($3.60/hr) vs 5x H100 ($11.00/hr) — still 4090
- Serving 10,000 users: 100x RTX 4090 ($18.00/hr) vs 25x H100 ($55.00/hr) — 4090 wins on cost but the operational overhead of 100 GPUs matters
3. Latency requirements are under 20ms TTFT
Financial services, real-time game AI, and interactive agents sometimes need sub-20ms time-to-first-token. The H100's HBM3 bandwidth (3.35 TB/s vs 1.01 TB/s) makes this achievable.
4. FP8 inference is critical
The H100's native FP8 support provides 2x throughput over FP16 with minimal quality loss. The 4090 lacks hardware FP8 — you're limited to INT8 or FP16.
Cost-Per-Token Breakdown
This is what actually matters for production inference budgets:
Llama 3 8B (common chatbot model):
| GPU | Hourly Cost | Tokens/hr | Cost per 1M tokens |
|-----|-------------|-----------|---------------------|
| RTX 4090 | $0.18 | 342,000 | $0.53 |
| A100 80GB | $1.49 | 720,000 | $2.07 |
| H100 SXM | $2.20 | 1,368,000 | $1.61 |
Llama 3 70B (enterprise reasoning model):
| GPU | Hourly Cost | Tokens/hr | Cost per 1M tokens |
|-----|-------------|-----------|---------------------|
| RTX 4090 (4-bit quant) | $0.18 | 48,000 | $3.75 |
| A100 80GB | $1.49 | 180,000 | $8.28 |
| H100 SXM | $2.20 | 420,000 | $5.24 |
Even for 70B inference, the quantized 4090 route costs less per token. The quality tradeoff from 4-bit quantization is minimal for most applications (within 1-2% on standard benchmarks).
The Smart Approach: Start With 4090s, Graduate to H100s
Most teams that end up on H100s started on 4090s. Here's a sensible scaling path:
- Prototype and validate on a single RTX 4090 ($0.18/hr). Test your model, measure latency, validate the user experience.
- Scale horizontally with multiple 4090s behind a load balancer as traffic grows.
- Upgrade to H100s only when you hit one of the thresholds above — large unquantized models, extreme latency requirements, or operational complexity from managing too many 4090s.
This approach keeps your burn rate low during the critical early stages and delays the 12x cost increase until your revenue justifies it.
Start inference on io.net — RTX 4090 from $0.18/hr, H100 from $2.20/hr. Deploy now