What a support bot costs to run, measured eight ways
Prompt caching cut the API cost about 40%. One compression setting quietly cost 4 points of accuracy. Self-hosting only beat the cheapest APIs above about 1 to 1.7 requests a second. This is the report an engagement delivers, run here on a public workload.
A customer support assistant
Each request carries a 1,385-token system prompt (policies, tone, 27 intent labels), help-centre articles retrieved for the question, and a real customer message from the public Bitext support dataset. That averages 2,757 input tokens and 66 output tokens. The model answers in JSON with an intent and a reply.
Quality is graded against the dataset's intent labels on 500 held-out questions, plus 250 GSM8K maths problems. Traffic arrives at random at rising rates until responses get too slow.
- Readable speed
- 95% of users see the first word within 2 s, then at least 7 words a second
- Strict speed
- 1.5 s to the first word, then at least 13 words a second
- Quality pass
- Intent accuracy within 2 points of the baseline, valid output 99% or more, and no significant change in which questions it gets right (exact McNemar test)
- Prices
- RunPod GPU list prices: L4 $0.59/hr, L40S $1.09/hr
Self-hosted: eight setups, one change at a time
Capacity is the highest traffic each setup handled at readable speed. Cost assumes the GPU stays that busy all hour.
| Run | Setup | Capacity (req/s) | Cost per 1,000 conversations | Intent accuracy | Quality |
|---|---|---|---|---|---|
| A | L4, defaults, a per-customer line at the top of the prompt | below 0.25 | n/a | 83.6% | baseline |
| B | L4, fixed prompt first, per-customer details after it | 0.5 | $0.33 | 83.4% | pass |
| C | B + FP8 weights | 0.5 | $0.33 | 82.4% | pass |
| D | B + 4-bit (AWQ) weights | 0.5 | $0.33 | 82.6% | pass |
| E1 | C + FP8 KV cache + larger batch settings | 1 | $0.16 | 83.0% | pass |
| E2 | D + FP8 KV cache + larger batch settings | 1.5 | $0.11 | 79.6% | fail (p = 0.01) |
| F | E1 + n-gram speculative decoding | 1 | $0.16 | 83.2% | pass |
| G | L40S, fixed prompt first, no other tuning | 5 | $0.06 | 83.0% | pass |
The same conversations on API models
Priced from each provider's list prices on 10 November 2026, using the measured token counts. Answer quality of these models was not tested.
| Model | Per 1,000, no caching | Per 1,000, with prompt caching |
|---|---|---|
| OpenAI gpt-6-luna | $0.31 | $0.18 |
| Anthropic Claude Haiku 5.5 | $0.31 | $0.18 |
| Google Gemini 3.1 Flash-Lite | $0.79 | $0.48 |
| OpenAI gpt-6.1-terra | $3.09 | $1.77 |
| Google Gemini 3.5 Flash | $4.73 | $2.86 |
| Anthropic Claude Sonnet 5.5 | $6.17 | $3.54 |
Sources: OpenAI, Anthropic, Google pricing pages. Cached prices ignore one-off cache writes and Gemini's small storage fee.
Six findings
- Prompt caching is the cheapest win. Putting the fixed system prompt first cut API cost by about 40%, and on the L4 it was the difference between unusable (A) and usable (B).
- Server settings doubled the cheap GPU's capacity. FP8 weights, FP8 KV cache and larger batch settings together (E1) took the L4 from 0.5 to 1 request a second with no measurable quality change. They were tested together, so the gain isn't split between them.
- Too much compression broke quality. 4-bit weights alone passed (D). Adding the FP8 cache on top (E2) was the fastest L4 setup and cut intent accuracy from 83.6% to 79.6%. Without the quality check, it would have looked like the winner.
- Speculative decoding didn't help here. Support replies rephrase the articles rather than copy them, so n-gram drafting found little to reuse (F).
- The cheapest GPU per hour isn't the cheapest per conversation. The untuned L40S (G) cost 2.7 times less per conversation than the best L4, when traffic keeps it busy.
- Self-hosting has to earn its place. The cheapest API models cost about $0.18 per 1,000 of these conversations with caching. The tuned L4 only beats that above about 0.9 requests a second, and the L40S above about 1.7 (6,000 conversations an hour), before counting engineering time.
What this doesn't tell you
- One model and one workload, with simulated traffic. Your prompts, traffic shape and quality bar will move these numbers.
- API models were priced, not run, so their answer quality and speed are untested here.
- The first traffic level of each run was slightly slower than the next, most likely warm-up.
- A first run on 9 November had setup errors: three configurations crashed and the first quality rule was too strict for the sample size. Both are fixed, and the first run's results are kept with notes.
Scripts, prompts and raw results: github.com/poojithdevan4D/teardown-01. Anyone can rerun them on Modal.
Want this run on your own traffic?
The same measurement on your model or API, your prompts and your quality bar: baseline in 5 business days, finished in 2 weeks, from $750 (₹50,000).