Which model should sort these complaints by product?

Scroll the table sideways for every column →

1. Qwen3-4B + LoRA (fine-tuned by ILLATE on 17,905 complaints; a client keeps the weights). Cost: modal-l4 at $0.80 an hour (https://modal.com/pricing ($0.000222 a second, $0.7992 an hour, rounded to $0.80); checked 8 Oct 2026), plus its container's 4 cores and 16 GiB at modal's $0.0000131 per core-second and $0.00000222 per GiB-second (https://modal.com/pricing; checked 8 Oct 2026): $0.3165 an hour, so $1.1165 an hour in all; one L4 GPU with 4 CPU cores and 16 GiB serves 26.34 requests a second (measured on these complaints at 64 requests in flight (p95 3,292 ms at that load), vLLM 0.31.0, 193 input tokens a call), planned at most 60% busy, so $0.020 per 1,000 calls at capacity. At least one is always on: $815 a month (730 hours); one carries up to about 41.5 million calls a month, and more are added above that. The container's 4 cores and 16 GiB are an assumption, ILLATE's planning size for a vLLM container, not measured on this run; Modal bills whichever is higher, what is requested or what is used. One-off: $3,000, if ILLATE builds it: ILLATE's fixed migration fee on 9 Oct 2026, nothing up front for the first three clients, and nothing owed unless the model meets the agreed margin. In-house instead: about 45 minutes on one H100 plus your engineers' time. Volume: below about 681,000 calls a month, one always-on server costs more than GPT-5.6 Luna (20 similar examples) at $1.20 per 1,000 calls, unverified price; below about 9.5 million calls a month, more than GPT-5.6 Luna (category list) at $0.086 per 1,000 calls, unverified price; below about 19.4 million calls a month, more than GPT-6 Luna (category list) at $0.042 per 1,000 calls, unverified price. Latency: one request in flight on one L4, 300 of these complaints, measured inside the server's container (no network); the saved answers' own timings are from batched offline generation and are not used. Invalid answers: constrained decoding: it cannot answer outside the label list, so zero invalid answers is by construction, not a measured quality.
2. GPT-5.6 Luna (20 similar examples; the 20 most similar training complaints as worked examples and a strict JSON-schema answer, reasoning none; pre-registered). Cost: gpt-5.6-luna at $0.20 input and $1.20 output per 1M tokens (unverified: OpenAI's model documentation for gpt-5.6-luna, read from a public copy of OpenAI's docs; not yet checked on https://openai.com/api/pricing; read 8 Oct 2026), times the 5,367.8 input and 20.2 output tokens per call these answers used, times 1.09: our OpenAI bill for 8-9 Oct 2026 came to about 9% more than these prices for the same tokens (one factor for both Lunas; the bill has no per-model split), a floor: it priced cached tokens at the full input rate, and OpenAI may have discounted them; real-time, no caching or batch discount. In production each call also needs the complaint embedded and your labelled complaints searched for the 20 most similar: a small CPU cost, not counted here. One-off: $750, if ILLATE sets it up: ILLATE's fixed switch-to-Luna package on 9 Oct 2026, offered when Luna passes your test, covering the search over your labelled examples, the prompt and schema, the request changes and a shadow run. $0 in fees if your team builds it, plus that work. Latency: OpenAI API round trip from a Modal container, 4 requests in flight; finding the 20 similar complaints was done beforehand in one batch and is not in these times.
3. GPT-5.6 Luna (category list; the 14 product names as filed, no examples, reasoning none). Cost: gpt-5.6-luna at $0.20 input and $1.20 output per 1M tokens (unverified: OpenAI's model documentation for gpt-5.6-luna, read from a public copy of OpenAI's docs; not yet checked on https://openai.com/api/pricing; read 8 Oct 2026), times the 333.5 input and 9.9 output tokens per call these answers used, times 1.09: our OpenAI bill for 8-9 Oct 2026 came to about 9% more than these prices for the same tokens (one factor for both Lunas; the bill has no per-model split), a floor: it priced cached tokens at the full input rate, and OpenAI may have discounted them; real-time, no caching or batch discount. One-off: no fee, the switch is a prompt and request change your team makes; ILLATE's switch-to-Luna package ($750 fixed on 9 Oct 2026, offered when Luna passes your test) sets Luna up with retrieved examples instead. Latency: OpenAI API round trip from a Modal container, 8 requests in flight, reasoning effort none.
4. TF-IDF + classifier (trained on the same complaints). Cost: modal at $0.0000131 per core-second and $0.00000222 per GiB-second: $0.0791 an hour (https://modal.com/pricing; checked 8 Oct 2026); one CPU server (1 core, 4 GiB) serves 1,011.5 requests a second (one complaint at a time on one core, measured on 300 of these complaints), planned at most 60% busy, so $0.000036 per 1,000 calls at capacity. At least one is always on: $58 a month (730 hours); one carries up to about 1.6 billion calls a month, and more are added above that. One-off: no fee, built in-house: your team trains it on a CPU with the same harness. ILLATE has no fixed price for building one; your team's time is not counted. Volume: below about 48,300 calls a month, one always-on server costs more than GPT-5.6 Luna (20 similar examples) at $1.20 per 1,000 calls, unverified price; below about 674,000 calls a month, more than GPT-5.6 Luna (category list) at $0.086 per 1,000 calls, unverified price; below about 1.4 million calls a month, more than GPT-6 Luna (category list) at $0.042 per 1,000 calls, unverified price. Latency: one complaint at a time, in process, on one Modal CPU core (2 threads), no network; refitted with the saved run's settings; it agreed with the saved answers on 99.6% of test items.
5. bge-small embeddings + classifier (trained on the same complaints). Cost: modal at $0.0000131 per core-second and $0.00000222 per GiB-second: $0.0791 an hour (https://modal.com/pricing; checked 8 Oct 2026); one CPU server (1 core, 4 GiB) serves 3.9 requests a second (one complaint at a time on one core, measured on 300 of these complaints), planned at most 60% busy, so $0.0094 per 1,000 calls at capacity. At least one is always on: $58 a month (730 hours); one carries up to about 6.1 million calls a month, and more are added above that. One-off: no fee, built in-house: your team trains it on a CPU with the same harness. ILLATE has no fixed price for building one; your team's time is not counted. Volume: below about 48,300 calls a month, one always-on server costs more than GPT-5.6 Luna (20 similar examples) at $1.20 per 1,000 calls, unverified price; below about 674,000 calls a month, more than GPT-5.6 Luna (category list) at $0.086 per 1,000 calls, unverified price; below about 1.4 million calls a month, more than GPT-6 Luna (category list) at $0.042 per 1,000 calls, unverified price. Latency: one complaint at a time, in process, on one Modal CPU core (2 threads), no network; encoder plus a linear classifier of the same shape, encoder max length 512 tokens.
6. bge-base embeddings + classifier (trained on the same complaints). Cost: modal at $0.0000131 per core-second and $0.00000222 per GiB-second: $0.0791 an hour (https://modal.com/pricing; checked 8 Oct 2026); one CPU server (1 core, 4 GiB) serves 1.7 requests a second (one complaint at a time on one core, measured on 300 of these complaints), planned at most 60% busy, so $0.022 per 1,000 calls at capacity. At least one is always on: $58 a month (730 hours); one carries up to about 2.7 million calls a month, and more are added above that. One-off: no fee, built in-house: your team trains it on a CPU with the same harness. ILLATE has no fixed price for building one; your team's time is not counted. Volume: below about 48,300 calls a month, one always-on server costs more than GPT-5.6 Luna (20 similar examples) at $1.20 per 1,000 calls, unverified price; below about 674,000 calls a month, more than GPT-5.6 Luna (category list) at $0.086 per 1,000 calls, unverified price; below about 1.4 million calls a month, more than GPT-6 Luna (category list) at $0.042 per 1,000 calls, unverified price. Latency: one complaint at a time, in process, on one Modal CPU core (2 threads), no network; encoder plus a linear classifier of the same shape, encoder max length 512 tokens.
7. GPT-6 Luna (category list; the 14 product names as filed, no examples, reasoning none). Cost: gpt-6-luna at $0.10 input and $0.50 output per 1M tokens (unverified: a third-party pricing article, not OpenAI; not yet checked on https://openai.com/api/pricing; read 4 Oct 2026), times the 333.5 input and 10.5 output tokens per call these answers used, times 1.09: our OpenAI bill for 8-9 Oct 2026 came to about 9% more than these prices for the same tokens (one factor for both Lunas; the bill has no per-model split), a floor: it priced cached tokens at the full input rate, and OpenAI may have discounted them; real-time, no caching or batch discount. One-off: no fee, the switch is a prompt and request change your team makes; ILLATE's switch-to-Luna package ($750 fixed on 9 Oct 2026, offered when Luna passes your test) sets Luna up with retrieved examples instead. Latency: OpenAI API round trip from a Modal container, 8 requests in flight, reasoning effort none.
8. MiniLM embeddings + classifier (trained on the same complaints). Cost: modal at $0.0000131 per core-second and $0.00000222 per GiB-second: $0.0791 an hour (https://modal.com/pricing; checked 8 Oct 2026); one CPU server (1 core, 4 GiB) serves 10.2 requests a second (one complaint at a time on one core, measured on 300 of these complaints), planned at most 60% busy, so $0.0036 per 1,000 calls at capacity. At least one is always on: $58 a month (730 hours); one carries up to about 16.1 million calls a month, and more are added above that. One-off: no fee, built in-house: your team trains it on a CPU with the same harness. ILLATE has no fixed price for building one; your team's time is not counted. Volume: below about 48,300 calls a month, one always-on server costs more than GPT-5.6 Luna (20 similar examples) at $1.20 per 1,000 calls, unverified price; below about 674,000 calls a month, more than GPT-5.6 Luna (category list) at $0.086 per 1,000 calls, unverified price; below about 1.4 million calls a month, more than GPT-6 Luna (category list) at $0.042 per 1,000 calls, unverified price. Latency: one complaint at a time, in process, on one Modal CPU core (2 threads), no network; encoder plus a linear classifier of the same shape, encoder max length 256 tokens.

Accuracy intervals: 10,000 bootstrap resamples of the test items. Paired intervals resample the same items for both models, as illate parity does. Macro-F1 averages F1 over every category in the label list that has test items, so a category an option never predicts counts as zero. Invalid: an answer outside the label list. Failed: no answer after retries. Both count as wrong. As filed: the same answers scored on the 14 labels before the mapping in section 08.

What each option costs a month

Scroll the table sideways for every column →

Highlighted: the cheapest at that volume, whatever its accuracy. On a tie, the more accurate option, as in section 01: at 5 million calls a month TF-IDF + classifier, bge-small embeddings + classifier and MiniLM embeddings + classifier cost the same, so TF-IDF + classifier, the most accurate of them, is highlighted. The recommendation only considers options within the margin (section 01). Per-call options: price times the tokens their answers used, no batch or cache discount. Hosted options: at least one server always on, more when the volume needs them, planned at most 60% busy, at the throughput in the notes above. Running costs only: one-off costs to build or switch are in the next table.

First year, one-off costs included

Scroll the table sideways for every column →

An assumption, not a quote: 12 months of the running cost above plus each option's one-off cost to build or switch to it, as the One-off notes under the options table say (who does the work, at what price, and the alternative). Your own team's time is not counted for any option. Highlighted as above: at 5 million calls a month TF-IDF + classifier, bge-small embeddings + classifier and MiniLM embeddings + classifier cost the same, so TF-IDF + classifier, the most accurate of them, is highlighted.

Scroll the table sideways for every column →

Scroll the table sideways for every column →

Confusion matrix, every category

Scroll the table sideways for every column →

Grey diagonal: correct. Shaded: mistakes, darker for more items. Column numbers match the row numbers. Cells are counts; each mistake's share of its true category is in the list above.

Scroll the table sideways for every column →

Dates, from this report to 23 Oct 2026

1. 9 Oct 2026Agree in writing: the 3-point margin, the label mapping in section 08, the interim model below, and who labels the shadow items.
2. 11 Oct 2026Write the prompts and the request changes, set up any search for similar examples they use, score GPT-5.6 Luna (20 similar examples) and the stronger setup section 01 names for GPT-6 Luna (category list) on your held-out items, and re-run this report with those answers.
3. 12 Oct 2026Shadow starts next to your current model (stage 1).
4. 20 Oct 2026Production moves off the old model to the interim: GPT-5.6 Luna (20 similar examples), the recommended option, or the needs_human_review queue (people triage every request: about 4,200 reviewer-hours a month at 500,000 calls, at 30 seconds an item) if its shadow gate hasn't passed by then. Test the switch while the old model still answers.
5. 23 Oct 2026Fine-tunes of gpt-4.1-nano, o4-mini, gpt-3.5-turbo and gpt-4 stop answering. The interim serves production on and after this date until the gates below pass; shadow continues next to it.
6. Then5%, 25% and 100% of live traffic, each when its gate passes: a gate is a check, not a date.

Labelling: the first gate needs 1,000 shadow items labelled by people. At 30 seconds an item (an assumption: time your reviewers on 20 items) that is about 8 reviewer-hours. Label them from shadow items your current model answered, before 20 Oct 2026: the first gate compares the new model with the answers production gave, and answers from the needs_human_review queue can't be compared (section 06, stage 1). Label a random sample of traffic, without showing either model's answer, so the comparison isn't tilted toward hard or easy cases.

Confidence and the review queue

No confidence scores were measured for GPT-5.6 Luna (20 similar examples) in this report. Measure them in shadow (illate defer picks a threshold on one labelled set and checks it on another) before the 5% stage; until then no answer carries a review flag.

Categories GPT-5.6 Luna (20 similar examples) gets right less than 50% of the time: Debt or credit management (0 of 5 right; mostly labelled Credit reporting or other personal consumer reports). Decide before the 5% stage whether any of these needs a rule.

Stages and gates

1. 1. ShadowGPT-5.6 Luna (20 similar examples) answers every request, but only production's answer is used: your current model, and from 20 Oct 2026 the interim. People label a random sample.Gate to 5%on at least 1,000 labelled shadow items that a model answered in production (your current model before 20 Oct 2026), illate parity --gold shadow-labels.jsonl --current production.jsonl --candidate new.jsonl --labels labels.json --mapping mapping.json --margin 0.03 --out parity-5pct says PASS: paired accuracy on your reviewers' labels for the same items, the new model within the 3-point margin of what production served. Items the needs_human_review queue answered can't be in that comparison: a person labelled them, so production is right on them by definition. If production is on the queue before 1,000 model-answered items are labelled, this part of the gate is an absolute floor instead: the lower end of the new model's 95% accuracy interval on those labelled items (the candidate row of the same parity report) at or above 86.5%. Also: accuracy of at least 86.5% (Qwen3-4B + LoRA 89.5% here, minus the margin); no invalid answers (none were measured here; investigate any); p95 latency inside your budget (measured here: 1,225 ms).
2. 2. 5% of live trafficA random 5% of requests get the new answer: one category, or defer (needs_human_review on this page) when its confidence is below the threshold you chose from the table above, and that item goes to a person from the start.Gate to 25%the share answered defer is within a few points of the rate in the table; for each category with at least 50 labelled items pooled across weeks, the 95% Wilson interval of its recall reaches section 03's figure minus the margin (smaller categories are read by a person, not gated); no mistake pattern that section 04 doesn't show.
3. 3. 25% of live trafficThe same checks on a larger slice, for at least one full weekly cycle.Gate to 100%every earlier gate still holds on the pooled labels, and the rollback in section 07 has been tested once by flipping the switch and seeing traffic reach the fallback.
4. 4. 100%Keep labelling a small random sample every week and re-run this report on it.Roll back (section 07) ifaccuracy pooled over four weeks falls below 86.5%; invalid answers rise above the gate; or p95 latency leaves your budget.

In the command: shadow-labels.jsonl holds your reviewers' labels, production.jsonl and new.jsonl each model's answers on the same ids, and labels.json and mapping.json are the label list and mapping in section 08. With --gold it compares accuracy on the labels; without it, illate parity would only measure agreement with production, mistakes included.

Check the numbers yourself

import json
import numpy as np

b = json.load(open("sample-decision-report-inputs.json"))


def paired(y, a, c, n_boot=10_000, seed=2):  # c minus a, as illate parity draws it
    d = (y == c).astype(float) - (y == a).astype(float)
    rng, n, means = np.random.default_rng(seed), len(d), []
    chunk = max(1, 2_000_000 // n)
    for s in range(0, n_boot, chunk):
        means.append(d[rng.integers(0, n, (min(chunk, n_boot - s), n))].mean(axis=1))
    lo, hi = np.quantile(np.concatenate(means), [0.025, 0.975])
    return d.mean(), lo, hi


def table(title, mapping, answers, ref):  # -1 is an answer outside the list: never right
    name = [mapping.get(x, x) for x in b["labels"]] + ["(outside the list)"]
    y = np.array([name[i] for i in b["gold"]])
    ans = {k: np.array([name[i] for i in v]) for k, v in answers.items()}
    print(f"# {title}: accuracy, then each option minus {ref}")
    for k, a in ans.items():
        diff, lo, hi = paired(y, ans[ref], a)
        print(f"{k:55} {np.mean(y == a):.4f}  {diff:+.4f} [{lo:+.4f}, {hi:+.4f}]")


table("section 02", b["mapping"], b["answers"], b["reference"])
if b["mapping"]:  # section 01: the same rule on the labels as filed
    table("as filed", {}, b["answers"], b["as_filed_reference"])
for k, v in b.get("served_answers", {}).items():  # section 01: one option as served
    table(f"{k} as served", b["mapping"], {**b["answers"], k: v}, b["reference"])
for k, (a, c) in b.get("setup_gains", {}).items():  # section 01: what a setup added
    table(f"{k} stronger setup", b["mapping"], {x: b["answers"][x] for x in (a, c)}, a)

The commands that produced this page

illate is ILLATE's command-line harness; a client receives it with the weights and scripts. The answer files and measurements sit on ILLATE's Modal volume; every one is fingerprinted below, and the ones whose names start with sample-decision-report- are published next to this page.

modal run scripts/modal_bench_task.py     # serving figures, measured on these complaints
modal run scripts/modal_decision_report.py
# mounts the illate-runs volume at /vol, writes the candidate list and mapping from MODELS and CFPB_MERGE in scripts/modal_bakeoff_data.py, then runs:

illate decision-report --gold /root/data/cfpb-product/test.jsonl --labels /root/data/cfpb-product/labels.json --mapping /tmp/decision/sample-decision-report-mapping.json --candidates-file /tmp/decision/sample-decision-report-candidates.json --prices /root/data/prices/2026-10.yaml --shutdown 2026-10-23 --html /tmp/decision/report.html --json /tmp/decision/report.json --bundle /tmp/decision/sample-decision-report-inputs.json --bundle-attach 'luna category-list prompt=/tmp/decision/sample-decision-report-luna-prompt.txt' --input 'Luna category-list prompt=/tmp/decision/sample-decision-report-luna-prompt.txt' --input 'fine-tune serving measurements=/vol/bench/cfpb-product/bench.json' --input 'CPU timings=/vol/bench/cfpb-product/cpu.json' --input 'fine-tune training record=/vol/client-cfpb-product/lora/run.json' --input 'intake manifest (training count, label conflicts)=/vol/client-cfpb-product/manifest.json' --bundle-attach 'luna nearest20 prompt=/tmp/decision/sample-decision-report-luna-nearest20-prompt.txt' --input 'Luna 20-example prompt=/tmp/decision/sample-decision-report-luna-nearest20-prompt.txt' --input 'GPT-5.6 Luna 20-example run status=/vol/openai_fair_more/cfpb-product/gpt-5.6-luna/nearest20/run-status.json' --input 'GPT-6 Luna 20-example run status=/vol/openai_fair_more/cfpb-product/gpt-6-luna/nearest20/run-status.json' --input 'GPT-6 Luna 20-example smoke test=/vol/openai_fair_more/smoke/cfpb-product/gpt-6-luna/nearest20/test.preds.jsonl'

Every file read, with its SHA-256

Scroll the table sideways for every column →

Label mapping applied before scoring

Scroll the table sideways for every column →

Why: On 25 Aug 2023 the CFPB renamed three of its products, and the filing date isn't in the complaint text, so a model can only guess which of a product's two names was in use; for routing they are the same product.

Seeds: 0 for each option's interval, 2 for paired differences, 10,000 resamples. illate parity --mapping with the same mapping and label list prints the same paired interval for any pair of these answer files.