GPT-5.6 Luna vs a fine-tuned model for ticket and intent classification

The pre-registered test: Luna at its best

Our first Luna runs, on 8 October 2026, gave it a plain prompt: on Banking77, 85.4% with the category list and 87.2% with one example of every category. A critic could fairly say we handicapped it. So before running anything else, we wrote down every setting, the run order, a $20 spending cap and the rule for what we would publish, and committed it. Then we ran five of the seven pre-registered Banking77 settings once each, on 8 October from 21:24 to 22:17 UTC, and changed nothing afterwards; the two left out for the budget are named below.

When Luna tied on Banking77, we did the same for the other two test sets: a second pre-registration with the same prompt, the same rule and a $50 cap, committed at 22:54 UTC on 8 October. The runs started 54 seconds later and finished at 23:57 UTC.

• The prompt. The category list, Structured Outputs holding every answer to the label list, and the 20 training examples most similar to each test item as worked examples (section 04 shows it).
• The runs. On Banking77: GPT-5.6 Luna and GPT-6 Luna, each at reasoning effort none and low, plus OpenAI's Decisions API (beta) on GPT-6 Luna. On CLINC150 and CFPB: each Luna at reasoning effort none. GPT-6 Luna on CFPB didn't fit the $50 cap and wasn't run, as the pre-registration expected.
• Luna's best. The highest accuracy among all complete Luna runs on each test set, including the ones from earlier on 8 October: 9 runs on Banking77, 4 on CLINC150 and 3 on CFPB. Taking the best flatters Luna slightly. That was deliberate.
• The rule. If our model leads Luna's best by under 2 points, or their 95% intervals overlap, we publish and drop the accuracy headline. If it leads by 2 to 5 points, we lead with the per-category results, not the gap.

What made the difference

• The examples, not the model or the reasoning. On Banking77, GPT-5.6 Luna went from 87.2% with one fixed example of every category to 93.8% with the 20 most similar ones: +6.6 points (+5.6 to +7.6). GPT-6 Luna went from 82.2% to 93.1%. On CLINC150, GPT-5.6 Luna went from 88.4% with the category list only to 94.6%: +6.3 points (+5.5 to +7.1).
• On long complaints, the examples barely moved the score. On CFPB, GPT-5.6 Luna went from 86.6% to 87.5%: +0.9 points (+0.1 to +1.7). On the 14 labels as filed they helped far more, from 51.2% to 62.3%. Most likely the examples carry the agency's old and new product names, which a category list can't teach.
• Reasoning effort low didn't help. On GPT-5.6 Luna it made no measurable difference on Banking77: 93.5% against 93.8%. On GPT-6 Luna it cost 1.9 points (+1.2 to +2.7): 91.1% against 93.1%.
• The newer model wasn't better. With the same 20 examples, GPT-5.6 Luna beat GPT-6 Luna by 0.7 points on Banking77 (+0.1 to +1.3). On CLINC150 the two were 0.2 points apart, which the test can't tell from zero (−0.2 to +0.6).
• The Decisions API was fast and cheap, but less accurate: 76.8% (75.3–78.2) on Banking77, 120 ms at the median. It got the bare category names as choices, with no descriptions and no examples. 10 of its 3,080 answers were refusals, counted as wrong.

The comparisons between Luna runs in this list weren't pre-registered. They use the same paired bootstrap, so read them as post-hoc. Two pre-registered Banking77 runs, each Luna with the schema and no examples, were left out before any full run because the $20 cap didn't cover all seven. We budgeted both jobs at deliberately high prices: $16.38 of a $20 cap for Banking77 and $43.17 of a $50 cap for CLINC150 and CFPB, smoke tests included. From our OpenAI bill for 8-9 Oct, they cost about $3.70 and $9.40. The 20-example prompt also added Structured Outputs and a slightly different instruction line. The schema-only runs that would separate those changes were the two left out, so the 6.6 points are the examples plus those changes. On 8 October, without a schema, GPT-5.6 Luna gave no invalid answers on Banking77, so the schema had none to fix. One unplanned event: Modal restarted the container once during the CLINC150 run on GPT-6 Luna. It resumed from the answers already saved; a few requests (2 or 3, going by our bill) were sent twice, and only the second answer exists.

Why GPT-5.6 Luna is the comparison that matters

Fine-tuned gpt-4.1-nano, o4-mini, gpt-3.5-turbo and gpt-4 models stop answering on 23 October 2026, and for fine-tuned gpt-4.1-nano OpenAI names gpt-5.6-luna as the recommended replacement base model (OpenAI deprecations page, checked 8 October 2026).

Luna can't be fine-tuned: its model page lists fine-tuning as not supported (GPT-5.6 Luna model page, checked 8 October 2026). So a team that moves a nano fine-tune to Luna replaces a trained model with a prompt. This page measures how close a prompt gets on classification, next to a fine-tuned small model and a much smaller embeddings classifier.

The named replacement has moved before. As recorded by a public mirror of OpenAI's page, it listed gpt-5-nano (25 April 2026), then gpt-5.4-nano (13 May 2026), then gpt-5.6-luna (4 August 2026); the dates are when the mirror's copy changed (history of the page in a public mirror, checked 8 October 2026). gpt-5-nano itself shuts down on 11 December 2026 (OpenAI deprecations page, checked 8 October 2026).

On 8 October 2026 we read these pages through verbatim public GitHub mirrors of OpenAI's docs; the links go to the live pages, except the change history. What changes in the request when you switch, and every dated fact: replacing a fine-tuned gpt-4.1-nano. Every retiring fine-tune and what to do before 23 October 2026: the shutdown checklist.

The prompts

Category list only (8 October). One system message: a task line, an instruction to answer with one category name, and every category name. Then the text to classify.

An example of every category (8 October). The same system message, then one worked example of every category from the training split, as a user message and the right answer, in label order: 77 examples for Banking77, 151 for CLINC150 and 14 for CFPB. Then the text to classify.

The 20 most similar training examples (pre-registered, all three test sets). A system message with the task line, an instruction to choose the one category that fits best, and every category name. Then the 20 training items most similar to the one being classified, each as a user message followed by its right answer as JSON, least similar first, so the closest sits just before the text to classify. Similarity is cosine over BAAI/bge-small-en-v1.5 embeddings, searched over the training split only: 9,000 messages for Banking77, 15,250 requests for CLINC150 and 17,905 complaints for CFPB. Those are the same rows our model trained on, never a dev or test row. A strict JSON schema whose enum is the test set's label list holds every answer to it: 77 labels, 151 with out-of-scope, and the 14 CFPB products as filed. Only the task line and the labels change between test sets.

On 8 October we didn't constrain the API models' output to the label list. Each answer was matched to a category generously (exact, then ignoring case and punctuation, then a close spelling match); an answer that matched nothing counts as invalid and wrong. Invalid answers were rare: at most 15 in 5,500. In the pre-registered runs, Structured Outputs held all 28,437 chat answers to the label list: 12,320 on Banking77 and 16,117 on CLINC150 and CFPB. Refusals from the Decisions API count as wrong.

Statistics. Accuracy comes with a 95% bootstrap confidence interval from 10,000 resamples of the test set. "Ours minus it" is a paired bootstrap: both models are scored on the same resampled items, which removes the noise from item difficulty. If that interval includes zero, the test can't tell the two models apart. Macro-F1 averages F1 over categories, so a small category counts as much as a big one; an answer outside the label list counts as wrong, not as a category of its own. On CLINC150, the off-topic and wrongly-rejected rates each have a 95% bootstrap interval over their own requests (1,000 off-topic, 4,500 real).

Off-topic requests

1,000 of the 5,500 test requests fit none of the 150 intents, and the right answer is "out of scope". The training split has only 250 of them among 15,250, which likely limits how many off-topic examples Luna's search can find. The trained models learned out-of-scope as one more class; Luna was told to answer oos when nothing fits. Two rates matter: how many off-topic requests a model catches, and how many real requests it wrongly turns away.

From our OpenAI bill for 8-9 Oct, GPT-5.6 Luna with examples comes to about $370 per million calls on Banking77 and about $1,200 on CFPB. So on Banking77 the CPU server costs less than Luna above roughly 160,000 calls a month, and the L4 with its container above roughly 2.2 million. On CFPB the L4 costs less above roughly 681,000. GPT-6 Luna costs about half as much per call ($0.18 per 1,000 on Banking77) and scored 0.7 points lower there. These figures leave out the cost and time of finding the 20 examples for each live request.

Low volume, or you'd rather not host a classifier: Luna with retrieved examples

If it passes your own test set, use it. You still need a labelled pool to retrieve from, and a small embeddings model or embeddings API to find the nearest examples for each request: our tests drew on the full training splits, 9,000 to 17,905 labelled rows. On long texts every call gets long too: about 5,400 input tokens a call on CFPB complaints. Pin the exact model ID and keep your test set, because OpenAI can change the model behind a name, and has named three replacements for nano fine-tunes since April 2026.

A flat bill on short, simple texts: an embeddings classifier

On Banking77 it tied ours, needs no GPU, and an engineer can train it in-house. It fell further behind on CLINC150, where it caught at most 62.9% of off-topic requests and Luna with examples was 4.0 points ahead of it, and on long CFPB complaints.

Long or messy text, high volume, or the cheaper options miss your bar: a model you own

A fine-tuned small model costs the same at 1 million calls a month as at 100 million. On one L4 it answered in 194 ms at the median one request at a time, and in 310 ms at about 50 requests a second (results page). It doesn't move unless you move it. With similar examples, Luna tied it on CLINC150. On long CFPB complaints it stayed 2.0 points ahead, with renamed products merged after seeing the errors, and the largest product alone accounts for more than that lead. Each Luna call there cost about three times as much as on short messages.

On your data

Whoever runs it for you, these 12 questions show what the report should contain. What ours looks like: a sample decision report on the same public CFPB complaints.

• We run the same bake-off on your labelled data at no charge: GPT-5.6 Luna with your most similar labelled examples, an embeddings classifier and a model you'd own, plus your current model's answers if you still have them.
• The check runs inside your cloud account or on a machine you control, so your data and the answers stay with you. You run the Luna column in your own OpenAI organization, with your own key; we never hold your key. We train only on labels a person or your records confirmed.
• If Luna with retrieved examples passes, the recommendation is Luna. We set it up and test it on your data for a fixed $750. If an embeddings classifier is enough, we say that too.
• If a model you own is needed, the migration is $3,000 fixed for one classification or routing task; the first three clients pay nothing up front and owe nothing unless the final model meets the margin we agree.

What this comparison doesn't show

• One run each. The intervals cover sampling noise in the test set, not run-to-run variation: a different training seed for ours, or a different day on the API for Luna. OpenAI can also change a model behind the same name.
• Public datasets. Banking77 and CLINC150 are clean, English and single-label; the CFPB complaints are real but public. Any of these models, including the base model we fine-tune, may have seen them during pre-training, and we can't rule that out. Your data is the test that counts.
• CFPB's result is on a knife-edge. Ours leads Luna's best by 104 of 5,117 complaints, and the 2-point line in our rule is 102.3. Two more right answers for Luna would have put CFPB in the "drop the accuracy headline" band too. The paired interval, +1.2 to +2.9, is above zero but includes values under 2.
• GPT-6 Luna with examples on CFPB is untested. It didn't fit the $50 cap. Running it now would be a new decision made after these results.
• Luna's best is the best of up to 9 runs per test set, which flatters it slightly. We chose that on purpose, before the results.
• A large labelled pool. Luna drew its examples from the full training split of each test set, 9,000 to 17,905 labelled rows, the same rows our model trained on. How it does with fewer labelled examples is untested.
• Post-hoc comparisons. The per-product table, the real-requests-only score on CLINC150 and the comparisons between Luna runs weren't pre-registered. Per-product F1 has no intervals.
• Not tested: reasoning effort medium; each Luna with the schema and no examples on Banking77 (pre-registered, left out for the budget); GPT-6 Luna with examples on CFPB; the Decisions API on CLINC150 and CFPB, or with choice descriptions; a fine-tuned encoder such as ModernBERT; and fine-tuned OpenAI models, because OpenAI refused to start a fine-tuning job on our account on 7 October 2026.
• Prices checked against our bill, not per model. The per-token prices come from mirrors of OpenAI's docs and a third-party article. Our OpenAI bill for 8-9 Oct came to about 9% more than those prices for the same tokens, across both Lunas, so the costs here include that 9%. The bill doesn't split by model, so each model's own price is still unverified. We priced every input token at the full rate; if OpenAI discounted the tokens it cached, the per-call cost on prompts that don't cache is higher than shown.
• The CFPB merge was chosen after seeing the confusions. In a migration, we agree a label mapping like this with you before training. The complaints were cut to 2,000 characters for every system alike.
• Latency was measured differently for the API and for our model (section 08).

To try the Luna prompt on your own data: embed your labelled examples and each incoming message with BAAI/bge-small-en-v1.5 (normalised, so cosine is a dot product), take the 20 nearest labelled examples, and send the prompt above with a strict JSON schema whose enum is your label list. Score it on a held-out set it never retrieves from.

The CFPB file is built by scripts/make_cfpb_client_file.py and goes through the same one-command check in scripts/modal_run.py that a client's file does. The 8 October prompts and answer matching are in src/illate/baselines/llm.py; the bootstrap and paired test in src/illate/bootstrap.py and src/illate/parity.py.

These scripts are in ILLATE's private repository; a migration hands yours over with the report. Public now: the parity check (github.com/illate/parity), the Banking77 adapter (illate/qwen3-4b-banking77-lora), the prompt templates above and bakeoff.json.