← Engineering notes

Zero invalid labels: constrained decoding for classification

A fine-tuned language model is still a free-text generator. For a classifier, every answer outside the label set is a production bug. Here is how we remove that failure mode, and what it costs.

5 Oct 2026Banking77 · 77 labelsQwen3-4B + LoRAvLLM 0.31 · L4

The failure

Our first smoke test of the fine-tuned model scored 39.5% on 200 test messages. The model wasn't confused about intent. It was inventing names: 68 of 200 answers weren't any of the 77 labels. It would write a plausible card_not_arriving where the label is card_arrival.

Parsing and fuzzy matching would hide some of that, but a classifier that sometimes returns a category that doesn't exist will eventually route a ticket to nowhere. We wanted it to be impossible, not unlikely.

The fix: decode through a label trie

We tokenize every label once and build a trie of token ids, each path ending in the end-of-turn token. At each decoding step, a logits processor masks every token that doesn't continue a valid path. Generation can only stop after a complete label, so the output is always one of the 77 names.

def allowed_next(trie, generated, end_id):
    node = trie
    for t in generated:
        if t not in node:
            return [end_id]
        node = node[t]
    return list(node) or [end_id]

# logits processor: -inf everywhere except allowed_next(...)

The model still chooses. The mask only removes options that aren't labels, so when it was going to write card_not_arriving, the highest-scoring valid continuation is usually the right label. In serving, vLLM's structured outputs do the same job with a choice list, so production behaves like the offline eval.

Result

Invalid answers on Banking77

Setup	Messages	Invalid	Accuracy
Fine-tuned, unconstrained (first smoke test)	200	68	39.5%
Same base model, no fine-tune, 77 labels listed in the prompt	3,080	44	63.1%
Fine-tuned, constrained, offline	3,080	0	94.0%
Fine-tuned, constrained, served by vLLM	3,080	0	93.9%

The smoke test used a much shorter training run on a small sample, so its accuracy isn't comparable with the full runs. The invalid count is the point: it went from a third of answers to none.

Cost

The price: a concurrency ceiling

On one L4, throughput rises up to 32 requests in flight and then falls, while p95 latency grows more than five-fold. Our reading is that the per-step label masking becomes the bottleneck at high batch sizes.

In flight	Throughput	p50	p95
16	50.1 req/s	310 ms	485 ms
32	86.6 req/s	351 ms	546 ms
64	56.2 req/s	780 ms	3,045 ms

So we cap each replica at 32 requests in flight and add replicas for higher peaks, rather than letting a queue build. That cap is written into each endpoint's limits. The full sweep is in the Banking77 write-up.

What we'd try next

Shorter label tokenizations, so each answer needs fewer constrained steps
Scoring all 77 labels in one forward pass instead of generating, which removes the per-step mask entirely
Repeating the sweep on an H100, where the ceiling should sit higher