ILLATE · parity check
cfpb-product: can a small model you own replace this task?
PASS
Compared against the free baseline, because no current-model answers were sent.
Difference -0.7 points (95% CI -1.9 to +0.5), agreed margin 3 points, 5,117 held-out test items.
Accuracy on the held-out test set
Bars start at 60.0%; whiskers show the 95% confidence interval.
| Model | Accuracy | 95% CI | Macro-F1 | Invalid |
|---|---|---|---|---|
| Free baseline (TF-IDF, CPU) | 67.7% | 66.4% to 69.0% | 46.7% | 0 |
| Small model you would own (Qwen3-4B LoRA) | 67.0% | 65.7% to 68.3% | 53.8% | 0 |
Where the small model is weakest
| Category | Small model F1 | Baseline F1 | Test items |
|---|---|---|---|
| Credit reporting or other personal consumer reports | 29.1% | 51.6% | 1134 |
| Credit card | 11.8% | 25.2% | 167 |
| Credit reporting, credit repair services, or other personal consumer reports | 75.4% | 75.9% | 2132 |
| Debt or credit management | 0.0% | 0.0% | 5 |
| Payday loan, title loan, personal loan, or advance loan | 0.0% | 0.0% | 23 |
Data
- 30,000 rows received, 25,582 used, 14 categories
- Split by category: 17,905 train, 2,560 dev, 5,117 test (test held out before training, used once)
- Dropped: 4418 duplicate text
- Training: 17,905 items, 44.7 min on one NVIDIA H100 80GB HBM3
Read before deciding
- 178 texts appear with different labels. Label noise caps the accuracy any model can reach; worth checking which label is right before training.
How this was measured
Accuracy intervals come from 10,000 bootstrap resamples of the test items. The verdict uses a paired bootstrap on the same items for both models: PASS only if the whole 95% interval of the difference sits above minus the margin agreed before the run. The check is open source: anyone can re-run it with illate-parity from github.com/illate/parity.
What happens next
- Check the numbers yourself: the parity tool is open source and the test items are yours.
- We train on your full data and either hand you the weights or serve the model behind an OpenAI-compatible endpoint, so the change in your code is the base URL.
- Rollout in stages (shadow, 1%, 25%, 100%) with your current model as fallback, and a defer option that passes low-confidence items back to it.
ILLATE · Poojith Devan · poojith@illate.dev · Run this check on your own data → · github.com/illate/parity