On this page
AI & Models
Decider 4B: Decisions Without Generating an Answer
A look at one-pass choice models, their inference costs, the v2 and v2.1 tradeoff, and how I would test one in a real workflow.
The small decision problem
A surprising amount of application work is a choice among known options. Which queue should get a ticket? Does this passage answer the question? Which of three safe actions should the agent try? Asking a chat model to compose and parse JSON for every such choice can be more machinery than the decision needs.
Decider 4B takes context and explicit choices, then returns a probability distribution over those choices in one forward pass. It does not generate free-form text. That makes it interesting as a small judgment step inside a larger program, not as a replacement for the program. The model author's description and code are at https://huggingface.co/Mapika/decider-4b.
What inference actually does
The model is a dense 4.2-billion-parameter derivative of Qwen3.5-4B-Base. A request supplies a state plus one or more questions, each with its own option list. Inference reads scores at designated answer positions and normalizes them into probabilities. Several questions can share one request.
For a fictional duplicate-charge ticket, code could present the options billing, technical support, and sales. It might separately ask whether the case merits refund review. The application would receive choices and probabilities; it would still decide whether to create a ticket, request evidence, or ask a person.
Memory and speed: read the workload
| Published figure | What it means |
|---|---|
| 8.4 GB | BF16 weight files; this is not total VRAM needed to serve requests |
| About 25 GB | The project's default CUDA-graph pool for the 4B serving configuration |
| 34.6 ms | Author-reported median v2 eager-PyTorch decision on short game states, NVIDIA B300 |
| 5.2 ms | Author-reported short three-question request with compilation and CUDA graphs, NVIDIA B300 |
| 72.8 / 190 requests/s | Author-reported BF16 HTTP serving with one / 64 clients on its stated setup |
These numbers come from the model author, not from my own GPU run. They are different workloads and serving modes, so they should not be combined into a single speedup claim. The weights alone understate VRAM needs; batch size, request length, runtime, and graph capture all matter. Sources: https://huggingface.co/Mapika/decider-4b and https://github.com/Mapika/decider/blob/main/docs/SERVING.md.
Because the model does not decode an answer token by token, tokens per second is a misleading headline. For an actual integration I would measure decision latency, requests per second at intended concurrency, peak VRAM, and whether the decision was correct.
v2 and v2.1 do not have the same strengths
The repository currently serves v2.1 by default, while v2 remains available as a pinned revision. The author's comparisons use matched inputs and seeds, but the tasks differ. On JevBench's public hard tier, v2 scored 67.6% accuracy and v2.1 scored 64.9%. In sampled live MiniWoB++ across 22 tasks, v2 scored 88.1% and v2.1 scored 93.2%.
Calibration also changes: the author reports hard-tier top-label expected calibration error of 0.104 for v2 and 0.184 for v2.1, where lower is better. A version choice therefore depends on the intended task and on how its probability threshold will be used. Pin the revision and evaluate your own examples before routing production work.
Where I would try it
My first candidates are constrained, reversible judgments: route a support item, classify page intent, choose which retrieved passage to inspect, or pick among safe browser actions. A conventional rule or search score remains the first baseline. The model earns its place only if it improves quality or latency on cases those baselines miss.
- Construct candidates and allowed actions in code; never ask the model to invent a target or endpoint.
- Record the option set, pinned revision, input class, latency, probability, and independently checked outcome.
- Use a threshold and a fallback for uncertain cases; send consequential decisions to a stronger check or a person.
- Measure peak VRAM and throughput on the intended GPU, prompt lengths, and concurrent load before deployment.
I have not completed an independent Decider 4B GPU benchmark for this article. The published figures show why the model merits a trial, not what it will do on another machine or workload. The useful next test is a labelled set from the target workflow, with correctness and calibration reported beside speed.