Decision Models in a System Built for Language Models: How a new class of bounded-output AI fits an expert-referenced grading system for community corrections supervision

Cliff Hurt Johnston, Ted Green and Madeline Warren

No wsjv6_v1, LawArchive from Center for Open Science

Abstract: A new class of AI models, often called decision models, answers each question by choosing from a fixed set of allowed answers rather than writing open-ended text. They are fast and inexpensive, and they cannot return content outside the answers they are given. Less than four weeks after the first of them, TypeSafe’s Jev, was released, one independent leaderboard was tracking more than 170, including models from Microsoft, OpenAI, Amazon and Cloudflare. We asked how well these models fit into an existing grading system for community corrections supervision visits, one built with expert human effort and developed around frontier large language models (LLMs). Using the expert reference standard from our previous study (10 visits, 43 rubric items, 429 unanimous YES/NO/NA judgments), we tested three decision models: TypeSafe Jev, OpenAI Decisions, and Microsoft Decision-1. Dropped into the system unchanged, they matched 65–77% of expert judgments. Adapting the task to each model raised agreement to 70–83%. The best decision-model workflows fell in the same range as three of five frontier LLM workflows, but below the best, GPT-6 Astra (90.2%). Every response was a valid label. Service calls took about a second or less and cost under half a cent per visit, against an estimated US$2.22 per visit for the leading LLM workflow at retail prices. One configuration returned identical answers across three runs, disagreements with the experts included. Decision models are a useful new building block for community corrections technology, but they do not outscore LLMs on a system designed for LLMs. Systems should be built around the strengths of the tools they use and tested against a human reference to choose among them. The next step is a scoring design built for decision models.

Date: 2026-10-15
References: Add references at CitEc
Citations:

Downloads: (external link)
https://osf.io/download/6ad121a165572fe62798762a/

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:osf:lawarc:wsjv6_v1

DOI: 10.31228/osf.io/wsjv6_v1

Access Statistics for this paper

More papers in LawArchive from Center for Open Science
Bibliographic data for series maintained by OSF ().