Laya is an open-source, non-autoregressive 'System 1' decision model from Convai Innovations. You give it a state (any text) plus typed questions — choice, score, or yes/no — and it returns structured answers with calibrated probabilities in a single forward pass, in 100+ languages. It's Apache-2.0, ~421M params, ~1GB inference memory, and runs locally at ~33ms/call.

How does Laya actually work under the hood?

It's a transformer used prefill-only (non-autoregressive): it reads the state + questions once and reads probabilities straight off dedicated output heads, instead of generating text token-by-token. Because there's no decode loop, one forward pass answers all questions at once. A small 'router' inspects the input (script/language) and picks the right checkpoint — English vs multilingual.

With reinforcement learning against strictly proper scoring rules (the project calls it RLCD). A strictly proper scoring rule (like log-loss or Brier score) is only maximized when the model reports its TRUE probability, so training directly rewards well-calibrated probabilities rather than just correct labels. Shipped checkpoints work zero-shot; fine-tuning on your own domain roughly doubles accuracy (0.362 → 0.766 on their 2,000-decision benchmark).

What does it take to run or fine-tune Laya?

Running: pip install laya, Python 3.10+, ~1GB memory; CPU-only works, a GPU makes it faster. Fine-tuning: a provided Kaggle notebook trains a custom checkpoint on free 2×T4 GPUs in ~4 hours — it builds the dataset, trains, fits calibration temperatures, evaluates, and pushes to the Hub.

Laya: A 421M Local Decision Model That Outruns the Cloud — and How It Actually Works

By Prahlad Menon Published 2026-09-24 5 min read

Two days ago I wrote about Kev, an open-source “decision model” that answers typed questions instead of generating prose. Here’s the next one in the same wave — Laya from Convai Innovations — and since the last post left the how a bit hand-wavy, this one actually opens the hood.

Same core idea as Kev, sharper execution on speed: a non-autoregressive System 1 decision engine, ~421M params, ~1GB memory, ~33ms per decision, locally, in 100+ languages. Apache-2.0.

First, the family tree (so this isn’t just noise)

There’s a closed cloud model called Jev that answers typed questions. Kev was an open reproduction built on Qwen. Laya is a different open take on the same problem, and it leans hard into one claim: a local decision model destroys a cloud one on latency because it never pays for a network round-trip.

Reported benchmark:

Laya: ~86 decisions/sec, P50 ≈ 9ms (single-call ≈ 33ms), local
Cloud API: ~3 decisions/sec, ~317ms round-trip

That ~20–30× gap isn’t clever kernels — it’s physics. A 9ms local function call beats a 317ms HTTP request every time, and you can’t optimize the internet away.

How it actually works

This is the part worth slowing down on, because “answers in one forward pass” sounds like magic and isn’t.

1. A normal LLM is autoregressive. To produce text it predicts one token, appends it, predicts the next, and loops — dozens or hundreds of forward passes for one answer. That loop is where the latency and cost live.

2. Laya is non-autoregressive / prefill-only. It runs the transformer over your input once (the “prefill” phase every LLM already does), and instead of starting a generation loop, it reads answers straight off dedicated output heads bolted onto the model. Think of it as: same transformer body, but the last layer is a set of small classifiers — one per question type — rather than a next-token predictor.

3. The questions are typed, so the output shape is known ahead of time:

choice → a probability over your N named options
score → a probability over an ordered scale
noul (yes/no) → a single probability the answer is “yes”

Because the shapes are fixed, the model can emit all answers for all questions in that single pass. No decoding, no JSON to parse out of prose, no regex salvage. You get numbers.

4. A router sits in front. It inspects the input (script, language) and picks the right checkpoint — English text → the English checkpoint, everything else → laya-multilingual. That’s why one API call works across 100+ languages without you choosing a model.

So the “trick” is architectural: move the work into the prefill and read probabilities off heads, instead of generating. That’s the whole reason it’s milliseconds instead of hundreds of milliseconds.

How it’s trained — and why “calibrated” is the key word

Here’s the genuinely interesting bit. Laya is trained with reinforcement learning against strictly proper scoring rules (the repo calls it RLCD).

Unpack that:

A strictly proper scoring rule — log-loss, Brier score — has a mathematical property: it’s only maximized when the model reports its true probability. If the real chance is 0.7 and the model says 0.9 to look confident, a proper scoring rule punishes it. So optimizing against it forces honest, calibrated probabilities, not just correct labels.
Training this way means when Laya says 0.87, that 0.87 is meant to be a real frequency you can act on — set a 0.80 threshold, route borderline cases, log confidence for drift. That’s the thing a chat LLM emitting {"answer":"yes"} can never give you cleanly.

This is exactly the theme running through our recent posts: Kev and Laya both argue that for a huge class of tasks you don’t want eloquence, you want a fast, calibrated decision. And it rhymes with the RF-SRC memory work we’ve been doing on the clinical side — small, purpose-built models that emit probabilities beat a giant generalist reluctantly classifying.

What models / sizes are involved

Two shipped checkpoints: laya (English) and laya-multilingual, plus a fine-tuned laya-typed-decisions.
~421M parameters, ~1GB inference memory — small enough for a laptop, edge box, or cheap VM.
Context: the multilingual checkpoint reads up to 8,192 tokens (ships defaulting to 1,024; raise max_len per call). Longer inputs cost proportionally more time — a ~6,300-token input took ~2.5s on an Apple GPU vs ~0.18s for short inputs — because cost tracks real input length, not the limit.

What it takes to run it

Genuinely a one-liner:

pip install laya

from laya import Router
router = Router()  # downloads a checkpoint on first use

state = "Hi, we were billed twice for March. Refund the duplicate today or we cancel."
questions = {
  "department": {"type": "choice", "instructions": "Which team handles this?",
    "criteria": {"billing": "invoices, payments, refunds",
                 "technical": "bugs, outages", "other": "everything else"}},
  "urgency":    {"type": "score",  "instructions": "How urgent?",
    "criteria": ["not urgent", "soon", "blocking"]},
  "churn_risk": {"type": "noul",   "instructions": "Do they threaten to cancel?"},
}

r = router.predict(state, questions)
print(r["answers"]["department"]["choice"])   # billing
print(r["answers"]["churn_risk"]["noul"])      # probability = yes

Python 3.10+, ~1GB RAM, CPU-only works (GPU just makes it faster).
Optional extras: laya[serve] (HTTP server), laya[mcp] (MCP), laya[langchain], laya[onnx] (ONNX Runtime), laya[fast] (GPU fast path).

What it takes to fine-tune it

This is where accuracy actually jumps. Zero-shot the base English checkpoint scored 0.362 on their 2,000-decision benchmark; the fine-tuned checkpoint hit 0.766 — roughly double.

A provided Kaggle notebook runs the entire loop on free 2×T4 GPUs in ~4 hours: build the dataset, train, fit calibration temperatures (so the probabilities stay honest), evaluate, and push to the Hub.
“Fit calibration temperatures” = a final rescaling step so the reported probabilities match observed frequencies — the practical companion to the proper-scoring-rule training.

So the real memory/compute story:

Inference: ~1GB, any laptop.
Fine-tuning: two free T4 GPUs (16GB each) for a few hours — no cluster, no frontier-lab budget.

The takeaway

The naming is getting sillier by the week (Jev → Kev → Laya). But the engineering argument keeps getting stronger: prefill-only transformers with probability heads, trained on proper scoring rules, give you fast, honest, local decisions — and they run on hardware you already have. If any part of your stack is a chat LLM cosplaying as a classifier, this is the pattern to watch.

👉 github.com/NandhaKishorM/laya