System 1 Decision Models: Laya, Token by Token

One email. Four questions. One forward pass.

That is the pitch. Not a chatbot that writes a paragraph and hopes you can parse a label out of it. A System 1 model: fast, typed, probabilistic judgement. No deliberative token stream. Just a distribution over options you defined.

Laya is that idea made concrete. Bidirectional encoder. [MASK] planted in front of every option. Shared scalar scorer. Softmax. Temperature. Done.

I am going to walk it the way the architecture notes do: start with why BERT, learn the vocabulary, build one support email into a sequence, follow every tensor through the forward pass, then see how the answer is read, how questions batch, and how RLCD trains the thing. Same structure. My words.

Part 1 · How BERT works, and why Laya needs it

Laya's backbone is a BERT-family encoder (ModernBERT-large or mmBERT-base). Almost everything weird and good about Laya is a consequence of a choice BERT made in 2018. So start there.

BERT reads. GPT writes.

Both are transformers. The difference is one line in the attention mask, and it changes everything downstream.

A decoder (GPT, Claude, Llama) is trained to predict the next token. To keep that task honest, each position may only look backwards. If position 3 could see position 4, predicting position 4 would be cheating. That is causal attention.

An encoder (BERT) is never asked to continue text. It is asked to understand a finished passage. Every position may look both ways. That is bidirectional attention.

Look at the word charge. The word that disambiguates it (invoice) sits to its right. Only the bidirectional model can use that. Causal attention leaves charge guessing.

This is why Laya is built on BERT and not on a small LLM. Laya is never asked to produce text. It is asked to judge a finished document. For judging, seeing the whole thing at once is strictly better.

Tokens, ids, and vectors

A transformer cannot read characters. Three conversions happen before any "thinking" does:

Every token becomes a row of $d$ numbers. $L$ is how many tokens the sequence has. $d$ is the model's hidden size, the width of the pipe everything flows through. For ModernBERT-large, $d = 1024$. Those two letters show up in every shape below.

Self-attention: what a layer actually does

At the start, the vector for charge is the same regardless of the sentence. It is a dictionary entry. Self-attention is what mixes context into it.

For every token, the layer builds a query (what am I looking for?), a key (what do I offer?), and a value (what do I contribute?). Then for each token $i$:

score(i, j) = query_i · key_j / √d
weights     = softmax(scores over all j)
output_i    = Σ_j weights_j · value_j

Stack enough of these and every position's vector becomes a summary of the whole sequence as seen from that position. The output of the stack is the last hidden state, shape [L, d]. In Laya that is literally:

h = self.encoder(input_ids=input_ids, attention_mask=attention_mask).last_hidden_state
# h : [batch, L, d]

The special tokens: [CLS], [SEP], [MASK], [PAD]

These are real vocabulary entries with learned embeddings. Not punctuation. Instructions to the model.

token	literally	what BERT learned	Laya uses it for
[CLS]	classify	position 0 scratchpad for a whole-sequence summary	feeds act_head; otherwise unused
[SEP]	separate	a boundary between fields	divides question / options / document
[MASK]	mask	a blank to fill from surrounding context	one per option: the slot where that option's score is read
[PAD]	pad	filler so ragged sequences form a rectangle	makes batches of different-length questions stackable

Why [MASK] is the key to the whole design

BERT's pretraining was masked language modelling. Hide ~15% of tokens behind [MASK]. Recover them from both sides of context.

Every BERT checkpoint spent its entire pretraining budget on one skill: turn a blank into a context-aware answer. Laya does not fine-tune that skill away. It repurposes it.

The reframe that makes Laya work: put a [MASK] in front of every answer option, inside the input, and read the hidden state at those positions. The question "does billing fit this email?" is delivered as exactly the kind of blank the model was pretrained to fill. Nothing is generated. The score is read off the slot.

Which BERT: ModernBERT and mmBERT

checkpoint	encoder	params	context	why it exists
laya	ModernBERT-large	421M	512	best English accuracy; ~50k English BPE, so it cannot read other scripts
laya-multilingual	mmBERT-base	322M	1024	100+ languages, smaller and faster; RoPE lets the encoder stretch toward 8192
laya-typed-decisions	ModernBERT-large	421M	1024	same English backbone, fine-tuned on specific workflows

Part 2 · Laya's vocabulary

Every term you will meet, defined once.

term	in the code	what it is
state	state	the thing being judged (string, dict, JSON, turns), serialized before tokenizing
question	questions[qid]	one typed judgement: type, instructions, usually criteria
choice	qtype = 0	pick one of K labels; criteria is {label: description}
score	qtype = 1	place the state on an ordered K-level rubric; criteria is a list
noul	qtype = 2	yes/no with a calibrated probability; internally [false, true]
option	render_options(q)	one candidate rendered to text: "billing: invoices, payments, refunds"
marker	markers, marker_pos	index where an option's [MASK] landed
marker_mask	marker_mask	which marker slots are real (different questions have different K)
head	self.head	two extra transformer layers after the encoder
scorer	self.scorer	MLP ending in Linear(d, 1); shared across all options
logit	logits	raw score for one option, before softmax
temperature	temperature_by_options	divisor on logits before softmax; >1 flattens over-confidence
confidence	confidence_from_probs	1 − H(p)/log K; how peaked the distribution is. Not the top probability
max_len	cfg["max_len"]	total token budget (512 or 1024)
head_max_len	cfg["head_max_len"]	slice reserved for question + options (192 or 256); the rest is state

Part 3 · Building the sequence

A worked example. This is the part that turns your Python dict into something a transformer can eat.

The input

state = {
  "from":    "user@acme.com",
  "subject": "Duplicate charge on invoice #4411",
  "body":    "Hi, we were billed twice for March. Please refund the "
             "duplicate today or we will cancel our plan."
}

questions = {
  "department": {
    "type": "choice",
    "instructions": "Which department should handle this request?",
    "criteria": {
      "billing":   "invoices, payments, refunds",
      "technical": "bugs, outages, system errors",
      "sales":     "pricing, new contracts",
      "other":     "everything else"
    }
  },
  # urgency: score, 3 levels
  # churn_risk: noul
  # refund_requested: noul
}

What the model receives as the state

Not the dict. serialize_state runs json.dumps on it first. The whole resulting string is what gets tokenized: braces, quotes, key names included. Field names cost real budget. Nested states burn tokens on punctuation. Flatter states leave more room for content.

Take the department question. Laya renders its four options to text:

[MASK] billing: invoices, payments, refunds
[MASK] technical: bugs, outages, system errors
[MASK] sales: pricing, new contracts
[MASK] other: everything else

The layout

Those options get sandwiched between the instructions and the document:

[CLS]  choice question: <instructions>  [SEP]
       [MASK] opt0  [MASK] opt1  [MASK] opt2  [MASK] opt3  [SEP]
       <the serialized state>  [SEP]

Three zones: question, options with MASK markers, state.

In the worked numbers, markers land at [12, 20, 30, 37]. The document does not start until position 43.

That is why bidirectional attention is load-bearing, not a preference. Under causal attention, the hidden state at position 12 would have seen zero tokens of the email. The score for billing would be computed with no knowledge of what it is scoring. The architecture would collapse. Because the encoder is bidirectional, position 12 attends forward across the state. That is why the layout can put options before the document at all.

What 106 is (and is not)

106 is not a dimension. Nobody chose it. It is simply how many tokens this one sequence turned out to be.

number	name	what it counts	where it comes from
106	$L$, sequence length	how many tokens are in this sequence	computed at request time; changes with every input
512	max_len	the ceiling $L$ may never exceed	checkpoint config
1024	$d$, hidden size	how many numbers are in each token's vector	fixed by the encoder; never changes

So h with shape [106, 1024] means: 106 tokens long, each one described by 1024 numbers. Perpendicular axes.

checkpoint	max_len	head_max_len	left for state
laya (English)	512	192	~320
laya-multilingual	1024	256	~768
laya-typed-decisions	1024	256	~768

The same email, all three question types

Everything above traced choice. The other questions go through identical machinery. The only difference is how criteria becomes option text. That is one function, render_options, with three rules:

row	question	type	K	L	markers
0	department	choice	4	106	[12, 20, 30, 37]
1	urgency	score	3	95	[11, 17, 22]
2	churn_risk	noul	2	96	[14, 24]
3	refund_requested	noul	2	95	[13, 23]

Every row carries the same ~62 state tokens. What differs is the header. And when you batch them, the shorter three pad up to 106. The batch's $L = 106$ because department is the longest row, not because 106 means anything on its own.

Where this goes wrong: Banking77

The budget is fixed. Ask a question with 77 options and truncation fires: roughly three tokens per label. "card_payment_fee_charged" and "card_payment_not_recognised" collapse to the same stub. Accuracy dies. That is a budget bug, not a capability story. Raise head_max_len, or do coarse-then-fine routing.

Part 4 · The forward pass, layer by layer

Four questions go in, so the batch dimension N = 4. Longest sequence is 106, so L = 106. Hidden size d = 1024. Widest question has four options, so kmax = 4.

Careful: those two fours are unrelated. logits[4, 4] is 4 questions by 4 option slots. The email is not a batch dimension. There is one email, copied into all four rows.

The narrowing at the gather is the design. A 421M-parameter model builds 106 rich token vectors, then keeps four of them: the ones planted at the markers.

The entire model, almost verbatim

Everything above is roughly these lines:

def forward(self, input_ids, attention_mask, marker_pos, marker_mask, qtype):
    h = self.encoder(input_ids=input_ids,
                     attention_mask=attention_mask).last_hidden_state   # [B, L, d]

    h = h + self.type_emb(qtype)[:, None, :]                            # question mode

    pad = ~attention_mask.bool()
    for layer in self.head.layers:
        h = layer(h, src_key_padding_mask=pad)                          # 2 more layers

    idx = marker_pos.clamp(min=0)[:, :, None].expand(-1, -1, h.size(-1))
    m   = torch.gather(h, 1, idx)                                       # [B, K, d]
    logits = self.scorer(m).squeeze(-1).float()                         # [B, K]
    logits = logits.masked_fill(~marker_mask, -1e4)                     # kill pads

    return logits, act_logits

Type embedding: three vectors that switch the mode

A learned vector for choice / score / noul is added to every token (same broadcast). The head knows what kind of judgement this is before it starts adapting features.

The head: two more layers, higher learning rate

Two extra Transformer encoder layers turn general language features into decision features. They train at a higher learning rate than the frozen-ish backbone.

The gather and the scorer

Instead of a classifier with a frozen number of classes, Laya plants one [MASK] per option, remembers those positions, gathers the hidden states there, and runs a shared scalar scorer:

gather hidden states at MASK positions  ->  [B, K, d]
shared MLP: d -> d -> 1                 ->  [B, K] logits
mask padded slots, softmax              ->  probabilities

Why that matters:

1. K is not baked into the weights. The scorer is Linear(d, 1). Two options or seventy use the same parameters. New label sets at inference time are allowed.
2. Every option sees every other option through bidirectional attention. billing is scored knowing technical and sales are on the table.
3. One forward pass scores all K options. No loop.
4. The option text is real semantics. "billing: invoices, payments, refunds" is read by the encoder, not just a dictionary key.

Compare that to a classic $d \times K$ classification head where K is frozen into the matrix:

The scorer itself is a small MLP that keeps a batch of option vectors and collapses width from $d$ down to 1:

Then: mask padded slots with $-10^4$, divide by temperature, softmax.

Part 5 · Reading the answer

One row of logits. Three ways of reading it. The mechanism never changes. Only the interpretation does.

Take department logits, divide by the fitted temperature for that bucket, then softmax:

Temperature is a single divisor fitted per (question type, option count) bucket. It never changes which option wins. It changes whether "0.9" means anything. And confidence ≠ top probability: confidence is 1 − H(p)/log K.

Three primitives, one mechanism

type	readout	you get
choice	argmax	top label + per-label probs + confidence
score	expectation over level indices	a point on the rubric (e.g. 1.84 / 2.0)
noul	$p[\text{true}]$	calibrated P(true) in $[0, 1]$

noul is internally a 2-option choice. score is a K-option choice read as an expectation. All three collapse to: score K options, softmax, read differently.

The payload ends with something like "output_tokens": 0. That field is the thesis compressed into one key. No tokens were generated, so there is nothing to parse and no way to hallucinate a label that was not in your criteria.

A product claim like "0.94 confidence" is only useful if you can gate on it:

if conf >= 0.85:
    route_automatically(dept)
else:
    escalate_to_human(dept)

Part 6 · Batching: four questions, one pass

The batch dimension is questions, not documents. That is unusual, and it is where the speed comes from.

Calling predict(state, questions) with four questions does not loop. It builds four sequences (same email in each), stacks them into one rectangle, and pads the shorter rows:

Always pack your questions together. The expensive part is the encoder. At inference, N questions on the same state = N rows = one forward pass.

Part 7 · How it learns: RLCD

Everything so far was inference on frozen weights. This is where those weights come from.

Why not just cross-entropy?

Ordinary classification maximises the probability of the correct label. That trains the argmax, and the argmax is the cheap part. Laya's product is the number beside it, because that is what people gate automation on.

A model can hit 95% accuracy while being wildly overconfident on the 5% it gets wrong. So Laya trains against a strictly proper scoring rule: a reward whose unique maximum is reporting your true belief. You cannot score better by exaggerating.

The core term is the logarithmic score, $\log q(\text{true})$:

Being right pays a little. Being confidently wrong costs a lot, and the cost grows without bound. That is why honesty is the optimal strategy.

Training data: distributions, not labels

Teacher labels are soft distributions, not one-hot:

You cannot learn to say 0.70 from data that only ever says 1.00.

The reward: three scoring rules stacked

term	what it enforces
log score	proper; floored so one disaster cannot dominate a batch
spherical score	also proper, but bounded in $[0, 1]$; stabilises the log term
RPS	ordinal awareness; applied only to score questions

For score questions, ranked probability score makes nearby level errors cheaper than distant ones:

The GRPO loop

Training (RLCD: reinforcement learning from calibrated decisions) uses that proper scoring rule as reward, plus a GRPO-style group of perturbed distributions as the baseline. No critic network required for that loop.

The group lives after the encoder: perturb logits, score, advantage against the group mean. Encode once, explore in logit space many times.

Calibration, fitted after training

Temperature is fitted post-hoc per (type, option-count) bucket so probabilities match reality better. The README's framing is the right one to leave with: Laya is a fast base to specialise, not a zero-shot oracle. The typed-decisions numbers belong to the fine-tune.

The six things to hold onto

1. Bidirectional attention is why the backbone is BERT. Options sit before the document; under causal attention they would see none of it.
2. [MASK] is a pretrained question slot. BERT learned to fill blanks. Laya plants one per option and reads the answer off it.
3. The scorer is shared and scalar. Linear(d, 1) per option means the label space lives in your request, not in the weights.
4. Questions are the batch dimension. N questions become N rows in one forward pass, not N passes.
5. The probability is the product. Argmax comes free. The temperature-calibrated distribution is what lets you gate automation on a threshold and mean it.
6. Training is where the capability is. Cross-entropy (or the proper-scoring-rule reward) does the learning; temperature does the honesty repair.

I have spent a lot of time on attention internals in earlier posts: dot product, self-attention, positional encodings. Laya sits on the other side of that stack. Once you can represent context, what is the cheapest honest way to turn it into a decision?

For a large class of tasks, it is not "ask the LLM to write the answer."

It is: define the options, pack them into the sequence, score the MASK slots, read a calibrated distribution.

That is a System 1 decision model. That is Laya.

---

References

• Nandakishor Mukkunnoth, Laya: Multilingual System 1 Decision Engine, laya.convaiinnovations.com
• Laya (code, router, benchmarks), github.com/NandhaKishorM/laya
• Nandakishor M, SalesRLAgent: A Reinforcement Learning Approach for Real-Time Sales Conversion Prediction and Optimization, arXiv:2503.23303