How to Fine-Tune Laya on Your Own Decision Data

Quick answer

• Fine-tuning is where Laya becomes competitive on a narrow task; its base checkpoints are near chance on the typed-decisions benchmark.
• Use a CSV for one choice question, or JSONL for mixed choice, score, and noul questions.
• Always run laya-train –dry-run first: it reports rows read, skipped questions, and the optimizer-update budget.
• Keep a calibration slice and a held-out test set. Fit temperatures before gating on confidence.
• Evaluate accuracy, calibration, and option-order stability; a checkpoint that is accurate but overconfident is not ready for a threshold.

What does fine-tuning Laya change?

Laya’s shipped checkpoints are trained for zero-shot typed decisions. On the project’s typed-decisions benchmark, the base checkpoints score around 0.36 against a 0.318 random baseline, while the fine-tuned checkpoint reaches 0.766 on the same 2,000 decisions. That is a large gain, and it is the reason fine-tuning is the main path to a competitive domain model rather than an optional extra.

Fine-tuning adapts the encoder and the decision head to your labels, your option descriptions, and your language. It does not turn Laya into a generative model, and it does not guarantee that a small dataset will work. The project’s own notes show that runs can collapse near the class prior, that seeds matter, and that a label-encoding positive control is a useful way to distinguish a hard task from a broken training budget.

Step 1 — prepare the data

For a single choice question, a CSV with one text column and one label column is enough. Each row is one decision, and the label column becomes a choice question over its distinct values in first-appearance order. Empty text or label rows are skipped and counted. For mixed question types, JSONL is the better shape: each line has a state, a questions object, and an expected answers object with one answer per question.

The JSONL expected object also works as an evaluation file, so the same cases can train and test a checkpoint. Include the hard cases: ambiguous inputs, multilingual text, long documents, and examples that look like one class but are labeled another. A dataset of easy examples produces a checkpoint that looks good on easy examples and fails where the product needs it.

{"state": "We were billed twice for March.",
 "questions": {
   "department": {
     "type": "choice",
     "instructions": "Which team should handle this?",
     "criteria": {"billing": "invoices and refunds", "technical": "bugs and outages"}
   },
   "urgent": {
     "type": "noul",
     "instructions": "Does this need an answer today?"
   }
 },
 "expected": {"department": "billing", "urgent": true}}

Step 2 — dry-run and size the job

A dry run builds the training items with the base checkpoint’s tokenizer and config without loading weights or training. It reports how many rows were read, how many items they produced, which questions were skipped and why, and how many optimizer updates the defaults will spend. That last number is more useful than epochs for comparing runs of different sizes, because a small dataset needs more epochs to reach the same number of updates.

Read the skipped reasons before starting a long run. Empty text, empty label, an invalid question, options that do not fit in the token budget, and options that collapse to the same tokens are all detectable before training. Fixing the data is cheaper than discovering it after an hour on a GPU.

# CSV: one row per decision, one text column, one label column
laya-train --data tickets.csv --text-column body --label-column department \
  --base english --out ./tickets-checkpoint

# JSONL: mixed questions, same shape as the evaluation file
laya-train --data train.jsonl --base english --out ./checkpoint --dry-run

# After the dry run looks right, train
laya-train --data train.jsonl --base english --out ./checkpoint

Step 3 — train with the right budget

The defaults are four epochs, a micro-batch of eight, and gradient accumulation of eight, for an effective batch of sixty-four. A full fine-tune on one T4 is the documented setup; --freeze-encoder trains only the decision head and fits a small GPU or CPU, but it gains much less. If you have many options per question, add --shuffle-options so the model does not learn a position bias.

The output directory contains the files laya.load expects, plus questions.json and a checkpoint_latest rewritten after each epoch. A crash costs at most an epoch. The config records the token budgets, training settings, and a calibration report; read the warnings, because a calibration slice that is too small leaves the temperature at 1.0 and makes the confidence numbers meaningless for gating.

Step 4 — evaluate and calibrate

Evaluate on a held-out set that was not used for training or threshold tuning. Report accuracy, Brier score, expected calibration error, and score error where relevant. For choice questions with many options, also measure how often the answer changes when the options are reversed; a model that is accurate but unstable under option order is fragile in production.

Calibration is part of the training run, not an afterthought. The tools fit one temperature per question type on a slice held out before training, and the fitted values go into the checkpoint config. Temperature scaling does not change the argmax or the accuracy; it changes the confidence. If you plan to gate on min_confidence, fit and evaluate before choosing the threshold.

Step 5 — serve the checkpoint

Load the checkpoint with laya.load and ask the same question set that was saved beside it. If you serve it over HTTP, point the server at the checkpoint directory and keep the question definitions versioned with the model. A checkpoint without its question set is not reproducible; the labels and instructions are part of the model’s behavior.

Pin the checkpoint revision or keep the directory in your own artifact store, and re-run the evaluation whenever the data, the questions, or the base model changes. A fine-tune is a new model version, not a configuration tweak. Treat it like a model release: changelog, evaluation report, rollout, and a way to roll back.

import json
import laya

agent = laya.load("./tickets-checkpoint")
questions = json.load(open("./tickets-checkpoint/questions.json"))

result = agent.predict(
    "We were billed twice for March. Please refund the duplicate.",
    questions,
)
print(result["answers"]["department"]["choice"])
print(result["answers"]["department"]["probabilities"])

Where this bites vibecoders

The assistant will fine-tune on the examples it just made up, report the training accuracy, and call it done. That measures memorization. Use real inputs, keep a test set you never train on, and check the calibration before you gate on confidence. The gain from fine-tuning is real, but only if the labels and the evaluation are real too.

Where AI coding assistants get this wrong

• Training and testing on the same examples.
• Using a dataset of easy cases and no ambiguous or multilingual inputs.
• Skipping the dry run and discovering bad rows after the job starts.
• Gating on confidence before fitting calibration temperatures.
• Ignoring option-order stability on questions with many labels.
• Serving a checkpoint without the questions.json it was trained with.

Checklist

• Collect labeled decisions from real inputs.
• Choose CSV for one choice question or JSONL for mixed types.
• Run laya-train –dry-run and read the skipped reasons.
• Keep a calibration slice and a held-out test set.
• Train, then evaluate accuracy, calibration, and option stability.
• Fit temperatures before using min_confidence.
• Version the checkpoint with its question set and evaluation report.

FAQ

How much data do I need to fine-tune Laya?

There is no universal number, but the project’s examples use thousands of decisions and warn that small datasets can collapse near the class prior. Start with a few hundred high-quality labels for a narrow task, run a dry run, and use a positive control to check whether the budget can learn an easy signal. More diverse hard examples usually help more than more easy ones.

Does fine-tuning need a GPU?

A full fine-tune realistically wants a GPU; the documented notebook uses two T4s, and a single T4 is a common target. --freeze-encoder fits a small GPU or CPU but gains much less. Inference is the part that runs comfortably on CPU.

Why is my fine-tuned model still at chance?

It can be the task, the labels, the budget, or the run setup. Use a positive control: prepend a tag that spells the answer and train on that copy. If the control learns, the training loop works and the real task is hard or the budget is wrong. If it does not, suspect the setup. Repeat with different seeds before concluding a task is unlearnable.

What is calibration and why does it matter?

Calibration means the predicted probabilities match observed frequencies: among answers at 0.8, about 80% should be correct. Laya’s shipped checkpoints can be overconfident, so a confidence threshold may not behave as expected. The training loop fits temperatures on a held-out slice; evaluate the result before gating.

Can I fine-tune the multilingual checkpoint?

Yes. Pass --base multilingual and train on the languages you need. Multilingual data is usually more variable, so include examples from each language and evaluate per language rather than only on the aggregate. A model that is good on average can still be poor on the language your users actually write.

Related topics

• Laya AI: The Complete Guide to the Open-Source Decision Model
• What Is Laya? The Open-Source Jev Alternative You Can Run Locally
• How to Run Laya Locally: Install, First Decision, CLI, HTTP, and MCP
• How to Evaluate a Decision Model Before You Ship It
• Laya Security: Pin Checkpoints, Verify Digests, and Protect laya-serve
• Jev vs Laya vs Clef: Which Decision Model Should You Use?

Sources

• Laya Documentation
• GitHub
• Hugging Face