Laya Local System One: Setup, Limits, and Jev Comparison

Choose among three checkpoints#

The repository lists three checkpoints:

The Router can select English or multilingual inference from the detected script and language. It can also select the typed-decisions checkpoint for recognized question schemas or an explicit model request.

This routing step exists because the English checkpoint can be confidently wrong on non-Latin scripts. Confidence gating cannot recover an input the checkpoint does not represent; choose the checkpoint before inference.

Install and run the Python package#

The project declares Python 3.9 or newer and installs from PyPI:

1
python -m venv .venv
2
source .venv/bin/activate
3
pip install laya

Load one checkpoint for a dedicated English pipeline:

1
from laya import load
2

3
agent = load(
4
    "convaiinnovations/laya",
5
    device="cuda",
6
)
7

8
state = {
9
    "subject": "Duplicate charge",
10
    "body": "I was charged twice and want a refund today.",
11
}
12

13
questions = {
14
    "department": {
15
        "type": "choice",
16
        "instructions": "Which team should handle this ticket?",
17
        "criteria": {
18
            "billing": "Payments, invoices, and refunds",
19
            "support": "Product help and bugs",
20
            "sales": "New purchases and upgrades",
21
        },
22
    },
23
    "refund_requested": {
24
        "type": "noul",
25
        "instructions": "Does the customer explicitly request a refund?",
26
    },
27
}
28

29
result = agent.predict(state, questions)
30
print(result)

For mixed-language traffic, use the router rather than repeatedly loading checkpoints by hand:

1
from laya import Router
2

3
router = Router(max_loaded=2)
4
result = router.predict(state, questions)
5

6
print(result["routing"]["model"])
7
print(result["answers"])

max_loaded=2 trades memory for fewer checkpoint reloads. The project reports multi-second reload times when alternating languages with only one checkpoint kept in memory, so include cold-load behavior in latency tests.

Read the benchmark beyond the headline#

The repository reports 32.8 to 39.5 milliseconds for a single question on a T4 GPU and 72.3 milliseconds for a batch of 10 with the multilingual checkpoint. Those are model-side measurements from the project’s benchmark, not end-to-end application latency.

More important, the project’s typed-decision results show a large gap between the base and fine-tuned models. On the published shared benchmark, the base checkpoints score around 0.34 to 0.36 accuracy, near the 0.318 random baseline and below the 0.461 majority baseline. The laya-typed-decisions checkpoint reports 0.766 accuracy on its target workflows.

That result does not mean the fine-tuned model will reach 0.766 on a different label set. It shows that domain and schema-specific training materially changes this kind of model. Build a labeled set for the actual decision and test the exact checkpoint you intend to deploy.

The project also warns that raw confidence can be overconfident and recommends temperature fitting. Score is currently weaker than Choice and Noul, and large option sets are constrained by the model’s token allocation.

Laya or Jev?#

Local inference is not automatically cheaper. Account for GPU or CPU capacity, memory, cold starts, monitoring, calibration, model distribution, and updates. A managed API is not automatically less secure either; evaluate data handling requirements rather than using deployment location as a proxy for the entire risk model.

Verification boundary for this guide#

The repository, package metadata, model card, source code, and published benchmark artifacts were inspected. The model weights were not downloaded and GPU inference was not executed in this environment. The reported latency and accuracy values therefore remain project-published results, labeled as such.

Use the Jev model guide for the managed alternative and the Jev TypeScript tutorial for an API workflow. If the goal is coding-agent cost control, review the Codex token-routing evidence before treating either decision model as an automatic proxy.

FAQ#