Laya Vision

Laya Vision makes typed, calibrated decisions about an image plus optional text, in one forward pass with no text generation. You give it a picture, some context and a set of questions; it answers each one as a multiple choice (choice), a yes/no probability (noul) or a graded level on a rubric you write (score), with probabilities you can act on at face value.

Try the demo: a Hugging Face Space that runs the published checkpoint on a free CPU, a few seconds per image. How to use it.

import laya
from PIL import Image

agent = laya.load_vlm("thaitea/laya-vision")
result = agent.predict(
    {"image": Image.open("photo.jpg"), "note": "customer says it arrived broken"},
    {
        "damage":   {"type": "score",  "instructions": "How much damage does the item show?",
                     "criteria": ["none", "cosmetic: scratches or dents", "functional: parts broken or missing", "destroyed"]},
        "category": {"type": "choice", "instructions": "What kind of item is this?",
                     "criteria": ["electronics", "clothing", "furniture", "food", "other"]},
        "outdoors": {"type": "noul",   "instructions": "Was the photo taken outdoors?"},
    },
)
a = result["answers"]
a["damage"]["score"], a["category"]["choice"], a["outdoors"]["noul"]   # expected level 0-3, top option, P(true)

Laya Vision is an independent fork of Laya that replaces its ModernBERT text encoder with a small vision-language model. Laya's predict(state, questions) API, output schema, RLCD training objective and temperature calibration are unchanged. It is an experimental research project, not affiliated with Convai Innovations, the authors of Laya.

Start here

  • Install: the Python package and what it needs.
  • Tutorials: the demo Space, and a first prediction in Python.
  • How-to guides: calibrate on your data, run jobs on Modal, evaluate, play games.
  • Concepts: how the model reads an answer, its architecture, calibration and training data.
  • Reference: the predict API and answer schema, checkpoints, file formats and every published result.

At a glance

Area What it covers
Question types choice, score (a rubric you write) and noul (yes/no), several per call
How it works One forward pass per image; a logit per option read from a marker token; nothing generated
Calibration Per-type temperatures fitted after training, and refitting them on your data
Checkpoints The recommended thaitea/laya-vision and what its numbers are backed by
Demo The published checkpoint in a Hugging Face Space, no install
Results The scorecard over 34 validation sets, games and latency, plus every experiment report

License and credits

  • Code: Apache 2.0, inherited from Laya (LICENSE).
  • Weights: CC BY-NC-SA 4.0, because the training data includes ScienceQA and CrisisMMD, which carry that license.

Laya Vision is an independent fork of Laya by Convai Innovations, Apache 2.0, and is not affiliated with them. The text decision model, its typed-question API (choice / score / noul), the RLCD training objective and the temperature calibration are theirs; this fork adds image input, the SmolVLM and ModernVBERT backbones, the rubric data and the game work. The design is described in the author's write-up. The original text model and its Colab/Kaggle notebooks live in the upstream repo. ModernVBERT is by its authors under MIT; SmolVLM by Hugging Face under Apache 2.0.