Laya Vision 201M (autoresearch, 2 h batch 64)¶

Checkpoint autoresearch/full/long-sep24-b64/best · code claude/eager-thompson-mui6to@e405ed2d · run 2026-09-24 04:28 UTC · val split

Generated by scripts/eval_report.py from autoresearch-full-long-sep24-b64.json, autoresearch-full-long-sep24-b64-latency.json.

Errors during the run: bench_latency: FileNotFoundError(2, 'No such file or directory')

What happened¶

• note · Across 34 datasets and 59,427 questions it answers 69.1% correctly (pooled). Best: dvqa at 92.4%; weakest: evalmuse at 23.3%.
• watch · Calibration: across all questions, its stated confidence and its actual accuracy differ by 4.1 points on average (ECE 0.041 pooled; under 0.03 means probabilities can be read at face value).
• watch · Poorly calibrated on 5 hard-label sets (ECE above 0.10): intergps 0.39, chartqa 0.28, visual7w 0.17, vqarad 0.13, ocrvqa 0.11.
• weak · Against human vote spreads it beats the always-predict-the-average baseline on 3 of 9 sets (not on ava, crisismmd, koniq, evalmuse, ferplus, vizwiz). (ECE is not meaningful on these sets: it compares confidence with the single most-voted answer, while the model is trained to spread probability like the voters.)
• good · Freeway: scores 23.7 against 0.0 for random play and 34.0 for the expert (normalized 0.70).
• watch · Breakout: scores 41.3 against 0.8 for random play and 218.0 for the expert (normalized 0.19).
• weak · Galaxian: scores 450.0 against 673.0 for random play (no expert baseline).
• good · ViZDoom basic: mean reward 75.8 (expert 75.8, random -121.9), kills in 100.0% of episodes.
• watch · Maze: solves 4×4: 46.0%, 6×6: 0.0%, 8×8: 0.0% (the shortest-path expert solves all; random play at most 2.0%).
• watch · Snake: eats 4.3 food per game on a 10×10 board (expert 25.1); games end by self 5, starved 1, wall 14.
• good · Acrobot: scores -138.7 against -500.0 for random play and -84.6 for the scripted expert (normalized 0.87); solved in 30.0% of episodes.
• watch · CartPole: scores 24.4 against 20.9 for random play and 500.0 for the scripted expert (normalized 0.01); solved in 0.0% of episodes.
• weak · LunarLander: scores -223.9 against -177.1 for random play and 273.2 for the scripted expert (normalized -0.10); solved in 0.0% of episodes.
• good · MountainCar: scores -145.4 against -200.0 for random play and -121.7 for the scripted expert (normalized 0.70); solved in 0.0% of episodes.
• note · Latency: 41 ms median per predict call on an L4 in bf16 (p90 46 ms).

Datasets¶

Calibrated answers, as predict returns them. Accuracy counts the most likely option against the labelled answer; ECE is the gap between stated confidence and accuracy. Sets marked * are scored against human vote spreads, where accuracy and ECE only see the most-voted answer: read those in Against human disagreement.

Held-out eval sets · mean 60.8%¶

Sets the model never trained on: photo quality, prompt alignment, CIFAR-10H, facial expressions, VizWiz answerability and POPE hallucination probes.

Rubric scoring · mean 65.9%¶

Graded-level questions (the score head): response quality, aesthetics, generated-image ratings, damage.

VQA (official val) · mean 71.2%¶

The three original post-training sets on their official validation splits.

Cauldron holdouts · mean 77.3%¶

Held-out rows of the 19 Cauldron subsets the model was post-trained on.

Against human disagreement¶

For sets where every image has many human votes: cross-entropy between the model's probabilities and the vote spread, lower is better, compared with a model that always predicts the dataset's average vote. The chart shows the baseline's cross-entropy divided by the model's: above the red line at 1, the model tracks how people vote on each image better than the average does; below it, worse.

Temperatures (choice, score, yes/no): 3.857, 2.100, 3.053.

Games¶

Each step the screen is the image and the options are the game's buttons; every policy plays the same seeded episodes.

Atari¶

Greedy play, 4,500-step cap. Normalized: 0 = random, 1 = expert.

ViZDoom · basic¶

A kill is +100 and every step costs, so waiting and missing go negative.

Maze¶

Share of mazes solved within 4× the shortest path; efficiency is shortest path over steps taken.

Snake¶

Food eaten per game, and how the games ended.

Classic control¶

Gymnasium CartPole, Acrobot, MountainCar and LunarLander from pixels, greedy; the screen ghosts the previous frame so motion is visible. Normalized: 0 = random, 1 = scripted expert; solved is the share of episodes reaching the environment's solved score.

Latency¶

One predict call with one question on a real validation image, preprocessing included, on an NVIDIA L4 in bf16: median 40.8 ms, p90 46.3 ms (n 165, mean views per image 1.0, mean input tokens 130.8).

Reading the numbers¶

• accuracy: share of questions where the most likely option is the labelled answer.
• ECE: expected calibration error, the average gap between stated confidence and accuracy (15 bins). 0.02 means a 70% answer is right about 68–72% of the time.
• NLL: negative log-likelihood of the right answer; punishes confident mistakes.
• cross-entropy against human votes: how far the model's probabilities are from the vote spread; compare with always predicting the dataset's average vote.
• levels off: for rubric scores, how far the model's expected level is from the voters' expected level.
• normalized (Atari, classic control): score rescaled so random play is 0 and the expert is 1.