How Do We Compare Vision-Language-Action Models?

Vision-Language-Action models, or VLAs, are becoming the policy interface for modern robot learning. A VLA takes visual observations, a language instruction, and sometimes proprioceptive robot state, then predicts robot actions.

In simple words: you tell the robot what to do, it sees the world, and it moves.

But there is a hard question behind every new VLA paper:

How do we know one VLA is actually better than another?

This question is not as simple as comparing two image classifiers on ImageNet. A VLA can have low training loss but still fail to pick up an object. It can succeed in one camera setup but collapse when the camera moves slightly. It can appear to understand “put the mug in the drawer” in one scene, then fail when the mug is blue instead of white.

So evaluating VLAs is really about evaluating behavior, not just prediction. The right question is not “did the model predict a plausible action?” It is “did the closed-loop robot system complete the task, recover from mistakes, and still work when the scene changes?”

This article gives a practical map for reading VLA results: which metrics matter, what they hide, and why benchmarks such as LIBERO are useful.

---

The Evaluation Checklist

A serious VLA evaluation should answer five questions:

Question	What it tests	Why it matters
Does it complete the task?	Closed-loop success rate	The robot either reaches the intended final state or it does not.
Does it generalize?	New objects, layouts, wording, lighting, and viewpoints	Memorized behavior is brittle outside the demo distribution.
Does it follow language?	Object references, spatial relations, paraphrases, and compositional instructions	A language-conditioned policy should not succeed by ignoring language.
Does it run fast enough?	Latency, throughput, control frequency, and memory	Real robots have timing constraints, not just accuracy targets.
Is the result reproducible?	Public protocol, seeds, rollout counts, dependencies, and baselines	Benchmark numbers are only useful when others can check them.

No single metric covers all five. Success rate is the anchor, but a credible comparison needs protocol details, robustness tests, baseline context, and deployment constraints.

---

The Main Comparison Points

A VLA is a policy. It controls an agent over time: observe, act, observe again, act again. That makes closed-loop task success the anchor metric.

A benchmark defines tasks, environments, instructions, and rollout episodes. Each episode is marked as success or failure. If we test “put the red mug on the plate” 50 times and the robot succeeds 42 times, the success rate is 84%.

Success rate is the most important number, but it can hide weakness. A model may score well because the benchmark is narrow, the camera is fixed, the object layouts are familiar, or the instructions are template-like.

Generalization and language

A strong VLA should not only memorize training tasks. Researchers test generalization across:

• object positions and instances
• spatial layouts
• language instructions
• lighting
• camera viewpoints
• task combinations
• longer task horizons

This is where many VLAs still struggle. They can look impressive in standard settings and become brittle under distribution shift.

Language grounding

VLAs are supposed to follow language, not just move toward familiar objects. Researchers ask:

• Does the model attend to the instruction?
• Can it distinguish “red cup” from “blue cup”?
• Can it follow “put apple in bowl” instead of “put apple on plate”?
• Can it handle paraphrases?
• Can it follow compositional instructions?

Good evaluation breaks shortcuts. If the same object is always in the same location, the model may ignore the instruction and still get high scores.

Robustness

Robustness asks whether performance survives realistic changes. Common perturbations include:

• Move the camera.
• Change the lighting.
• Change object positions.
• Change background texture.
• Rephrase the instruction.

This is the motivation behind robustness-focused variants such as LIBERO-Plus.

The two rollouts below show why robustness needs closed-loop evaluation. They are not a benchmark by themselves; they are examples of how small distribution shifts can change policy behavior.

---

What Makes a VLA Comparison Credible?

The same headline number can mean different things depending on protocol. Most VLA evaluations follow a simple flow:

1. Define the benchmark or real-world robot setup.
2. Test zero-shot or fine-tune on target demonstrations.
3. Run closed-loop rollouts where actions affect future observations.
4. Score each rollout as success or failure.
5. Compare against relevant baselines.

A credible report makes the protocol visible, especially task split, rollout count, demonstration data, control frequency, seeds, simulator settings, and baselines.

One average score is not enough. Technical readers need per-task success, long-horizon completion, language accuracy, latency, and common failure modes. Baseline context also matters: 90% success means little unless the comparison uses the same setup.

Robotics results are sensitive to small changes in seed, camera setup, simulator version, or action horizon. A VLA result is only meaningful when the protocol is clear enough to reproduce or challenge.

---

The Practical Standard

For Verapulse, the standard is simple: a VLA result should be useful to someone trying to build or evaluate a real robot policy.

That means we care about closed-loop task success, data requirements, adaptation cost, control-loop latency, and whether the evaluation protocol is clear enough to repeat.

In short: A good VLA benchmark does not just say the model looked smart. It shows where the policy works, where it breaks, what it costs, and how to check the claim.