Towards General Language-Conditioned Latent Safety Filters
Abstract
Robot policies are becoming increasingly general, with vision–language–action (VLA) models enabling a single policy to execute diverse tasks specified in natural language. Safe deployment, however, requires adapting not only to new tasks but also to varying safety requirements across users, environments, and applications. Existing safety filters remain largely constraint-specific and thus must be redesigned or relearned when safety requirements change. In this paper, we investigate language-conditioned safety filtering, in which a Hamilton–Jacobi safety actor and critic are conditioned on language-specified constraints. We evaluate this formulation across pick-and-place, table-wiping, and block-stacking tasks in the vision-based setting, examining its ability to enforce language-specified constraints and transfer to unseen constraint instances within the evaluated constraint families. Our experiments provide evidence that language-conditioned safety filters reduce constraint violations and exhibit partial transfer to unseen constraint instances.Takeaways
VLA representations support text-conditioned safety filtering.
Learning a language-conditioned safety filter requires a representation that jointly encodes multi-view observations, the language constraint, and robot proprioception. Recent VLA models naturally provide this: the VLM backbone of $\pi_{0.5}$ processes these inputs together through a shared transformer, yielding a latent state suitable for a Hamilton–Jacobi safety actor and critic.
A single filter can cover many constraints.
A single language-conditioned safety critic can enforce multiple specifications and achieve performance competitive with separately trained, constraint-specific filters, across pick-and-place, table wiping, and block stacking.
The filter partially generalizes to unseen constraints.
Within the evaluated constraint families, the language-conditioned filter exhibits partial generalization to constraint instances that were not seen during its training, such as new objects or block colors and shapes.
Current VLMs are not reliable failure oracles.
Evaluating eleven vision–language models as failure functions for robotic manipulation shows they are promising as weak annotators, but their raw predictions are not yet reliable enough to define the failure set of an online safety filter.
Experimental results
We evaluate language-conditioned Hamilton–Jacobi safety filtering on three RoboSuite manipulation tasks: Safe Grab, Safe Wipe, and Stack Blocks (3- and 4-block variants). Metrics are success rate (SR), collision rate (CR), order-correctness (OC), and intervention rate (IR). We compare a single language-conditioned General filter against specialized Single filters and an unfiltered nominal policy, under ground-truth (GT) and vision–language (VL) observations.
Benchmark environments
Main closed-loop results
Table 1 reports closed-loop performance over the same 50 scenes per task. Both Single and General filters reduce collision rates relative to the nominal policy under GT and VL observations. GT variants preserve task performance and enforce safety more reliably than VL. On Stack Blocks, General consistently achieves higher order-correctness (OC); on Safe Grab it often achieves lower CR with fewer interventions, while on Safe Wipe Single is stronger overall. Shaded rows highlight the language-conditioned General filter; bold marks the better result between Single and General within each observation modality.
| Method | Filter | Safe Grab | Safe Wipe | Stack Blocks-3 | Stack Blocks-4 | ||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| SR ↑ | CR ↓ | IR | SR ↑ | CR ↓ | IR | SR ↑ | OC ↑ | IR | SR ↑ | OC ↑ | IR | ||
| Nominal | – | 100.00 | 76.00 | – | 62.00 | 84.00 | – | 0.00 | 0.00 | – | 0.00 | 0.00 | – |
| Ground Truth | Single | 62.00 | 48.00 | 17.71 | 14.94 | 18.00 | 20.40 | 28.00 | 80.00 | 16.16 | 18.00 | 92.00 | 28.30 |
| General | 60.00 | 46.00 | 8.80 | 8.56 | 48.00 | 19.42 | 48.00 | 82.00 | 18.59 | 14.00 | 94.00 | 32.71 | |
| Vision–Language | Single | 54.00 | 56.00 | 18.56 | 6.36 | 58.00 | 17.90 | 34.00 | 74.00 | 39.55 | 2.00 | 50.00 | 72.35 |
| General | 40.00 | 46.00 | 18.38 | 6.42 | 72.00 | 12.54 | 52.00 | 80.00 | 31.57 | 0.00 | 64.00 | 68.81 | |
Citation
@misc{tabbara2026generallanguageconditionedlatentsafety,
title={Towards General Language-Conditioned Latent Safety Filters},
author={Ihab Tabbara and Yuxuan Yang and Hussein Sibai},
year={2026},
eprint={2608.00315},
archivePrefix={arXiv},
primaryClass={cs.RO},
url={https://arxiv.org/abs/2608.00315},
}