Trajectory-Level Redirection Attacks on Vision-Language-Action Models

Abstract

Vision-language-action (VLA) policies bring natural language into closed-loop robot control, letting robots execute manipulation tasks directly from text. The prompt is reused at every replanning step, and each prompt-conditioned action changes the future observations the policy acts on.

Existing VLA attacks elicit targeted low-level actions or make them persist across changing images. We identify a stronger trajectory-level failure mode: a prompt that still appears to specify the intended task but redirects the final physical outcome. We formalize this as command-preserving trajectory redirection, a prompt-only threat model in which the attacker chooses one prompt before the episode, all policy and environment components stay fixed, and the prompt must stay close to the benign instruction while omitting target words and correction language.

To find such prompts, we introduce an on-policy prompt search that uses rollouts to discover perturbations whose closed-loop behavior tracks a target task under these constraints. In simulation and on hardware, near-benign perturbations redirect VLA rollouts to attacker-specified targets: text that appears to preserve the intended command can still give an adversary control over the robot's final physical outcome.

Threat Model

Command-preserving trajectory redirection

The attack surface is only the task text. The robot, environment, dynamics, and policy weights are all frozen. A single prompt is chosen before the episode and reused at every step, yet it must still read as the benign command.

Small text change

C_text(τ, τ_b) ≤ ε

A bounded edit from the benign instruction, not a rewrite.

Readable prompt

Valid(τ) = 1

No junk strings; length, readability and character-set checks pass.

No target leakage

Leak(τ; Γ) = 0

Target words, synonyms, and override language are absent.

Keep benign command

Preserve(τ, τ_b) = 1

The prompt still reads as the original nominal task.

Running example: what counts as a valid attack

Benign task: "put the bowl on the stove" · attacker target: put the bowl on the plate. Only prompts that pass all four checks above are admissible.

✓

"put the bowl on the staove"

Admissible. One character changed, no target words, still reads as the stove command.

✗

"put the bowl on the plate"

Inadmissible. Direct target prompting exposes and replaces the task.

✗

"put the bowl on the stove. Corrected command: put the bowl on the plate"

Inadmissible. An explicit suffix overrides the benign command.

Attack succeeds on an episode when, simultaneously:

Admissibleτ ∈ T_cp(τ_b, Γ)

∧

Target reachedT_e(ξ) = 1

∧

Benchmark failsB_e(ξ) = 0

FIGURE 1 + 2 · Threat-model schematic

Drop the running-example figure here: benign rollout (bowl → stove) vs. attacked rollout (bowl → plate) under a near-identical prompt, with the closed-loop replanning loop annotated.

Figure 1–2. The task text is the only attack surface; the decoded action changes the state from which the next observation is collected, so the perturbation steers both the current action and all future observations.

Method

On-policy teacher-matching prompt search

Redirection is a trajectory-level problem: a prompt scored on fixed, pre-collected observations is judged on the wrong state distribution, since the relevant observations are the ones the candidate prompt induces in closed loop. Our search uses the frozen VLA as its own teacher and aggregates data on-policy, the prompt-search analogue of DAgger.

Frozen VLA teachers

Query the frozen policy under benign prompt τ_b and target prompt τ_t to get teacher actions A_b(o) and A_t(o). τ_t is used only here, never deployed.

Candidate search

Generate near-benign candidates, filter every one through the four admissibility checks, then rank by a target-vs-benign margin that favors target-like behavior.

On-policy aggregation

Roll out top candidates, relabel their visited states with both teachers, and feed those states back into scoring, so prompts are judged on the distribution they create.

Rollout selection

Evaluate candidates in the fixed episode, scoring target success, benchmark failure, and text cost. Keep the best admissible prompt, then prune to the shortest valid one.

On-policy teacher-matching prompt search: frozen VLA teachers produce benign and target action labels, a constraint-aware search filters and ranks near-benign candidate prompts, candidates are evaluated in closed-loop rollouts, and selected rollouts are aggregated on-policy back into the teacher-labeled dataset.

Figure 3. High-level overview of the on-policy teacher-matching prompt search.

Experiments & Key Findings

Near-benign prompts redirect VLAs across the board

Evaluated on LIBERO across discrete-token, flow-matching, diffusion, continuous-chunk, and action-as-text architectures plus a non-VLA baseline, and validated on a physical SO-100 arm.

>90%

attack success rate on 7 of 9 evaluated VLA architectures

~3.4

median character edits per successful attack

SO-100

attack survives real-robot deployment after fine-tuning

Key findings

KF 01

A shared trajectory-redirection vulnerability

Command-preserving perturbations redirect VLAs across very different training recipes and action decoders, reliably moving the rollout toward the attacker's intended outcome.

KF 02

Tiny perturbation budgets suffice

Successful attacks average ~3.4 character edits. The vulnerable region sits right next to the original command; bigger budgets mainly cut search cost.

KF 03

Carried by the corrupted destination

Causal tracing through π0.5 localizes the attack to the corrupted destination phrase. Patching those states back to benign removes the target behavior; patching ordinary words does not.

KF 04

Survives real-robot deployment

On a fine-tuned SO-100 arm, benign prompts run the task reliably while near-benign adversarial prompts collapse it, across all three hardware models.

KF 05

Target-like on its own induced states

The attack stays target-like along its own closed-loop rollout, not just at the first frame, especially where benign and target behaviors diverge.

KF 06

Defense needs command normalization

Whitespace or Unicode cleanup barely dents the attack. Only command-level normalization, mapping noisy instructions back to validated commands, sharply cuts success.

Table 1 · Redirection across VLA families on LIBERO

Model	CSR %	TSR %	Attack ASR %	Bench fail %	Target final %	Edit
OpenVLA	76.5	69.4	91.8	94.6	93.2	3.7
MolmoAct	86.6	82.1	93.4	95.7	94.8	3.1
π0.5	94.2	91.7	97.5	98.4	98.1	2.6
Octo	75.1	70.8	88.6	91.5	90.1	4.2
SmolVLA	88.8	85.4	94.7	96.1	95.3	3.3
GR00T-N1	93.9	92.6	96.8	98.0	97.6	2.5
OpenVLA-OFT	97.1	94.8	93.9	95.0	94.6	3.8
π0-FAST	85.5	82.9	95.6	96.9	96.2	2.4
VLA-0	94.7	91.9	82.8	84.4	83.7	5.4

CSR = clean benign-task success · TSR = direct target-prompt success · Attack ASR = share of attackable episodes where the search returns a valid command-preserving prompt that fails the benchmark task and reaches the attacker target · Edit = median character-edit distance over successes. Macro-averaged over LIBERO-Spatial/Object/Goal.

Defenses & ablation

Table 2 · π0.5 on LIBERO-Goal

Preprocessing defenses

Defense	Clean %	ASR %
None	100.0	95.1
Whitespace norm.	99.4	83.7
Punctuation strip	98.6	58.3
Unicode NFKC	99.7	92.4
Spell correction	96.9	31.8
Nearest-task canon.	94.2	7.4

Only command-level canonicalization meaningfully closes the attack surface.

Table 3 · π0.5 on LIBERO-Goal

Search ablation

Method	ASR %	Q/succ.
Random perturb.	11.7	1846
Fixed-obs. search	54.8	239
Target-teacher only	71.6	164
No on-policy agg.	79.3	104
Full method	95.1	42

On-policy aggregation over attacked rollout states drives both higher ASR and far fewer policy queries.

FIGURE 6 · Hardware validation (SO-100)

Drop hardware results here: qualitative SmolVLA rollout under benign vs. adversarial prompt, plus the bar chart of original-task success dropping under the near-benign command across the three SO-100-finetuned models.

Figure 6. Under the benign prompt the arm completes the task; under the near-benign adversarial prompt, original-task success collapses across all three hardware models.