1University of California, Berkeley 2Amazon ∗Work done during an internship at Amazon
Preprint · 2026
Reinforcement learning (RL) in simulation can acquire precise dexterous control, but it relies on privileged state that is unavailable at deployment. Latent RL Prior distills large-scale RL rollouts into a flow-matching action decoder, freezes it, and learns only a visual interface that maps camera images to the decoder’s latent context, using demonstrated actions as the sole supervision.
10×
fewer demonstrations on door lifting: 50 vs. 500 for the full-data VLA baselines
5×
fewer demonstrations on bimanual assembly: 1,000 vs. 5,000 for the full-data VLA baselines
32.6%
door-lifting success at an unseen 5.0 kg mass with 50 demos, vs. 12.9% for π0.5 with 500
8 / 8
decoder layers with lower contact-force probe error than a scratch decoder, for every force target and budget
Baselines are full-parameter fine-tuning of π0.5 and GR00T N1.7. Efficiency refers to downstream visual demonstrations; the action prior is pretrained on 100,000 successful simulated RL rollouts.
Learning dexterous manipulation from limited visual demonstrations is challenging because policies must acquire both perceptual grounding and precise control, while reinforcement learning (RL) in simulation can acquire rich control behaviors from large-scale interaction with privileged information. We introduce Latent RL Prior, a two-stage framework that reuses this control knowledge for data-efficient visual policy learning. During privileged pretraining, we distill large-scale RL rollouts into a flow-matching action decoder conditioned on proprioception and latent task context. During visual adaptation, we freeze the decoder and learn a visual interface that maps images to its latent context using action supervision alone, without privileged labels or latent targets. The same decoder can further be shared across arms by learning coordinated visual contexts for bimanual manipulation. Across three simulated tasks, our method outperforms full-parameter fine-tuning of π0.5 and GR00T N1.7 in the low-data regime; on door lifting and bimanual assembly, it surpasses full-data baselines using 10× and 5× fewer demonstrations, respectively. Further analysis shows that direct visual-to-latent conditioning improves adaptation performance and training stability over routing visual predictions through the pretrained state encoder. Robustness to unseen mass variation and greater linear accessibility of contact-force information in decoder representations further provide evidence of retained RL-acquired control knowledge.
Training has two stages that share one action decoder. Task context reaches the decoder through cross-attention, while proprioception and flow time enter through adaptive layer normalization (AdaLN). The source of the context can therefore change from privileged state to images while the pretrained proprioceptive processing and action generation stay intact.
Stage 1
An RL teacher, fine-tuned from the released SimToolReal checkpoint on procedural pose alignment with randomized object shape, size, and density, provides 100,000 successful rollouts.
A state tokenizer and Q‑Former map the privileged history to latent context tokens. A transformer action decoder cross-attends to them and is trained with conditional flow matching.
Stage 2
The action decoder is frozen. A DINOv2 image encoder and a visual Q‑Former map camera images directly to the decoder’s context tokens (latent transfer).
Training uses the same flow-matching action loss: no privileged-state labels, no latent targets, and no trajectories paired with the RL data.
Extension
Each arm receives its own group of queries. Both groups are processed jointly by one visual Q‑Former, so self-attention can coordinate the arms.
The same frozen single-arm decoder then generates each arm’s actions from that arm’s context and proprioceptive history.
Change the context source; keep the decoder
Pretraining: Zt = Qψ(Tη(Pt))
Adaptation: Zt = Qϕ(Evis(It))
Shared decoder: vθ(Atτ, τ, Zt, Rt) ≈ At1 − At0
Pt: privileged history · It: image history · Rt: proprioceptive history · At: action chunk · τ: flow time. The decoder vθ is trained in stage 1 and frozen afterwards.
All experiments run in MuJoCo Warp with KUKA iiwa arms and SHARPA dexterous hands. A single pretrained action decoder is reused for every downstream task.
Single arm · Unseen object
Pick up an unseen brush and match its target pose.
Success criterion. All four object–target keypoint distances must be within 1.5 cm (strict) or 5 cm (relaxed) for 10 policy steps, not necessarily consecutive.
Videos replay collected demonstration trajectories in simulation; they are not closed-loop evaluations of the adapted policy. Download this video
| Dataset | Episodes | Arm–hand | Cameras |
|---|---|---|---|
| Pretraining | 100,000 | single | – (privileged state) |
| Target pose alignment | 4,000 | single | F, S, H |
| Door lifting | 500 | single | F, S, H |
| Bimanual assembly | 5,000 | bimanual | F, HL, HR |
F: front, S: side, H: in-hand (L/R: left/right). Only the pretraining rollouts contain privileged information; the visual datasets do not. Downstream demonstrations come from a separate SAPG policy trained for each task.
Does the RL prior reduce the number of visual demonstrations needed?
Door lifting
50 vs. 500
With 50 demonstrations, our policy outperforms both VLA baselines trained on the full 500-demonstration dataset.
10× fewer demonstrations
Bimanual assembly
1,000 vs. 5,000
With 1,000 demonstrations, the shared single-arm prior surpasses both full-data baselines on a two-arm task.
5× fewer demonstrations
Target pose alignment
ε = 1.5 cm
Higher success at every budget under the strict threshold. Under the relaxed 5 cm threshold the gap narrows with more data, becoming comparable to π0.5 at 4,000 demonstrations.
Largest gains for precise control
Takeaway: The RL prior helps most when visual demonstrations are scarce or the task demands precise control.
Where does the gain come from, how should vision access the prior, and what does the prior retain after adaptation?
How should visual observations connect to the frozen prior?
Latent transfer maps image features directly to the decoder’s context tokens. Explicit transfer instead predicts a privileged-shaped state and routes it through the frozen pretrained state encoder. Both use action-only supervision.
Takeaway: Reusing the prior depends on how vision interfaces with it. Conditioning the decoder’s latent context directly gives higher success and more consistent training than explicit transfer.
Which modules should change during visual adaptation?
Takeaway: Adapt the image encoder conservatively and keep the action decoder frozen.
Does robustness acquired during RL pretraining survive visual adaptation?
RL pretraining randomizes object density, whereas each visual dataset contains a single object mass. Policies are evaluated on heavier objects without further training.
Door lifting at 5.0 kg
Both methods perform similarly at the reference mass.
Takeaway: As mass increases, the RL-prior policy degrades more gradually than π0.5, despite using fewer demonstrations.
Do the decoder’s representations retain physical information learned through RL?
Linear ridge probes predict contact forces from the hidden state of each decoder layer of the adapted pose-alignment policy, using a separate force-annotated dataset. Lower RMSE means the force information is more linearly accessible. The comparison is a decoder of the same architecture trained from scratch.
Layer 4 · probe RMSE
Figure 9 data: mean over three policy seeds on identical IID probe test episodes; RMSE in newtons. Both policies fine-tune the image encoder at 0.2× its base learning rate for 100k steps. The vertical scale is fixed within each force target.
Takeaway: Contact forces are more linearly decodable from the RL-prior decoder at every layer and budget. With 200 demonstrations, it already matches or outperforms the scratch decoder trained on 4,000.
The mass and force analyses are consistent with retained control knowledge from RL pretraining; they do not isolate a causal mechanism for the demonstration-efficiency gains.
If you find this work useful, please cite it. This entry will be replaced with the arXiv citation once the paper is posted.
@misc{wang2026latent,
title = {Latent RL Priors for Data-Efficient Dexterous Manipulation},
author = {Yixiao Wang and Peiyi Chen and Mustafa Mukadam and
Shreekant Gayaka and Masayoshi Tomizuka and Son Tran},
year = {2026},
note = {Preprint; arXiv version forthcoming}
}