Abstract
Effective long-horizon planning using latent world models remains a formidable challenge, and temporal abstraction is a promising direction for addressing it. We investigate whether this abstraction can improve LeWorldModel on long-horizon, goal-conditioned control tasks. To this end, we introduce Hi-LeWM, an extension that freezes the pretrained low-level LeWM and adds a high-level predictor that plans over latent subgoals. We evaluate Hi-LeWM on PushT and OGBCube across increasing goal offsets; our results suggest that hierarchy does not automatically improve performance. At short horizons, the best configuration benefits from a one-step high-level plan, while longer horizons reveal a mismatch between the learned macro-action space and the inference-time distribution, which we trace to the quality of CEM-selected subgoals as the main bottleneck. This is because unconstrained search can select latent macro-actions that appear favorable under the learned model but yield poor control targets. We find that constraining the search around macro-actions encoded from training trajectories, combined with appropriate subgoal execution timing, recovers effective hierarchical regimes, improving over flat LeWM by +11.3% at medium-range horizons and +14.7% at the longest PushT horizon. Overall, temporal abstraction can benefit long-horizon planning in LeWM, but only when paired with a constrained high-level search space.
Background: flat planning in LeWorldModel
LeWM plans directly over primitive actions in a compact JEPA-style latent space. That is exactly what makes long horizons hard.
An encoder maps the current observation and the goal image to latents and . A frozen action-conditioned predictor rolls the latent forward under a candidate action sequence, and planning picks the sequence whose predicted terminal latent lands closest to the goal latent.
The optimization runs with the cross-entropy method inside a receding-horizon loop: only a short prefix of each plan is executed before replanning.
Two things degrade as the goal moves further away. Model error compounds across imagined rollouts, and the optimizer has to search a longer sequence of primitive actions under a fixed budget. LeWM stays at a single temporal scale throughout, which is efficient at short horizons and fragile at long ones.
Method: add a level, freeze everything else
Hi-LeWM keeps the pretrained LeWM stack fixed and learns only two modules on top: a macro-action encoder and a high-level predictor.
The setting is deliberately restrictive. The low-level planner is untouched, so any gain has to come from the added high-level model and its planner. We follow the waypoint-based interface of recent hierarchical latent-planning work: latent waypoints are encoded observations sampled at temporally separated states along a trajectory, and the macro-action between two waypoints is a compressed summary of the primitive actions executed in between.
What is trained
- Macro-action encoder g
- 2-layer transformer, 4 heads, d_model 192
- Macro-action latent
- dℓ = 32
- Max action chunk
- 15 steps
- High-level predictor
- depth 6, 16 heads, FF 2048, adaLN
- Waypoints
- N = 5, random sorted, span ≤ 15
- Optimizer
- AdamW, lr 5e-5, wd 1e-3, bf16
- Batch / epochs
- 128 / up to 15
- Parameters
- 30.5M total, 12.5M trainable
Joint end-to-end training of the low-level predictor and encoder alongside the high level was tested and gave no measurable gain, so the frozen setup was kept for its lower cost.
Results
Goal offsets of 25, 50 and 75 environment steps on PushT and OGBCube. Best configuration per sweep, mean and standard deviation over three seeds, 50 episodes per run.
Reading the numbers
Flat LeWM degrades sharply with distance on PushT: 94.0% at d=25, 52.7% at d=50, 18.0% at d=75. That collapse is the regime hierarchy is supposed to rescue.
It does not. Naive Hi-LeWM lands at 89.3%, 38.7% and 15.3%, which is 14.0 points below the flat planner at d=50 and still 2.7 points below it at d=75. The d=25 result is not a hierarchy win either: the best sweep there uses a high-level horizon of hh=1, so the strongest short-horizon configuration is a one-step subgoal planner rather than a real temporal abstraction.
OGBCube behaves differently. Its flat baseline stays comparatively stable across offsets (65.3%, 52.0%, 53.3%), so it is a useful robustness check but not diagnostic of horizon-induced collapse. It is reported in the appendix for that reason.
Flat LeWM baseline sweep. Action block size 5, CEM budget 300 x 30 in all runs. Shaded cells in the paper mark the best setting; here they are highlighted in red.
| Goal | Budget | h | r | PushT (%) | Cube (%) |
|---|---|---|---|---|---|
| d=25 | 50 | 5 | 5 | 94.0 ± 2.0 | 65.3 ± 4.2 |
| d=50 | 100 | 5 | 1 | 36.0 ± 7.2 | 48.0 ± 2.0 |
| d=50 | 100 | 5 | 5 | 52.7 ± 5.0 | 52.0 ± 3.5 |
| d=50 | 100 | 10 | 5 | 22.0 ± 8.0 | 45.3 ± 1.2 |
| d=75 | 150 | 5 | 5 | 18.0 ± 2.0 | 53.3 ± 7.0 |
| d=75 | 150 | 10 | 5 | 10.7 ± 1.2 | 40.7 ± 8.3 |
| d=75 | 150 | 15 | 5 | 3.3 ± 2.3 | 41.3 ± 4.2 |
Where the hierarchy breaks
Two diagnostics on PushT at d=50 separate the low-level controller from the high-level planner. The controller is fine. The planner is not.
The frozen low level can execute good targets
Replacing the high-level planner with oracle subgoals, that is, true future latents taken from the dataset, lifts success to 73.3 ± 4.2% at d=50, well above both the flat planner and naive Hi-LeWM. Executability is therefore not the bottleneck: when intermediate targets are aligned with the task trajectory, the frozen low-level planner reaches them.
The qualitative picture matches. Oracle targets are coherent configurations of the T block and the pusher. CEM-generated targets are diffuse, poorly localized, and inconsistent with any reachable intermediate state.

The high level exploits its own objective
Rolling the high-level model forward without executing anything isolates the planner. Under teacher forcing and under open-loop rollouts driven by true macro-actions, prediction error grows in the expected, mild way. Driven by CEM-selected macro-actions, the pattern inverts: the first stage becomes four times worse while the second stage becomes the most accurate of all.
That is the signature of optimizer exploitation. CEM is scored only on the terminal latent, so it is free to pick macro-actions that arrive at a favorable final prediction through an intermediate waypoint that is useless as a control target. The hierarchy then hands exactly that broken waypoint to the low-level planner.
High-level forecast diagnostic, latent MSE. Lower is better.
| Stage | Teacher-forced | Open-loop true | Open-loop CEM |
|---|---|---|---|
| Stage 1 | 0.081 | 0.081 | 0.347 |
| Stage 2 | 0.104 | 0.216 | 0.011 |

Intervention: put the search on rails
If the failure is a support mismatch, the fix belongs at inference time: keep high-level search near macro-actions the data could actually have produced.
Empirical-macro CEM
Instead of sampling candidate sequences from an unconstrained Gaussian, we build a bank of macro-action sequences encoded from real training trajectories, then sample candidates as local residual perturbations around those anchors.
CEM still optimizes, but it refits only the residual distribution from the elite residuals; the empirical anchor itself is never optimized. Each iteration resamples anchors and includes one zero-residual candidate, so an unmodified empirical macro-action sequence is always evaluated. For staged planning, sampling whole sequences from the bank preserves the temporal correlation between successive macro-actions and worked better than sampling each stage independently.
Staged execution
The second knob is timing. Online Hi-LeWM replans the high level periodically and passes only the first waypoint down. Staged execution runs high-level CEM once at the start of the episode, stores the ordered latent targets, and advances the active target on a fixed schedule while the low-level planner keeps closing the loop.
Staged execution preserves a coherent temporal scaffold but cannot correct a bad high-level rollout. That trade shows up directly in the numbers: it is the best option at d=50 (64.0%) and a liability at d=75 (22.0%, below online constrained search at 32.7%).

Macro-action representations
No parameterization of the high-level action space wins everywhere.
The reference model (first row) is best at d=50. Shrinking the continuous latent to dℓ=8 does not help, and fixed-stride waypoints cost a little. Raising it to dℓ=192 to match the latent-state dimension was tried early and made high-level planning much harder, since it enlarges the space CEM has to search.
At d=75 the vector-quantized variant with 128 codes is the strongest at 34%, which is consistent with the support argument: a discrete codebook is another way to keep high-level search inside a small set of data-supported abstractions. VQ-16 does not help, so the discretization can also be too aggressive.
VQ-128 is not the main model because it changes the trained representation rather than only the inference procedure, and it drags in codebook learning, chunk reconstruction, commitment and codebook losses, and a separate discrete planning interface. That is a different question from the one this paper asks.
Best PushT success rate per high-level representation design.
| Waypoints | Macro-action | Size | d=50 | d=75 |
|---|---|---|---|---|
| random | continuous | dℓ=32 | 46 | 20 |
| fixed stride | continuous | dℓ=32 | 42 | 20 |
| fixed stride | continuous | dℓ=8 | 42 | 20 |
| random | VQ | 16 codes | 38 | 20 |
| random | VQ | 128 codes | 44 | 34 |
Discussion
The picture is conditional rather than positive or negative. At d=25 flat LeWM is near saturation and the best hierarchical sweep degenerates to a one-step planner, so hierarchy buys nothing. At d=50 constrained search with staged execution reaches 64.0%, 11.3 points above flat. At d=75 staged execution fails but constrained online replanning reaches 32.7%, 14.7 points above flat.
The failure mode is neither low-level executability nor temporal abstraction on its own. It is that adding a high level creates a continuous latent search space whose support the optimizer is free to leave, and the terminal-latent objective rewards it for doing so.
Scale matters for interpretation. Zhang et al. report substantially stronger PushT numbers with a higher-capacity DINO-WM setup, so none of this rules out hierarchy in general. It identifies the conditions under which this particular interface transfers to a compact frozen LeWM: subgoals have to stay data-supported, executable, and matched to the temporal scale at which they are used.
Limitations
- PushT carries the main analysis because varying the goal offset gives controlled horizon growth. OGBCube does not collapse the same way, and no third environment was tested.
- Retraining-based representation choices, VQ macro-actions in particular, are left to future work. The interventions studied here are deliberately inference-time only.
- Results are best-of-sweep per model and offset over three seeds and 50 episodes, so they carry the usual selection optimism of a small sweep.
- The latent forensics section is a single-checkpoint exploratory track, not part of the paper's protocol.
Paper and reproduction
Reproduction
The implementation builds on stable-worldmodel, which supplies the environment interfaces, the MPC components and the baseline world-model infrastructure. Evaluation episodes are built from expert demonstrations: a start observation is sampled from a trajectory and the goal is a future observation from the same trajectory, separated by d environment steps.
Training the final models took 7.56 hours on a single A100 for PushT and 12.80 hours on a single H100 for Cube. Evaluation used 50 episodes per run, with individual runs between roughly 5 and 58 minutes depending on environment and planning budget.
Decoder probe
A lightweight query-based transformer decoder maps latents back to 224x224 pixels. It is used only for diagnostics and figures, never for acting. Reconstructions from true latents reach MSE 0.0040, PSNR 36.89 dB, SSIM 0.9913; from predicted latents 0.0097, 33.08 dB, 0.9834. Predicted latents therefore stay visually decodable, which is why the failures are attributed to planning rather than to a collapsed representation.

BibTeX
@misc{caselli2026mindgappromisespitfalls,
title={Mind the Gap: Promises and Pitfalls of Hierarchical Planning in LeWorldModel},
author={Niccolò Caselli and Francesco Massafra and Samuele Punzo and Salvatore Lo Sardo and Ippokratis Pantelidis and Sathya Kamesh Bhethanabhotla},
year={2026},
eprint={2607.12547},
archivePrefix={arXiv},
primaryClass={cs.RO},
url={https://arxiv.org/abs/2607.12547},
}
References
- LeWorldModel: Stable End-to-End Joint-Embedding Predictive Architecture from Pixels. arXiv:2603.19312, 2026.
- Hierarchical Planning with Latent World Models. arXiv:2604.03208, 2026.
- stable-worldmodel: A Platform for Reproducible World Modeling Research and Evaluation. 2026.
- The Cross-Entropy Method for Combinatorial and Continuous Optimization. Methodology and Computing in Applied Probability, 1999.
- Diffusion Policy: Visuomotor Policy Learning via Action Diffusion. IJRR, 2023.
- OGBench: Benchmarking Offline Goal-Conditioned RL. 2025.
- Learning Hierarchical World Models with Adaptive Temporal Abstractions from Discrete Latent Dynamics. ICLR, 2024.
- Deep Hierarchical Planning from Pixels. NeurIPS, 2022.
- DINO-WM: World Models on Pre-trained Visual Features enable Zero-shot Planning. 2025.
- IMWM: Intuition Models Complement World Models for Latent Planning. arXiv:2606.01626, 2026.
- Neural Discrete Representation Learning. NeurIPS, 2017.