A robot crossing a building has to decide where to go and how to move its legs at the same time. H-JEPA, a world-model architecture from researchers at Yann LeCun's Advanced Machine Intelligence (AMI) Labs and academic partners, tackles that by planning at several levels of abstraction. It is a different approach from today's agentic and generative AI systems, because it models the physical world rather than generating text or pixels.
Compared with flat JEPA world models, H-JEPA improved accuracy on navigation tasks while needing less planning compute.
Why JEPA World Models Need a Hierarchy
Joint-Embedding Predictive Architectures (JEPAs) encode images or video into compact latent representations instead of predicting every pixel. That makes them efficient and well suited to tasks like navigation and object manipulation, where generating pixels is unnecessary. In action-conditioned JEPAs, the model also receives actions, so it can simulate the outcome of different action sequences and compare them with a goal.
Flat versions hit two related problems:
- Cost and drift: Predicting a distant outcome means simulating many small steps, which raises compute and lets small errors compound.
- One representation, two jobs: The same latent space must track fine physical dynamics and judge progress toward a faraway goal.
How H-JEPA Learns
H-JEPA trains on sequences of observations and the actions linking them. The lowest level encodes visual observations and predicts the next latent state from earlier states and actions. Higher levels take the representations from the level below and predict transitions over longer time intervals.
- State encoder: Represents the environment at that level's timescale.
- Action encoder: Summarizes actions over that level's interval.
- Predictor: Learns how the represented state changes.
The whole hierarchy trains end to end. Each level compares its prediction with the observed future representation, and a regularization mechanism stops the model from collapsing to the same representation for every input. Training signals from higher levels also flow through the lower-level encoders.
The time intervals between levels are fixed in advance, but what each level keeps is learned. Higher levels can hold on to slow, predictable features and drop fast-changing detail. In the maze, the top level keeps the robot's location and discards its leg configuration.
How H-JEPA Plans
- Encode: The current and target observations are encoded at every level.
- Plan at the top: The high-level planner searches for actions whose predicted outcomes land closer to the goal in its latent space.
- Pass subgoals down: Predicted states become subgoals for the next level, which searches for actions to reach them.
- Reach primitive actions: Each level refines the plan until the lowest level outputs executable actions.
- Replan: After a short run of actions, the system updates the plan using new observations and any deviation from the predicted path.
What the Experiments Showed
| Test | What It Covers | Result |
|---|---|---|
| FourRoom, AntMaze, Cube | Simulated navigation and manipulation | Adding levels up to three generally raised success and cut planning compute |
| AntMaze (3 levels) | Visual maze navigation vs. HWM | Nearly twice HWM's success rate |
| DROID | Real robot manipulation videos | Better offline planning fidelity than baselines, with less planner compute |
The team also tested Push-T and FourRoom Distractors, comparing against LeWM (a single-level JEPA) and HWM (a hierarchical model that shares one latent space). In AntMaze, the robot's position stayed recoverable at higher levels while leg-configuration detail faded. Higher levels also gave smoother distance-to-goal estimates along corridors, which helps the planner route around walls.
On DROID, pure predictive training kept static scene details and lost information about the moving robot. Adding an inverse dynamics objective, which predicts the action connecting two observed states, helped preserve action-relevant information.
Who Benefits
H-JEPA is most relevant to robotic navigation, warehouse automation, and manipulation systems that must tie long-term goals to precise movements. The researchers also propose linking higher-level representations to language, so that agents could eventually receive goals as instructions instead of target images. That is a direction, not a demonstrated result.
If you follow how AI companies are positioning their technology, our AI power map of chips, models, and cloud dominance gives useful background.
Why This Matters
Most of the AI attention goes to language models, but robots and physical systems need models of how the world changes. H-JEPA shows that separating "where am I going" from "how do I move" can improve accuracy and cut compute at once. The same idea of splitting high-level direction from low-level execution also shows up in agent design, as in our guide to reducing AI agent costs.
Official source: H-JEPA project page.
Frequently Asked Questions (FAQ)
What is H-JEPA?
H-JEPA is a hierarchical world-model architecture that learns separate representations at different timescales, so an AI system can plan a long-term route and the precise movements for it.
How is a JEPA different from a generative model?
A JEPA encodes observations into compact latent features instead of predicting granular details like pixels, which makes it more efficient for tasks such as navigation and manipulation.
How does H-JEPA improve on flat JEPA models?
It avoids simulating many tiny steps for distant goals and stops forcing one representation to track both fine physics and long-term progress, improving accuracy while using less planning compute.
Does more hierarchy always help?
No. The experiments show that more levels are not always better, and results depend on the scope of the training data.
Can H-JEPA take language instructions?
Not yet. The researchers propose connecting higher-level representations to language as future work, so goals could be given as instructions instead of target images.
No comments:
Post a Comment