Grounding LLMs with JEPA-based world models trained in simulation — has this been tried? [D]
Heat trend
Collecting trend data
The percentage is based on available heat signal, not comment count or independent people.
A discussion explores whether training JEPA-based world models within physics simulations like MuJoCo could ground LLMs. The idea is for the model to predict future state representations in an abstract embedding space, with prediction failures indicating incorrect physics, unlike next-token prediction. This approach aims to move LLMs beyond statistical relationships to a more intuitive understanding of physics, addressing a limitation akin to the Mary's Room problem.
LLMs describe physics well but don't "understand" it in any grounded sense — they've learned statistical relationships between tokens like "falls" and "gravity", not actual physical intuition. This is basically the Mary's Room problem: Mary knows every physical fact about color but has never seen one. LLMs are Mary.
The idea I've been thinking about:
- Train a JEPA-style model inside a physics simulation (think MuJoCo or a simple 2D env). Instead of predicting pixels or tokens, the model predicts representations of future states in an abstract embedding space. If it gets physics wrong, the predictor fails — the loss is unforgiving in a way next-token prediction never is.
- The embedding space that emerges should encode actual physical structure — object permanence, momentum, trajectories — because that's what makes prediction possible. Not surface-level textures, just the principles.
- Freeze those representations and attach them to an LLM-style reasoning model as a conditioning signal. The LLM now has both linguistic physics knowledge AND grounded physical intuition it can actually "run" forward — closer to a computational primitive than a propositional fact.
The hypothesis is this makes downstream learning significantly faster — the LLM doesn't have to rediscover that objects fall, it has representations that already encode that.
V-JEPA does something adjacent for video (predict future frame representations, not pixels), and DreamerV3 uses a latent world model for efficient RL — but the specific combo of JEPA-style prediction + sim-grounded physics representations + LLM attachment doesn't seem to have been done cleanly.
Questions for the community: - Is there prior work I'm missing that does exactly this? - What's the right interface between grounded JEPA representations and a language model? Just concatenate to the prompt embedding? Cross-attention? - Would the sim-to-reality gap kill the transfer, or are the representations abstract enough to survive it?
Interested in whether this is worth building a small prototype for.