Grounding LLMs with JEPA-based world models trained in simulation — has this been tried? [D]
热度趋势
趋势数据积累中
百分比基于当前可用热度信号，而非评论数或独立用户人数。
一项讨论探讨了在MuJoCo等物理模拟环境中训练基于JEPA的世界模型是否能为大型语言模型（LLMs）提供基础。核心思想是让模型在抽象嵌入空间中预测未来状态的表示，如果物理预测错误，模型就会失败，这与下一个词元预测不同。这种方法旨在使LLMs超越统计关系，获得对物理更直观的理解，从而解决类似于“玛丽的房间”问题所揭示的局限性。
LLMs describe physics well but don't "understand" it in any grounded sense — they've learned statistical relationships between tokens like "falls" and "gravity", not actual physical intuition. This is basically the Mary's Room problem: Mary knows every physical fact about color but has never seen one. LLMs are Mary.
The idea I've been thinking about:
- Train a JEPA-style model inside a physics simulation (think MuJoCo or a simple 2D env). Instead of predicting pixels or tokens, the model predicts representations of future states in an abstract embedding space. If it gets physics wrong, the predictor fails — the loss is unforgiving in a way next-token prediction never is.
- The embedding space that emerges should encode actual physical structure — object permanence, momentum, trajectories — because that's what makes prediction possible. Not surface-level textures, just the principles.
- Freeze those representations and attach them to an LLM-style reasoning model as a conditioning signal. The LLM now has both linguistic physics knowledge AND grounded physical intuition it can actually "run" forward — closer to a computational primitive than a propositional fact.
The hypothesis is this makes downstream learning significantly faster — the LLM doesn't have to rediscover that objects fall, it has representations that already encode that.
V-JEPA does something adjacent for video (predict future frame representations, not pixels), and DreamerV3 uses a latent world model for efficient RL — but the specific combo of JEPA-style prediction + sim-grounded physics representations + LLM attachment doesn't seem to have been done cleanly.
Questions for the community: - Is there prior work I'm missing that does exactly this? - What's the right interface between grounded JEPA representations and a language model? Just concatenate to the prompt embedding? Cross-attention? - Would the sim-to-reality gap kill the transfer, or are the representations abstract enough to survive it?
Interested in whether this is worth building a small prototype for.