World models vs. LLMs: Yann LeCun's billion-dollar bet

The big picture

Modern deep learning exploded with convolutional nets and supervised learning — AlexNet on ImageNet in 2012 being the watershed. But that approach leans heavily on human-labeled data, which is nothing like how humans actually learn.

Self-supervised learning changed the game. Next-token prediction in transformers uses raw text as its own label, and that unlocked the leap from GPT-1 to GPT-3 to ChatGPT. This generative, autoregressive recipe works astonishingly well for language — but it hits serious problems when naively applied to images and video.

Why generative video goes "blurry and wrong"

A straightforward "GPT for video" — predicting the next frame's pixels autoregressively — quickly collapses into blurry nonsense. The model has to average over many plausible futures in a gigantic continuous output space.

This raises the key question: do self-supervised models really need to be generative at all — or do they just need to learn useful internal representations?

Joint embeddings (and the collapse problem)

Joint-embedding methods encode inputs into vectors and learn to make embeddings of related views similar — e.g. two augmentations of the same image — without reconstructing pixels. The classic example is Siamese networks for signature verification: a contrastive loss pulls genuine pairs together and pushes fraudulent ones apart in embedding space.

Done naively at scale, though, joint embeddings suffer representation collapse: the network can just output the same vector (say, all ones) for every input — perfect similarity, zero learning.

Barlow Twins, DINO, and the breakthrough

Inspired by Horace Barlow's redundancy-reduction hypothesis, LeCun and collaborators introduced Barlow Twins: two encoders process two distorted views, and the loss drives the cross-correlation matrix of their outputs toward the identity. Diagonal entries → 1 (the same neuron agrees across views); off-diagonals → 0 (different neurons stay decorrelated). That both avoids collapse and forces neurons to capture distinct factors of variation.

The road to JEPA

JEPA didn't appear from nowhere — it's the culmination of a decade of joint-embedding research. Here's the lineage, from the original Siamese networks to today's LeJEPA:

From representations to world models: JEPA

In his 2022 position paper, A Path Towards Autonomous Machine Intelligence, LeCun argued that today's systems are nowhere near human learning efficiency. A teenager learns to drive in ~20 hours because they already carry rich world models of physics, objects, and causality. He defines "common sense" as exactly that: a collection of world models predicting what's likely, plausible, or impossible.

JEPA (Joint-Embedding Predictive Architecture) formalizes this. An encoder maps observations to states; a predictor maps the state at time t (optionally plus an action) to the state at t+1 — in embedding space, not pixel space. For video, it predicts the next frame's embedding, freeing it from chasing unpredictable details (leaves in the wind) and focusing on salient structure.

JEPA for action and control

Condition the predictor on an action (say, control signals to a robot arm) and JEPA becomes a learned dynamics model: state at t + action → predicted state at t+1 in latent space.

In V-JEPA 2, this powers robotic control: encode a goal image into a goal state, then search over hypothetical action sequences using the learned world model to find one whose predicted future matches the goal. It's classical optimal control — planning over a dynamics model — except both the representation and the dynamics are learned jointly instead of hand-designed.

Two architectures, visualized

LLMs vs. JEPA world models, side by side

The likely future isn't one winning — it's both: LLMs as the language/reasoning interface, world models as the grounded core for agents that act in the physical world.

The bet against LLM-centric AGI

LeCun's controversial claim: you can't build a truly reliable agentic system without a world model that predicts the consequences of its actions before acting. Current LLMs and vision-language-action models act via autoregressive token generation without internal simulation of outcomes — so they can't natively plan with safety guarantees.

His billion-dollar bet, via the newly formed AMI Labs ($1.03B seed at a $3.5B pre-money valuation, March 2026), is that JEPA-style world models — not ever-bigger LLMs — will be the foundation for autonomous, safe, and efficient AI agents, starting in industrial, robotic, and healthcare domains where hallucinations are most dangerous.