What Is a World Model?

World models teach physical AI systems how to behave in the real world. 

Earlier physical AI approaches were based on explicitly programming physical rules. This works well inside controlled environments like game engines and early robotics simulators but doesn’t work in the real world, in changing and unexpected scenarios. World models learn real-world dynamics from data using modern architectures and training techniques to simulate the world accurately. 

Building a world model involves these steps:

Data Curation
Data curation prepares large-scale image, video, sensor, and action data for model training. It includes filtering, annotation, classification, deduplication, segmentation, and quality checks to remove low-value or redundant samples. The goal is to create a cleaner, more representative dataset that improves training efficiency and model accuracy.

Data Search
Data search helps developers find specific scenarios within massive multimodal datasets. Embedding models enable semantic search across video, images, actions, and events, making it easier to retrieve rare or task-specific examples. This can significantly accelerate data selection for training, post-training, and evaluation.

Data Evaluation
Data evaluation determines whether a dataset is suitable for training a world model. It measures factors such as quality, diversity, temporal consistency, action-effect relationships, domain coverage, safety, and relevance. These assessments help identify which data should be retained, repaired, augmented, or replaced.

Pre-training teaches a world model general representations of how environments look, behave, and change over time. Depending on the architecture, training data can include large-scale video, images, text, 3D data, sensor observations, and sometimes actions or trajectories. The resulting model understands spatial structure, motion, object interactions, and patterns useful for prediction, generation, reasoning, or planning. World models are typically multimodal or omnimodal, meaning they work across multiple data types. 

Pre-training typically involves four key choices:

Training Data and Data Mixture
Developers select and balance the modalities, domains, and scenarios used for training. Some world models learn primarily from large-scale unlabeled video, while multimodal models may jointly train on images, text, video, 3D, sensor, or action data. The appropriate mixture depends on whether the model is intended for visual understanding, generation, simulation, or embodied control.

Representation and Tokenization
Raw inputs must be transformed into representations the model can efficiently process. Generative models may compress images and video into discrete or continuous latent representations, while predictive architectures learn directly in embedding space.

Training Objective
The objective determines what the model learns to predict. Approaches include predicting future observations or latent representations, reconstructing masked information, generating future video, or predicting outcomes conditioned on actions. Action conditioning is particularly important when the model must simulate how a physical AI system's behavior changes its environment.

Architecture Design
The architecture determines how the model represents and predicts the world. 

• Autoregressive Models
  Autoregressive models predict future information conditioned on previous states. This makes them well suited to sequential prediction and interactive environments, where each new state depends on prior observations and actions.
• Diffusion Models
  Diffusion models learn to generate high-dimensional outputs, such as images or video, through iterative denoising. They can produce high-quality visual simulations and may also be combined with autoregressive temporal prediction; the two approaches are not mutually exclusive.
• Joint Embedding Predictive Architectures
  Joint embedding predictive architectures, such as JEPA, predict representations of future or missing observations rather than reconstructing every pixel. This allows models to focus on higher-level structure and dynamics while avoiding the need to model every visual detail.
• Mixture of Experts
  Mixture-of-Experts architectures route different inputs through specialized subsets of a larger network. They can increase model capacity and specialization while controlling the amount of computation used for each input. MoE is an architectural scaling technique rather than a defining characteristic of world models.
• Hybrid and Multimodal Architectures
  World models increasingly combine techniques rather than fitting neatly into one category. For example, a model may use autoregressive temporal prediction with diffusion-based generation or combine specialized components for understanding and generation, as in Mixture-of-Transformers (MoT) architecture.
• 3D Foundation Models and Spatial Representations
  3D foundation models and representations such as neural fields or Gaussian splats are part of the broader spatial AI ecosystem. They are not world models, but they can provide explicit geometry and persistent scene representations that complement world models for reconstruction and simulation in physical AI.

Mid-training continues training a pretrained model on targeted datasets before task-specific post-training. It is commonly used to improve capabilities, introduce new modalities, expand domain knowledge, or strengthen areas such as physical reasoning and action understanding. Mid-training helps bridge general pre-training and downstream specialization.

Post-training adapts a world model for a specific domain, capability, or downstream task using additional data and feedback. Developers can specialize models for applications such as robotics, autonomous vehicles, simulation, visual reasoning, or action generation. Starting from a pretrained model typically requires far less data and compute than training from scratch.

Self-Supervised Fine-Tuning
Self-supervised fine-tuning adapts a pretrained model using unlabeled domain-specific data. The model learns patterns and representations directly from the data without requiring manual labels. It is especially useful for domain adaptation, such as specializing a general model using warehouse or driving videos.

Supervised Fine-Tuning (SFT)
Supervised fine-tuning trains a model on labeled examples that demonstrate the desired input-output behavior. It can teach a world model to predict future states, understand scenes, generate actions, or follow task-specific instructions. SFT is commonly used when developers have high-quality labeled or demonstration data.

Reinforcement Learning (RL)
Reinforcement learning improves model behavior using reward signals based on outcomes or task performance. Instead of learning only from static examples, the model learns which predictions or actions lead to better results. This is useful for improving planning, decision-making, and interactive behavior in robotics and autonomous systems.