How a Video Model Works: A Practitioner's Read of Veo 3.1 and Seedance 2.0

Five seconds of generated video: a woman in a red jacket with three buttons, walking left to right. I clicked "extend scene." The next five seconds came back. Same woman, same jacket. Two buttons. The sleeve cuff was slightly off too.

The model hadn't continued the clip. It had re-generated one that looked similar.

That moment broke my mental model of what a video model is. I'd been thinking of "extend" as the model picking up where it left off, frame 121 conditioned on frame 120, the way an LLM continues a sentence. What was actually happening was much weirder, and once I understood it, every other behavior of these models started making sense.

I've been generating clips with Veo 3.1 and Seedance 2.0 for several months. Both labs publish polished marketing pages and almost no architecture documentation. Most explainers online are either two years out of date or paper summaries written for ML PhDs. This post is the in-between read I wanted and couldn't find: a practitioner's mental model of what a 2026-era video model is actually doing under the hood.

The current public leaderboard (Artificial Analysis Video Arena¹) is dominated by Chinese labs as of spring 2026: as of 2026-04-19, the top five with-audio entries are Dreamina Seedance 2.0 720p (ByteDance), HappyHorse-1.0 (Alibaba), and two Kling 3.0 variants (Kuaishou), with Google's Veo 3.1 holding the #5 slot. xAI's grok-imagine-video and other entrants trail. The architecture story below applies across all of them. They are variations on a shared template.

In this post:

1. The frame-by-frame fallacy: why thinking of video as N images is the wrong starting point
2. The four building blocks: autoencoder, transformer, conditioning, sampler
3. Spatiotemporal patches: the unit of work: how 3D patches replace 2D pixels
4. The diffusion transformer for video: denoising in latent space
5. Conditioning streams: text, image, audio, reference clips: what each input costs and buys
6. Audio-video joint generation: what's actually new in Veo 3 and Seedance 2
7. Inference: the 41-second 1080p clip: distillation and what's still slow
8. What the marketing pages don't tell you: identity drift, motion priors, the long-horizon problem
9. What's actually next: long-horizon coherence, region-level editing, granular audio control
10. What I'd do differently: treat the prompt as a brief, not a command

A note before we open the hood: neither Google nor ByteDance has published a full technical report for Veo 3.1 or Seedance 2.0. The architecture-level claims below are inferred from product surface area (DeepMind's Veo page², ByteDance's Seedance 2.0 page³), the Seedance 1.0 paper⁴, the broader DiT and latent-diffusion literature, and what's consistent across the field. I'll flag where the evidence is thin.

---

The frame-by-frame fallacy

If you came to video models from image models, the natural mental model is: "An image model generates one frame. A video model generates many frames in sequence."

This is wrong in a way that matters.

A frame-by-frame video model would have to solve a brutal coordination problem. Frame 17 needs to know what was in frame 16 (so the cup is in roughly the same place), but also what's coming in frame 23 (so the camera pan lands smoothly), and what the audio track is doing (so the lip sync isn't off by 80ms). Solve this naively and you get the 2022-era video models: the cup teleports, the face morphs, motion is jittery, every clip looks like a fever dream.

Modern video models sidestep this by never working on a single frame at a time. They work on a block of space-time, typically a few seconds of video, all frames considered jointly. The denoising operation runs across the entire block in parallel. Coordination across time isn't a separate problem; it's the same problem as coordination across space, just on a different axis.

The frame-by-frame view: "generate frame 1, then condition on it to generate frame 2, then..."

The actual model: "generate this 5-second block of latent space-time, all at once, conditioned on the prompt."

That's the whole shift. The rest of this post is the consequences.

---

The four building blocks

Strip away the marketing and a modern video model has four parts:

1. A spatiotemporal autoencoder. Compresses raw RGB pixels (and now audio waveforms) into a much smaller latent representation. Think 100-200× compression depending on the model: the LTX-Video paper⁵ reports a 1:192 ratio. The model never operates on pixels directly because pixels are too expensive and too redundant. This is the latent-diffusion idea from Rombach et al. 2022⁶, extended along the time axis.

2. A diffusion transformer (DiT). The actual generative engine. It takes a latent block full of noise and a text prompt, and it iteratively removes the noise until what's left is a clean latent block that decodes to a coherent video.

3. Conditioning streams. Text (always), reference images, audio, prior video clips, camera trajectories. Each gets encoded and injected into the transformer at every denoising step.

4. A sampler. The control loop that runs the denoising for some number of steps (anywhere from 1 to 50, depending on whether the model is distilled).

Every current video model fits this template: Veo 3.1, Seedance 2.0, HappyHorse, Kling 3.0, Wan, Hunyuan, Runway Gen-4.5. The differences are in the size of each block, the training data, the sampling schedule, and the conditioning vocabulary.

The Seedance 1.0 paper makes this explicit: it lists "an efficient architecture design with proposed training paradigm, which allows for natively supporting multi-shot generation and jointly learning of both text-to-video and image-to-video tasks" as the second of its four core technical contributions (arXiv:2506.09113). Translation: one transformer, multiple input modes, multiple output modes, trained end-to-end.

---

Spatiotemporal patches: the unit of work

The autoencoder's job is to take an RGB video and turn it into a much smaller cube of numbers that the transformer can chew on.

For a single image, this is well-understood: a VAE compresses, say, a 1024×1024 image into a 128×128 latent grid. The transformer treats each cell of that grid as a "patch": a token in transformer parlance.

Video extends this in the obvious way: the latent isn't a 2D grid, it's a 3D cube. Width × height × time. A patch is no longer a square of latent space; it's a small cube of latent space-time. A 5-second clip at 24fps becomes maybe 8 latent frames after temporal compression, and each latent frame is maybe 32×56 in a 1080p configuration. Total tokens: a few thousand to a few tens of thousands, depending on resolution and length.

	Image model	Video model
Latent shape	2D grid (height × width)	3D cube (height × width × time)
Patch	2D square	3D cube of space-time
Tokens for 1080p	~4K	~16-50K depending on length
Compression ratio	~64×	~100-200× including time
Attention cost	Quadratic in token count	Quadratic in token count

The third row is why video models are so much more expensive than image models. A 5-second clip is roughly 4-12× more tokens than a single image, and attention scales quadratically. A video model isn't 24× more expensive than an image model. It's roughly 16-150× more expensive, depending on length and how aggressively the architecture factorizes attention.

This is why every serious video model uses some form of factorized attention: full attention across space within a frame, sparser attention across time, sometimes hybrid windows. Pure full 3D attention doesn't scale.

---

The diffusion transformer for video

The transformer takes the noisy latent cube and the conditioning, and at each denoising step it predicts the noise that should be subtracted. Run this for N steps and you go from pure noise to a clean latent that the autoencoder decodes back to RGB pixels and (in Veo 3 / Seedance 2) audio.

The architecture is a Diffusion Transformer (DiT), the family introduced in Peebles & Xie's 2022 paper⁷ that essentially every frontier video lab has converged on. The original work was for images. The video adaptation keeps the same idea (replace the U-Net backbone with a transformer over latent patches) and changes the attention pattern.

In an image DiT, attention is over a 2D grid: every patch attends to every other patch. In a video DiT, you have three reasonable choices:

1. Full 3D attention. Every spatiotemporal patch attends to every other one. Maximum quality, prohibitive cost beyond a few seconds.
2. Spatial-then-temporal factorization. Each layer alternates: one block does attention within a frame (spatial), the next does attention across time at the same spatial position (temporal). Cheap, but loses cross-cutting interactions.
3. Hybrid windowed attention. Full attention within local space-time windows, sparser global attention across windows. Most production models live here.

The Seedance 1.0 abstract claims "superior spatiotemporal fluidity with structural stability," which is what you'd expect a hybrid attention scheme to optimize for: motion that flows because temporal context is preserved, structure that holds because spatial context is preserved.

The denoising itself is the standard story: train the model to predict the noise that was added to a clean latent at some random timestep, and at inference, walk the timestep schedule backwards from pure noise to a clean signal. The video-specific twist is that "clean" now means "temporally coherent," which is enforced entirely by the training data and the attention pattern, not by any explicit motion prior.

---

Conditioning streams: text, image, audio, reference clips

A video model's job description is "produce a video that satisfies the conditioning." The conditioning is where the product surface lives.

The minimum conditioning is a text prompt, encoded by a language model (typically a T5 variant or a custom encoder), and injected via cross-attention at every transformer block. Standard since Imagen and Stable Diffusion.

Modern video models add far more conditioning streams. The Veo 3.1 page lists most of them by name:

• Reference images ("ingredients"). One or more images that the model should incorporate as scene, character, or object. Each is encoded via the same image encoder used at training time and concatenated to the conditioning sequence.
• Style images. A separate stream that influences the aesthetic without dictating the content.
• Character images. Specifically tagged so the model preserves identity across the generated clip.
• Prior video. For "extend your scene": the last second of a previous clip becomes the first second of the new one, conditioning the rest.
• First/last frame. Two image conditioning streams, with the model required to interpolate plausibly between them.
• Camera trajectory. A sequence of camera-position vectors that the model interprets as "move back, zoom in, pan up."

Seedance 2.0 lists a similar surface: "Supporting images, audios and videos as references, Seedance 2.0 enables creators to transform an idea into visuals with full control over performance, lighting, shadow, and camera movement" (Seedance 2.0 page).

Each conditioning stream is technically cheap: a few extra tokens in the cross-attention. The cost is in the training. Every conditioning mode the model supports has to have been seen during training, with paired examples. This is why the gap between models isn't really architecture anymore. It's data and post-training.

What looks like model capability is usually data capability. The model can do "first frame to last frame interpolation" because someone curated millions of clips where the first and last frames are tagged and the middle is the supervision target. There's no special architecture for it.

---

Audio-video joint generation

This is the genuine 2025-2026 leap. Veo 2 generated silent video. Veo 3 generates video with synchronized audio: footsteps, dialogue, ambient noise, music. Seedance 2.0 makes audio-video joint generation its headline capability.

Pre-2025, audio was bolted on. You generated video with a video model, then ran a separate model (or a human) to dub it. The two pipelines didn't know about each other, and it showed: lip sync was always slightly off, sound effects didn't match camera motion, ambient audio felt generic.

Joint generation means the same model produces both, in the same pass, with shared conditioning. There are two architectural ways to do this and the public evidence doesn't conclusively distinguish between them:

Approach	What it does	What it costs
Single shared transformer8	Audio waveforms are encoded to latents alongside video latents. The DiT denoises both jointly. One transformer, one denoising loop.	Higher per-step compute, simpler training pipeline, tightest A/V sync
Two-tower with cross-attention	Separate audio and video transformers, with cross-attention layers binding them.	Lower compute, more complex training, looser sync

Veo 3 and Seedance 2 both market "native" audio, which suggests the single-transformer approach. But neither has confirmed it publicly. Either way, the result is qualitatively different from dubbing: when a hand slaps a table in the video, the audio model knows because it shared the same denoising trajectory, not because some downstream system tried to align timestamps.

The practitioner consequence: prompting changes. With dubbed video models, audio was an afterthought you fixed in post. With Veo 3 and Seedance 2, audio is part of the prompt, and the model produces wildly different scenes if you mention sound. "A man walking through a forest" is a different generation than "a man walking through a forest, leaves crunching underfoot, distant bird calls."

The first time it landed for me was a public space scene. Two near-identical prompts, run back-to-back.

"A man walking into a bar, jazz playing softly."

Warm low light. Half-empty room. The camera tracked him slowly. A single bartender at the back, polishing a glass.

"A man walking into a bar, thumping electronic music."

Neon palette, packed dance floor, the camera cut faster. The bartender was off-screen entirely. Same subject, same opening clause, different film.

I'd swapped three words at the end of the prompt. The model had restructured the entire scene. Lighting, pacing, crowd, even camera grammar. None of which I'd specified. The audio noun pulled the visual world in tow.

That's the consequence of joint generation. Audio isn't a layer the model adds at the end. It's a conditioning stream that shapes the denoising trajectory from the first step. When you say "jazz playing softly," the model doesn't plan a soundtrack and paint pixels around it. It works back from the audio to figure out what kind of visual world that audio implies. Genres carry composition rules. The model has learned them.

---

Inference: the 41-second 1080p clip

The Seedance 1.0 paper reports generating a 5-second 1080p clip in 41.4 seconds on an NVIDIA L20 (arXiv:2506.09113). The L20 is a data-center inference GPU positioned below the H100/A100 tier. 41.4 seconds for 5 seconds of video is roughly 8× off real-time. The paper reports a ~10× speedup from distillation, so the base model would take on the order of 7 minutes for the same clip.

In Seedance 1.0, that ~10× speedup came largely from distillation alongside system-level optimizations, not architecture changes. The base diffusion model trains with a 50-step denoising schedule (or similar). The distilled model is trained to match the base model's output in 4-8 steps, sometimes fewer.

	Base model	Distilled model
Denoising steps	30-50	1-8
Quality	Best	Near-base
Latency	Long (minutes)	Short (seconds)
What it enables	Research, eval	Product

Production-quality latencies require distillation. Veo 3.1 and Seedance 2.0 almost certainly ship distilled inference: the latencies they market wouldn't be possible otherwise. The base models exist somewhere inside the labs and are probably what's used to generate evaluation reference videos.

What's still slow:

• Long clips. Inference scales roughly linearly with clip length and superlinearly with token count, depending on how aggressively the architecture factorizes attention. 30-second clips at 4K are still firmly in the "research preview, expect to wait" category for both labs.
• Complex multi-shot scenes. "Native multi-shot generation" works, but inference cost compounds with every additional shot.
• High-fidelity audio. Audio decoding adds a separate pass after the joint denoising. The decoder cost is non-trivial at high fidelity.

---

What the marketing pages don't tell you

The capability lists on the Veo and Seedance pages are real. They also paper over the failure modes that anyone using these models daily runs into.

Identity drift. Character consistency works for short clips and breaks subtly over longer ones. Faces shift. Outfits change a button (literally: see the opening). The character image conditioning is a strong prior, not a hard constraint. The longer the clip, the more denoising steps have a chance to drift, and "extend scene" is the worst case because the model is essentially generating a fresh clip with the last frame as a hint.

Motion priors are domain-specific. Walking, running, hand gestures are well-modeled because there's enormous training data. Specialized motions are weaker: a particular dance style, a sport's specific technique, sign language. The model has seen the general shape but not the specific grammar.

The long-horizon problem. Both Veo 3.1 and Seedance 2.0 are best in the 5-10 second range. "Extend your scene" features stitch shorter clips together, which works for shot-to-shot continuity but doesn't solve the fundamental issue: the model has no concept of a story. Extended clips are coherent locally, drift globally.

Prompt sensitivity is high and asymmetric. Small word changes produce wildly different generations, but other small changes do nothing. There's no obvious mapping from "what I said" to "what changed in the output." This is a context engineering problem wearing a video coat.

Audio is the new prompt-engineering frontier. Now that audio is jointly generated, the prompt vocabulary doubles. "A door slamming" affects not just the soundtrack but the visual emphasis, the camera position, the framing. Most prompting guides haven't caught up.

---

What's actually next

The capability list above describes today, not the frontier. Three things text-to-video diffusion models genuinely don't yet do well:

Long-horizon coherence beyond 30 seconds. Current models are sharp at 5-10 seconds and start drifting noticeably past 15. "Extend scene" stitches with overlap, which is why the buttons fall off the jacket (see the opening). Multi-minute clips with persistent characters, consistent physics, and a narrative arc that doesn't collapse are the next real wall. The most credible architecture path I've seen is LongScape⁹, which combines intra-chunk diffusion denoising with inter-chunk autoregressive generation under a Mixture-of-Experts router. It's still a research preview rather than a productised model, but it's the first concrete attempt to compose the two paradigms instead of forcing one to do both jobs. The obvious naive paths (longer context windows, hierarchical generation) all have well-known failure modes.

Editable, composable generation. Today: one prompt produces one clip. Take it or regenerate the whole thing. The frontier is region-level editing: select a part of a generated clip, change one element, regenerate just that part while preserving everything else. The model becomes a substrate (closer to Photoshop's content-aware fill on a video timeline) rather than a one-shot output device. LazyDrag¹⁰ is the most promising 2025 result here: stable drag-based editing on multi-modal DiTs via explicit correspondence maps, training-free. Still a research demo, not a polished tool. Nothing is productised at the level the eventual users will demand.

Granular audio control. Joint audio-video generation now works at the marketing-page level (Veo 3, Seedance 2.0). What it doesn't do yet: separate prompted stems for dialogue, music, and sound effects, each editable independently after generation. The current pattern is "one audio bag, take what you get." Real production workflows need layered, addressable audio. The architectural extension is straightforward (more conditioning streams, more decoder heads) but the training data curation is brutal.

A note on world models. Systems like Genie 3 from DeepMind generate interactive 3D environments rather than fixed clips. They share some plumbing with video models (latent representations, transformer backbones) but produce a fundamentally different output: an explorable simulation, not a finished video. Different product, different evaluation, different use cases. Out of scope here so the architecture story stays clean.

What's not changing: the four-block architecture is stable. The work for the next two years is in long-horizon training objectives, region-level editing, audio decomposition, and inference cost reduction. Not in replacing DiT or rebuilding from scratch. Video diffusion is in its post-Imagen phase. The plumbing is settled. The product surface is the action.

What I'd do differently

I'd been prompting video models the way I prompt LLMs. Specific. Instructional. Minimal. "A man walks left to right. Wide shot. Two seconds." Workmanlike results. Acceptable shots that needed three or four regenerations to get one that didn't feel canned.

The shift was treating the prompt as a director's brief, not a build command. Mood. Reference image. Camera language. Audio. Time of day. Lens implication ("shallow depth of field, golden hour, anamorphic"). The same scene, but specifying every layer of conditioning the model actually has, instead of leaving most of those streams to default.

A six-line brief produces a noticeably better clip than a one-line command asking for the "same" shot. Not because the model is smarter when given more text. Because the model is conditioning on text plus image plus audio plus camera vectors at every denoising step (see the conditioning section above), and a one-line prompt is leaving three of those four streams unspecified. The default for "unspecified" is generic. The default for "specified" is closer to what you actually wanted.

If I were starting over: write the prompt like a brief, not a command. Treat unspecified conditioning as wasted leverage.

---

Related: Context Engineering > Prompt Engineering · Reactivity over Prediction

This is the first of a planned three-part series on the modern multimodal model stack. The companions on vision-language models and image diffusion are coming next.

Footnotes

1. Artificial Analysis, Video Generation Arena (link). Standings cited as of spring 2026; the leaderboard turns over fast and the spread between top models is small. ↩

2. Google DeepMind, Veo product page (link). No architecture paper has been published for Veo 3.1; capability and conditioning surface inferred from this page. ↩

3. ByteDance Seed, Seedance 2.0 product page (link). No architecture paper has been published; capability surface and "native" audio-video framing inferred from this page. ↩

4. ByteDance Seed team, Seedance 1.0: Exploring the Boundaries of Video Generation Models, arXiv:2506.09113 (link). The most detailed publicly available architecture reference for any current frontier video model. Most of the architecture-level claims in this post that go beyond product pages are anchored here. ↩

5. HaCohen et al., LTX-Video: Realtime Video Latent Diffusion, arXiv:2501.00103, December 2024 (link). Reports a 1:192 spatiotemporal compression ratio in the autoencoder, demonstrating how aggressive latent compression unlocks real-time inference on top of a DiT backbone. ↩

6. Rombach et al., High-Resolution Image Synthesis with Latent Diffusion Models, arXiv:2112.10752 (link). Established the latent-diffusion paradigm that every modern video model extends along the time axis. ↩

7. Peebles & Xie, Scalable Diffusion Models with Transformers, arXiv:2212.09748 (link). Introduced the Diffusion Transformer (DiT) architecture that essentially every frontier video lab has converged on. ↩

8. Wang et al., AV-DiT: Efficient Audio-Visual Diffusion Transformer for Joint Audio and Video Generation, arXiv:2406.07686, June 2024 (link). The first published architecture to formalize the single-shared-DiT approach for joint audio-video generation, processing both modalities as a single token sequence. ↩

9. LongScape: Advancing Long-Horizon Embodied World Models with Context-Aware MoE, arXiv:2509.21790, September 2025 (link). Combines intra-chunk diffusion denoising with inter-chunk autoregressive causal generation under a Mixture-of-Experts router; the most credible architectural path so far for multi-minute coherent video. ↩

10. LazyDrag: Enabling Stable Drag-Based Editing on Multi-Modal Diffusion Transformers via Explicit Correspondence, arXiv:2509.12203, September 2025 (link). Replaces fragile implicit attention matching with explicit correspondence maps, enabling stable training-free region-level editing on MM-DiTs. ↩