Trace Anything: ByteDance Turns Any Video Into a 4D Trajectory Field in One Forward Pass
Most video models still treat a clip as what it looks like: a stack of 2D frames. Trace Anything (ICLR 2026, Apache-2.0) treats it as what it is — stuff moving through 3D space over time — and recovers that structure from ordinary video in a single pass.
The idea: a trajectory field
The paper’s core object is a trajectory field: a dense mapping that assigns every pixel in every frame to a continuous, parametric 3D trajectory. Three spatial dimensions plus time is where the “4D” comes from. Rather than a sparse set of tracked points (classic point tracking) or a frame-by-frame depth map, you get the full 3D path of essentially every visible point in the scene across the whole clip.
Each pixel’s path is encoded as a small set of control points defining a parametric curve, so the representation is compact and smooth rather than a per-frame lookup table.
Why the “single forward pass” part matters
Prior approaches to this kind of 4D reconstruction lean on iterative optimization, auxiliary estimators (separate depth, flow, or pose networks), and a lot of compute per video. Trace Anything predicts the entire trajectory field in one feed-forward pass — a video, an image pair, or even an unstructured set of images goes in; the trajectory field comes out. No optimization loop, no stack of helper models.
That efficiency is the practical unlock. One-pass inference is what turns a research representation into something you could actually run over real footage at scale.
The results, briefly
The team trained on large-scale 4D data (including data from a new platform they built) and report three things:
- State of the art on their new trajectory-field estimation benchmark, and competitive on established point-tracking benchmarks.
- Large efficiency gains from the one-pass paradigm — no iterative optimization, no auxiliary estimators.
- Emergent abilities they did not train for directly: goal-conditioned manipulation (useful for robotics), motion forecasting, and spatio-temporal fusion.
That third point is the interesting one. When a representation is the “right” one for a domain, capabilities fall out of it that you didn’t explicitly supervise. Predicting where every point goes in 3D turns out to be most of what a robot planner or a motion-forecasting model wants anyway.
Where this is useful
- Robotics / manipulation — goal-conditioned control gets a dense, physical, 3D motion prior straight from video.
- Motion capture — markerless 3D motion from ordinary footage, no rig.
- 3D / 4D reconstruction — a continuous trajectory field is a strong backbone for scene reconstruction over time.
- Video understanding and editing — knowing the real 3D motion of every pixel is a far richer substrate than optical flow for analysis, retiming, and effects.
It also rhymes with a theme we keep writing about: the medium wants to be structured, not just rendered. We’ve argued that explainer video becomes controllable when you represent it as code with a deterministic timeline (see our MotionStudio writeup). Trace Anything is the analysis-side version of the same instinct — recover the underlying structure of motion instead of staying at the pixel surface.
The caveats
It’s a research release, Apache-2.0, with official PyTorch inference code and an interactive 3D viewer — not a productized SDK. Expect to do integration work. Trajectory-field quality on your footage will depend on how far it sits from the training distribution (lighting, motion blur, non-rigid and transparent surfaces are the usual hard cases). And “any video∗” carries an asterisk in the paper for good reason — read the limitations before you bet a pipeline on it.
But as a statement of where video representation is heading — from flat frames to dense 4D motion you can recover in one pass — it’s one of the more exciting things out of ByteDance Seed this year.
Paper: arxiv.org/abs/2510.13802 · Code: github.com/ByteDance-Seed/TraceAnything · Project page: trace-anything.github.io