Mixamo LLM Mocap: Turn Any Video Into 3D Character Animation With AI

What if you could film yourself doing a martial arts combo on your phone and have it automatically animated on a 3D character in Blender? Mixamo LLM Mocap does exactly that — no motion capture suit, no manual keyframing, and the entire pipeline is scriptable enough that an AI agent can run it end-to-end.

Repo: github.com/squall01337/mixamo-llm-mocap

What It Does

The pipeline takes a video (filmed or AI-generated) and outputs a clean FK animation on any Mixamo character:

1. Single performer — kicks, punches, dance moves, any motion
2. Two performers fighting — both tracks extracted from the same video, retargeted onto different characters, positioned at the actual distance they stood

The output is honest Mixamo FK: hips translate, everything else is quaternions at 30fps. Clips drop into any Mixamo workflow without cleanup.

The 10-Stage Pipeline

video plate (locked camera, T-pose bookends)
│
├─ 1. estimate_pose_gvhmr.py    → SMPL-X mesh recovery → 33 landmarks + pelvis height
├─ 2. analyze_landmarks.py      → numeric beat detection
├─ 3. action_specs/<name>.json  → motion as data (support schedule, fists, rest blends)
├─ 4. lift_to_mixamo.py         → direction-preserving retarget to YOUR rig
├─ 5. apply_mixamo_fk.py        → FK aim + foot planting (via Blender MCP)
├─ 6. qa_clip.py                → automated gate: no explosions, pops, foot skate
├─ 7. compare_reference.py      → frame-by-frame vs video → which frames differ
├─ 8. compare_pair.py           → two-character: separation, reach, intrusion
├─ 9. run_in_blender.py         → mesh-vs-mesh collision check
└─ 10. render_preview.py        → preview + side-by-side showcase

For two performers, stages 1-7 run once per fighter (--person left|right), then the system builds one scene with both characters and verifies they interact correctly.

Why This Is Different

Motions Are Data, Not Code

A new motion is a small JSON spec, not a script rewrite:

{
  "name": "spinning_kick",
  "beats": [
    {"frame": 0, "support": "left", "fist_right": false},
    {"frame": 15, "support": "none", "airborne": true},
    {"frame": 30, "support": "right", "fist_right": true}
  ]
}

The action_specs/ folder has worked examples: kung-fu forms, combos with jumps, two-fighter duels.

Real Ground Contact

Planted feet solve to ground height with zero skate — the support ankle pins through each stance. Jumps use the estimator’s actual pelvis arc.

QA Gate, Not Vibes

Problems are caught numerically before a human looks:

• Exploded bones
• Hip pops
• Foot skate
• Drifting roots
• Broken rest poses

Closed Refinement Loop

compare_reference.py measures the retarget against the source video frame-by-frame on what your eye actually reads:

• Hand height relative to face
• Distance between hands
• Limbs inside torso
• Gaze direction

Reports exact frame windows that diverge. “His hands are too high” becomes numbers, and over-corrections get caught before shipping.

Two Characters, One Scene

Two-performer plates split by screen side (robust when tracker IDs swap on contact). Each retargets onto a different Mixamo character with their own proportions, placed at the actual distance from the footage.

compare_pair.py verifies:

• Separation between characters
• Strike reach accuracy
• Limb intrusion

A Blender BVH pass checks actual skinned meshes for collision — because Mixamo characters are thicker than humans, and choreography with 2cm near-misses will collide when retargeted.

Requirements

Component	Source	Notes
Mixamo character	mixamo.com	Download FBX Binary, T-pose
Blender 5.1+	blender.org
Blender MCP add-on	blender.org/lab/mcp-server	Enable “Allow Online Access”
GVHMR (pose estimator)	github.com/zju3dv/GVHMR	Clone into tools/GVHMR
GVHMR checkpoints	HuggingFace	~5GB
SMPL-X body model	smpl-x.is.tue.mpg.de	Free research registration
GPU	~8GB VRAM	Developed on RTX 4080

Quick Start

1. Build Your Rig Scene

blender --background --python pipeline/setup_rig.py -- --fbx ybot.fbx --out ybot_rest.blend

This measures your character into rig_profile.json (rest pose, bone lengths, hip/ground heights).

2. Run the Pipeline

# Extract landmarks from video
python pipeline/estimate_pose_gvhmr.py --video plates/combo.mp4 --out plates/landmarks.json

# Analyze beats
python pipeline/analyze_landmarks.py --landmarks plates/landmarks.json

# Create action spec (schema in docs/PIPELINE.md)
# → action_specs/combo.json

# Retarget to Mixamo rig
python pipeline/lift_to_mixamo.py --spec action_specs/combo.json

# Apply in Blender
python pipeline/run_in_blender.py all action_specs/combo.json

# QA check
python pipeline/qa_clip.py --spec action_specs/combo.json

# Compare to reference video
python pipeline/compare_reference.py --spec action_specs/combo.json

# Render preview
python pipeline/render_preview.py action_specs/combo.json --showcase

3. For Two Performers

Add --person left|right to the estimate step, create one spec per fighter, then run compare_pair.py.

AI Agent Compatibility

This is explicitly designed for AI agents to operate:

• Beat decisions come from numbers (analyze_landmarks.py), never from eyeballing frames
• Every stage is CLI or socket call — fully scriptable
• Pitfalls documented — docs/PITFALLS.md encodes every mistake so the next operator doesn’t repeat them

An AI agent can:

1. Generate a video prompt
2. Run the full pipeline
3. Read the QA output
4. Iterate on the action spec
5. Ship the final animation

No human needs to touch Blender.

Video Plate Requirements

For best results, your source video needs:

• Locked camera — no panning/zooming
• T-pose bookends — start and end in T-pose for calibration
• Clean background — helps pose estimation
• Good lighting — shadows confuse the estimator

See docs/PROMPTING.md for AI video generation prompts that survive retargeting.

Documentation

Doc	Coverage
INSTALL.md	Every dependency, Windows-proven
PIPELINE.md	Full workflow + action_spec schema
RIG.md	Mixamo rig conventions
PITFALLS.md	Every mistake already made
PROMPTING.md	Writing video prompts that retarget well

Use Cases

Game Development:

• Prototype character animations from reference videos
• Convert martial arts tutorials into game-ready clips
• Create NPC fight choreography

Film/VFX:

• Previs from actor reference
• Retarget stunt doubles onto digital characters
• Quick animation tests before full mocap sessions

AI Content Creation:

• Generate video with AI → automatically animate 3D character
• Fully autonomous animation pipeline
• Batch processing of reference footage

Limitations

• Locked camera only (no tracking shots yet)
• Best with full-body visible footage
• Complex cloth/props may confuse pose estimation
• ~8GB VRAM requirement

Why This Matters

Traditional mocap requires expensive suits, studio time, and cleanup work. Markerless mocap exists but usually needs specific camera setups.

This pipeline:

• Works with any video (including AI-generated)
• Outputs production-ready Mixamo FK
• Has built-in QA so you know when something’s wrong
• Runs entirely via CLI — perfect for AI agent integration

The combination of GVHMR for pose estimation, Blender MCP for the 3D work, and a QA-gated pipeline means you can go from “I want that move” to “here’s the animation file” without touching the software yourself.

---

Links:

• Repo: github.com/squall01337/mixamo-llm-mocap
• GVHMR (pose estimator): github.com/zju3dv/GVHMR
• Mixamo characters: mixamo.com
• Blender MCP: blender.org/lab/mcp-server
• SMPL-X model: smpl-x.is.tue.mpg.de