Resource2Skill: Turning YouTube Tutorials Into Agent Skills (No Novel Models Required)

By Prahlad Menon Published 2026-08-13 5 min read

We’ve been covering video understanding models lately — VST (continuous reasoning during playback), CoPE-VideoLM (codec primitives for efficiency), Video-Use (agentic video editing).

Microsoft’s Resource2Skill sounds like it fits that category: “AI agents that watch tutorials and execute them.”

But here’s what most coverage misses: there’s no novel video understanding here. The “watching” is Gemini looking at screenshots. The innovation is the pipeline that structures those screenshots into executable skills.

Let me be precise about what this is and isn’t.

What Resource2Skill Actually Does

It’s a three-stage framework:

Stage 1: Distillation (offline, uses Gemini)

Take a YouTube tutorial (e.g., “How to make a glassmorphism card in CSS”)
Extract keyframes at moments where the screen changes
Send frames + transcript to Gemini (gemini-2.5-flash by default)
Gemini outputs structured markdown: skill name, steps, code snippets
Store in a “Skill Wiki” with text, code, and visual components

Stage 2: Retrieval (runtime, uses BM25 + GPT)

Agent receives a task (“Build a landing page for a coffee shop”)
BM25 search over skill wiki finds relevant entries
GPT-5.4/5.5 reads candidates and picks which skills to compose

Stage 3: Execution (runtime, uses MCP)

Selected skill code runs against real software backends
PowerPoint via python-pptx, Blender via bpy, web via Playwright, etc.
Agent iterates until task complete

The Models (No Magic Here)

For distillation:

Gemini 2.5 Flash — Google’s model, via API
Receives: keyframes (images) + transcript (text)
Outputs: structured skill markdown

For agent execution:

GPT-5.5 (best performance)
GPT-5.4 (main evaluation model)
GPT-5.4 Mini / Nano (smaller variants)

These are standard API models. You can use them today. The paper doesn’t introduce new architectures — it orchestrates existing models into a skill-building pipeline.

The “Video Understanding” Is Just Screenshots

Let’s be clear about what “watching a tutorial” means here:

YouTube video
    ↓
Extract keyframes (frames where pixels change significantly)
    ↓
Send frames + transcript to Gemini
    ↓
Gemini writes structured markdown describing the procedure
    ↓
Store in wiki

Gemini isn’t “watching” video. It’s looking at a handful of screenshots and reading a transcript. The same thing you’d get from a blog post with embedded images.

The paper’s ablation confirms video matters — removing video drops performance from 68.9% to 59.4%. But that’s because screenshots capture visual state (tool palettes, canvas layouts, color choices) that text descriptions miss. It’s not because there’s sophisticated temporal reasoning happening.

What Actually Works (I Ran It)

I cloned the repo and ran their agent on a simple task:

python cli.py agent \
  --domain web \
  --task "Build a landing page for a coffee shop called Bean There" \
  --model gpt-5.4 --reasoning medium --max-iter 20

Result: 12 iterations, 15 tool calls, success.

The agent:

Searched the pre-built skill wiki for “coffee shop landing hero menu”
Found relevant design patterns (distilled from web design tutorials)
Inspected visual references from those skills
Generated a full HTML/CSS page with hero, menu cards, pricing, testimonials, footer
Rendered a screenshot

The output was genuinely good — dark coffeehouse aesthetic, gradient cards, responsive CSS.

But I didn’t distill anything. I used Microsoft’s pre-built skill library that ships with the repo. The distillation step requires a Gemini API key, which I didn’t have configured.

The Real Results

Across seven domains with 80 tasks each:

Domain	Without Skills	With Skills	Δ
Web	68.7%	82.4%	+13.7
Excel	58.6%	76.4%	+17.8
PowerPoint	55.4%	64.8%	+9.4
Blender	29.5%	44.1%	+14.6
CAD	48.7%	55.7%	+7.0
UE5	29.1%	67.3%	+38.2
REAPER	73.2%	77.3%	+4.1
Average	51.9%	66.9%	+15.0

The UE5 result is striking: +38.2 percentage points. Without skills, GPT-5.4 rarely assembles a working Unreal Engine scene through the Python API. With distilled skills from tutorials, it consistently produces scorable output.

Why This Matters (Pipeline > Models)

The contribution isn’t “we built a video understanding model.” It’s:

Structured skill format — Each skill has text (what it does), code (how to execute it), and visuals (what it looks like). The agent can inspect any view.
Hierarchical organization — Skills are taxonomized by domain (PowerPoint: Layout → Typography → Motion). BM25 search uses taxonomy paths, not just content.
Unified offline/online pipeline — The same distillation code that builds the library offline can run online when the agent encounters a capability gap.
MCP execution — Skills run through Model Context Protocol against real software backends, not simulated environments.

The models are commodity. The pipeline is the product.

The Honest Limitation

To distill your own skills, you need:

Gemini API access (for video analysis)
Azure OpenAI or OpenAI access (for agent execution)
Domain-specific backends (LibreOffice for PPT, Blender for 3D, etc.)

The repo ships pre-built skill libraries on HuggingFace, so you can run the agent side without Gemini. But the “watch a tutorial, learn a skill” loop requires the full stack.

Try It

git clone https://github.com/microsoft/resource2skill
cd resource2skill
python3.11 -m venv .venv && source .venv/bin/activate
pip install -r requirements.txt
pip install "mcp>=1.26,<2.0"  # version matters

# Install Playwright for web domain
python -m playwright install chromium

# Validate
python cli.py validate-domain --domain web

# Run agent (uses pre-built skill library)
python cli.py agent \
  --domain web \
  --task "Build a landing page for a nonprofit called Quartz" \
  --model gpt-5.4 --reasoning medium --max-iter 20

Note: The repo defaults to --reasoning low but Azure OpenAI’s GPT-5 deployment may only support medium. Adjust accordingly.

Bottom Line

Resource2Skill is a skill extraction and execution framework, not a video understanding model. The “video understanding” is Gemini looking at keyframes — useful, but not novel.

The actual contribution is showing that:

Tutorial content can be systematically converted into agent-executable skills
Structured skill libraries outperform “just prompt the model” by +15 points average
The same pipeline works across wildly different domains (web, Excel, Blender, audio)

Every YouTube tutorial is potential skill library material. That’s the insight. The models are just plumbing.

Links:

Related Posts: