Microsoft Skill Recorder: Teach AI Agents by Doing Tasks Once

By Prahlad Menon 6 min read

Microsoft just released Skill Recorder, an open-source desktop app that watches you do a task once and turns it into a skill that AI agents can repeat forever.

This is programming by demonstration for the agent era: instead of writing SKILL.md files by hand or explaining workflows to an LLM, you just… do the thing. The agent figures out what you did and how to generalize it.

GitHub: microsoft/skill-recorder

How It Works

  1. Record — Hit ⌘⇧R (Mac) or Ctrl+Shift+R (Windows) and do your task normally. Skill Recorder captures your screen, active windows, browser URLs, and optionally your voice narration.

  2. Analyze — Click Analyze and GitHub Copilot reconstructs what you did as a clear intent plus an ordered list of steps. Review and edit until it reads right.

  3. Create — Generate a reusable Skill (agent runs on demand) or Automation (runs on a schedule or trigger).

The key insight: recording yourself submitting one expense report can teach the agent to submit all expense reports. It generalizes from your single example.

What Gets Captured

Everything stays local until you explicitly hit Analyze:

  • Window tracking — Which apps/windows you switched between
  • Browser URLs — Pages you visited (macOS)
  • Screen video — Low-rate snapshots when the screen changes
  • Clipboard — Short previews of copied text (to tie steps together)
  • Narration (optional) — Voice transcribed on-device via Whisper (99 languages supported)

When you Analyze, this data goes to GitHub Copilot’s cloud for processing.

⚠️ Don’t record secrets. Passwords, API keys, tokens—keep them out of recordings.

The Output

Skill Recorder produces two things:

Skills — A SKILL.md procedure file that agents can execute on demand. The agent prefers native tools (like gh CLI or web_fetch) over replaying UI clicks.

Automations — The same procedure, but triggered by schedule or event. Set it and forget it.

Why This Matters

The friction in agent adoption isn’t capability—it’s teaching. Writing detailed SKILL.md files is tedious. Prompting an agent to figure out your workflow is hit-or-miss.

Skill Recorder inverts the problem: instead of describing what you want, you demonstrate it. The agent watches, learns, and generalizes.

This is especially powerful for:

  • Repetitive workflows you do weekly/monthly
  • Multi-app tasks that touch browser + desktop apps
  • Processes too complex to explain but easy to demonstrate
  • Non-technical users who can’t write agent instructions

Installation

macOS and Windows 11 (x64 and ARM64) are supported. It’s a source build—nothing installed globally:

# macOS / Ubuntu (get commit from latest release page)
commit="<release-commit>"; curl -fsSL "https://raw.githubusercontent.com/microsoft/skill-recorder/$commit/install.sh" | SKILL_RECORDER_COMMIT="$commit" bash
# Windows (PowerShell)
$commit="<release-commit>"; $env:SKILL_RECORDER_COMMIT=$commit; irm "https://raw.githubusercontent.com/microsoft/skill-recorder/$commit/install.ps1" | iex

First launch on macOS will prompt for Screen Recording permission.

How Skills Get Executed

Once you’ve created a skill, how does an agent actually run it? The SKILL.md file goes into a skills folder that the target agent monitors:

Microsoft Scout: Place skills in ~/.copilot/skills/ on your machine. Scout discovers them automatically and can run them on demand when you ask.

Copilot Cowork: Drop SKILL.md files into your OneDrive at /Documents/Cowork/skills/. Cowork discovers custom skills at the start of each session—you can have up to 50. When you ask Cowork to do something that matches a skill, it executes the steps using your permissions.

Copilot Studio: Import skills into your Studio workspace for enterprise automation flows.

The execution model is agentic, not macro-based. When Cowork runs a skill, it:

  • Uses native Microsoft 365 tools (send email, create document, post to Teams)
  • Falls back to local browser automation when needed (via Edge on your device)
  • Asks for approval before sensitive actions (sending emails, posting messages)
  • Inherits your permissions—it can only access what you can access

Automations add triggers: run daily at 9 AM, run when a matching email arrives, run when you’re @mentioned in Teams.

GitHub: microsoft/skill-recorder
License: MIT
Requires: GitHub account with Copilot access

Frequently Asked Questions

Is Skill Recorder free?

Yes. Skill Recorder is open source under the MIT license. However, the Analyze step requires GitHub Copilot access, which is a paid GitHub feature (free for students, educators, and open-source maintainers). The recording and local processing are completely free.

Does Skill Recorder upload my screen recordings?

Not while recording. All capture happens locally. Only when you click “Analyze” does Skill Recorder send data to GitHub Copilot’s cloud. This includes window titles, URLs, screen snapshots, clipboard previews, and narration text. Don’t record passwords, tokens, or other secrets.

What platforms does Skill Recorder support?

macOS is the primary platform. Windows 11 (x64 and ARM64) is also supported. Linux (Ubuntu) has partial support. Browser URL capture currently works on macOS only.

How does the narration feature work?

Narration is optional. If enabled, Skill Recorder transcribes your voice on-device using OpenAI’s Whisper model (252 MB download on first use). It supports 99 languages. The transcription adds context to help Copilot understand what you’re doing.

Can the generated skills work with any AI agent?

Skills are output as SKILL.md files, which is a common format. The primary integration targets are Microsoft Scout, Copilot Cowork, and Copilot Studio. Other agents that understand SKILL.md procedures (like OpenClaw or Claude with MCP) could potentially use them with adaptation.

How do I run a skill after creating it?

Place the SKILL.md file in the appropriate skills folder: ~/.copilot/skills/ for Microsoft Scout, or /Documents/Cowork/skills/ in OneDrive for Copilot Cowork. The agent discovers it automatically. Then just ask the agent to do the task—it matches your request to the skill and executes the steps using native tools, asking for approval before sensitive actions like sending emails.

Does the agent replay my exact clicks?

No. Skill Recorder prefers native tools over UI replay. If you recorded yourself using the GitHub web UI, the resulting skill might use the gh CLI instead. The goal is robust automation, not macro recording.

Can non-technical users create skills?

Yes, that’s the point. You don’t need to write code or understand agent architectures. Just do your task while recording, click Analyze, review the generated steps, and create the skill. The voice narration feature helps add context without typing.

How does generalization work?

When you record yourself filling out one expense report, Copilot infers the pattern and creates a skill that can fill out any expense report. It identifies which parts are variable (the expense details) and which are fixed (the form location, submit button). You can review and adjust the generalization before creating the skill.