ART: Reinforcement Learning for Multi-Step Agents

By Prahlad Menon 4 min read

Imagine your AI agent actually improving over time instead of hallucinating the same errors forever.

That’s the pitch behind ART (Agent Reinforcement Trainer) from OpenPipe — an open-source framework for training multi-step agents with reinforcement learning. They claim a Qwen model trained with ART outperforms o3 on email tasks. Let’s dig in.

What ART Actually Is

ART is a GRPO (Group Relative Policy Optimization) framework designed specifically for agentic workflows — not single-turn chat, but multi-step tool-calling agents that need to learn from entire trajectories.

The architecture splits into two parts:

Client (runs anywhere, including your laptop):

  • Executes your agent logic
  • Collects trajectories (message histories)
  • Assigns rewards based on task success
  • Sends training data to the backend

Backend (requires GPU):

  • Runs vLLM for inference
  • Trains LoRA adapters with GRPO
  • Hot-swaps updated weights mid-run
Your Agent Code → ART Client → Trajectories + Rewards
                                      ↓
                              ART Backend (GPU)
                                      ↓
                              Trained LoRA → Better Agent

The Training Loop

  1. Inference: Your agent runs multiple rollouts in parallel, making decisions and calling tools
  2. Trajectory Collection: Each rollout’s full message history is captured
  3. Reward Assignment: Your code scores each trajectory (did it succeed? how well?)
  4. GRPO Training: The backend uses reward differences between rollouts to update weights
  5. Checkpoint: New LoRA is saved and loaded into vLLM
  6. Repeat: Agent now performs slightly better, generating new training data

This is RL as it should work for agents — learning from experience, not just memorizing examples.

The o3 Claim

OpenPipe’s headline result: a Qwen 2.5 14B model trained with ART beats o3 on email retrieval tasks. They call it ART•E (Agent Reinforcement Training for Email).

The task: search through an inbox to find specific emails based on natural language queries.

Their benchmark shows:

  • Base Qwen 2.5 14B: ~40% accuracy
  • After ART training: ~85% accuracy
  • o3: ~75% accuracy

The catch? This is on their specific email retrieval benchmark. Your mileage will vary on different tasks. But the principle holds — RL can push smaller models past larger ones on specialized tasks.

Two Ways to Run It

Use W&B Training as the backend. Your Mac runs the client, W&B handles GPU infrastructure.

pip install openpipe-art
from art import TrainableModel
from art.serverless.backend import ServerlessBackend

backend = ServerlessBackend(api_key="your_wandb_api_key")

model = TrainableModel(
    project="my-agent",
    name="agent-001",
    base_model="OpenPipe/Qwen3-14B-Instruct"
)

await model.register(backend)

Cost: W&B has a free tier. Full training runs typically cost $15-200 in GPU time.

Option 2: Local Backend (Requires NVIDIA GPU)

If you have a Linux machine with CUDA:

pip install openpipe-art[backend]
from art.local.backend import LocalBackend

backend = LocalBackend(path="./.art")

model = TrainableModel(
    project="my-agent",
    name="agent-001",
    base_model="Qwen/Qwen2.5-7B-Instruct"
)

await model.register(backend)

Note: LocalBackend uses vLLM + Unsloth, which require CUDA. Apple Silicon (M1/M2/M3) won’t work for local training — use ServerlessBackend instead.

Hands-On: Train an Agent to Play 2048

The fastest way to test ART is their 2048 notebook. Here’s what you’ll need:

Prerequisites

  1. W&B Account (free): wandb.ai
  2. W&B API Key: Settings → API Keys
  3. Google Colab (free tier works)

Steps

  1. Open the 2048 notebook

  2. Set your W&B API key in the environment variables cell

  3. Click “Run all”

  4. Watch metrics in your W&B workspace

The model (Qwen 3.6 27B) learns to play 2048 over ~2 hours. You’ll see the reward curve climb as it figures out tile-merging strategies.

Testing on Your Own Agent

Here’s a minimal template for training your own agent:

import art
from art import TrainableModel, gather_trajectory_groups
from art.serverless.backend import ServerlessBackend

# 1. Define your agent task
async def run_agent(model, task_input):
    """Your agent logic here — tool calls, multi-step reasoning, etc."""
    messages = [{"role": "user", "content": task_input}]
    
    # Agent loop
    while not done:
        response = await model.chat.completions.create(
            messages=messages,
            tools=your_tools
        )
        # Process response, call tools, update messages
        # ...
    
    return result

# 2. Define reward function
def compute_reward(result, expected):
    """Score the trajectory — 1.0 for success, 0.0 for failure, or gradients."""
    if result == expected:
        return 1.0
    elif partial_match(result, expected):
        return 0.5
    return 0.0

# 3. Training loop
async def train():
    backend = ServerlessBackend(api_key="...")
    
    model = TrainableModel(
        project="my-task",
        name="agent-v1",
        base_model="OpenPipe/Qwen3-14B-Instruct"
    )
    await model.register(backend)
    
    for iteration in range(100):
        trajectories = []
        
        # Run multiple rollouts
        for task in training_tasks:
            result = await run_agent(model, task.input)
            reward = compute_reward(result, task.expected)
            
            trajectory = model.get_trajectory()
            trajectory.set_reward(reward)
            trajectories.append(trajectory)
        
        # Train on this batch
        groups = gather_trajectory_groups(trajectories)
        await model.train(groups)

What You Need for Good Results

From the FAQ, ART works best when:

  1. Base model succeeds at least 30% of the time — RL can’t teach tasks that are completely out of distribution

  2. Rewards are quantifiable — you need a clear signal (success/failure, score, LLM-as-judge)

  3. Tasks are repeatable — GRPO runs many parallel rollouts, so your environment can’t have side effects

Supported Models

Serverless (W&B Training):

  • OpenPipe/Qwen3-14B-Instruct (recommended for beginners)
  • Qwen/Qwen3-30B-A3B

Local Backend:

  • Qwen 2.5 family (7B, 14B, 32B, 72B)
  • Llama 3.1/3.2/3.3 family
  • Most vLLM-compatible models with LoRA support

Gemma 3 is not currently supported.

Integrations

ART plays well with:

  • LangGraph: Native integration for training LangGraph agents
  • MCP: MCP•RL teaches models to master MCP servers
  • W&B Weave: Automatic trace logging
  • Langfuse: Observability integration

The Bigger Picture

ART represents a shift in how we think about agent development:

Old way: Prompt engineer → Deploy → Watch it fail → Prompt engineer more → Repeat

ART way: Define task + reward → Train on rollouts → Deploy improved model → Collect production failures → Retrain

The trained model is smaller, faster, and specialized. A 14B parameter model beating o3 isn’t magic — it’s the difference between a generalist and a specialist.

Try It

  1. Quickest: Run the 2048 notebook in Colab
  2. Real task: Try the ART•E email agent
  3. Your agent: Install openpipe-art and adapt the template above

Links: