Collaborative Gym · Stanford SALT Lab

CollabSkill ranking

A Bayesian rating framework inspired by how multiplayer games rank their players. Each completed evaluation episode - one agent, one human worker, one task, one standardized score - updates our beliefs about both team members.

Team out come is modeled as additive (agent skill + human skill). For each entry we track the posterior mean μ and uncertainty σ, and rank by the conservative score μ − 3σ so entries with few episodes don't over-rank. The faded bar segment shows the 3σ uncertainty margin between the ranked score and μ.

Claude Cowork

claude-sonnet-4.6

76.74μ 77.21 · σ 0.16

Claude Code

1.0

74.78μ 75.25 · σ 0.16

Codex

gpt-5.2-codex

71.27μ 71.73 · σ 0.16

Manus

manus-1.6-lite

71.20μ 71.65 · σ 0.15

Gemini CLI

gemini-3-auto

69.55μ 70.01 · σ 0.15

Integrating Human-Agent Collaboration Skill Measure Into Co-Gym (paper)

Evaluating Human-Agent Collaboration with Real Workers at Scale

In Collaborative Gym, we validated that a dual-control, non-turn-taking paradigm benefits human-agent collaboration across three example tasks: travel planning, writing related work sections, and analyzing tabular data. Over the past year, many commercial agents (e.g., Claude Code) have adopted this paradigm — and people are increasingly applying them to economically valuable, real-world work.

We introduce the CollabSkill Framework to extend Co-Gym v1 along two dimensions: the tasks on which human-agent teams are evaluated, and the ability to disentangle skill contributions from both humans and AI agents.

More realistic tasks

Collecting human-agent collaboration data from real-world tasks

Infrastructure for Collecting Human-Agent Collaboration Data with Real Workers

CollabSkill provides infrastructure for collecting human-agent collaboration data at scale. A unified task schema links prompts, reference files, and deliverables to O*NET occupational categories — supporting heterogeneous tasks across sectors and enabling workers to be matched to tasks aligned with their occupational background, reducing variability in task familiarity.

We involve real human workers rather than LLM-based simulators, which tend to be excessively cooperative and offer no insight into how real workers engage with or perceive AI agents.

Deeper contribution analysis

Estimating CollabSkill for human and agent via additive latent-skill decomposition

A Bayesian Framework for Disentangling Skill Contributions from Humans and AI Agents

A core challenge in human-agent evaluation is the wide variability in AI literacy across human workers. Rather than treating humans as noise, CollabSkill assigns latent skill values to both the human and the agent, modeling outcomes as an additive decomposition of agent skill, human skill, and observation noise.

Initializing each entity's latent skill with a Gaussian prior yields a closed-form posterior via the Kalman filter measurement update, producing a CollabSkill score for each side.