Oqoqo

What is Oqoqo?

Oqoqo is a platform for building evals and private benchmarks around real agent work. You define tasks and rubrics, pick agents like Claude Code, Codex, or Cursor, and run each trial in its own cloud sandbox. Results show pass rates, token use, and full trajectories, so you see where agents get stuck.

Top Features:

• Custom benchmarks: write tasks and rubrics in plain language, versioned per experiment.
• Treatment comparisons: test skills, MCP servers, CLIs, and SDKs against raw agents.
• Trajectory view: inspect every tool call, command, error, and stopping point.

Use Cases:

• Agent readiness: check whether coding agents can actually use your API.
• Model selection: run identical tasks across agents and models to compare.
• Regression checks: trigger experiments from CI whenever a change could break agents.

Who Can Use Oqoqo?

• Devtool companies: see how agents handle their SDKs, docs, and MCP servers.
• AI engineers: pick agent and model combinations based on real evidence.
• Product managers: spot interface friction that slows agents and prioritize fixes.

Pricing

• Free ($0): 100 runs monthly with unlimited members, projects, and experiments.
• Pro ($20 per month): 300 runs monthly at roughly seven cents per run.
• Ultra ($60 per month): 1,000 runs monthly at six cents per run.

Pros and Cons

Pros:

• Wide agent support: works with eleven agents, including Claude Code and Codex.
• Isolated runs: each trial gets its own sandboxed machine for reproducibility.
• Flexible access: launch experiments from the web app, CLI, or MCP server.

Cons:

• Run caps: unused monthly plan runs expire instead of carrying over.
• Niche audience: mainly useful for teams building products that agents use.
• Setup effort: good rubrics and task sets take time to design well.

FAQs:

1) What can I evaluate with it?

Skills, MCP servers, CLIs, SDKs, APIs, docs, and the workflows built on them.

2) Which agents does it support?

Claude Code, Codex, Cursor, GitHub Copilot, OpenCode, Qwen Code, Hermes, and several others.

3) Is there a free plan?

Yes, the free plan includes 100 runs monthly with no credit card.

4) Can I bring my own models?

Yes, connect your existing model providers and choose a model per agent.

5) Can I buy extra runs?

Yes, top-up packs range from 25 to 5,000 runs and never expire.