Multi-agent systems and sub-agents on camelRun

View as Markdown · Docs index for coding agents

The delegate built-in lets an agent hand a task to a sub-agent and get its answer back as the call's result, while it keeps the conversation. This is the OpenAI Agents SDK's agent.as_tool() and LangGraph's supervisor. The runtime runs it, and it is durable: a sub-agent is an agent of its own, and a parent interrupted while it waits (camelRun restarted under it) still collects the sub-agent's answer when its run resumes. For a short tour, see multi-agent orchestration on camelRun.

The agents built-in starts sub-agents in the background instead: the model goes on at once, and each one's answer arrives as a message of its own when it ends. See Background sub-agents.

camelRun has no handoffs: an agent cannot pass its conversation to another agent. See Coming from the OpenAI Agents SDK.

Sub-agents: delegate

Give an agent the delegate builtin and say whom it may delegate to: its agents, by definition key or id. The SDKs add the builtin when you pass the settings.

await agents.runtime.upsertDefinition("researcher", {
  name: "Researcher", description: "Finds sources and summarizes them",
  systemPrompt: "You research a question and answer with sources.", builtins: ["web_search", "web_fetch"],
});

const lead = await agents.upsert("research-lead", {
  instructions: "You plan research and write the report. Delegate each question to the researcher.",
  delegate: { agents: ["researcher"] },
});
const run = await lead.run("Compare the three largest EV makers' 2025 margins.");

A definition takes the same settings, for every agent made from it:

{ "name": "Research lead", "systemPrompt": "…", "builtins": ["delegate"], "delegate": { "agents": ["researcher", "writer"] } }

The model gets a delegate tool that lists the agents with what each is for (the definition's description, unless the entry gives its own):

The call's result is { agentId, requestId, status, text, output?, error? }. status is completed, input_required (the sub-agent waits on a person) or failed. The model sees a failed or waiting sub-agent as a failed call, with the reason.

Who the sub-agent is

An entry in agents is one of these:

A sub-agent the runtime makes is a real agent:

• It is linked to its parent: parentAgentId and parentRunId in GET /v1/agents/{id} and the agent list, and in the metadata of its run.
• It acts for whom the parent acts (subject, context) and uses the parent's key scope. It is billed to the same account. Its tool servers' identity tokens also name its parent and its chain's first agent (par and root; identity.parentAgentId and rootAgentId in the SDKs).
• It has its own workspace. Pass it what it needs in the task, or mount a shared volume in both definitions.
• It lives a day, as a scratch agent does. Read its history in the meantime with its id, from run.toolCalls:

const call = run.toolCalls.find(call => call.tool === "delegate");
const child = await agents.get(call.agentId!);
console.log(await child.history());

Parallel sub-agents

The model may call delegate several times in one response. The calls run at once, up to maxParallel (default 4) per run; the rest wait their turn. This is map-reduce: one call per item, and the model combines the answers.

delegate: { agents: ["researcher"], maxParallel: 8 }

Limits

Sub-agents are agents, so everything that bounds agents bounds them, and the parent's limits cover them:

Parallel sub-agents split what the parent has left: each holds its share until it ends, and a later call gets an even share of what no running sub-agent holds. So together they stay within the parent's limit, though in that one response each may finish past its own share.

An agent cannot delegate to an agent already working on its chain (it would wait on itself), and an agent's own key cannot change any of these settings. The model calls delegate directly, never from js_exec.

Cancelling

Aborting the parent's run (agent.abort(), POST /v1/agents/{id}/abort) aborts the sub-agents it is waiting on, and cancels the parent's queued runs. The abort answers once the sub-agents have ended (at most 10 seconds): what they spent by then counts in the aborted run's usage.subagentCostUsd and against the parent's spend limit. Deleting the parent does the same, and also deletes the sub-agents the runtime made for it. A named agent is aborted only while it runs the parent's task, never another.

When a run is interrupted

A sub-agent is keyed by the parent's tool call, and its run by the call too. If the parent is interrupted while it waits (camelRun restarted under it), the parent's run resumes and makes the call again. That call finds the same sub-agent and run, and waits for its answer; the task never runs twice. The sub-agent, if it was interrupted too, resumes like any agent.

The runtime records each sub-agent before it sends the sub-agent its task, so an abort or delete of the parent reaches it even before the parent's run resumes. Should the parent's run end without its answer, the sub-agent is aborted, and what it spent still counts against the parent's spend limit.

Watching sub-agents

A sub-agent's progress can appear on its parent's event stream. Ask for it: subagents: true in the SDKs, or ?subagents=1 on the events URL. Without it you get none of these events.

const lead = await agents.upsert("research-lead", { delegate: { agents: ["researcher"] }, subagents: true });
for await (const part of lead.stream("…")) {
  if (part.type === "subagent_start") console.log(`→ ${part.name} (${part.agentId})`);
  if (part.type === "subagent_end") console.log(`← ${part.agentId}: ${part.status}`);
}

In a browser, the watcher (@camelai/run/watch, with subagents: true) folds them into state.subagents, by delegate call, and a chat made with watch: { subagents: true } shows each sub-agent as a collapsible transcript under its call in <AgentChat> (part.subagent for a tool renderer of your own). A browser token that lists its events needs these types listed too.

A sub-agent's events are relayed through its parent's stream, except after camelRun moves one of them mid-run (a restart); its own stream and history always have everything.

Background sub-agents

The agents builtin gives the model five tools. It takes the same delegate settings (whom it may start, maxDepth, maxParallel) and needs them, as delegate does. An agent may have both builtins.

const lead = await agents.upsert("research-lead", {
  instructions: "Start a researcher for each question, and write the report as their answers come in.",
  builtins: ["agents"], delegate: { agents: ["researcher"], maxParallel: 8 },
});

The sub-agent is the same as a delegated one: the same targets, chain and depth limit, identity and key scope, and it lives a day. Its run's spend limit is a share of what its parent agent may still spend (its spendLimit): what no running sub-agent holds, split evenly among the slots maxParallel leaves. So sub-agents started together share it rather than each getting all of it. A stateless run (POST /v1/runs) cannot start one, because nothing would hear it end.

Notifications

When a sub-agent's run ends (completed, failed, aborted, or waiting on a person), camelRun sends its parent a prompt with request id child_<the child's request id>. It queues behind the parent's running turn, or starts a turn if the parent is idle. Its message is a user message with a source of its own, in history, events and exports:

{ "role": "user", "content": [{ "type": "text", "text": "Found 3 sources: …" }],
  "source": { "kind": "agent", "agentId": "client_…", "name": "researcher-1" }, "requestId": "child_spawn_…",
  "metadata": { "agentId": "client_…", "name": "researcher-1", "status": "completed", "output": { … }, "usage": { … } } }

The model reads it in a block only camelRun writes, never as the person:

<agent_notification name="researcher-1" status="completed">
Found 3 sources: …
</agent_notification>

The sub-agent's text can't close the block or open another: such tags in it, and in any user message or tool result, are neutralized. Your UI should render a message with source as a notification, not as the user.

• A sub-agent whose ending a wait_agent call answered adds no notification.
• A parent waiting on a person (an approval, ask_user) gets its notifications once they answer, because a new message would supersede what it asked.
• A sub-agent that waits on a person notifies with status input_required, and metadata.inputs lists what it asked. Answer its inputs as any agent's (POST /v1/agents/{child}/inputs/{id}, see Human input). It resumes, and its next ending notifies again.

Messages

send_message goes between an agent and its own sub-agents and parent, no one else:

• Parent to sub-agent: the text joins the sub-agent's running turn, or starts a turn that keeps its history. A sub-agent that had finished works again, and its answer comes back as a new notification. The sub-agent reads <agent_message name="parent">…</agent_message>.
• Sub-agent to parent: to: "parent", which camelRun resolves through the run's signed chain. It reaches the parent as a notification does, without the sub-agent's turn ending, as <agent_message name="researcher-1">…, with source and metadata.kind: "message". The parent's stream has subagent_message ({ agentId, name, text }). A sub-agent camelRun made has send_message even without the builtin.

A turn sends at most 20 messages, and the turns that messages start count against the loop guard below, so two agents messaging each other stop.

Stopping background sub-agents

• interrupt_agent, or POST /v1/agents/{child}/abort for one sub-agent: its turn is aborted, and its parent's notification says aborted.
• Aborting the parent (agent.abort(), POST /v1/agents/{id}/abort) aborts its running sub-agents too. { "children": "keep" } (abort({ children: "keep" })) leaves them running.
• An abort never drops a notification: one waiting for its turn lands in history without a turn (its run ends aborted).
• Deleting the parent aborts its running sub-agents and deletes the ones camelRun made. Named agents it started ({ agent: "…" } targets) stay.

Exactly once

camelRun records each sub-agent before it sends the task, keyed by the spawn_agent call, so a resumed turn finds the same sub-agent and never starts a second. Its ending is delivered as its run ends, or, if that delivery is interrupted, a few seconds later. The parent takes each ending once, so an interruption delays a notification and never repeats it.

Cost and loop guards

• What the sub-agent spent is charged to the parent's spend limit when its notification lands (or its wait_agent answers it), and shows in that run's usage.subagentCostUsd.
• Spend limits apply to the turns notifications start. A parent at its limit gets the notification in history, without a turn: the run ends with stopped: "spend_limit".
• Notifications and messages may start at most 100 turns per chain of agents (counted by its first agent) and clock hour, so agents that keep starting each other stop. Past it, a notification or message lands in history without a turn, and its run ends with stopped: "agent_loop_limit", naming the limit. Send the agent a message to go on.
• At most maxParallel sub-agents run at once per parent. Past it, spawn_agent fails and the model waits for one.

Events

With ?subagents=1 (subagents: true in the SDKs), the parent's stream has:

• subagent_start (with background: true) when a sub-agent starts;
• subagent_event with its events while they are relayed through the parent (see above);
• subagent_end (with background: true, name and status) when its notification lands or a wait_agent answers it;
• subagent_message when it messages its parent.

Its own stream and history always have everything. A browser token minted with the children scope (plus the others it needs: { "scopes": ["events", "history", "children"] }) reads the agent's sub-agents too, by their ids, so a chat can reload a sub-agent's transcript with its parent's token.

Workers your tools prepare

A tool can start a sub-agent for the model, so the model never names an agent. Your tool does the authorization and setup with your API key, and answers with a spawn directive. Use it for workers per customer object (a bot, a repo, a ticket) that keep their history across tasks.

import { serveTools, spawnAgent } from "@camelai/run/server";
import { tool, schema } from "@camelai/run";

export default { fetch: serveTools({
  work_on_bot: tool({
    description: "Hand a task to the bot's worker", input: schema.Object({ bot: schema.String(), task: schema.String() }),
    execute: async ({ bot, task }, { identity }) => {
      await assertOwnsBot(identity!.rootAgentId ?? identity!.agent, bot);
      const worker = await agents.upsert(`bot-worker-${bot}`, { instructions: "You maintain one bot." });
      return spawnAgent({ agent: worker.id, task, name: `bot-${bot}` });
    },
  }),
}, { runtime: "https://run.camelai.com", tenant: "acme" }) };

If you answer MCP yourself, the directive is _meta["camelrun/spawn"]: { agent, task, name?, output? } on the tool's result. camelRun takes it only from a tool server with auth "runtime" (a definition's or the agent's own). It checks that the agent is your tenant's, is not already in the calling agent's chain, and that the depth and maxParallel limits allow it (the calling agent's delegate settings, or the defaults). An agent that starts sub-agents only through your tools can set those limits alone, with delegate: { maxParallel, maxDepth } and neither the delegate nor the agents builtin, which gives the model no tool of its own to start them. The model's call then answers { agentId, name }, and the worker's answer arrives as a notification.

A worker is an agent of its own, so its runs go one at a time: a task given to a busy worker queues behind its current run (the call answers queued: true), and each task notifies when it ends. Deleting the parent leaves such workers alone. Aborting it aborts the worker only while it runs the parent's task.

In the SDKs and UIs

• agentNotice(message) (Python agent_notice) tells a history message camelRun wrote apart: { kind: "notification" | "message", agentId, name, status?, error?, output?, usage?, inputs? }, or undefined for anyone else's.
• stream() parts and watcher events include subagent_message. abort({ children: "keep" }) keeps background sub-agents running.
• <AgentChat> shows notifications and agent messages as collapsible notices (AgentNotice), never as the user's bubbles. The chat model's UserChatMessage.agent says which (labels.agentNotification and labels.agentMessage name them).
• Tool servers read identity.parentAgentId and identity.rootAgentId in a sub-agent's calls.

Coming from the OpenAI Agents SDK

Coming from LangGraph