Multi-agent systems and sub-agents on camelRun

View as Markdown · Docs index for coding agents

The delegate built-in lets an agent hand a task to a sub-agent and get its answer back as the call's result, while it keeps the conversation. This is the OpenAI Agents SDK's agent.as_tool() and LangGraph's supervisor. The runtime runs it, and it is durable: a sub-agent is an agent of its own, and a parent interrupted while it waits (camelRun restarted under it) still collects the sub-agent's answer when its run resumes. For a short tour, see multi-agent orchestration on camelRun.

The agents built-in starts sub-agents in the background instead: the model goes on at once, and each one's answer arrives as a message of its own when it ends. See Background sub-agents.

For a plan with many sub-agents — a sweep, a migration, research cross-checked across sources — the workflow built-in has the model write a JavaScript script that coordinates them, and only the script's result comes back. See Workflows.

camelRun has no handoffs: an agent cannot pass its conversation to another agent. See Coming from the OpenAI Agents SDK.

Sub-agents: delegate

Give an agent the delegate builtin and say whom it may delegate to: its agents, by definition key or id. The SDKs add the builtin when you pass the settings.

await agents.runtime.upsertDefinition("researcher", {
  name: "Researcher", description: "Finds sources and summarizes them",
  systemPrompt: "You research a question and answer with sources.", builtins: ["web_search", "web_fetch"],
});

const lead = await agents.upsert("research-lead", {
  instructions: "You plan research and write the report. Delegate each question to the researcher.",
  delegate: { agents: ["researcher"] },
});
const run = await lead.run("Compare the three largest EV makers' 2025 margins.");

A definition takes the same settings, for every agent made from it:

{ "name": "Research lead", "systemPrompt": "…", "builtins": ["delegate"], "delegate": { "agents": ["researcher", "writer"] } }

The model gets a delegate tool that lists the agents with what each is for (the definition's description, unless the entry gives its own):

The call's result is { agentId, requestId, status, text, output?, error? }. status is completed, input_required (the sub-agent waits on a person) or failed. The model sees a failed or waiting sub-agent as a failed call, with the reason.

Who the sub-agent is

An entry in agents is one of these:

A sub-agent the runtime makes is a real agent:

• It is linked to its parent: parentAgentId and parentRunId in GET /v1/agents/{id} and the agent list, and in the metadata of its run.
• It acts for whom the parent acts (subject, context) and uses the parent's key scope. It is billed to the same account. Its tool servers' identity tokens also name its parent and its chain's first agent (par and root; identity.parentAgentId and rootAgentId in the SDKs).
• It has its own workspace. Pass it what it needs in the task, or mount a shared volume in both definitions.
• It lives a day, as a scratch agent does. Read its history in the meantime with its id, from run.toolCalls:

const call = run.toolCalls.find(call => call.tool === "delegate");
const child = await agents.get(call.agentId!);
console.log(await child.history());

Parallel sub-agents

The model may call delegate several times in one response. The calls run at once, up to maxParallel (default 4) per run; the rest wait their turn. This is map-reduce: one call per item, and the model combines the answers.

delegate: { agents: ["researcher"], maxParallel: 8 }

Limits

Sub-agents are agents, so everything that bounds agents bounds them, and the parent's limits cover them:

Parallel sub-agents split what the parent has left: each holds its share until it ends, and a later call gets an even share of what no running sub-agent holds. So together they stay within the parent's limit, though in that one response each may finish past its own share.

An agent cannot delegate to an agent already working on its chain (it would wait on itself), and an agent's own key cannot change any of these settings. The model calls delegate directly, never from js_exec.

Cancelling

Aborting the parent's run (agent.abort(), POST /v1/agents/{id}/abort) aborts the sub-agents it is waiting on, and cancels the parent's queued runs. The abort answers once the sub-agents have ended (at most 10 seconds): what they spent by then counts in the aborted run's usage.subagentCostUsd and against the parent's spend limit. Deleting the parent does the same, and also deletes the sub-agents the runtime made for it. A named agent is aborted only while it runs the parent's task, never another.

When a run is interrupted

A sub-agent is keyed by the parent's tool call, and its run by the call too. If the parent is interrupted while it waits (camelRun restarted under it), the parent's run resumes and makes the call again. That call finds the same sub-agent and run, and waits for its answer; the task never runs twice. The sub-agent, if it was interrupted too, resumes like any agent.

The runtime records each sub-agent before it sends the sub-agent its task, so an abort or delete of the parent reaches it even before the parent's run resumes. Should the parent's run end without its answer, the sub-agent is aborted, and what it spent still counts against the parent's spend limit.

Watching sub-agents

A sub-agent's progress can appear on its parent's event stream. Ask for it: subagents: true in the SDKs, or ?subagents=1 on the events URL. Without it you get none of these events.

A delegate sub-agent's progress report is also its call's progress: a tool_execution_update of the delegate call, with details: { type: "progress", tool: "delegate", message }, which every watcher gets and <AgentChat> shows under the call. The model waiting on the call sees none of it: its answer is the call's result.

const lead = await agents.upsert("research-lead", { delegate: { agents: ["researcher"] }, subagents: true });
for await (const part of lead.stream("…")) {
  if (part.type === "subagent_start") console.log(`→ ${part.name} (${part.agentId})`);
  if (part.type === "subagent_end") console.log(`← ${part.agentId}: ${part.status}`);
}

In a browser, the watcher (@camelai/run/watch, with subagents: true) folds them into state.subagents, by delegate call, and a chat made with watch: { subagents: true } shows each sub-agent as a collapsible transcript under its call in <AgentChat> (part.subagent for a tool renderer of your own). A browser token that lists its events needs these types listed too.

A sub-agent's events are relayed through its parent's stream, except after camelRun moves one of them mid-run (a restart); its own stream and history always have everything.

Background sub-agents

The agents builtin gives the model five tools. It takes the same delegate settings (whom it may start, maxDepth, maxParallel) and needs them, as delegate does. An agent may have both builtins.

const lead = await agents.upsert("research-lead", {
  instructions: "Start a researcher for each question, and write the report as their answers come in.",
  builtins: ["agents"], delegate: { agents: ["researcher"], maxParallel: 8 },
});

The sub-agent is the same as a delegated one: the same targets, chain and depth limit, identity and key scope, and it lives a day. Its run's spend limit is a share of what its parent agent may still spend (its spendLimit): what no running sub-agent holds, split evenly among the slots maxParallel leaves. So sub-agents started together share it rather than each getting all of it. A stateless run (POST /v1/runs) cannot start one, because nothing would hear it end.

Notifications

When a sub-agent's run ends (completed, failed, aborted, or waiting on a person), camelRun sends its parent a prompt with request id child_<the child's request id>. It queues behind the parent's running turn, or starts a turn if the parent is idle. Its message is a user message with a source of its own, in history, events and exports:

{ "role": "user", "content": [{ "type": "text", "text": "Found 3 sources: …" }],
  "source": { "kind": "agent", "agentId": "client_…", "name": "researcher-1" }, "requestId": "child_spawn_…",
  "metadata": { "agentId": "client_…", "name": "researcher-1", "status": "completed", "output": { … }, "usage": { … } } }

The model reads it in a block only camelRun writes, never as the person:

<agent_notification name="researcher-1" status="completed">
Found 3 sources: …
</agent_notification>

The sub-agent's text can't close the block or open another: such tags in it, and in any user message or tool result, are neutralized. Your UI should render a message with source as a notification, not as the user.

• A sub-agent whose ending a wait_agent call answered adds no notification.
• A parent waiting on a person (an approval, ask_user) gets its notifications once they answer, because a new message would supersede what it asked.
• A sub-agent that waits on a person notifies with status input_required, and metadata.inputs lists what it asked. Answer its inputs as any agent's (POST /v1/agents/{child}/inputs/{id}, see Human input). It resumes, and its next ending notifies again.

Progress

A sub-agent camelRun made (a spawn_agent or delegate child) has report_progress({ text }): a line or two on how its task is going. A report is status, not a message. The parent is never interrupted for it, and it never starts a turn or counts against the loop guard:

• Latest wins. Each running sub-agent has one current report; a new one replaces it. It is at most 500 characters, and a turn reports at most 100 times.

• It rides on the parent's next tool result. When the parent's model gets the result of any tool it called, camelRun adds the reports it has not seen yet, latest only:

  <agent_updates>
  Progress your running sub-agents reported (latest each; no answer needed):
  - researcher-1 (12s ago): Read 3 of 5 sources; now comparing margins
  </agent_updates>

  Only camelRun writes this block: the tags are neutralized in a report's text and anywhere else. In history it is a part of that tool result of its own, { type: "text", text: "<agent_updates>…", agentUpdates: true }, so your UI can show it apart from the tool's output.

• list_agents and wait_agent show it as progress and progressAt of each running sub-agent; what they showed is not repeated with the next result.

• A waiting parent can watch it. wait_agent does not end for a report, unless the model asks: with untilProgress: true, a report it has not seen ends the wait too, with updated: ["researcher-1"] and each one's latest progress. It answers no sooner than 15 seconds into the wait, so a sub-agent reporting each step costs its parent one model call per 15 seconds at most. The model reads how the work goes, and corrects a sub-agent that went off track with send_message, which reaches it while it works.

• An ending supersedes it. Once a sub-agent's run ends, its reports are shown no more: its notification (or wait_agent) carries its answer.

• Watchers see it at once, as subagent_progress (see Events).

Use send_message to "parent" instead for something the parent must act on: that interrupts it.

Messages

send_message goes between an agent and its own sub-agents and parent, no one else:

• Parent to sub-agent: the text joins the sub-agent's running turn (one that was just started, in the same response, too), or starts a turn that keeps its history. A sub-agent that had finished works again, and its answer comes back as a new notification. The sub-agent reads <agent_message name="parent">…</agent_message>.
• Sub-agent to parent: to: "parent", which camelRun resolves through the run's signed chain, without the sub-agent's turn ending. A parent that is running takes it into that turn at its next step, after the tool calls it is making: it starts no turn. A wait_agent in progress returns at once, with messaged: ["researcher-1"], so the model reads the message next. An idle parent gets it as a turn of its own, as a notification. Either way the model reads <agent_message name="researcher-1">…, with source and metadata.kind: "message", and the parent's stream has subagent_message ({ agentId, name, text }). The call answers into: "its running turn" or "a new turn of its own". A sub-agent camelRun made has send_message even without the builtin.

A turn sends at most 20 messages, and the turns that messages start count against the loop guard below, so two agents messaging each other stop.

Stopping background sub-agents

• interrupt_agent, or POST /v1/agents/{child}/abort for one sub-agent: its turn is aborted, and its parent's notification says aborted.
• Aborting the parent (agent.abort(), POST /v1/agents/{id}/abort) aborts its running sub-agents too. { "children": "keep" } (abort({ children: "keep" })) leaves them running.
• An abort never drops a notification: one waiting for its turn lands in history without a turn (its run ends aborted).
• Deleting the parent aborts its running sub-agents and deletes the ones camelRun made. Named agents it started ({ agent: "…" } targets) stay.

Exactly once

camelRun records each sub-agent before it sends the task, keyed by the spawn_agent call, so a resumed turn finds the same sub-agent and never starts a second. Its ending is delivered as its run ends, or, if that delivery is interrupted, a few seconds later. The parent takes each ending once, so an interruption delays a notification and never repeats it.

Cost and loop guards

• What the sub-agent spent is charged to the parent's spend limit when its notification lands (or its wait_agent answers it), and shows in that run's usage.subagentCostUsd.
• Spend limits apply to the turns notifications start. A parent at its limit gets the notification in history, without a turn: the run ends with stopped: "spend_limit".
• Notifications and messages may start at most 100 turns per chain of agents (counted by its first agent) and clock hour, so agents that keep starting each other stop. Past it, a notification or message lands in history without a turn, and its run ends with stopped: "agent_loop_limit", naming the limit. Send the agent a message to go on.
• At most maxParallel sub-agents run at once per parent. Past it, spawn_agent fails and the model waits for one.

Events

With ?subagents=1 (subagents: true in the SDKs), the parent's stream has:

• subagent_start (with background: true) when a sub-agent starts;
• subagent_event with its events while they are relayed through the parent (see above);
• subagent_end (with background: true, name and status) when its notification lands or a wait_agent answers it;
• subagent_message when it messages its parent;
• subagent_progress (with background: true) when it reports progress.

Its own stream and history always have everything. A browser token minted with the children scope (plus the others it needs: { "scopes": ["events", "history", "children"] }) reads the agent's sub-agents too, by their ids, so a chat can reload a sub-agent's transcript with its parent's token.

Workers your tools prepare

A tool can start a sub-agent for the model, so the model never names an agent. Your tool does the authorization and setup with your API key, and answers with a spawn directive. Use it for workers per customer object (a bot, a repo, a ticket) that keep their history across tasks.

import { serveTools, spawnAgent } from "@camelai/run/server";
import { tool, schema } from "@camelai/run";

export default { fetch: serveTools({
  work_on_bot: tool({
    description: "Hand a task to the bot's worker", input: schema.Object({ bot: schema.String(), task: schema.String() }),
    execute: async ({ bot, task }, { identity }) => {
      await assertOwnsBot(identity!.rootAgentId ?? identity!.agent, bot);
      const worker = await agents.upsert(`bot-worker-${bot}`, { instructions: "You maintain one bot." });
      return spawnAgent({ agent: worker.id, task, name: `bot-${bot}` });
    },
  }),
}, { runtime: "https://run.camelai.com", tenant: "acme" }) };

If you answer MCP yourself, the directive is _meta["camelrun/spawn"]: { agent, task, name?, output? } on the tool's result. camelRun takes it only from a tool server with auth "runtime" (a definition's or the agent's own). It checks that the agent is your tenant's, is not already in the calling agent's chain, and that the depth and maxParallel limits allow it (the calling agent's delegate settings, or the defaults). An agent that starts sub-agents only through your tools can set those limits alone, with delegate: { maxParallel, maxDepth } and neither the delegate nor the agents builtin, which gives the model no tool of its own to start them. The model's call then answers { agentId, name }, and the worker's answer arrives as a notification.

A worker is an agent of its own, so its runs go one at a time: a task given to a busy worker queues behind its current run (the call answers queued: true), and each task notifies when it ends. Deleting the parent leaves such workers alone. Aborting it aborts the worker only while it runs the parent's task.

In the SDKs and UIs

• agentNotice(message) (Python agent_notice) tells a history message camelRun wrote apart: { kind: "notification" | "message", agentId, name, status?, error?, output?, usage?, inputs? }, or undefined for anyone else's.
• stream() parts and watcher events include subagent_message and subagent_progress; the watcher's SubagentView.progress is a delegate sub-agent's latest report. abort({ children: "keep" }) keeps background sub-agents running.
• A tool result's <agent_updates> part is not the tool's output: the tool_result stream part's output and the chat's ToolPart.result (text, content, data) leave it out, and ToolPart.result.agentUpdates has it.
• <AgentChat> shows notifications and agent messages as collapsible notices (AgentNotice), never as the user's bubbles. The chat model's UserChatMessage.agent says which (labels.agentNotification and labels.agentMessage name them).
• Tool servers read identity.parentAgentId and identity.rootAgentId in a sub-agent's calls.

Coming from the OpenAI Agents SDK

Coming from LangGraph