--- title: 'Concepts' description: 'Durable keyed agents, runs, events, where tools run, and people in the loop' --- [View as Markdown](https://camelai.com/docs/camelrun/concepts.md) ยท [Docs index for coding agents](https://run.camelai.com/llms.txt) ## Agents are durable and keyed An agent is a conversation that lasts. Its history, files, configuration (model, instructions, tools) and pending work live in camelRun, not in your process. Most agents are asleep at any moment and cost nothing. A message wakes one on whichever node has room, and it picks up where it left off. You name an agent with a **key**, your own id for it, such as `support-triage`, `user-123` or `thread-9f2c`. Keys are 1 to 80 letters, digits, `_` and `-`. ```ts const agent = await agents.upsert("user-123", { model, instructions, tools }); ``` * The same key is the same agent, from any process, until you delete it with `agent.delete()`. * `upsert` makes the agent if there is none, and otherwise brings it to the configuration you pass. A changed model, instructions or tool list applies between turns, and the history carries over. * Who the agent acts for (`subject`, `context`), its definition and its mounts are fixed when it is made. Changing them is a 409 that names the field. Delete the agent and upsert again to start over. * Over REST, the key is the `Idempotency-Key` header of `POST /v1/agents`. Every agent also has an **id** (`client_...`), which is safe to log and store, and a **token**, which lets its holder run that agent and nothing else. You rarely need the token, because upserting by key gives you the agent again. Agents made without a key are scratch agents that expire after a day unless you set `ttlSeconds`. ## Runs A message to an agent starts a **run**: the model loop, with every tool call it makes, until the model answers or something stops it. An agent has one run at a time. A message sent during a run waits for it (`whileRunning: "queue"`, the default) or joins it (`whileRunning: "steer"`, read by the running turn after its current step). ```ts const run = await agent.run("Summarize ticket 123", { user: "alice" }); ``` | Field | | | ----------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `id` | The run's id. Pass `idempotencyKey` to choose it; sending the same key again returns the same run | | `status` | `completed`, `input_required` (it waits on a person, see `inputs`), or `failed` | | `text` | The final reply | | `output` | With `output: schema`, the answer in that shape. See [Structured output](/docs/camelrun/structured-output) | | `inputs` | What it waits on, each with `answer()` | | `error` | `{code, message, uncertain?}` when it failed | | `toolErrors` (Python `tool_errors`) | Tool calls that didn't complete (timed out, connection lost, nobody serving the tools). The model was told and carried on | | `toolCalls` (Python `tool_calls`) | Every tool call the run made (the first 100), calls from `js_exec`'s code included: `{tool, toolCallId, innerCallId?, ok, code?}`. Arguments and results are in history | | `files` | Files the run wrote | | `usage` | What its model calls used | `run()` has no timeout: runs can take minutes, and a run waiting on a person can wait for days. Pass an `AbortSignal` (Python: `timeout=`) to stop waiting; the run itself goes on until it ends or you call `agent.abort()`. ## Forking A **fork** is a new agent made from an existing one: the same configuration (model, instructions, definition, tools, built-ins, who it acts for, key scope, spend and run limits), a copy of its history, and a copy of its workspace files. From then on each is its own: a turn, a summary or a file written in one never shows in the other. Fork to try another direction without losing the first, to branch a conversation where a user edits an earlier message, or to start many agents from one prepared one. ```ts const branch = await agent.fork({ key: "thread-9f2c-b" }); const earlier = await agent.fork({ atMessage: 4 }); // history through message 4 ``` ```python branch = await agent.fork(key="thread-9f2c-b") earlier = await agent.fork(at_message=4) ``` * **Where it ends.** By default the copy ends with the agent's last finished turn, never inside one that is running or waiting on a person, so forking a busy agent is safe. `atMessage` ends it earlier: a history index keeps that message and the tool results that answer it, and a run's request id keeps that run's whole turn. A message in a turn that hasn't ended is a 409 `FORK_POINT_RUNNING`. * **Long conversations.** The copy keeps the source's summaries, so the fork's model sees what the source's would at that point. History keeps every message. * **Files.** The fork's `/workspace` is a fork of the source's volume as it is when you fork, not as it was at `atMessage`. Other volumes the source mounts stay shared. Files its history refers to stay available to the fork, whatever becomes of the source. * **Not copied:** schedules, channel bindings, pending inputs, runs and their events, and what the source has spent. * **Its own:** `subject`, `context`, `systemPromptAppend` and `modelHeaders` can be given for the fork instead of the source's, so a fork for another conversation names that conversation. * **Keys.** A fork made with a `key` lives until deleted, and forking again with the same key returns the same fork. Without one it's a scratch agent that lives a day, unless you set `ttlSeconds`. * `forkedFrom: {agentId, atMessage}` on the fork (`GET /v1/agents/{id}`, the console) says where it came from. A fork is an agent like any other: it counts toward your agents and storage. The copy itself costs only storage, and shares stored content with the source. Over REST it's `POST /v1/agents/{id}/fork`; in the console, **Fork** on the agent's page, or **Fork from here** under a reply. ## The result is the truth; events are for display An agent's **events** stream every step: text as the model writes it, tool calls and their progress, and inputs asked and answered. Use them to show the agent working, not to decide what happened. * The run's result (`run()`, the `run.completed` webhook, or `GET /v1/agents/{id}/requests/{requestId}`) is recorded durably and always arrives, even if your process restarts or the connection drops. * The stream replays from memory on reconnect, as far back as it can. Where it can't (a restarted node, a long disconnection), the SDK gets a snapshot of the running turn instead. Reply to your user from the run, and render the stream as progress. See [Events](/docs/camelrun/events). ## Where tools run Pick by where your code runs: | Your situation | Use | How | | ---------------------------------------------------------------------------------- | ---------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | A long-lived server or worker, and the tools touch its state | **Attached tools** | `tools` on `upsert`. The SDK holds a connection and camelRun calls your functions over it. One process serves an agent's tools at a time, and a call running when it restarts is lost | | Serverless functions, several instances, or one backend serving many users' agents | **Served tools** | `serveTools(tools)` on an HTTPS endpoint of yours, named in a definition with `auth: { type: "runtime" }`. camelRun calls it with a signed token saying who each call is for. Safe through your deploys and restarts | | A third-party API | **OpenAPI** or **MCP** sources in a definition | camelRun calls the API itself, with credentials it stores sealed | | Web search, fetching pages, scheduling, asking the user | **Built-ins** | `builtins: ["web_search", "web_fetch", "schedule", "ask_user"]` | | Other agents doing tasks for this one | The **`delegate`** built-in | `delegate: { agents: ["researcher"] }`. See [Multi-agent](/docs/camelrun/multi-agent) | The same `tool({...})` definitions work attached and served, so you can start attached and move to served without changing a tool (see [Several processes, workers and deploys](/docs/camelrun/tools#several-processes-workers-and-deploys)). Every agent also has file tools over its [files](/docs/camelrun/files), and `js_exec`, a sandbox where the model writes JavaScript that calls its tools. See [Tools](/docs/camelrun/tools). ## Processes and connections A process that serves an agent's tools **attaches** to it: it holds the agent's event stream and answers its tool calls. Only one process at a time can, so a tool call never lands in the wrong place. A second process that tries gets `APPLICATION_CONNECTED`, unless it passes `takeover: true`. Any number of processes can **run** an agent without serving its tools: an agent upserted without local tools, or with `attach: false`, follows the stream read-only. That's how serverless functions and webhook handlers run agents whose tools are served elsewhere. A run of an agent with attached tools while no process serves them is refused with `APPLICATION_NOT_CONNECTED`, after a few seconds' grace. Pass `allowDisconnected: true` to run anyway; calls to those tools then fail and show in `toolErrors` as `not_connected`. Schedules and channels are never refused, so agents that must work with nobody attached should use served tools. ## People in the loop A run can stop and wait for a person: the model asks a question (`ask_user`), a tool needs approval before it runs, or a tool asks for a form. The run ends with `status: "input_required"` and its `inputs`, the agent sleeps, and answering the last input resumes it. ```ts if (run.status === "input_required") run = await run.inputs[0].answer(true, { from: "alice" }); ``` See [Human input](/docs/camelrun/human-input). ## Idempotency Everything that changes something can be retried safely: * `upsert` by key, and `run` or `POST /v1/agents/{id}/prompt` by `idempotencyKey` (the request id). The same key returns the same agent or run. * Every other POST takes an `Idempotency-Key` header. A retry with the same key, path and body gets the first answer again for a day. The same key with a different body is a 409. * Every tool call carries `context.idempotencyKey`, the same for every attempt of that call. Key your side effects by it. A tool call whose answer was lost (the connection dropped, or its deadline passed) is never sent again. The model is told its outcome is unknown, so it can check before repeating anything. ## Waking later An agent can be woken by a schedule (`agent.schedule({ text, inSeconds, at, everySeconds })`, or the `schedule` built-in so the agent sets its own), by a [channel](/docs/camelrun/channels), or by your own code from a [webhook](/docs/camelrun/webhooks). ## Definitions A **definition** is a reusable configuration: model, instructions, tool sources (MCP servers, OpenAPI specs, built-ins), limits and mounts. Make agents from one with `definition: "def_..."` on `upsert`, and roll a change out to all of them with `apply: "all"`. See [Definitions](/docs/camelrun/definitions).