Use vLLM, Ollama or your own model server in camelRun

View as Markdown · Docs index for coding agents

Any server that speaks OpenAI Chat Completions, OpenAI Responses or Anthropic Messages can be a provider of your own: a hosted API the catalog lacks, a model the catalog hasn't caught up with, a gateway, or your own vLLM, Ollama or LM Studio. Name the provider, list its models, and your agents use them like catalog models, as <name>/<model id>. A provider can be your account's, or one key scope's, for one customer's agents only.

await agents.runtime.setProvider("acme-llm", {
  type: "openai-completions",
  baseUrl: "https://llm.acme.example/v1",
  apiKey: process.env.ACME_LLM_KEY,
  models: [{ id: "acme-70b", contextWindow: 131072, maxOutputTokens: 8192, pricing: { input: 0.6, output: 0.8 } }],
});
const agent = await agents.upsert("support", { model: "acme-llm/acme-70b" });

The provider

PUT /v1/providers/{name} adds a provider or replaces it:

• The name is 1 to 40 lowercase letters, digits and -, and can't be a built-in provider's (openai, groq, openrouter and the rest of GET /v1/providers).
• The key and headers are never shown again. GET /v1/providers lists your providers after the built-in ones: under custom their address, header names and models, and under key the key's last four characters.
• A rotated key, new address or new headers reach agents at their next model call, with no reconfiguration. A model's declaration (its context window, pricing, compat) is taken when an agent takes the model: agents already on it take a changed declaration when configured with it again (PATCH /v1/agents/{id}/configuration with {"model": "<name>/<id>"}).
• DELETE /v1/providers/{name} deletes one; its agents fail at their next call until it is set again or they move to another model.
• A key scope can have its own entry for your provider (PUT /v1/key-scopes/{scope}/providers/{name}, {apiKey?, baseUrl?, headers?}), such as each customer's own deployment. What the entry gives replaces the provider's own (its headers all of the provider's); what it leaves out is the provider's, the key included.

In a key scope

A key scope can have providers of its own, for its agents only: each customer's own endpoint, under the same name in every scope.

curl -X PUT https://run.camelai.com/v1/key-scopes/org_42/model-providers/custom \
  -H "Authorization: Bearer $CAMELAI_API_KEY" -H "Content-Type: application/json" \
  -d '{"type": "openai-responses", "baseUrl": "https://bedrock-mantle.us-west-2.api.aws/openai/v1",
       "apiKey": "'"$ORG_42_BEDROCK_API_KEY"'", "models": [{"id": "openai.gpt-5.6-terra", "contextWindow": 200000, "reasoning": true}]}'

An agent in org_42 (keyScope: "org_42") then names custom/openai.gpt-5.6-terra.

• The body, the name's rules and the limits are those of your own providers: up to 20 per scope, besides your account's 20.
• Only agents in the scope see its providers. An agent elsewhere that names the model gets Unknown model, and its calls never use the scope's key. In its scope, a provider shadows one of your account's with the same name.
• A model is resolved in the agent's scope when it is made or configured, and in the new scope when PATCH /v1/agents/{id}/configuration moves it (give the model there too). Definitions name your account's providers only.
• GET /v1/key-scopes/{scope}/model-providers lists the scope's (never their key or header values), DELETE …/model-providers/{name} deletes one, and deleting the scope deletes them all. GET /v1/models?keyScope={scope} lists models as the scope's agents see them.

Models

Each model is {id, contextWindow, maxOutputTokens?, input?, reasoning?, pricing?, compat?}:

GET /v1/models lists your providers' models before the catalog's, all available. A model whose id the catalog knows for the provider's API (a gateway in front of Anthropic's or OpenAI's models) is called as the catalog calls it, such as which thinking settings it takes. What you declare wins, and its price is yours.

Servers that differ from OpenAI's

For Chat Completions servers. camelRun streams, sends tools as OpenAI function tools and reads tool calls as they arrive in pieces. It copes with what many servers leave out:

• A stream without usage costs nothing and counts no tokens. If you meter end users' credits, their call is charged an estimate instead. Set compat.supportsUsageInStreaming: false if the server refuses stream_options.
• A server that ends its streams without finish_reason needs compat.supportsFinishReason: false. Without it such a stream fails, rather than risk taking a reply cut off mid-way for a whole one.

Temperature and output length work as for catalog models: see Output length and temperature.

Where the server can be

camelRun calls the address from the public internet, through the same outbound checks as MCP servers and web_fetch, when you save the provider and again at every call. It must be https, and its host must resolve to public addresses only, so localhost and your LAN are unreachable. Put a server on your own machine behind a public https address, such as a tunnel, and keep it behind a key.

For example, with Ollama behind a Cloudflare Tunnel and a Cloudflare Access service token:

await agents.runtime.setProvider("home-ollama", {
  type: "openai-completions",
  baseUrl: "https://ollama.example.com/v1",
  apiKey: null,
  headers: { "CF-Access-Client-Id": process.env.ACCESS_CLIENT_ID!, "CF-Access-Client-Secret": process.env.ACCESS_CLIENT_SECRET! },
  models: [{ id: "qwen3:32b", contextWindow: 40960, reasoning: true, compat: { maxTokensField: "max_tokens" } }],
});
const agent = await agents.upsert("local-helper", { model: "home-ollama/qwen3:32b" });

If the tunnel's address changes, save the provider again with the new baseUrl; agents follow at their next call.

A self-hosted runtime can reach servers on its own network through AGENT_OUTBOUND_ALLOW_ORIGINS.

Billing

Your own providers never use the platform's keys, and their tokens are never charged to prepaid credit. Usage records what the declared pricing says each run cost, which feeds spend limits, the usage pages, and the usage.recorded and run.completed webhooks.

A model the catalog doesn't have yet

Groq is a built-in provider: set its key (PUT /v1/providers/groq/key) and use groq/<model> for any model in GET /v1/models. For one the catalog lacks, add Groq's API under a name of your own:

curl -X PUT https://run.camelai.com/v1/providers/groq-preview \
  -H "Authorization: Bearer $CAMELAI_API_KEY" -H "Content-Type: application/json" \
  -d '{"type": "openai-completions", "baseUrl": "https://api.groq.com/openai/v1", "apiKey": "'"$GROQ_API_KEY"'",
       "models": [{"id": "new-model-preview", "contextWindow": 131072, "maxOutputTokens": 8192,
                   "pricing": {"input": 0.2, "output": 0.6}}]}'

Then name it groq-preview/new-model-preview.