Operate a self-hosted installation

Run operational commands from the checked-out camelAI repository on the VM. Keep .env.selfhost readable only by the operator account because it contains identity, signing, integration, and AI-provider secrets.

Check health

Use both the readiness endpoint and the host doctor:

curl --fail --silent http://127.0.0.1:3001/api/selfhost/health
bun run selfhost:doctor

The health response includes a versioned capability contract. Builds, deploys, notebooks, and SQL should report state: "configured".

configured means the Docker image and workerd namespace are wired. It does not mean a functional job ran during the lightweight HTTP check. The response therefore includes a verification command for each attached runtime.

Missing required runtime bindings make readiness fail with HTTP 503. Intentionally disabled outbound email and SMTP do not make the service unhealthy.

Run attached-runtime smokes

Run all four smokes after initial installation, after an upgrade, and after a Docker or host-networking change:

bun --env-file=.env.selfhost run selfhost:container:smoke:mount
bun --env-file=.env.selfhost run selfhost:container:smoke:project
bun --env-file=.env.selfhost run selfhost:container:smoke:analysis
bun --env-file=.env.selfhost run selfhost:container:smoke:db-query

The mount smoke verifies bidirectional synchronization between local R2 storage and a sandbox container. The project smoke launches a build container and executes a small build. The analysis smoke starts Jupyter and executes a notebook. The database-query smoke loads and exercises the supported query drivers. Each command removes its smoke container afterward.

The mount smoke does not require /dev/fuse, SYS_ADMIN, or an unconfined AppArmor profile. A supported self-host installation must not grant those privileges to containers that execute generated code.

Inspect containers and logs

For the default bundled-Pomerium automatic-TLS deployment:

docker compose \
  --env-file .env.selfhost \
  -f docker-compose.selfhost.yml \
  -f docker-compose.selfhost.caddy.yml \
  -f docker-compose.selfhost.pomerium.yml \
  -f docker-compose.selfhost.pomerium-loopback.yml \
  ps

docker compose \
  --env-file .env.selfhost \
  -f docker-compose.selfhost.yml \
  -f docker-compose.selfhost.caddy.yml \
  -f docker-compose.selfhost.pomerium.yml \
  -f docker-compose.selfhost.pomerium-loopback.yml \
  logs --tail=200 caddy pomerium app

If you use external Pomerium or Cloudflare Access, omit both Pomerium overlay files but keep the Caddy overlay as the restricted HTTP origin.

On an AWS-provisioned host, also inspect:

sudo cloud-init status --wait
sudo journalctl -u camelai-selfhost --no-pager
sudo camelai-selfhost-compose logs --tail=200 caddy pomerium app

Back up

Create a backup of durable application state, local Artifact repositories, Pomerium's databroker, and Caddy's certificate/account state:

bun run selfhost:backup

Backups are stored under .selfhost/backups/<timestamp>.

.env.selfhost is intentionally excluded because it contains secrets. Store a separate encrypted copy with access controls appropriate for your organization. Also retain any provided TLS material and the release manifest used by the installation.

Copy backups off the VM and test restores on a separate node on a regular schedule. A backup that exists only on the application disk does not protect against disk or instance loss.

Restore

Stop the stack before restoring:

docker compose \
  --env-file .env.selfhost \
  -f docker-compose.selfhost.yml \
  -f docker-compose.selfhost.caddy.yml \
  -f docker-compose.selfhost.pomerium.yml \
  -f docker-compose.selfhost.pomerium-loopback.yml \
  down

bun run selfhost:restore -- .selfhost/backups/<timestamp>

Restore the matching .env.selfhost, any provided TLS material, code release, and image digests before starting. Then run the doctor, start the stack, and execute all four runtime smokes.

Upgrade

Download selfhost-release.json and selfhost-upgrade-bootstrap.mjs from the matching camelAI GitHub Release. Keep both files outside the mutable repository checkout. For example:

RELEASE=selfhost-vX.Y.Z
curl --fail --location \
  --output /secure/path/selfhost-release.json \
  "https://github.com/qaml-ai/camelAI/releases/download/${RELEASE}/selfhost-release.json"
curl --fail --location \
  --output /secure/path/selfhost-upgrade-bootstrap.mjs \
  "https://github.com/qaml-ai/camelAI/releases/download/${RELEASE}/selfhost-upgrade-bootstrap.mjs"

Run:

bun run selfhost:upgrade -- \
  --release selfhost-vX.Y.Z \
  --manifest /secure/path/selfhost-release.json

The upgrader:

1. Requires a clean tracked worktree.
2. Verifies the target Git revision and all eight digest-pinned images.
3. Backs up durable volumes.
4. Saves the previous checkout and .env.selfhost.
5. Hands control to the target release's upgrader.
6. Pulls and starts the target images.
7. Runs the doctor and all four attached-runtime smokes.
8. Prints the exact rollback command.

For an installation whose current upgrader predates target-code handoff, run the standalone bootstrap once:

node /secure/path/selfhost-upgrade-bootstrap.mjs \
  --repo "$PWD" \
  --release selfhost-vX.Y.Z \
  --manifest /secure/path/selfhost-release.json

Use the exact rollback path printed by a successful upgrade:

bun run selfhost:upgrade -- --rollback .selfhost/releases/<timestamp>

AWS-provisioned hosts install a wrapper around the same verified upgrader:

sudo camelai-selfhost-upgrade \
  --release selfhost-vX.Y.Z \
  --manifest /secure/path/selfhost-release.json

Protect the Docker host

The application must have read-write access to /var/run/docker.sock to create build, notebook, query, and egress containers. Treat application-container control as root-equivalent VM access.

• Do not expose the Docker API over TCP.
• Keep the camelAI application and local Artifact ports on loopback.
• Restrict host administration to SSM or a narrow management network.
• Apply operating-system and Docker security updates on a tested schedule.
• Monitor disk use under Docker's data root and the backup directory.
• Restrict outbound traffic according to your provider and package-registry needs.
• Protect .env.selfhost, Pomerium and Caddy secrets, TLS keys, and backups.

The app service uses Linux host networking because the attached container egress sidecars communicate through loopback. Changing it to a nested Compose bridge prevents the runtime containers from becoming ready.

Outbound email

Outbound email is disabled. Setting EMAIL_FROM_ADDRESS, WORKSPACE_EMAIL_DOMAIN, or SMTP-looking environment variables does not enable delivery.

Until a supported internal SMTP transport is implemented:

• Provision users through Pomerium or Cloudflare Access.
• Deliver invitation URLs through an approved internal channel.
• Route support requests through your existing support process.
• Do not add a fake SMTP sink that accepts and discards messages.

Troubleshooting

Pomerium cannot read its configuration

Keep .selfhost/pomerium/config.yaml and secret files owned by the operator and mode 0600. Use the repository's Pomerium Compose overlay, which retains the minimal DAC_OVERRIDE capability needed to read those protected bind mounts on Linux.

A runtime container never becomes ready

Confirm:

• The host is x86_64.
• Docker has at least 4 CPUs and 7.5 GiB usable memory.
• The application container uses host networking.
• The Docker socket is mounted read-write.
• The configured runtime and container-egress images come from one release.
• Host firewall rules do not block Docker bridge control traffic.

Do not substitute Cloudflare's stock cloudflare/proxy-everything sidecar for the manifest's SELFHOST_CONTAINER_EGRESS_IMAGE. The release image contains a bridge-bypass entrypoint for hosts where the stock sidecar intercepts Docker control traffic.

If the VM runs a host firewall, inspect its Docker bridge policy and add the narrowest rule that permits container-to-host control traffic. For UFW, the equivalent broad interface rule is sudo ufw allow in on docker0; review it against your network policy before applying it. A host without a blocking firewall does not need that rule.

Run bun run selfhost:doctor, then run the failing runtime's smoke command.

App deployment reports an S3FS or /dev/fuse mount failure

Upgrade to selfhost-v0.1.4 or later. Earlier releases could ask local sandbox containers to use Cloudflare's production S3FS mount path, which is not available to stock self-host containers.

Do not add /dev/fuse, SYS_ADMIN, or apparmor=unconfined to the sandbox containers. Current releases use the Sandbox SDK's FUSE-free local R2 synchronization path instead. Verify it directly:

bun --env-file=.env.selfhost run selfhost:container:smoke:mount
bun run selfhost:doctor

This is a persistent version/configuration error on an affected release, not a transient outage. Repeated deployment retries will not fix it.

Caddy cannot obtain or renew a certificate

Confirm that:

• The selected DNS credential can read the zone and edit TXT records.
• SELFHOST_TLS_DNS_PROVIDER matches where the authoritative zone is hosted.
• The Route 53 hosted-zone ID is correct, or the Cloudflare token is scoped to the correct zone.
• Outbound HTTPS and DNS resolution work from the VM.

Run bun run selfhost:configure, inspect docker compose ... logs caddy, and retry only after correcting the credential or zone. Do not work around DNS validation by exposing Pomerium's plaintext upstream.

The control plane returns a redirect loop

Confirm that:

• SELFHOST_PUBLIC_BASE_URL and SELFHOST_MAIN_HOSTNAME describe the same host.
• POMERIUM_AUTHENTICATE_URL uses a distinct host.
• The original Host header reaches Pomerium and camelAI.
• The certificate covers both hosts.
• The VM can resolve the public hostnames.

Public apps redirect to sign-in

The control-plane route and deployed-app wildcard route must remain separate. Regenerate bundled Pomerium configuration with:

bun run selfhost:configure

For an external proxy, configure the wildcard app route without forced authentication and let camelAI enforce each app's public or private setting.