Self-hosting troubleshooting

Diagnostic guides for the oz-agent-worker daemon and its task execution. Use this page when a worker won’t start, won’t connect, tasks stay queued, or tasks fail.

Note

The steps below apply to the managed architecture (oz-agent-worker daemon). For unmanaged deployments, refer to the documentation for the environment running oz agent run (e.g., GitHub Actions, Kubernetes).

---

Worker won’t start

Docker backend

Cause: Docker isn’t running, or the daemon platform isn’t supported.

Fix:

1. Verify Docker is running: docker info.
2. Confirm the daemon platform is linux/amd64 or linux/arm64. Windows containers are not supported.
3. If the worker runs inside Docker, confirm the /var/run/docker.sock mount is correct and the mounting user has permission to the socket.

Kubernetes backend

Cause: The worker Deployment could not start, reach the Kubernetes API, or create its startup preflight Job. Missing Secrets, insufficient RBAC, and admission policies are common causes.

Fix:

1. Run kubectl describe pod -n NAMESPACE WORKER_POD. For Helm installs, if the pod reports CreateContainerConfigError, confirm the Secret configured by warp.apiKeySecret.name exists and contains the key configured by warp.apiKeySecret.key.
2. Check the worker logs for Kubernetes API or preflight diagnostics: kubectl logs -n NAMESPACE WORKER_POD.
3. Confirm the worker’s namespace has these permissions: create, get, list, watch, delete on jobs; get, list, watch on pods; get on pods/log; list on events.
4. Confirm the task namespace allows pods with a root init container, unless you enabled native image volumes with kubernetesBackend.useImageVolumes=true.
5. If your cluster restricts image sources, set kubernetesBackend.preflightImage to an allowlisted image. The default is busybox:1.36.
6. To pull the preflight image from a private registry, configure imagePullSecrets in kubernetesBackend.podTemplate.

A successful preflight confirms that the cluster can create the configured pod shape. It does not validate every task image, task Secret, setup command, or runtime network dependency.

Direct backend

Cause: The oz CLI isn’t installed or isn’t on the worker’s PATH.

Fix:

1. Install the Oz CLI on the worker host. See Installing the CLI.
2. If the CLI isn’t on PATH, set oz_path in the config file to the absolute path of the oz binary.

---

Worker won’t connect

Cause: The API key is invalid, expired, or the host cannot reach the Automation Platform‘s backend.

Fix:

1. Confirm your API key is correct, not expired, and has team scope.
2. Regenerate the API key in Settings > Cloud platform > API keys if you suspect it’s invalid.
3. Ensure the host has outbound internet access to oz.warp.dev:443.
4. Check that no firewall rules are blocking WebSocket connections to wss://oz.warp.dev.
5. Increase log verbosity with --log-level debug to see connection details.

See Security and networking for the full list of outbound endpoints the worker needs.

---

Tasks not being picked up

Cause: The worker isn’t running, the --host value doesn’t match the worker’s --worker-id, or the worker and task belong to different teams.

Fix:

1. Confirm the worker is running and connected. Check the worker logs for Successfully connected to server.
2. Verify the --host (or worker_host) value you passed matches your --worker-id exactly. Case-sensitive.
3. Ensure the worker’s team matches the team creating the task.

---

Metrics not appearing

Cause: The worker is running, but metrics do not reach Prometheus or your collector.

Fix:

1. For Prometheus, confirm OTEL_METRICS_EXPORTER=prometheus, bind to 0.0.0.0 in Docker or Kubernetes, and run curl -s localhost:9464/metrics.
2. Confirm no firewall or network policy blocks the metrics port.
3. For OTLP, confirm the collector endpoint is reachable and the protocol is correct.
4. For Helm, confirm metrics.enabled=true and check the Kubernetes Service or PodMonitor. A PodMonitor requires the Prometheus Operator CRDs.
5. Restart the worker with --log-level debug and check for metrics errors.

See Monitoring for the full setup guide.

---

Task failures

Cause: The task environment, resources, or dependencies failed.

Fix (all backends):

1. Review task logs in the cloud agent dashboard or via session sharing.
2. Use --no-cleanup to keep the container, Job, or workspace indefinitely for inspection. With Kubernetes cleanup enabled, failed Jobs remain for 24 hours by default.
3. Use --log-level debug to see detailed execution logs.

Docker backend (task failures)

1. Verify Docker is running (docker info).
2. If using a custom image, confirm it is glibc-based (not Alpine/musl) and that its architecture matches the worker’s Docker daemon platform.

Kubernetes backend (task failures)

The Helm chart’s worker.resources configures the worker Deployment, not task Jobs. Task containers have no worker-defined CPU or memory defaults. Configure the task container in kubernetesBackend.podTemplate or use a workload-specific runner instance shape. There is no universal task size; choose values based on the workload’s measured needs.

Start with the Pod state and events:

kubectl get jobs,pods -n NAMESPACE
kubectl describe pod -n NAMESPACE TASK_POD

Pod stays Pending with FailedScheduling

Verify: In kubectl describe pod, look for PodScheduled=False, FailedScheduling, Insufficient cpu, or Insufficient memory. Check the Pod’s requests and scheduling constraints against eligible node capacity.

Fix: Right-size requests, lower worker.maxConcurrentTasks, or add compatible capacity. Configure autoscaling for nodes that satisfy the Pod’s selectors, affinity, and tolerations. Raising only a limit does not make a Pending Pod schedulable.

Running container terminates with OOMKilled

Verify: Confirm the task container’s termination reason is OOMKilled. Compare peak memory use with its limit, then check Pod events and node pressure.

Fix: Reduce peak memory use or increase task memory in the runner instance shape or pod_template. An instance shape sets the memory request to the same value as the limit, so confirm an eligible node can schedule the larger request.

Pod is Evicted

Verify: Read the Pod reason and events for memory, disk, or node pressure.

Fix: Restore node headroom, reduce concurrency, or add capacity before rerunning the task. For voluntary disruption, see Protect active task pods from disruption.

Task exits with code 143

Exit code 143 generally indicates SIGTERM; it does not prove an out-of-memory failure.

Verify: Check the container termination reason and Pod events for eviction, preemption, node drain, the Job deadline, or manual deletion.

Fix: Address the recorded cause. Treat the failure as OOM only when Kubernetes reports OOMKilled.

Other Kubernetes failures

• Image pull errors - Confirm the image and architecture, registry access, and imagePullSecrets.
• CreateContainerConfigError or FailedMount - Use Pod events to find the missing Secret, ConfigMap, service account, volume, or key.
• Init container failure - Check each init container’s status and logs. Custom init containers must finish before task starts.
• Network failure - Test DNS, TLS, and the destination from a task Pod, not the worker Pod.
• Missing task credentials - Provide repository, registry, and application credentials through your approved Secret integration. The worker API key is not a task credential.

Direct backend (task failures)

1. Verify the Oz CLI is accessible.
2. Verify the workspace root directory has write permissions for the user running the worker.

---

Image pull failures

Docker backend (image pull)

1. If using a private registry, ensure Docker credentials are available to the worker. See Private Docker registries.
2. Try pulling the image manually on the worker host: docker pull <image>.

Kubernetes backend (image pull)

1. Configure imagePullSecrets in the pod_template section of your worker config.
2. Verify the Secret exists in the task namespace and contains valid credentials.

Both backends (image pull)

• Verify the image exists and the tag is correct.
• Check network connectivity from the worker/cluster to the registry.

---

Related pages

• Self-hosting overview — Architecture and decision guide.
• Self-hosted worker reference — CLI flags and config schema, including every flag mentioned here.
• Security and networking — Outbound endpoints the worker needs.
• Agent Session Sharing — Attach to running tasks to debug interactively.