> For the complete documentation index, see [llms.txt](https://fleet.hackmap.win/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://fleet.hackmap.win/guides/deployment.md). # Deployment There's a canned stack at [`deploy/docker-compose/`](https://github.com/sarperavci/fleet/tree/main/deploy/docker-compose/README.md) — Redis (AOF) + master + Caddy + Prometheus + one demo worker. `cp .env.example .env`, fill in two tokens with `openssl rand -hex 32`, `docker compose up -d`, done. Read on for what the pieces are doing and how to take them apart. The rest of this guide reflects what's worked in production, reframed for the generic framework. ## Shape One master host. Public IP, behind a reverse proxy with TLS. Redis on the same host or a managed instance. N worker hosts. No inbound ports needed; workers only dial out. The master and workers don't need to be in the same network. As long as workers can reach `MASTER_URL` and the master can reach Redis, the deploy works. ## Master deployment The simplest functional setup uses Docker Compose with two services: Redis and the master. ```yaml # docker-compose.yml on the master host services: redis: image: redis:7-alpine command: redis-server --appendonly yes --save 60 1 volumes: - redis-data:/data restart: unless-stopped master: image: your-registry/fleet-master:latest environment: REDIS_URL: redis://redis:6379/0 FLEET_ADMIN_TOKEN: ${FLEET_ADMIN_TOKEN} FLEET_WORKER_TOKEN: ${FLEET_WORKER_TOKEN} ports: - "127.0.0.1:8000:8000" depends_on: - redis restart: unless-stopped volumes: redis-data: ``` Put Caddy or Nginx in front of port 8000 to terminate TLS and forward to the master. The master itself doesn't speak HTTPS — that's deliberate, because TLS is a deployment concern and varies per environment. The Redis AOF setting matters. Without it, a Redis restart wipes everything: configs, gen counters, output streams, queued tasks. The original farm hit this exact bug because the default compose had `--save ""` and `--appendonly no`. Don't repeat that mistake. ## Worker deployment One Docker container per worker. The container needs the framework, the specific plugin you're running, and an init process (`tini`). ```dockerfile FROM python:3.13-slim-bookworm RUN apt-get update && apt-get install -y tini && rm -rf /var/lib/apt/lists/* COPY . /app WORKDIR /app RUN pip install --no-cache-dir -e . -e examples/url-pinger ENTRYPOINT ["/usr/bin/tini", "--", "fleet", "worker"] ``` For browser-based plugins, the base image needs the Chromium stack. Reference the `turnstile_farm/docker/worker.Dockerfile` in the original repo — it installs Thorium, Brave, Xvfb, and the X11 libs in one layer. Run on each worker host: ```yaml services: worker: image: your-registry/fleet-worker:latest environment: MASTER_URL: https://fleet.example.com FLEET_WORKER_TOKEN: ${FLEET_WORKER_TOKEN} REDIS_URL: redis://localhost:6379/0 # only if your plugin uses redis directly command: ["--type", "url-pinger", "--id", "host1"] init: true # alternative to tini in the Dockerfile restart: unless-stopped ``` The `--id` should be unique per worker. The hostname is a reasonable default if you don't set it. Two workers with the same ID will fight for the same WebSocket and the same config. ## Bootstrap order Doesn't matter. Start the master first or last, doesn't matter. Workers will retry the WebSocket connection until they succeed. The master will accept registrations from workers that arrive in any order. The only thing that needs to be up-front is Redis. The master will fail to start if it can't connect. ## Updating For the master: ```bash docker compose pull master docker compose up -d master # --no-deps if you want to skip redis recreate ``` The `--no-deps` matters. Without it, Compose may recreate the Redis container, which wipes state if your volume isn't mounted correctly. AOF + a named volume is the durable setup; verify both before relying on a master update being non-destructive. For workers, rolling restarts work fine. Workers reconnect, sync state, and resume. The master sees a momentary `last_seen` gap and then a re-register. The original farm has a "redeploy\_v2.sh" script that captures a container's runtime config via `docker inspect`, pulls the new image, and recreates with the same env/limits. It's worth stealing the pattern for your own setup if you're not on Kubernetes. ## Auth in production Always set `FLEET_ADMIN_TOKEN` and `FLEET_WORKER_TOKEN` explicitly. The auto-generated tokens are fine for a smoke test but they rotate on every master restart, which means every worker disconnects. Tokens should be at least 32 hex characters. Generate with `openssl rand -hex 32`. The same worker token is used by every worker. If one worker host is compromised, the attacker can impersonate any worker — push fake outputs, fake stats, fake state. Per-worker tokens are on the roadmap; for now, treat the worker token like a shared secret and rotate when you suspect compromise. To rotate worker tokens without dropping any connection: 1. Generate the new token (`openssl rand -hex 32`). 2. Add it to `FLEET_WORKER_TOKENS` on the master alongside the existing primary. Both CSV (`new-tok, third-tok`) and JSON list (`["new-tok","third-tok"]`) are accepted. 3. Roll the new token out to workers one at a time, replacing their `FLEET_WORKER_TOKEN`. 4. Once every worker is on the new token, swap `FLEET_WORKER_TOKEN` on the master to the new value and drop the old one from `FLEET_WORKER_TOKENS`. The old token stops being accepted on the next master process restart. The admin token grants full control: read every config, patch every config, pop every stream, see every worker's stats. Keep it out of source control. Use a secrets manager. For multi-tenant or external-team setups, set `FLEET_TOKEN_SCOPES` so each token can only act on a subset of automation types: ```bash export FLEET_TOKEN_SCOPES='{ "team-a-token-abcd...": ["cf-scraper", "cf-cookie-harvester"], "ops-token-9999...": ["*"] }' ``` `*` grants every type. The admin token has `*` implicitly. Per-automation REST routes return 403 when the caller's token doesn't include the path's automation type. Bad JSON degrades to admin-only. Optionally bound submission throughput per token by setting `FLEET_SUBMIT_RPS_PER_TOKEN` (default 0 = unlimited). Over-cap requests get HTTP 429. The limiter is in-memory, so it's per-master-process; scale that with the rest of your master tier. ## TLS The master speaks plain HTTP and WebSocket. Run it behind a reverse proxy that terminates TLS — Caddy is the smallest path; nginx and Traefik also work. Critical: proxy must forward both `/` (HTTP) and the `/api/v1/automations/.../control` WebSocket upgrade. ### Caddy ```caddy fleet.example.com { reverse_proxy master:8000 } ``` That single line handles ACME, HTTPS, HTTP/2, and WebSocket upgrades. ### nginx ```nginx server { listen 443 ssl http2; server_name fleet.example.com; ssl_certificate /etc/letsencrypt/live/fleet.example.com/fullchain.pem; ssl_certificate_key /etc/letsencrypt/live/fleet.example.com/privkey.pem; location / { proxy_pass http://master:8000; proxy_set_header Host $host; proxy_set_header X-Real-IP $remote_addr; proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; proxy_set_header X-Forwarded-Proto https; # WebSocket upgrade — required for worker control sockets. proxy_http_version 1.1; proxy_set_header Upgrade $http_upgrade; proxy_set_header Connection $connection_upgrade; proxy_read_timeout 86400s; } } map $http_upgrade $connection_upgrade { default upgrade; "" close; } ``` Workers then point at `wss://fleet.example.com/...` automatically — `MASTER_URL=https://fleet.example.com` and the Agent rewrites the scheme to `wss://` for the WebSocket. Don't expose `/metrics` publicly. Either restrict by network policy / firewall, or add a second listener (`/metrics`) behind an internal-only IP, or scrape it via a Prometheus that lives in the same VPC. ## Monitoring The master exposes `/healthz` for liveness. Use it for your load balancer. The dashboard at `/` is the operator UI. Behind admin auth. Use it for ad-hoc inspection. ### Traces For distributed tracing, point an OTLP-compatible collector at the workers (the OpenTelemetry SDK ships in the base wheel): ```bash export OTEL_EXPORTER_OTLP_ENDPOINT=http://collector:4318 export OTEL_SERVICE_NAME=fleet-worker ``` Spans are emitted around the hot paths: `slot.run_continuous`, `slot.run_one`, `slot.reserve`, and `api.submit_task`. Every span carries `fleet.automation_type`, `fleet.worker_id`, `fleet.slot_id`, and (where applicable) `fleet.task_id` + `fleet.attempt` as attributes. If `opentelemetry-api` isn't installed or the env isn't set, the wrapper is a no-op and there's zero overhead. ### Metrics For real monitoring, the master exposes `/metrics` in Prometheus text-exposition format: `fleet_master_up`, `fleet_workers_total{automation_type,state}`, `fleet_worker_last_seen_age_seconds`, `fleet_worker_slots/cpu_pct/rss_mb`, `fleet_stream_length`, `fleet_queue_size`, `fleet_queue_dlq_length`, `fleet_pool_size`. The compose stack in `deploy/docker-compose/` includes a pre-wired Prometheus that scrapes it. Keep that endpoint off the public Internet — the Caddyfile in the stack 404s `/metrics` at the edge and Prometheus connects over the compose network. Worker logs are stdout. `docker logs -f turnstile-worker` is the canonical view. Centralize them with Loki, Datadog, or whatever you already have. ## Scaling out Add worker hosts. Each new worker gets a unique `--id`, connects, and waits for an admin PATCH to enable it. There's no automatic load balancing because there's no shared queue at the type level — each worker has its own slot count, and you set `slots` per worker. For BatchAutomation, scaling is more natural: bump the number of workers, each pulls from the same `tasks::pending` BLPOP queue, throughput scales linearly until the queue producer is the bottleneck. For ContinuousAutomation, scaling means adding workers and giving each one a `slots` count that adds up to your target. If you need a homogeneous fleet, write a script that patches all workers with the same desired count. --- # Agent Instructions This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com. ## Querying This Documentation If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question. Perform an HTTP GET request on the current page URL with the `ask` query parameter, and the optional `goal` query parameter: ``` GET https://fleet.hackmap.win/guides/deployment.md?ask=&goal= ``` `ask` is the immediate question: it should be specific, self-contained, and written in natural language. `goal` is optional and describes the broader end goal you are ultimately trying to accomplish on behalf of the user. GitBook uses it to tailor the answer towards what is most useful for that goal. The response will contain a direct answer to the question and relevant excerpts and sources from the documentation. Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.