> For the complete documentation index, see [llms.txt](https://fleet.hackmap.win/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://fleet.hackmap.win/concepts/workers-and-slots.md). # Workers and slots Workers are processes. Slots are concurrent units of work inside one worker. Both terms get overloaded in fleet conversations so it's worth nailing them down. ## Worker A `fleet worker --type ` invocation. One process, pinned to one automation type. It owns: A WebSocket to the master. A periodic HTTP poll to the master (the safety net for WebSocket drops). A reconcile loop, which compares the latest pushed config to the locally observed one and applies a plan. A recycle loop, which ages out old slots and respawns missing ones. A pool of slot tasks — one per active slot. A periodic stats reporter that aggregates per-slot metrics and sends them to the master. The worker has a stable `worker_id` (typically the hostname; overridable with `--id`). The master uses this ID to address it. Two workers with the same ID will fight for the same WebSocket and the same config; don't do this. A single host can run multiple workers of different types side by side. Each is its own process with its own WS connection. On a 16-core box you might have one worker for `turnstile-solver` and another for `account-creator`, both reporting to the same master. ## Slot A unit of concurrent work inside a worker. Created and destroyed by the reconcile loop based on the config's `slots` field. For a `ContinuousAutomation`, a slot is an `asyncio.Task` running `run_slot(ctx)`. The slot ends when that coroutine returns or raises. The framework then runs the plugin's `cleanup` hook. For a `BatchAutomation`, a slot is an `asyncio.Task` running an internal loop that blocks on `BLPOP tasks::pending`. Each pop yields one task; the slot calls `run_one(task, ctx)` and loops. The slot ends when `ctx.shutdown` fires. Slot IDs are integers, stable across the lifetime of one slot, starting at 0. They reset when the worker restarts. They are not unique across workers — two workers each with 3 slots will both have slots 0, 1, 2. ## How many slots per worker? Depends on what the slot does. Some heuristics: A browser-based slot eats about 250-500 MB of RAM and one CPU core when busy. On a 4-core / 8 GB VPS, 4 slots is plausible; 8 is pushing it. An HTTP-based slot (httpx in a loop) is cheap. Hundreds per worker is fine. The bottleneck becomes the proxy, not the host. A scrape-and-parse slot with a heavy parsing step (Playwright + DOM extraction) is somewhere in between. Profile and measure. The framework doesn't enforce any limit. If you ask for 1024 slots, it will try. Your OS will tell you when it's unhappy. ## Recycle `recycle_seconds` in the config sets the maximum age of any slot before it gets torn down and replaced. Default is 0 — no recycling. You'd typically set this to a non-zero value when slot freshness matters. Browser sessions accumulate fingerprint history that anti-bot systems use to raise the friction score. Killing the browser every 10 minutes resets that history. The same logic doesn't apply to an HTTP-only slot, so leave `recycle_seconds=0` for those. The recycle loop staggers the initial fleet so they don't all expire at once. With `slots=8` and `recycle_seconds=600`, the slots are seeded with virtual ages 0, 75, 150, ..., 525 — so one slot gets recycled every \~75 seconds rather than all 8 at the 10-minute mark. ## Failure Slot crashes are normal. The plugin's `run_slot` or `run_one` raises an exception; the framework logs it, runs `cleanup`, and removes the slot from the pool. The next recycle tick notices `len(slots) < desired.slots` and the heal step respawns it. Worker crashes are also normal. Docker `restart: unless-stopped` brings the process back. The worker reconnects with the same `worker_id`, the master sends it the existing config, reconcile runs `cold_start`, and the slots come back. Master crashes are recoverable as long as the master's Redis has AOF on. Workers don't care: they cache the last applied config, refuse any older gen, and keep running until either the master comes back or the worker itself restarts. ## Graceful drain To take a worker out of rotation without losing in-flight work, `POST /api/v1/automations/{type}/workers/{id}/drain`. The master sends a `Drain` frame over the worker's control WebSocket; the worker stops reserving new tasks, lets the active `run_one` / `run_slot` finish, runs each slot's `cleanup` hook, and then exits. Its `state` reports `DRAINING` until the WS drops. Use drain before `DELETE /api/v1/automations/{type}/workers/{id}` if you care about the current task. Use the DELETE without drain when the worker is already gone (host died, container OOM-killed) and you just want the master to stop showing it. ## Per-slot stats Every Stats frame the worker emits carries a `per_slot` list: one entry per slot with `slot_id`, `age_seconds` (uptime since the slot's `started_at`), and `counters` (the slot's `ctx.metrics` snapshot). This is what surfaces "one bad slot" — sort by `age_seconds` to find slots that should have recycled but didn't, or by counter values to find a slot doing zero useful work while its peers churn. The master writes every Stats frame into a 240-entry-per-worker ring buffer; query via `GET /api/v1/automations/{type}/workers/{id}/stats?since=&limit=`. See [observability](/operations/observability.md). --- # Agent Instructions This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com. ## Querying This Documentation If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question. Perform an HTTP GET request on the current page URL with the `ask` query parameter, and the optional `goal` query parameter: ``` GET https://fleet.hackmap.win/concepts/workers-and-slots.md?ask=&goal= ``` `ask` is the immediate question: it should be specific, self-contained, and written in natural language. `goal` is optional and describes the broader end goal you are ultimately trying to accomplish on behalf of the user. GitBook uses it to tailor the answer towards what is most useful for that goal. The response will contain a direct answer to the question and relevant excerpts and sources from the documentation. Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.