> For the complete documentation index, see [llms.txt](https://fleet.hackmap.win/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://fleet.hackmap.win/operations/scaling.md). # Scaling The framework scales horizontally by adding worker hosts and vertically by bumping `slots` per worker. Each axis has different limits. ## Vertical: more slots per worker `slots` is set per worker via the config. Bumping it triggers the reconcile loop's `scale_up` action: new slots are spawned without disturbing existing ones. Scaling down is the symmetric `scale_down` — the oldest slots are torn down first. The hard limit is `slots <= 1024` in `BaseConfig`. The practical limit is whatever your host can run. Some rules of thumb from the original fleet: A browser slot: 250-500 MB resident, \~0.5-1 CPU core when busy. On a 4-core / 8 GB host, 4 slots is comfortable, 8 is the upper end. Beyond that, the Linux PID cgroup or RAM becomes the limit. An HTTP-only slot (httpx loop, no parsing): negligible RAM, mostly IO-bound. 100+ slots per host is reasonable. The bottleneck moves to the upstream — proxy throughput, target rate limits. A CPU-heavy slot (parsing, hashing, regex over large inputs): one per core. Bumping past `cores * 2` makes everything slower. The framework doesn't enforce these. It will try to spawn whatever you ask for, log errors when spawning fails, and the heal step will keep retrying. If you ask for 100 browser slots on a 4 GB host, you'll see `OOMKilled` containers and eventually `cannot allocate memory` in the worker logs. ## Horizontal: more worker hosts Each new worker host runs `fleet worker --type X --id host-N`. The master tracks them independently; each has its own config and its own `slots` count. For BatchAutomation, this is straightforward: every worker BLPOPs from the same `tasks::pending` queue. Throughput scales linearly with worker count until either the producer is the bottleneck or Redis is. For ContinuousAutomation, scaling is per-worker. You set `slots` on each, and the total is the sum. There's no built-in "total fleet target" you can patch — the framework treats each worker as independent. If you want a homogeneous fleet, write a script that PATCHes the same config to every worker: ```bash for id in $(curl -s -H "Authorization: Bearer $ADMIN" \ http://master/api/v1/automations/X/workers | jq -r '.[].worker_id'); do curl -X PATCH http://master/api/v1/automations/X/workers/$id/config \ -H "Authorization: Bearer $ADMIN" \ -H "Content-Type: application/json" \ -d '{"slots": 4, "enabled": true}' done ``` ## Master capacity One master per fleet. The master is FastAPI + Uvicorn, single-process by default. It can comfortably handle: A few hundred concurrent WebSockets. A few thousand HTTP requests per second (mostly `pop` and PATCH). ZSET pushes at the rate Redis can sustain — single-threaded Redis tops out around 100k ops/sec on modern hardware, and each `ctx.emit` is one `ZADD` plus a pubsub publish. Beyond that, you're outside the framework's design target. Sharding by automation type onto separate master processes works but is manual. Federation across multiple masters isn't built. ## Redis capacity Redis is the bottleneck before the master is, in most workloads. The hot paths: `ZADD stream:` on every output emit. `ZPOPMIN` on every consumer pop. `HGET`/`HSET` on worker state updates (every \~15 seconds per worker). `PUBLISH event:...` on every emit. A single Redis instance can handle a fleet of \~50 workers each producing a few outputs per second. Past that, consider: Splitting Redis by automation type. The store reads `wkr::*` and `stream:` — these can live on different Redis instances if the master code is patched accordingly. Moving to Redis Cluster for horizontal scale. The framework's keys are not cluster-aware today; you'd need to add hash tags. Replacing Redis with a more scalable store for streams (Kafka, NATS JetStream). This is a meaningful framework change. In practice, none of the workloads we've run have come close to either Redis or master limits. Browser farms are slot-count-bound, not throughput-bound. ## Stream cap and TTL Default 50,000 entries and 3600-second TTL per stream. Both are tunable via the push call's parameters; neither is configurable at runtime through the API today. If your stream has thousands of entries waiting unconsumed, either the consumer is slow or the cap is too low. Bumping the cap is a temporary fix; making the consumer keep up is the real fix. If your stream is small (always under a few hundred entries), you're fine. The bounded design is to prevent runaway producers from filling Redis; it's not a tuning knob you need to think about in the steady state. ## When you've outgrown the framework If you find yourself wanting: Per-region routing, with workers in region X only seeing tasks from region X. A scheduler that decides when to run things based on time / external events. Strict exactly-once delivery on the stream. Multi-master federation. Sub-millisecond latency. You've outgrown the framework. Build the missing piece outside it, or pick a different tool. The framework is designed for fleets in the low hundreds of workers, total outputs in the millions per day, latency in the seconds-to-minutes range. It's good at that shape and bad at most others. --- # Agent Instructions This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com. ## Querying This Documentation If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question. Perform an HTTP GET request on the current page URL with the `ask` query parameter, and the optional `goal` query parameter: ``` GET https://fleet.hackmap.win/operations/scaling.md?ask=&goal= ``` `ask` is the immediate question: it should be specific, self-contained, and written in natural language. `goal` is optional and describes the broader end goal you are ultimately trying to accomplish on behalf of the user. GitBook uses it to tailor the answer towards what is most useful for that goal. The response will contain a direct answer to the question and relevant excerpts and sources from the documentation. Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.