> For the complete documentation index, see [llms.txt](https://fleet.hackmap.win/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://fleet.hackmap.win/concepts/architecture.md).

# Architecture

Three roles: one master, many workers, one Redis. The master runs a FastAPI app, owns the Redis store, and serves a dashboard. Workers are separate processes that dial out to the master over WebSocket and HTTP. Redis holds the persistent state. Workers never connect to each other; everything that needs to move between automations goes through Redis at the master.

A worker is pinned to one automation type and runs N concurrent slots inside it. The master can host many automation types side by side. Plugin code only runs inside workers; the master is automation-agnostic and discovers what to manage from installed Python entry-points.

## Topology

```mermaid
flowchart LR
  Client["client / dashboard / CLI"]
  Master["master<br/>(FastAPI + Uvicorn)"]
  Redis[("redis<br/>(AOF)")]
  W1["worker A1<br/>type: hello-world<br/>N slots"]
  W2["worker A2<br/>type: hello-world<br/>N slots"]
  W3["worker B1<br/>type: cf-cookie-harvester<br/>N slots"]

  Client -- "REST /api/v1<br/>+ /metrics" --> Master
  Master -- "blpop / hset /<br/>zadd / lrange" --> Redis
  W1 -- "WS control<br/>HTTP poll /config" --> Master
  W2 -- "WS control<br/>HTTP poll /config" --> Master
  W3 -- "WS control<br/>HTTP poll /config" --> Master
```

Workers never speak Redis directly. The master is the only Redis client. That keeps the Redis password off every worker host and means adding a federation or sharding layer later is just changing what the master does per frame.

## Master

A FastAPI app behind Uvicorn. It does five things:

Serves the dashboard at `/`.

Serves a REST API under `/api/v1` for managing automations, workers, configs, and streams. Admin endpoints require an admin bearer token; worker-facing endpoints require a worker bearer token.

Accepts a WebSocket connection from each worker at `/api/v1/automations/{type}/workers/{id}/control`. Workers send Register, State, Stats, and Output frames. The master sends ConfigChanged frames when an admin patches a config.

Owns the Redis store. Workers never touch Redis directly. Output streams, worker records, and BatchAutomation task queues all live in Redis under well-known keys.

Loads automation classes at startup via Python entry-points. The master sees every installed plugin and can serve its config schema as JSON to the dashboard for auto-form rendering.

### Batch task lifecycle

```mermaid
sequenceDiagram
  autonumber
  participant C as client
  participant M as master
  participant R as redis
  participant W as worker (slot)

  C->>M: POST /api/v1/.../tasks
  M->>R: rpush q:{type}:{tier}
  Note over W: blpop on q:{type}:{high,normal,low}
  W->>R: blpop pops task body
  R-->>W: envelope { id, payload, attempt }
  W->>R: zadd processing { id: now+vis }
  Note over W: run_one(payload)
  alt success
    W->>R: zrem processing
    W->>M: WS Output frame
    M->>R: stream_push (bounded)
  else error & attempt < max
    W->>R: zadd scheduled { body: now+backoff }
  else error & attempt >= max
    W->>R: rpush dlq:{type}
  end
  Note over M: master sweeper requeues<br/>visibility-expired tasks
```

## Worker

A long-running Python process pinned to one automation type. Started with `fleet worker --type <name>`.

On startup it imports the entry-point for that type, opens a WebSocket to the master, sends a Register frame with hardware info, and starts polling `/api/v1/automations/<type>/workers/<id>/config` every 30 seconds. The WebSocket is the low-latency notification channel; the HTTP poll is the safety net in case the WS notification is missed.

When a config arrives with a higher `config_gen` than the current observed config, the worker runs the reconcile diff and applies a plan. Plans are short: one of `stop_all`, `cold_start`, `restart`, `scale_up`, `scale_down`, `restart_pages`, or `noop`.

Slots are the unit of concurrent work. For a `ContinuousAutomation`, each slot is a Python task running the plugin's `run_slot` coroutine. For a `BatchAutomation`, each slot is a worker pulling tasks from a Redis BLPOP queue and running `run_one` per task.

The recycle loop on each worker runs every few seconds. It does two things: tear down any slot that has aged past `recycle_seconds`, and respawn any slot that was dropped due to a previous failure (the heal step). The heal step is what prevents the "I have zero browsers but my config says five" failure mode that an earlier prototype suffered from in production.

### Reconcile loop

```mermaid
flowchart LR
  N["new config arrives<br/>(WS ConfigChanged<br/>or HTTP poll)"] --> G{"gen &gt; current_gen?"}
  G -- "no" --> Skip[ignore]
  G -- "yes" --> D["diff applied vs desired"]
  D --> P{plan}
  P -- "noop" --> Skip
  P -- "stop_all" --> Stop[drain & stop slots]
  P -- "cold_start" --> Start[spawn slots]
  P -- "scale_up" --> Up[spawn extra slots]
  P -- "scale_down" --> Down[drain extras]
  P -- "restart_pages" --> Recycle[per-slot reset]
  P -- "restart" --> Restart[stop_all → cold_start]
  Stop & Start & Up & Down & Recycle & Restart --> Heal["heal: respawn any<br/>dropped slot to match<br/>target_workers"]
```

## Redis

The only persistent state. Layout in [redis-schema](/reference/redis-schema.md).

The master writes everything. Workers send their state and outputs over WebSocket frames; the master persists them. Workers don't speak Redis. This means the Redis password never leaves the master, and adding a federation layer later is just changing what the master does on each frame.

Streams are bounded by length and TTL. Default 50,000 entries and 3600 seconds. The bounded layout means a producer can run forever without a consumer and Redis memory doesn't blow up; the oldest get evicted.

## Plugins

A plugin is a pip package with a `fleet.automations` entry-point. The package contains an automation class (subclass of `ContinuousAutomation` or `BatchAutomation`) and a Pydantic config class (subclass of `BaseConfig`).

When the master starts, `importlib.metadata.entry_points(group="fleet.automations")` returns every installed plugin. The master loads each one. The `@register("name")` decorator on the automation class binds it to its type name. The dashboard renders one tab per registered automation.

Workers do the same thing but only load the entry-point matching their `--type` flag.

## Where the failure modes live

The master is a single point of failure. There's no built-in HA. The state is in Redis, so master restarts are recoverable as long as Redis has AOF on. Workers keep running through master restarts because they cache the last applied config locally and reject any reset attempt with `gen <= current_gen`.

Workers are not. If a worker process dies, its slots die with it. Docker `restart: unless-stopped` brings it back. The worker reconnects, sends Register, and the master tells it to cold-start with the same config.

Slots crash all the time. The framework treats this as expected. The heal step respawns them.

Network partitions between worker and master are handled by the WS reconnect logic and the 30-second HTTP poll. A partition that lasts hours doesn't break anything: the worker keeps running its current config, and the master keeps holding the desired one. They reconcile when the partition heals.


---

# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.

## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.

Perform an HTTP GET request on the current page URL with the `ask` query parameter, and the optional `goal` query parameter:

```
GET https://fleet.hackmap.win/concepts/architecture.md?ask=<question>&goal=<endgoal>
```

`ask` is the immediate question: it should be specific, self-contained, and written in natural language.
`goal` is optional and describes the broader end goal you are ultimately trying to accomplish on behalf of the user. GitBook uses it to tailor the answer towards what is most useful for that goal.

The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.