> For the complete documentation index, see [llms.txt](https://fleet.hackmap.win/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://fleet.hackmap.win/reference/backends.md).

# Backends

The master owns one `Backend` instance for the whole process. Workers never speak the backend protocol directly — they go through the master over WebSocket and the master persists everything. So "which backend you pick" is purely a master-side decision.

## Built-in backends

| URL scheme                                                     | implementation    | deps                           | use                                         |
| -------------------------------------------------------------- | ----------------- | ------------------------------ | ------------------------------------------- |
| `memory://` (or `mem://`)                                      | `InMemoryBackend` | none                           | tests, demos, ephemeral runs                |
| `sqlite:///path/to/db` (or `sqlite://:memory:`, `file:///...`) | `SqliteBackend`   | stdlib `sqlite3`               | single-host deploys, no Redis               |
| `redis://...` (or `rediss://...` for TLS)                      | `RedisBackend`    | `redis>=5.0.0` (already a dep) | multi-host, the canonical production option |

The CLI takes `--backend-url` (or `$FLEET_BACKEND_URL`) and routes through `fleet.core.backend.backend_from_url`. Third-party packages can register additional schemes via the `fleet.backends` entry-point.

## Picking one

**`memory://`** is for tests and `fleet master --backend-url memory://` smoke checks. It's also what `pytest --backend=memory` (the default) uses. Nothing persists across restarts. Pub/sub `event_publish` is a no-op (no subscribers in-process).

**`sqlite://`** is the right choice for "one VPS, one master, a handful of workers." One file on disk, AOF-equivalent durability from sqlite's WAL, no extra services to run. WAL mode + `busy_timeout=5000` gives concurrent reads without contention; writes serialize through a single connection guarded by `threading.Lock`. Every method dispatches via `asyncio.to_thread` so the event loop never blocks. `event_publish` is still a no-op — sqlite doesn't have pub/sub. See feature 1.5.

**`redis://`** is what the original CAPTCHA-solving fleet (8 hosts / 42 workers / 200+ ops/min) was built on. Use it when you need horizontal scaling, the atomic reserve-driven batch queue path, or a real pub/sub layer for `ctx.events.publish` / `subscribe`. AOF must be enabled (`--appendonly yes`) — otherwise a Redis restart wipes configs, queues, streams, everything. The bundled compose stack sets this correctly.

## What every backend implements

The `Backend` protocol covers:

* **kv** — `kv_get` / `kv_set` / `kv_del` / `kv_setnx` (the last is the atomic primitive idempotency keys are built on).
* **pool** — `pool_put` / `pool_list` / `pool_remove` / `pool_release` / `pool_claim_any` / `pool_size`.
* **lock** — `lock_acquire` / `lock_release`.
* **counter** — `counter_incr` / `counter_get` / `counter_set` / `counter_del`.
* **queue** (with 3 priority tiers + visibility timeout) — `queue_push` / `queue_size` / `queue_peek` / `queue_reserve` / `queue_ack` / `queue_nack` / `queue_sweep_expired` / `queue_dead`.
* **dlq** — `dlq_peek` / `dlq_pop` / `dlq_length` / `dlq_clear`.
* **stream** (bounded by length + TTL) — `stream_push` / `stream_pop` / `stream_peek` / `stream_length` / `stream_trim`.
* **events** — `event_publish` (no-op for memory + sqlite, Redis Pub/Sub otherwise).
* **workers** — register / get / list / remove / get\_config / set\_config / set\_state / set\_stats + `worker_stats_push` and `worker_stats_history` for the ring buffer behind the stats history endpoint.
* **catalog** — `catalog_set` / `catalog_get` / `catalog_all`.

The protocol is in `src/fleet/core/backend.py`. Conformance is verified by the backend suite in `tests/core/test_backend.py` — run it against any backend with `pytest --backend=<memory|redis|sqlite>`.

## Sqlite schema

If you want to poke at a sqlite-backed fleet with `sqlite3` directly:

```
kv(ns, key, value BLOB, expires_at)
pool_items(name, item_id, payload BLOB, tags JSON, expires_at)
pool_claims(name, item_id, until)
locks(name, token, expires_at)
counters(name, value)
queues(id, name, tier, body BLOB)          # tier in (high, normal, low)
queue_processing(name, task_id, body, deadline)
queue_scheduled(id, name, available_at, body)
queue_dead(id, name, body)
streams(id, name, score, body)
workers(automation_type, worker_id, hardware JSON, config JSON, config_gen,
        state, last_seen, last_error, stats JSON)
worker_stats_history(automation_type, worker_id, ts, frame JSON)
catalog(automation_type, doc BLOB)
```

WAL mode means a long-running `SELECT` doesn't block writes — read freely while the master is live. For backups, copy the `.db`, `.db-wal`, and `.db-shm` files together while the master is stopped, or use sqlite's online backup API.

## Redis schema

See [redis-schema](/reference/redis-schema.md) for the live key layout.

## Switching backends

Switching means losing state — no migration tooling exists. Practical sequence:

1. `docker compose down` (or systemctl stop) the master.
2. Drain workers first if you care: `POST /api/v1/automations/{type}/workers/{id}/drain` per worker.
3. Change `FLEET_BACKEND_URL` (and add any needed `depends_on:` / service entries for the new backend).
4. Bring the master up against the new backend — empty queues, empty pools, no worker history.
5. Workers reconnect, send `Register`, and the master treats them as fresh.

If you need to preserve workload state, do an export from the old backend (DLQ peek + pool list + queue peek) and a re-submit / re-`pool_put` against the new one. The dashboard's inspector views show you exactly what's in flight at the moment you cut over.


---

# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.

## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.

Perform an HTTP GET request on the current page URL with the `ask` query parameter, and the optional `goal` query parameter:

```
GET https://fleet.hackmap.win/reference/backends.md?ask=<question>&goal=<endgoal>
```

`ask` is the immediate question: it should be specific, self-contained, and written in natural language.
`goal` is optional and describes the broader end goal you are ultimately trying to accomplish on behalf of the user. GitBook uses it to tailor the answer towards what is most useful for that goal.

The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.