> For the complete documentation index, see [llms.txt](https://fleet.hackmap.win/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://fleet.hackmap.win/reference/redis-schema.md). # Redis schema Every key Fleet writes. Plugins should not touch most of these directly — the typed `Backend` API and the `ctx.*` primitives wrap them. The schema is documented here for operational tasks: backups, manual inspection, ad-hoc cleanup, sizing. Keys are prefixed `fleet:` (primitives) or `wkr:` / `wkrs:` / `stream:` / `event:` (worker state, streams, event bus). If you run another app against the same Redis DB, these prefixes keep you out of trouble. ## Per-worker state ``` wkr:::hardware string JSON Hardware payload wkr:::config string JSON merged config dict wkr:::gen string monotonic int as ASCII wkr:::state hash state=IDLE|RUNNING|ERROR, last_seen=, last_error= wkr:::stats string JSON last Stats frame ``` `config_gen` increments on every PATCH to `/config`. Workers reject pushes with `gen <= current_gen` to prevent regression from a stale master. ## Indices ``` wkrs: set every worker_id of this type ever registered wkrs:all set every "|" for cross-type listings ``` Workers are never removed from these sets automatically. A worker that's been offline for weeks still appears in `wkrs:`. Filter by `last_seen` if you need active-only. ## Output streams ``` stream: zset score= member= ``` Bounded by default to 50,000 members and 3600 seconds of TTL. Push trims both. Pop is `ZPOPMIN`, peek is `ZRANGE 0 N`, length is `ZCARD`. ## Catalog ``` fleet:catalog hash automation_type -> JSON {config, output, task_payload} ``` The master writes this on startup by introspecting registered automations. Plugins read it (via the REST API or the backend) to look up an automation's declared schemas. See [contracts](/concepts/contracts.md). One entry per registered automation type. Cleared and re-populated on every master restart. ## Pools ``` fleet:pool: hash item_id -> JSON {p (b64 payload), t (tags), x (expires_at)} fleet:pool::claim: string "1", with TTL = hold_seconds ``` A pool is a hash of items keyed by item\_id. Each item carries an optional per-item expiry — Fleet prunes expired items lazily on every put and on every list. Claims are separate keys with a short TTL. A claim is an exclusive lease: while the key exists, no other consumer can claim the same item. When the slot crashes mid-work, the claim key expires and the item becomes claimable again. See [primitives](/concepts/primitives.md) and the [pool walkthrough](/guides/cf-cookies-pool.md). ## Locks ``` fleet:lock: string , with TTL = hold_seconds ``` Token-based mutex. `lock_acquire` does `SET NX EX` with a random token; `lock_release` runs a Lua script that deletes the key only if the stored token matches the caller's. This prevents the race where a lock expires, a different slot acquires it, and the original holder's later `release` accidentally drops the new holder's lock. ## Counters ``` fleet:counter: string integer ``` Atomic `INCRBY`. No TTL — counters live until you delete them. ## KV ``` fleet:kv:: string bytes (typically JSON), with optional TTL ``` Namespaced key-value store. The `ns` segment is the plugin-supplied namespace; the master never inspects it. Use it to scope keys per-automation (`scraper-inflight`, `harvester-state`, etc.) so different plugins don't step on each other. ## Task queues (BatchAutomation) ``` fleet:queue: list JSON {"id", "payload"} — pending tasks fleet:queue::processing zset task_id -> visibility deadline (reserved/in-flight) fleet:queue::processing:body hash task_id -> body (kept until ack) fleet:queue::scheduled zset body -> ready_at (nack'd retries, delayed) fleet:queue::dead list dead-lettered bodies (DLQ) ``` The pending list for a batch automation. `ctx.submit(QUEUE, payload)` does `RPUSH`. The master also accepts submissions over REST at `POST /api/v1/automations//tasks` — payloads are validated against the automation's declared `TaskPayload` schema before being pushed. Reserve is atomic: a worker slot pops the head and registers it in `:processing` (with a visibility deadline) in a single `LPOP`+`ZADD`+`HSET` Lua call, so a crash between pop and register can't lose the task. A body with no usable `id` is delivered once and left unregistered rather than looping forever. Retry and dead-letter are built in. On a `run_one` exception the slot `nack`s the task with exponential backoff — the body moves to `:scheduled` and is drained back onto the queue when `ready_at` passes (atomically, so concurrent reservers don't duplicate it). After `max_attempts` the task lands in `:dead`. The master also sweeps `:processing` every 10s and re-queues any reservation past its visibility deadline, so a crashed slot that never `nack`s still recovers. See [batch automation](/guides/batch-automation.md). ## Event bus ``` event: pubsub channel JSON payloads ``` No persistence. Subscribers receive messages published while they're subscribed, miss everything else. Common topics: `event:.emit` — published on every `ctx.emit` from any worker of that type. Currently used internally by the stream-push pipeline; plugins can subscribe if they want a low-latency signal. Plugin-defined topics — anything you publish via `ctx.events.publish("topic", {...})`. ## Cleaning up To remove all state for an automation type: ```bash redis-cli --scan --pattern 'wkr:my-type:*' | xargs redis-cli del redis-cli del wkrs:my-type stream:my-type fleet:queue:my-type redis-cli hdel fleet:catalog my-type ``` To remove a pool (e.g., to invalidate every cookie at once): ```bash redis-cli del fleet:pool:cf-cookies redis-cli --scan --pattern 'fleet:pool:cf-cookies:claim:*' | xargs redis-cli del ``` To remove one worker: ```bash redis-cli del \ wkr:my-type:host1:hardware \ wkr:my-type:host1:config \ wkr:my-type:host1:gen \ wkr:my-type:host1:state \ wkr:my-type:host1:stats redis-cli srem wkrs:my-type host1 redis-cli srem wkrs:all 'my-type|host1' ``` These are manual operations — there's no admin API for them yet. ## Persistence The master assumes AOF is on. Configs, gens, the worker indices, and pool contents need to survive Redis restarts; if they don't, every worker disconnects and reconnects with cached gens, the master can't push updates until its gen exceeds the workers' cached values, and any pooled state (cookies, accounts, fingerprints) is gone until producers refill. Streams and event-bus messages can be lost without correctness consequences — workers will just produce new entries. Counters, KV, queue contents, and pool contents are operationally painful to lose. The framework does not enable AOF for you. Set it in your Redis config: ``` appendonly yes appendfsync everysec save 60 1 ``` --- # Agent Instructions This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com. ## Querying This Documentation If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question. Perform an HTTP GET request on the current page URL with the `ask` query parameter, and the optional `goal` query parameter: ``` GET https://fleet.hackmap.win/reference/redis-schema.md?ask=&goal= ``` `ask` is the immediate question: it should be specific, self-contained, and written in natural language. `goal` is optional and describes the broader end goal you are ultimately trying to accomplish on behalf of the user. GitBook uses it to tailor the answer towards what is most useful for that goal. The response will contain a direct answer to the question and relevant excerpts and sources from the documentation. Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.