> For the complete documentation index, see [llms.txt](https://fleet.hackmap.win/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://fleet.hackmap.win/operations/observability.md).

# Observability

Three signals out of the box: metrics (Prometheus text format), traces (optional OpenTelemetry export), and a built-in stats history ring buffer for ad-hoc inspection. Logs are stdout — emit them however your platform already does (json mode flips the default formatter, see below).

## Metrics

The master serves Prometheus text-exposition format at `GET /metrics`. No auth in-process — the endpoint is meant to be scraped over a private network. The bundled Caddyfile in `deploy/docker-compose/` 404s `/metrics` at the public edge for this reason.

Metric names:

| metric                               | labels                         | meaning                                                      |
| ------------------------------------ | ------------------------------ | ------------------------------------------------------------ |
| `fleet_master_up`                    | —                              | always 1 while the master is responding                      |
| `fleet_workers_total`                | `automation_type`, `state`     | count of workers in each state (IDLE/RUNNING/ERROR/DRAINING) |
| `fleet_worker_last_seen_age_seconds` | `automation_type`, `worker_id` | seconds since the worker last sent a frame                   |
| `fleet_worker_slots`                 | `automation_type`, `worker_id` | slot count from the latest Stats frame                       |
| `fleet_worker_cpu_pct`               | `automation_type`, `worker_id` | most-recent CPU% sampled by the worker                       |
| `fleet_worker_rss_mb`                | `automation_type`, `worker_id` | RSS in MB                                                    |
| `fleet_stream_length`                | `automation_type`              | output stream length                                         |
| `fleet_queue_size`                   | `automation_type`              | summed across all 3 priority tiers                           |
| `fleet_queue_dlq_length`             | `automation_type`              | DLQ size — alert on this                                     |
| `fleet_pool_size`                    | `pool`                         | pool item count                                              |

Useful alerts:

* `fleet_queue_dlq_length > 0` for 5 minutes → tasks are failing systematically.
* `max_over_time(fleet_worker_last_seen_age_seconds[2m]) > 60` → worker stopped reporting.
* `rate(fleet_queue_size[5m]) > 0` and `fleet_workers_total{state="RUNNING"} == 0` → producers are filling the queue with no consumers.

The compose stack in `deploy/docker-compose/` pre-wires a Prometheus that scrapes `master:8000/metrics` every 15s. See feature 4.1.

## Stats history

Every `Stats` frame the master receives is appended to a per-worker ring buffer in the backend (`worker_stats_push`). Read via:

```
GET /api/v1/automations/{type}/workers/{id}/stats?since=<unix_ts>&limit=240
```

Default ring size is 240 frames per worker — at the worker's 15-second emit cadence that's the last \~60 minutes. Each frame has the full Stats shape: cpu/rss/slots, the aggregated counters from `ctx.metrics`, and the `per_slot` array with each slot's `slot_id`, `age_seconds`, and per-slot counters.

The `per_slot` view (feature 4.3) is what you want for "one bad slot in the fleet" investigations — sort by `age_seconds` descending to find slots that have been alive far longer than `recycle_seconds` (the recycle loop is broken), or by counter values to find the slot doing zero useful work.

This is held in the active backend, so when you swap from `memory://` to `redis://` or `sqlite:///...` the same query just works. See features 4.3 + 4.4.

## Traces (OpenTelemetry)

The OpenTelemetry SDK ships in the base wheel. Enable tracing and point an OTLP/HTTP collector at the workers and master:

```bash
export OTEL_EXPORTER_OTLP_ENDPOINT=http://collector:4318
export OTEL_SERVICE_NAME=fleet-worker   # or fleet-master
# any other OTEL_* env you need: OTEL_RESOURCE_ATTRIBUTES, OTEL_TRACES_SAMPLER, ...
```

If `opentelemetry-api` isn't installed or no `OTEL_EXPORTER_*` env is set, the instrumentation falls back to `contextlib.nullcontext` and costs nothing.

Spans currently emitted:

| span                  | when                                          | attributes                                                  |
| --------------------- | --------------------------------------------- | ----------------------------------------------------------- |
| `slot.run_continuous` | once per slot lifetime (ContinuousAutomation) | `fleet.automation_type`, `fleet.worker_id`, `fleet.slot_id` |
| `slot.reserve`        | each BLPOP wait inside a batch slot           | as above                                                    |
| `slot.run_one`        | each batch task                               | as above + `fleet.task_id`, `fleet.attempt`                 |
| `api.submit_task`     | each `/tasks` POST on the master              | `fleet.automation_type`, `fleet.task_id`, `fleet.priority`  |

Exceptions raised inside a `slot.run_one` are attached to the span via `record_exception` so the trace surfaces the failure type alongside the latency. See feature 4.5.

You can also activate it without any `OTEL_EXPORTER_*` by setting `FLEET_OTEL_ENABLED=1` — useful if you've configured a collector via the standard OTel SDK auto-config but want to be explicit.

## Logs

Default formatter is human-readable; set `FLEET_LOG_FORMAT=json` to switch to single-line JSON, suitable for Loki/Datadog/whatever consumes structured logs.

`fleet.core.logging.bind_log_context(...)` threads contextvars (`automation_type`, `worker_id`, `slot_id`, `task_id`) into every log record produced while the binding is active. The slot runner already binds these around `run_slot` / `run_one`, so plugin log lines from your code show up with the right scoping fields without you doing anything.

## Picking signals per problem

* **Capacity / throughput / SLO** → Prometheus.
* **One bad slot or one bad worker** → stats history `per_slot` + worker drain.
* **Why does this one task take so long** → OTel traces, find the slow span.
* **Why did this specific run fail** → logs (with the bound `task_id`), then DLQ inspection via the dashboard.


---

# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.

## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.

Perform an HTTP GET request on the current page URL with the `ask` query parameter, and the optional `goal` query parameter:

```
GET https://fleet.hackmap.win/operations/observability.md?ask=<question>&goal=<endgoal>
```

`ask` is the immediate question: it should be specific, self-contained, and written in natural language.
`goal` is optional and describes the broader end goal you are ultimately trying to accomplish on behalf of the user. GitBook uses it to tailor the answer towards what is most useful for that goal.

The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.