> For the complete documentation index, see [llms.txt](https://fleet.hackmap.win/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://fleet.hackmap.win/operations/troubleshooting.md).

# Troubleshooting

Symptoms that have actually happened, with their causes and fixes.

## Worker says "ws connected" but the dashboard shows it as offline

The worker's WebSocket is up but the master isn't getting state frames. Three causes:

The worker can't reach Redis directly (only relevant if your plugin uses `ctx.stream` or `ctx.events`). Symptom: the worker logs Redis connection errors. Fix: set `REDIS_URL` on the worker host.

The stats reporter hasn't fired yet. It runs every 15 seconds. If the worker came up 5 seconds ago, it's normal. Wait 30 seconds and refresh.

The worker token is the admin token (or vice versa). Symptom: the WS connects but the HTTP poll returns 403. Fix: set `FLEET_WORKER_TOKEN` to the worker token, not the admin token.

## Worker has zero slots but the config says N

The config is enabled but `len(slots)` is 0. Three causes:

The worker just restarted and hasn't done its first reconcile yet. Wait 5 seconds.

The config has `gen=0` but the worker has a cached `gen=N`. This happens when the master's Redis is wiped and the worker has been running through the wipe. The worker refuses to apply `gen <= current_gen`. Fix: PATCH the config a few times to bump the master's gen above the worker's. The first PATCH that exceeds the worker's cached gen takes effect.

The plugin's `run_slot` raised before `ctx.shutdown` was set, and the heal loop is failing to respawn. Check the worker logs for `slot factory failed` or `heal spawn failed`. The cause is plugin-specific.

## "RuntimeError: can't start new thread"

The worker container's pids cgroup is saturated. Almost always means a child process leak. The original Turnstile farm hit this when Chromium subprocesses were reparented to PID 1 (the Python agent) and never reaped.

Fix:

Add `tini` to the worker Dockerfile and use it as the entrypoint, or set `init: true` in the compose service.

If using `fleet-browser`, the pool's `_kill_processes_for_userdir` cleanup runs on slot stop and catches most leaks. Make sure your Dockerfile is recent.

Check `ps -ef | wc -l` inside the container. A healthy browser worker has a few hundred processes. If you're seeing 5000+, the leak is real.

## "WebSocket connection failed: HTTP 403"

The worker token is wrong. Triple-check `$FLEET_WORKER_TOKEN` matches on both ends.

If it's the `?token=` query param: URL-encoding. The token shouldn't contain reserved characters; `openssl rand -hex 32` outputs only `[0-9a-f]` which is always safe.

## Master crashes on startup with "loaded automations: \[]"

No plugins are installed. The master itself doesn't include any automations — they all come from pip-installed entry-points. Install at least one:

```bash
pip install -e examples/hello-world
```

If a plugin is installed (`pip show <name>` works) but doesn't appear, the entry-point line in its `pyproject.toml` is wrong. Re-check:

```toml
[project.entry-points."fleet.automations"]
my-name = "my_package:MyAutomation"
```

The dotted path must point at the class object, not the module. `my_package` and `my_package:MyAutomation` are different.

## All workers go to IDLE and stop emitting

The master pushed a config with `enabled=false`. Either someone admin-patched it or the master restarted and Redis state was wiped (default config has `enabled=false`).

Check the config:

```bash
curl -s -H "Authorization: Bearer $ADMIN" \
  http://master/api/v1/automations/X/workers/Y | jq .config
```

If `enabled` is false, PATCH it true. If the config is empty, the master lost state. See [Redis schema](/reference/redis-schema.md) for the persistence section — AOF should be on.

## Stream is empty even though workers say they're running

Three causes:

The workers are running but not emitting. Check worker logs for slot-level errors.

A consumer is popping faster than producers can keep up. Check the producer rate (`current_stats.metrics.counter.<name>`) vs. the pop rate.

The stream cap is too aggressive. Default 50k / 3600s. If you have a burst of 100k emits in 5 minutes with no consumer, the oldest 50k were evicted before any pop.

`peek` instead of `pop` to verify entries exist without consuming.

## Dashboard shows wrong worker count

The master's worker list comes from `SMEMBERS wkrs:<type>`. Workers are never removed automatically. A worker that's been offline for weeks still shows up.

The dashboard doesn't filter by `last_seen` today. Until it does, look at the `last_seen` timestamp on each row — anything more than a couple minutes ago is probably gone.

To clean up:

```bash
redis-cli srem wkrs:my-type ghost-worker-id
redis-cli del wkr:my-type:ghost-worker-id:hardware \
              wkr:my-type:ghost-worker-id:config \
              wkr:my-type:ghost-worker-id:gen \
              wkr:my-type:ghost-worker-id:state \
              wkr:my-type:ghost-worker-id:stats
redis-cli srem wkrs:all 'my-type|ghost-worker-id'
```

## "validation failed for type='register'"

The worker is sending a frame the master can't decode. Almost always means a protocol-version mismatch — the worker is on a newer/older version of the framework than the master.

The framework versions all components together. Mixing `fleet-core` versions across master and workers is unsupported. Pin them to the same git SHA or pypi version.

## Worker process disappears with no error

`OOMKilled` by the Linux OOM killer. The slot count exceeds available RAM. `dmesg` on the host shows it.

Either reduce `slots` or move to a larger host. The framework doesn't know about cgroup limits; if you tell it 16 slots and the host can only hold 8, it will try and the kernel will kill the process.


---

# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.

## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.

Perform an HTTP GET request on the current page URL with the `ask` query parameter, and the optional `goal` query parameter:

```
GET https://fleet.hackmap.win/operations/troubleshooting.md?ask=<question>&goal=<endgoal>
```

`ask` is the immediate question: it should be specific, self-contained, and written in natural language.
`goal` is optional and describes the broader end goal you are ultimately trying to accomplish on behalf of the user. GitBook uses it to tailor the answer towards what is most useful for that goal.

The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.