> For the complete documentation index, see [llms.txt](https://fleet.hackmap.win/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://fleet.hackmap.win/guides/cf-cookies-pool.md). # CF cookies pool walkthrough End-to-end walkthrough of the cleanest cross-automation pattern Fleet supports: two plugins on different machines coordinating through a typed shared resource, with no raw Redis anywhere in the plugin code. The setup is the canonical motivating case: bypassing Cloudflare's clearance challenge by running a fleet of headed browsers that harvest cookies and a much larger fleet of HTTP scrapers that consume them. Browsers are slow and expensive; cookies are cheap to reuse. Separating them lets each scale independently. This guide implements both halves on top of the framework's pool, queue, and submit primitives. ## Architecture `cf-cookie-harvester` is a `BatchAutomation`. It only does work when given a task. The task says "harvest a cookie for site X". The harvester opens a browser, waits for the CF challenge to clear, captures the cookies and the user agent, then writes a typed entry into the shared pool. `cf-scraper` is a `ContinuousAutomation`. Each slot runs a steady scrape loop. On every iteration it checks the pool for a fresh cookie for its target site. If the pool is thin, it submits a harvest task. Either way, it claims a cookie with a short hold, makes the HTTP request using the proxy and user-agent that minted the cookie, and emits the result. The pool decouples the two. The harvester doesn't know who's consuming; the scraper doesn't know who's producing. Adding a different harvester (Playwright, undetected-chromedriver, a manual queue of cookies pasted by an operator) doesn't change a single line of the scraper. ## The three packages A contracts package, a harvester plugin, a scraper plugin. The contracts package is the only thing both plugins depend on. ### cf-contracts ```python # cf_contracts/__init__.py from pydantic import BaseModel from fleet.core import Pool, Queue, Stream class CfCookie(BaseModel): cookies: dict[str, str] user_agent: str proxy_url: str | None site: str class CfCookieTags(BaseModel): site: str fresh: bool = True class HarvestRequest(BaseModel): site: str class ScrapeResult(BaseModel): url: str status: int body_len: int COOKIE_POOL = Pool[CfCookie, CfCookieTags]("cf-cookies", payload=CfCookie, tags=CfCookieTags) HARVESTER = Queue[HarvestRequest]("cf-cookie-harvester", payload=HarvestRequest) SCRAPER_OUT = Stream[ScrapeResult]("cf-scraper", payload=ScrapeResult) ``` No runtime code. Schemas plus three typed references. The references are the strings that couple the two plugins, declared exactly once. ### cf-cookie-harvester ```python # cf_cookie_harvester/__init__.py import asyncio from cf_contracts import COOKIE_POOL, CfCookie, CfCookieTags, HarvestRequest from fleet.core import BaseConfig, BatchAutomation, register from fleet_browser import slot as browser_slot from fleet_cloudflare import harvest_clearance class HarvesterConfig(BaseConfig): cookie_ttl_seconds: int = 1500 @register("cf-cookie-harvester") class CookieHarvester(BatchAutomation[HarvesterConfig]): Config = HarvesterConfig TaskPayload = HarvestRequest Pools = {"cookies": COOKIE_POOL} async def run_one(self, task, ctx): site = task.payload.site url = task.payload.url or f"https://{site}/" pool = ctx.pool(COOKIE_POOL) async with ctx.lock(f"harvest:{site}", hold_seconds=70, wait_seconds=2) as got_it: if not got_it: return if await pool.where(site=site, fresh=True).size() > 0: return async with browser_slot(ctx, page_url=url) as page: result = await asyncio.to_thread(harvest_clearance, page, url, 60) if not result.success: ctx.metrics.incr(f"harvest_failed_{(result.error or 'UNKNOWN').lower()}") return await pool.put( payload=CfCookie( cookies=result.cookies, user_agent=result.user_agent, proxy_url=ctx.proxy_url, site=site, ), tags=CfCookieTags(site=site, fresh=True), ttl_seconds=ctx.config.cookie_ttl_seconds, ) ctx.metrics.incr("cookies_harvested") ``` `harvest_clearance` (from `fleet-cloudflare`) drives the browser through the challenge and stops the load the instant CF hands out the cookie, so each successful harvest costs around 500 KB of proxy traffic rather than fetching the whole page body. Three primitives at work: `ctx.lock("harvest:", ...)` stops a thundering herd. If twenty scrapers all hit an empty pool simultaneously and all submit harvest tasks, only one harvester actually runs. The rest acquire the lock, find a fresh cookie already in the pool, and exit immediately. `ctx.pool(COOKIE_POOL).where(site=..., fresh=True).size()` is the post-lock recheck. Without this, two harvesters that both got into the locked region (because the lock TTL expired) would still produce two cookies. The check inside the lock is the actual correctness guarantee. `pool.put(...)` writes the typed cookie. The framework validates the payload against `CfCookie`, validates the tags against `CfCookieTags`, encodes everything, and writes to Redis. The plugin never sees a key. ### cf-scraper ```python # cf_scraper/__init__.py from urllib.parse import urlparse import asyncio import random import httpx from cf_contracts import COOKIE_POOL, HARVESTER, HarvestRequest, ScrapeResult from fleet.core import BaseConfig, ContinuousAutomation, register class ScraperConfig(BaseConfig): site: str requests_per_minute: int = 60 pool_low_watermark: int = 1 @register("cf-scraper") class CfScraper(ContinuousAutomation[ScraperConfig]): Config = ScraperConfig Output = ScrapeResult async def run_slot(self, ctx): pool = ctx.pool(COOKIE_POOL) site = ctx.config.site host = urlparse(site).hostname or site delay = 60.0 / max(1, ctx.config.requests_per_minute) while not ctx.shutdown.is_set(): # request a top-up if the pool is thin. fire-and-forget. if await pool.where(site=site, fresh=True).size() <= ctx.config.pool_low_watermark: await ctx.submit(HARVESTER, HarvestRequest(site=site)) async with pool.where(site=site, fresh=True).claim(hold_seconds=20) as item: if item is None: await asyncio.sleep(0.5) continue jar = httpx.Cookies() for name, val in item.payload.cookies.items(): jar.set(name, val, domain=host) async with httpx.AsyncClient( proxy=item.payload.proxy_url, cookies=jar, headers={"User-Agent": item.payload.user_agent}, timeout=15, ) as client: url = f"{site}/?_={random.randint(0, 1_000_000)}" try: resp = await client.get(url) except Exception: ctx.logger.exception("request failed") continue if resp.status_code in (403, 503): await item.mark_bad(f"status {resp.status_code}") continue await ctx.emit(ScrapeResult( url=str(resp.url), status=resp.status_code, body_len=len(resp.text), )) try: await asyncio.wait_for(ctx.shutdown.wait(), timeout=delay) except asyncio.TimeoutError: pass ``` Five things to notice about this slot loop: The watermark check (`size() <= pool_low_watermark`) plus `ctx.submit(HARVESTER, ...)` is the demand signal. The scraper drives the harvester, not the reverse. When traffic is quiet, the pool stays at the watermark level and the harvester is idle. `pool.where(...).claim(...)` is destructive in the sense that no other slot can claim the same item during `hold_seconds`, but it's not destructive in the data sense — the cookie stays in the pool, ready for the next free slot. The auto-release fires on context exit. The HTTP request uses both the cookie's proxy URL and its user agent. CF binds clearance cookies to the egress IP and to the UA fingerprint of the browser that obtained them; sending them with a different proxy or UA fails almost every time. `item.mark_bad(...)` evicts a cookie that the target server rejected. It won't be handed out again. The next loop iteration sees a thinner pool, triggers a new harvest, and the system self-heals. `ctx.emit(ScrapeResult(...))` is typed. The framework checks the payload is an instance of `ScrapeResult` (the automation's declared `Output`), serializes it, and writes onto `stream:cf-scraper`. Downstream consumers — another automation, an admin client doing `POST /output/pop` — get strongly-typed data. ## Running it Install all three packages and the framework on a master host and on each worker host. The master needs no per-plugin config; it learns about both automations through entry-points. ```bash pip install -e '.[browser,cloudflare]' pip install -e examples/cf-contracts pip install -e examples/cf-cookie-harvester pip install -e examples/cf-scraper ``` Start the master. ```bash FLEET_ADMIN_TOKEN=admin FLEET_WORKER_TOKEN=worker fleet master ``` Start one harvester worker. On a host that can run headed Chromium. ```bash fleet worker --type cf-cookie-harvester --id harvester-1 curl -X PATCH http://localhost:8000/api/v1/automations/cf-cookie-harvester/workers/harvester-1/config \ -H "Authorization: Bearer admin" -H "Content-Type: application/json" \ -d '{"enabled": true, "slots": 2, "cookie_ttl_seconds": 1500}' ``` Start scraper workers. As many as your traffic plan calls for, on whatever machines. ```bash fleet worker --type cf-scraper --id scraper-1 curl -X PATCH http://localhost:8000/api/v1/automations/cf-scraper/workers/scraper-1/config \ -H "Authorization: Bearer admin" -H "Content-Type: application/json" \ -d '{"enabled": true, "slots": 20, "site": "https://example.com", "requests_per_minute": 60}' ``` Pull scraped results. ```bash curl -X POST http://localhost:8000/api/v1/automations/cf-scraper/output/pop \ -H "Authorization: Bearer admin" -H "Content-Type: application/json" \ -d '{"n": 10}' ``` ## What's happening under the hood When the scraper's first slot starts, the pool is empty. The watermark check fires, the scraper submits a `HarvestRequest`. The master validates the JSON payload against `HarvestRequest`'s schema (taken from the catalog), serializes it, and `RPUSH`es onto `fleet:queue:cf-cookie-harvester`. The harvester's BatchAutomation slots `BLPOP` from that queue. The first one to wake takes the task, deserializes it back to a typed `HarvestRequest`, and runs `run_one`. It acquires `fleet:lock:harvest:example.com`, runs the browser, writes a `CfCookie` into `fleet:pool:cf-cookies`, releases the lock. The scraper's next loop iteration calls `pool.where(site="example.com", fresh=True).claim(hold_seconds=20)`. Behind the scenes the framework lists pool items, filters by tags in process, and atomically `SET NX EX`'s a claim key for the first matching unclaimed item. The slot gets back a typed `CfCookie`, uses it, the context manager releases the claim on exit. Twenty scraper slots running in parallel never claim the same cookie at the same time, but they share the pool — a single fresh cookie can serve many of them in sequence. Cloudflare rotates the clearance cookie based on the egress IP; the pool design respects that by storing the proxy URL alongside. If a slot gets 403 from a cookie, it calls `mark_bad`, the cookie is evicted, the next iteration sees a thinner pool, submits another harvest. No human intervention. No manual recovery. ## What you'd build next Two automations like this are enough for a small fleet. Real production setups grow more components and the same primitives keep working: A `cf-stats` automation that subscribes to the scraper's output stream and writes aggregates somewhere durable. A `cf-cookie-monitor` automation that periodically `peek`s the pool and emits a Prometheus-style metric for how many fresh cookies are in flight. A second harvester (`cf-harvester-playwright`) that puts into the same pool but uses a different browser stack. The scraper doesn't know the difference; whichever produces faster wins. Adding any of these is a new pip package plus an entry-point line. None of the existing plugins change. --- # Agent Instructions This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com. ## Querying This Documentation If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question. Perform an HTTP GET request on the current page URL with the `ask` query parameter, and the optional `goal` query parameter: ``` GET https://fleet.hackmap.win/guides/cf-cookies-pool.md?ask=&goal= ``` `ask` is the immediate question: it should be specific, self-contained, and written in natural language. `goal` is optional and describes the broader end goal you are ultimately trying to accomplish on behalf of the user. GitBook uses it to tailor the answer towards what is most useful for that goal. The response will contain a direct answer to the question and relevant excerpts and sources from the documentation. Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.