Stealth & Anti-Detection

VoidCrawl uses a minimal-footprint stealth strategy inspired by zendriver / nodriver. Stealth is enabled by default. The guiding rule: present a real browser consistently — don’t fake things.

For the hardest gates there is a second, opt-in layer that shrinks the CDP control channel itself — see Minimal CDP footprint. It is off by default because it trades away response capture.

TL;DR (at a glance)

What VoidCrawl does, and why each piece exists:

Layer	What we do	Why
Launch flags	Drop chromiumoxide’s --enable-automation/--disable-extensions; add --disable-blink-features=AutomationControlled plus a curated zendriver/nodriver-derived set	The biggest automation signal lives in launch flags. AutomationControlled is what makes navigator.webdriver a native false — we do not patch it in JS.
CDP surface	cdp_mode="normal" by default; "minimal" is opt-in per session	The domains chromiumoxide enables eagerly (Runtime, Network, …) are themselves tells. Minimal drops them, at the cost of response capture.
No JS injection	addScriptToEvaluateOnNewDocument is empty	Each injected script is itself a fingerprint. We patch nothing in page-world JS.
UA / Client Hints	Real UA (Headless stripped), with navigator.platform + userAgentData (Sec-CH-UA) derived from that UA so they agree	A Linux UA with platform === "Win32" or empty brands is a bot tell.
GPU	--headless=new + ANGLE + --disable-gpu-sandbox -> hardware WebGL	Legacy headless renders WebGL with SwiftShader (software) — a strong bot signal.

Managed Cloudflare Turnstile (the hard case):

Mode	Result
Headful	Best option; still profile/IP/environment dependent. Docker headful, nodriver, and bare Docker Chrome currently all stall on the same Cloudflare canary from this host.
Headless	Usually gated — use headful + warm profile + clean exit for Turnstile-walled targets.

All defaults are overridable by the caller (see Overriding the defaults).

Philosophy: less is more

Most automation tools try to spoof every fingerprint — fake plugins, fake WebGL, fake UA. This backfires against modern WAFs (Akamai, Cloudflare, PerimeterX) because:

1. Spoofed values are inconsistent. A hardcoded Chrome/131 UA on a Chromium 148 build is an instant flag. A fake WebGL renderer that doesn’t match the real GPU is trivially caught.
2. The spoofing itself is detectable. Every Page.addScriptToEvaluateOnNewDocument call is a fingerprint. Overriding navigator.plugins with a Proxy behaves differently from the real PluginArray prototype — and detectors check for exactly that.
3. The automation signal is in the launch flags, not JS. chromiumoxide’s defaults include --enable-automation, which tells every WAF “I’m automated” before a page loads.

VoidCrawl’s approach: don’t fake anything. Launch with clean flags, let Chrome report its real values, and only ensure those values are internally consistent.

Lesson learned the hard way: VoidCrawl used to inject two JS patches — deleting navigator.webdriver and force-opening shadow DOMs. Both were removed. Deleting navigator.webdriver made it undefined (real Chrome reports false — undefined is the tell). Force-opening shadow DOMs broke Cloudflare Turnstile, which renders its challenge in a closed shadow root and tamper-checks it: forcing it open failed the challenge with ERROR 600010. We inject zero page-world JS today. To reach into a closed shadow root without tampering, use the AX-tree locators (ax_box_in_frame / click_ax_in_frame) — the browser-computed accessibility tree descends into closed roots, so a trusted compositor click can drive the widget with no shadow patch. See Challenge escalation.

The automation signal is in the launch flags

After disable_default_args() (which strips chromiumoxide’s toxic defaults) we re-add a curated set. Flags are stored without the leading -- (chromiumoxide prepends it; a literal -- would produce the inert ----flag — a bug we fixed, which had silently disabled the whole list).

Removed (toxic defaults)

Flag	Why it’s bad
--enable-automation	Literally opts in to automation detection
--disable-extensions	Real Chrome always has extension support

Anti-automation flags we add

Flag	Purpose
--disable-blink-features=AutomationControlled	Removes the automation-controlled Blink feature, so navigator.webdriver is a native false — not a JS patch
--disable-infobars	Suppresses the “Chrome is being controlled by automated test software” bar
--disable-features=IsolateOrigins,site-per-process,TranslateUI	Keeps cross-origin frames in-process so evaluate_js_in_frame can reach them, and drops the Translate UI
--no-pings, --disable-component-update, --disable-session-crashed-bubble, --disable-search-engine-choice-screen, --homepage=about:blank	Suppress automation-ish background behaviour and first-run UI
--no-first-run, --no-service-autorun, --no-default-browser-check, --password-store=basic, --use-mock-keychain	Profile / first-run hygiene

Plus the safe noise-reducers kept from chromiumoxide’s own list (--disable-background-networking, --disable-background-timer-throttling, --disable-backgrounding-occluded-windows, --disable-breakpad, --disable-client-side-phishing-detection, --disable-component-extensions-with-background-pages, --disable-default-apps, --disable-dev-shm-usage, --disable-hang-monitor, --disable-ipc-flooding-protection, --disable-popup-blocking, --disable-prompt-on-repost, --disable-renderer-backgrounding, --disable-sync, --force-color-profile=srgb, --metrics-recording-only) — 33 flags in total, defined as DEFAULT_CHROME_ARGS in crates/core/src/session.rs.

Cross-origin frames

--disable-features=IsolateOrigins,site-per-process covers ordinary cross-origin frames, but Chrome field-trial-isolates a few origins (notably google.com, hence reCAPTCHA’s bframe) out-of-process regardless. Reaching those is an explicit per-session opt-in — extra_args=["--disable-site-isolation-trials"] — rather than a global default, so the browser’s isolation posture is unchanged for callers who don’t need it.

UA / platform / Client-Hints consistency

This is the only “override” we apply, via CDP Emulation.setUserAgentOverride (not page-world JS) in Page::apply_stealth. We probe the browser’s real UA, strip any Headless token, and from that one string derive a coherent identity so UA, navigator.platform, and navigator.userAgentData all agree:

Signal	Value (for the real Linux/Chrome UA)
navigator.userAgent	real build, HeadlessChrome -> Chrome
navigator.platform	Linux x86_64 (Win32 / MacIntel for those UAs)
Sec-CH-UA-Platform	Linux / Windows / macOS
userAgentData.brands / fullVersionList	Chromium/Google Chrome at the UA’s major/full version + a GREASE entry

A mismatch here (e.g. the old hardcoded platform: "Win32" on a Linux UA, or empty brands) is itself a bot signal — both flagged by the rebrowser bot-detector, now green.

GPU acceleration (hardware WebGL)

A headless browser that renders WebGL with SwiftShader (Chrome’s software fallback) advertises itself: WEBGL_debug_renderer_info returns "ANGLE (… SwiftShader …)", which Cloudflare and others weight as “no real GPU -> likely a bot/VM.”

VoidCrawl forces hardware rendering:

• --headless=new — the legacy --headless forces SwiftShader; the new mode runs the full browser stack and can use a real GPU.
• --use-angle=vulkan + --enable-gpu + --ignore-gpu-blocklist — route WebGL through ANGLE on the real GPU.
• --disable-gpu-sandbox — lets the GPU process reach the DRM render node. This is the lever — on a host with a working driver it’s usually all you need (no VK_DRIVER_FILES juggling).

Verified on AMD (RADV): renderer becomes ANGLE (AMD, Vulkan … (AMD Radeon … RADV …)), radv) — hardware, not SwiftShader. The defaults are vendor-generic (ANGLE uses whatever Intel/AMD/NVIDIA driver the machine has); nothing is hardcoded per vendor.

In Docker, hardware GPU additionally needs Mesa drivers in the image + /dev/dri passthrough — see /voidcrawl/guides/docker/ and /voidcrawl/guides/docker/. Without a GPU passed through, the container falls back to SwiftShader.

To force software rendering (or pick a different backend), override --use-angle (see below).

What we don’t touch (and why)

We inject no page-world JS, and we leave these alone:

Signal	Why
navigator.webdriver	The launch flag already yields a native false. A JS patch (deleting it -> undefined, or a redefined getter) is itself detectable.
navigator.plugins	Real Chrome populates it; faking creates inconsistencies.
navigator.userAgent	We use the real UA (Headless stripped) — no version mismatch.
WebGL vendor/renderer	The real GPU string (once hardware-accelerated) beats any fake.
window.chrome.runtime, navigator.permissions, canvas	Default behavior is already correct; spoofing adds detectable noise.
Shadow DOM mode	We do not force-open it (it broke Turnstile). Interacting with a challenge widget works via real compositor clicks at pixel coordinates regardless of shadow mode — locate the target inside a closed root with ax_box_in_frame / click_ax_in_frame (see Challenge escalation).

Headful vs headless (and managed Turnstile)

For the toughest WAFs, headful is required — headless has detectable differences that survive every JS patch:

• Different rendering/compositing pipeline.
• Missing / non-default screen, media, and input-related properties.
• The managed-challenge score is simply lower.

Concretely, against managed Cloudflare Turnstile / full-page challenge canaries, headful is necessary but not sufficient:

Mode	Outcome
Host/container headful	Best baseline, but still depends on IP reputation, profile warmth, sandbox/container posture, and launch surface
Docker headful from this host	Currently stalls at Just a moment… for VoidCrawl, nodriver, and bare Chrome alike
Headless	Usually stalls; no token

import os
from voidcrawl import BrowserPool

# WAF / managed-Turnstile targets -- headful:
os.environ["CHROME_HEADLESS"] = "0"
async with BrowserPool.from_env() as pool:
    async with pool.acquire() as tab:
        await tab.navigate("https://waf-protected-site.com")
        await tab.wait_for_network_idle(timeout=15.0)
        html = await tab.content()

# Unprotected / bulk targets -- headless is fine and faster (default).

For a headless farm that still needs to clear Turnstile, run the headful GPU container (/voidcrawl/guides/docker/) rather than headless.

The full-page Managed Challenge / Challenge Page interstitial (“Just a moment…”, served by the edge in front of a route) is the hard gate. Headful + GPU is only the starting point; use a warm persisted profile and clean network exit before treating a library as the limiting factor.

Minimal CDP footprint (full-page Managed Challenge)

The full-page Cloudflare Managed Challenge interstitial is sensitive to the CDP control channel and to environment (IP/profile/container/sandbox). The current Docker canary from this host is challenged for VoidCrawl, nodriver, and bare Chrome alike, so use it as a parity benchmark rather than a claim that CDP minimization alone can pass every gate.

chromiumoxide eagerly enables a set of CDP domains on every target, and each one is a tell a clean browser never sends. cdp_mode="minimal" skips them:

What is measured, and what is not

Against 2captcha.com/demo/cloudflare-turnstile-challenge on an AMD-GPU Linux host:

Setup	"normal"	"minimal"
Headful	reaches the destination (3/3)	reaches the destination (3/3)
Headless	walled — Just a moment…	walled, identically
Docker headful, 40 concurrent ops	40/40, 3.23s	40/40, 3.15s

On this target, from this host, the two modes are behaviourally equivalent — and the throughput difference is noise. Minimal is not demonstrated to beat normal here.

The reasoning holds (every domain below is a documented automation tell, and enabling fewer cannot make detection more likely), so treat minimal as “a smaller CDP surface”, not a bypass. Whether it changes an outcome is target- and environment-specific — measure it on yours.

CDP enable	Why it’s a tell	In minimal mode
Runtime.enable	Emits Runtime.consoleAPICalled and serializes console arguments — the canonical CDP-automation signal, and observable from page JS	skipped — Runtime.evaluate still works in the main world
Network.enable	A clean browser doesn’t subscribe	skipped (lose response capture / network-idle)
Performance.enable, Log.enable	Eager instrumentation	skipped (no loss)
Target.setAutoAttach(waitForDebuggerOnStart)	Automation-shaped; observable via child-target startup timing	skipped (lose OOPIF auto-attach)
Isolated-world addScriptToEvaluateOnNewDocument	A persistent injected script	skipped (lose evaluate_function)

from voidcrawl import BrowserConfig, BrowserSession

async with BrowserSession(
    BrowserConfig(headless=False, stealth=True, cdp_mode="minimal")
) as b:
    page = await b.new_page("about:blank")
    await page.navigate("https://site-behind-a-cloudflare-challenge.com")
    # The interstitial clears on its own; poll the title rather than awaiting
    # network idle, which needs the Network domain:
    while "just a moment" in (await page.eval_js("document.title")).lower():
        await asyncio.sleep(1)

The mode is per session, so one process can run a minimal session against a walled target and a normal session for capture work at the same time. In Rust: BrowserSessionBuilder::minimal_cdp(), or .cdp_mode(CdpMode::Minimal).

What minimal mode costs

Nothing errors silently, but a lot stops working: response capture (network_capture_arm / wait, CDP request-header capture), wait_for_network_idle, cross-origin evaluate_js_in_frame, evaluate_function, and OOPIF auto-attach. Those helpers raise CdpDomainNotEnabled rather than returning empty data — see Escalating a minimal session.

"normal" is the default precisely because these losses are invisible until you go looking for missing data.

Measure it on your own targets

scripts/bench_antibot_cdp.py runs both modes against the same URL and reports passed / challenged / blocked plus a fingerprint snapshot, so you can check the trade rather than taking the table above on faith:

uv run python scripts/bench_antibot_cdp.py \
  --url https://<cloudflare-managed-challenge-target> \
  --engine voidcrawl --cdp-mode normal --cdp-mode minimal --runs 3 --headful

Add --engine nodriver --engine zendriver to compare against those drivers when they’re installed. Targets are operator-supplied; none are committed to the repo.

The older process-global VOIDCRAWL_STEALTH_NO_RUNTIME=1 environment variable still selects minimal mode when no explicit cdp_mode is given. Prefer cdp_mode: the variable applies to every session in the process, so it cannot express “minimal here, normal there”.

Escalating a minimal session

A minimal session is not a dead end. Network.enable and Runtime.enable are runtime CDP commands, not launch flags, so a tab can start quiet, clear a wall, and then be escalated deliberately. This is exactly what Chrome DevTools does when you open its Network panel — and why that panel says “reload to record”.

async with BrowserSession(BrowserConfig(cdp_mode="minimal")) as b:
    page = await b.new_page("https://walled-site.com")
    ...                                # quiet CDP surface: no Network domain

    await page.escalate_network()       # explicit, greppable
    await page.reload()                 # capture starts here
    caps = await page.expect_responses({"api": "*/api/*"})

Capture starts at the enable

Network.enable only observes traffic that happens after it. Escalating after a response has already arrived will not retroactively capture it — escalate, then re-navigate (or capture the next navigation).

escalate_network() deliberately does not reload for you: only you know whether re-navigating a page that just cleared a challenge is safe.

Escalation is never automatic. A session launched minimal was launched that way on purpose, so the network helpers fail closed with CdpDomainNotEnabled — naming the method that fixes it — rather than quietly making the session louder or handing back empty data.

Method	Enables	Page-observable?
await page.escalate_network()	Network — response capture, request-header capture, network-idle waits, set_headers	No. It injects nothing and changes no JS-visible state, so it is the cheap one to turn on late
await page.escalate_runtime()	Runtime — frame-scoped evaluate_js_in_frame	Yes. The most-cited CDP automation tell. Do not escalate Runtime on a tab that still has a challenge to pass

Both are idempotent and are no-ops under cdp_mode="normal", so mode-agnostic code can call them freely.

Tab routing

await tab.instrumentation_state() on any Page or PooledTab reports what that handle has enabled, which gives a simple routing rule: send gates to quiet tabs, capture to escalated ones, and reach for a fresh tab rather than trying to “un-taint” one.

state = await tab.instrumentation_state()
if state.low_cdp:
    await tab.navigate("https://site-behind-a-managed-challenge.com")
else:
    # Already instrumented -- use it for capture / debug work.
    response = await tab.goto("https://example.com", capture_endpoints=True)

Field	Meaning
low_cdp	True while neither high-signal domain is enabled — the state a cdp_mode="minimal" tab starts in
network_enabled	True after Network.enable. Seeded True under cdp_mode="normal"
runtime_enabled	True after Runtime.enable. eval_js uses one-shot Runtime.evaluate and does not set this

Escalation is per-tab and does not leak, so a fresh tab from the same session is still quiet.

It is a routing signal, not a security boundary

It reports what this handle enabled since it wrapped the target. A tab adopted from another CDP client may carry instrumentation VoidCrawl cannot see, so prefer a fresh tab for the highest-sensitivity gate. Pool tabs are recycled and may already have been escalated by earlier work — check rather than assume.

Overriding the defaults

Every default flag is overridable by the caller — useful to force a GPU backend, disable acceleration, add a proxy bypass, etc. Caller args are merged by switch key, so a caller value replaces the matching default (we don’t emit duplicate switches — Chrome’s per-switch precedence is inconsistent):

from voidcrawl import BrowserConfig

# Force software rendering (e.g. to compare, or on a GPU-less box):
cfg = BrowserConfig(extra_args=["--use-angle=swiftshader"])

# Disable the GPU entirely:
cfg = BrowserConfig(extra_args=["--disable-gpu"])

The same applies through the MCP server / pool config (extra_args).

Waiting for readiness (event-driven)

JS-heavy sites and challenge pages aren’t ready at page load. Two event-driven waits — no polling, no sleeps:

async with pool.acquire() as tab:
    await tab.navigate(url)
    # Chrome's networkIdle lifecycle event (returns event name, or None on timeout):
    await tab.wait_for_network_idle(timeout=15.0)
    # …or an in-page MutationObserver for a specific selector:
    await tab.wait_for_selector("#results", timeout=15.0)

Why networkIdle is unreliable for SPAs

networkIdle fires after zero in-flight requests for 500ms — but WebSockets, SSE/long-polling, analytics beacons, and lazy-loading keep the network active, so on many modern apps it never fires. Prefer wait_for_selector("<the element you actually care about>"): it resolves the moment that element is inserted, regardless of network.

Real-world results

Target	Approach	Result
Akamai WAF (BusinessWire)	chromiumoxide defaults (--enable-automation)	403
Akamai WAF (BusinessWire)	+ heavy JS spoofing + fake UA	403
Akamai WAF (BusinessWire)	disable_default_args + clean flags + real UA	Success
Managed Cloudflare / Turnstile gates	headful, hardware GPU, consistent UA, no JS injection, plus opt-in cdp_mode="minimal"	Best available posture; final result depends on profile/IP/environment
Docker Cloudflare canary from this host	VoidCrawl, nodriver, and bare Chrome	All challenged (Just a moment…)
Managed Cloudflare / Turnstile gates	headless	Usually gated

The lesson, twice over: the flags + a consistent real browser matter more than JS patches — and a wrong JS patch is worse than none.