爬虫-Crawl4AI

发表于 分类于 本文字数： 254k 阅读时长 ≈ 7:42

这篇讲什么

  • Crawl4AI 0.9.5 在 Python 3.14.10 上逐条实测：它是一个面向 LLM 时代的异步抓取器——抓完直接给你 markdown，AsyncWebCrawler.arun() 一次调用就把「起浏览器、等 JS、滚动、抽正文、转 markdown」全做完。
  • 本篇把每个参数面都按 inspect.signature 打全并逐个实测：BrowserConfig 44 个参数、CrawlerRunConfig 100 个参数、CrawlResult 29 个字段、CacheMode 5 种、深度抓取策略与过滤器链、四种结构化抽取、markdown 生成器与两种内容过滤器、会话与 cookie、反检测的三个适配器。
  • 三条只有实测才知道的坑：默认 cache_mode=BYPASS（不是「启用缓存」）；它自带一个结构性反爬判据，小页面会被误判成「被拦截」（success=False 但正文其实抓到了，还会顺手打一行云广告）；robots.txt 默认完全不遵守，check_robots_txt=True 才拒绝。
  • 与系列其它篇的口径差别：抓的靶站沿用 ssr1/spa1（外加本机靶站做协议级演示），但抓取对象是「一页 → 一段 markdown」，没有 Scrapy 的 Pipeline、也没有 Crawlee 的队列存储；要断点续爬得自己接 cache_mode 与 session_id。

装哪一种

官方 quick-start 只给一句 pip install crawl4ai，然后补一句 crawl4ai-setup 装浏览器；全功能写法是 pip install "crawl4ai[all]"。这两种装法在这台机器上差了 22 个包和 800 MB，段代码里把两个环境各跑一遍取数：

install_two_ways.py
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
# -*- coding: utf-8 -*-
"""两个官方安装口径的差集：pip install crawl4ai 与 pip install Crawl4AI[all]。

本机建了两个独立虚拟环境（一个最小装、一个 [all]），这段脚本用 subprocess
分别去两个环境里取「发行包数 + 包名集合 + 目录体积」，再算差集。
"""

import json
import os
import subprocess
import sys

# 本机建了两个虚拟环境：一个只跑 pip install crawl4ai，一个跑 pip install "Crawl4AI[all]"。
# 换机器时只改 VENV_ROOT 这一行就行。
VENV_ROOT = r"E:\Desktop\Test"
BASE_PY = os.path.join(VENV_ROOT, "crawl4ai-base-env", "Scripts", "python.exe")
ALL_PY = os.path.join(VENV_ROOT, "crawl4ai-env", "Scripts", "python.exe")

CHILD = r"""
import importlib.metadata as md
import os
import sys

sp = os.path.dirname(os.path.dirname(sys.executable)) + "\\Lib\\site-packages"
dists = sorted(d.metadata["Name"] for d in md.distributions() if d.metadata["Name"])
size = sum(os.path.getsize(os.path.join(r, f))
           for r, _, fs in os.walk(sp) for f in fs)
print(json.dumps({"n": len(dists), "names": dists,
                  "gb": round(size / 1e9, 2),
                  "ver": md.version("Crawl4AI")}))
"""


def probe(python_exe, label):
    out = subprocess.run([python_exe, "-X", "utf8", "-c", "import json\n" + CHILD],
                         capture_output=True, text=True, encoding="utf-8").stdout.strip()
    data = json.loads(out.splitlines()[-1])
    print("%-12s 版本 %s | 发行包 %d 个 | site-packages %.2f GB"
          % (label, data["ver"], data["n"], data["gb"]))
    return set(data["names"])


base_names = probe(BASE_PY, "最小装")
all_names = probe(ALL_PY, "[all]")
only_all = sorted(all_names - base_names)
only_base = sorted(base_names - all_names)
print("只在 [all] 里的包 %d 个：" % len(only_all))
for i in range(0, len(only_all), 6):
    print("   ", ", ".join(only_all[i:i + 6]))
print("只在最小装里的包：", only_base or "无")
1
2
3
4
5
6
7
8
最小装          版本 0.9.5 | 发行包 97 个 | site-packages 0.70 GB
[all]        版本 0.9.5 | 发行包 119 个 | site-packages 1.55 GB
只在 [all] 里的包 22 个：
    PySocks, annotated-doc, mpmath, narwhals, outcome, pypdf
    safetensors, scikit-learn, selenium, sentence-transformers, setuptools, shellingham
    sortedcontainers, sympy, threadpoolctl, torch, transformers, trio
    trio-websocket, typer, websocket-client, wsproto
只在最小装里的包： 无
  • 最小装 97 个发行包、site-packages 0.70 GB；[all] 是 119 个、1.55 GB。多出来的 22 个包基本就是 torch 全家桶：torch、transformers、sentence-transformers、scikit-learn、safetensors、mpmath、sympy，加上 selenium、trio、pysocks、typer、pypdf 这些零头。也就是说 [all] 那 800 MB 是为了语义抽取（CosineStrategy）和 PDF 解析准备的。
  • 常规抓取不需要 [all]：最小装已经带上了 playwright、patchright、aiohttp、shapely/alphashape、tiktoken、trimesh（这些是必需依赖，不是 extras），所以「装哪个」的判断依据是你要不要跑语义模型。
  • 两个环境的 crawl4ai 版本完全一致（0.9.5），说明 extras 只影响可选依赖，不影响核心版本。装之前建议先跑一次 pip 的 --dry-run 看它会不会动核心包——这个系列里已经吃过几次 pip 静默降级的亏。

版本号怎么取

这一条只有一行代码，但踩中的人不少：crawl4ai.__version__ 不是版本号字符串，而是一个子模块对象。

version_is_a_module.py
1
2
3
4
5
6
7
8
9
10
11
12
13
14
# -*- coding: utf-8 -*-
"""crawl4ai.__version__ 是个模块对象，不是版本号字符串。"""

import importlib.metadata

import crawl4ai

print("import crawl4ai; crawl4ai.__version__ 的类型 ->", type(crawl4ai.__version__).__name__)
print("它 repr 的开头 ->", repr(crawl4ai.__version__)[:9], "……（后面是绝对路径，不打印）")

from crawl4ai.__version__ import __version__

print("from crawl4ai.__version__ import __version__ ->", __version__)
print("importlib.metadata.version('Crawl4AI') ->", importlib.metadata.version("Crawl4AI"))
1
2
3
4
import crawl4ai; crawl4ai.__version__ 的类型 -> module
它 repr 的开头 -> <module ' ……（后面是绝对路径，不打印）
from crawl4ai.__version__ import __version__ -> 0.9.5
importlib.metadata.version('Crawl4AI') -> 0.9.5
  • 打印 crawl4ai.__version__ 会得到 <module 'crawl4ai.__version__' from '…\__version__.py'>，写日志或做版本断言时这是废的。
  • 正确写法有两条：from crawl4ai.__version__ import __version__（拿到 0.9.5），或者 importlib.metadata.version("Crawl4AI")。注意包名大小写：PyPI 上的规范名是 Crawl4AI，version("crawl4ai") 在有些环境里也能过（distribution 名大小写不敏感），但按官方写法用 Crawl4AI 更稳。
  • 上面那段代码没有 repr 全量打印，只打了前 9 个字符——因为完整的 repr 里带 site-packages 绝对路径，换一台机器就不一样了，写进文章就没法复现。

四个命令行

包里带了四个可执行入口，官方文档把它们描述成「装好之后的自检三板斧」。这段把三个跑完（第四个用离线保护跑，避免真下大模型），退出码与关键输出原样贴出：

cli_four_entries.py
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
# -*- coding: utf-8 -*-
"""四个官方命令行各跑一次，看它们到底做了什么。

- crawl4ai-setup 带 CRAWL4AI_MODE=api：跳过浏览器安装，只做建目录 + 初始化数据库 + 迁移
- crawl4ai-doctor：真的起浏览器抓一次 crawl4ai.com（要联网）
- crawl4ai-migrate：先备份再迁移，备份文件名带时间戳（下面归一化成 <时间戳>）
- crawl4ai-download-models：带 HF_HUB_OFFLINE=1 跑，让它在校验阶段就失败，不真下模型
"""

import os
import re
import subprocess
import sys

HERE = os.path.dirname(sys.executable)
ENV = dict(os.environ, CRAWL4AI_NO_CLOUD_NOTICE="1", PYTHONIOENCODING="utf-8")


def run(label, args, extra_env=None, limit=6):
    env = dict(ENV, **(extra_env or {}))
    p = subprocess.run([os.path.join(HERE, args[0] + ".exe")] + args[1:],
                       capture_output=True, text=True, encoding="utf-8",
                       errors="replace", env=env, timeout=300)
    text = (p.stdout or "") + (p.stderr or "")
    lines = [l.rstrip() for l in text.splitlines() if l.strip()]
    print("$ %s  -> 退出码 %d" % (args[0], p.returncode))
    for line in lines[:limit]:
        line = re.sub(r"crawl4ai\.db\.backup_\d{8}_\d{6}", "crawl4ai.db.backup_<时间戳>", line)
        line = re.sub(r"[A-Za-z]:\\\\?[^ ]*\.crawl4ai", r"<家目录>\.crawl4ai", line)
        line = re.sub(r"C:\\Users\\[^ ]*", "<家目录>", line)
        print("   |", line[:150])


run("setup 走 api 分支（不装浏览器）", ["crawl4ai-setup"], {"CRAWL4AI_MODE": "api"})
run("doctor（联网抓一次）", ["crawl4ai-doctor"], limit=3)
run("migrate", ["crawl4ai-migrate"], limit=3)
run("download-models（离线保护）", ["crawl4ai-download-models"],
    {"HF_HUB_OFFLINE": "1", "HF_HUB_DISABLE_TELEMETRY": "1"}, limit=4)
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
$ crawl4ai-setup  -> 退出码 0
   | [INIT].... → Running post-installation setup...
   | [SETUP]... ℹ CRAWL4AI_MODE=api detected. Skipping Playwright browser installation.
   | [INIT].... → Starting database initialization...
   | [COMPLETE] ● Database initialization completed successfully.
   | [COMPLETE] ● Post-installation setup completed!
$ crawl4ai-doctor  -> 退出码 0
   | [INIT].... → Running Crawl4AI health check...
   | [INIT].... → Crawl4AI 0.9.5
   | [TEST].... ℹ Testing crawling capabilities...
$ crawl4ai-migrate  -> 退出码 0
   | [COMPLETE] ● Database backup created at: <家目录>\.crawl4ai\crawl4ai.db.backup_<时间戳>
   | [INIT].... → Starting database migration...
   | [COMPLETE] ● Migration completed. 3 records processed.
$ crawl4ai-download-models  -> 退出码 1
   | [LOG] Welcome to the Crawl4AI Model Downloader!
   | [LOG] This script will download all the models required for Crawl4AI.
   | [LOG] Downloading text classifier...
   | Traceback (most recent call last):
  • crawl4ai-setup 不是「只在第一次装浏览器」。它内部的命令是 playwright install --with-deps --force chromium 加 patchright install --with-deps --force chromium，--force 意味着每次都重下浏览器（约 2 × 118 MiB 再加 headless shell）。上面那次跑加了 CRAWL4AI_MODE=api，才走成「只建家目录 + 初始化 SQLite + 跑迁移」这条快路，日志里 Skipping Playwright browser installation 就是证据。
  • crawl4ai-doctor 要联网。它不是静态检查浏览器目录，而是真的起浏览器抓一次 https://crawl4ai.com，抓通才打 ✅ Crawling test passed!。所以离线环境里 doctor 失败不代表你装坏了。
  • crawl4ai-migrate 会先备份再迁移，备份文件名带时间戳（输出里已归一化成 <时间戳>），跑一次多一个文件，别惊讶家目录里为什么躺了一排 crawl4ai.db.backup_*。
  • crawl4ai-download-models 会去 HuggingFace 拉文本分类器，网络受限时退出码 1、报 huggingface_hub.errors.LocalEntryNotFoundError。这里用 HF_HUB_OFFLINE=1 复现，没有真的下载模型。
  • 附带一条：库和 CLI 默认会往输出里打「Crawl4AI Cloud is live」的广告横幅，这个横幅一天只打一次、看家目录里的标记文件，所以对可复现输出是有害的——本篇所有块都在脚本开头写了 os.environ.setdefault("CRAWL4AI_NO_CLOUD_NOTICE", "1")（认 1/true/yes）。

浏览器这一层

文档说 crawl4ai 支持 chromium/firefox/webkit，还提了 patchright 那条undetected 路线。实际默认走谁、本机装了些什么、browser_type 写错会怎样，这段全部量出来：

browser_types.py
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
# -*- coding: utf-8 -*-
"""浏览器这一层：默认引擎是谁、本机有哪些浏览器、browser_type 写错会怎样。"""

import asyncio
import inspect
import os
import re
import sys

import crawl4ai.browser_adapter as BA
from crawl4ai import AsyncWebCrawler, BrowserConfig, CrawlerRunConfig
from crawl4ai.async_crawler_strategy import AsyncPlaywrightCrawlerStrategy

adapter_names = sorted(n for n in dir(BA) if n.endswith("Adapter"))

CACHE = os.path.join(os.environ["LOCALAPPDATA"], "ms-playwright")


def silent():
    return CrawlerRunConfig(verbose=False)


def first_line(text):
    line = text.splitlines()[0] if text.splitlines() else ""
    return re.sub(r"[A-Za-z]:\\Users\\.*?ms-playwright", "<浏览器缓存目录>", line)[:110]


async def one(label, **browser_kwargs):
    try:
        async with AsyncWebCrawler(config=BrowserConfig(verbose=False, **browser_kwargs)) as cr:
            r = await cr.arun("https://ssr1.scrape.center", config=silent())
            print("%-34s success=%s 状态 %s markdown 行数 %d"
                  % (label, r.success, r.status_code, len(r.markdown.splitlines())))
    except Exception as exc:
        print("%-34s 抛 %s: %s" % (label, type(exc).__name__, first_line(str(exc))))


print("适配器名字 =", adapter_names)
mod = sys.modules[AsyncPlaywrightCrawlerStrategy.__module__]
hits = [l.strip() for l in inspect.getsource(mod).splitlines()
        if "self.adapter = " in l or "use_undetected=" in l]
print("源码里决定用哪个适配器的两行：")
for l in hits[:2]:
    print("   ", l)
print("本机浏览器缓存目录 =", sorted(e for e in os.listdir(CACHE) if not e.startswith(".")))
print("crawl4ai-setup 只装 chromium，所以 firefox/webkit 的目录不在上面这行里")

asyncio.run(one("browser_type=chromium（默认）"))
asyncio.run(one("browser_type=firefox", browser_type="firefox"))
asyncio.run(one("browser_type=webkit", browser_type="webkit"))
asyncio.run(one("browser_type=safari（不存在的值）", browser_type="safari"))
asyncio.run(one("channel=chrome", channel="chrome"))
asyncio.run(one("channel=msedge", channel="msedge"))
1
2
3
4
5
6
7
8
9
10
11
12
适配器名字 = ['BrowserAdapter', 'PlaywrightAdapter', 'StealthAdapter', 'UndetectedAdapter']
源码里决定用哪个适配器的两行：
    self.adapter = browser_adapter or PlaywrightAdapter()
    use_undetected=isinstance(self.adapter, UndetectedAdapter)
本机浏览器缓存目录 = ['chromium-1248', 'chromium_headless_shell-1248', 'ffmpeg-1013', 'winldd-1007']
crawl4ai-setup 只装 chromium，所以 firefox/webkit 的目录不在上面这行里
browser_type=chromium（默认）          success=True 状态 200 markdown 行数 74
browser_type=firefox               抛 Error: BrowserType.launch: Executable doesn't exist at <浏览器缓存目录>\firefox-1555\firefox\firefox.exe
browser_type=webkit                抛 Error: BrowserType.launch: Executable doesn't exist at <浏览器缓存目录>\webkit-2370\Playwright.exe
browser_type=safari（不存在的值）         success=True 状态 200 markdown 行数 74
channel=chrome                     success=True 状态 200 markdown 行数 74
channel=msedge                     success=True 状态 200 markdown 行数 74
  • 默认引擎是 playwright，不是 patchright：源码里写死 self.adapter = browser_adapter or PlaywrightAdapter()，只有显式传 UndetectedAdapter() 才会 use_undetected=True 去 import patchright.async_api。好消息是 patchright 与 playwright 的 browsers.json 是同一个 revision 1248，切 undetected 不需要再下一次浏览器。
  • 本机浏览器缓存目录里只有 chromium-1248、chromium_headless_shell-1248、ffmpeg-1013、winldd-1007 四项，firefox/webkit 根本不在（crawl4ai-setup 只装 chromium）。
  • browser_type 写错的表现分两种，这条必须记住：firefox/webkit 抛的是「浏览器可执行文件不存在」那句 Executable doesn’t exist，而 safari 这种根本不存在的取值却 success=True——它不校验取值，悄悄按 chromium 跑了。channel="chrome"、channel="msedge" 同理，能跑但跑的是本机那个引擎。
  • 所以：想确认「我真的用了想要的浏览器」，唯一可靠的办法是回页面里读指纹（navigator.userAgent、navigator.plugins.length），本篇「反检测那三层」一节就是这么做的。

本机靶站

后面的章节大量使用一个进程内起的小靶站（robots.txt、服务端计数、表单回显、懒加载、SPA、嵌套链接都在里面）。协议级的行为只能在能自控的靶站上测：在线练习站不会给你 Disallow: /private/，也不会告诉你「这一枪到底到没到服务端」。先把靶站自己验一遍，后面所有输出的判据都在这里：

fixture_smoke.py
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
# -*- coding: utf-8 -*-
"""先确认靶站自己是通的：每个端点用 urllib 打一遍，回的东西符合预期再往下走。"""

import asyncio
import inspect
import json
import os
import re
import threading
from http.server import BaseHTTPRequestHandler, HTTPServer

# 云广告一天只打一次，打不打取决于家目录里的标记文件，会把输出变得不可复现，所以先静音
os.environ.setdefault("CRAWL4AI_NO_CLOUD_NOTICE", "1")

from crawl4ai import (AsyncWebCrawler, BrowserConfig, CacheMode, CrawlerRunConfig)
from crawl4ai.async_crawler_strategy import AsyncPlaywrightCrawlerStrategy


PORT = 8060
BASE = "http://127.0.0.1:8060"
HITS = {}


def html(title, body):
    return ("<!doctype html><html><head><meta charset='utf-8'><title>%s</title>"
            "<meta name='description' content='probe'><style>.nav{color:red}</style>"
            "</head><body>"
            "<div class='nav'>NAV 这是靶站的导航区，故意写长一点，"
            "免得页面被 crawl4ai 的结构性判据当成空壳</div>"
            "%s"
            "<div class='filler'>FILLER 这是靶站的页脚区，同样是有实际文字的标记</div>"
            "</body></html>" % (title, body))


LIST_BODY = ("<h1>列表页</h1><ul class='items'>"
             + "".join("<li><a href='/d/%d'>片名%d</a><span class='price'>%.1f</span></li>"
                       % (i, i, i * 1.5) for i in range(1, 6))
             + "</ul><a class='next' href='/list?page=2'>下一页</a>")

LAZY_BODY = ("<h1>懒加载列表</h1><div id='list'></div>"
             "<div style='height:4000px'></div>"
             "<script>window.addEventListener('scroll',function(){"
             "if(window.__on)return;window.__on=1;"
             "for(var i=0;i<20;i++){var d=document.createElement('div');"
             "d.className='item';d.textContent='滚动条目'+i;"
             "document.getElementById('list').appendChild(d);}});</script>")

SPA_BODY = ("<h1>前端渲染的列表</h1><div id='app'></div><script>setTimeout(function(){var s='';"
            "for(var i=0;i<8;i++){s+='<div class=row>渲染行'+i+'</div>'}"
            "document.getElementById('app').innerHTML=s;},200);</script>")

TABLE_BODY = ("<table class='movies'><tr><th>片名</th><th>年份</th></tr>"
              + "".join("<tr><td>片名%d</td><td>%d</td></tr>" % (i, 1990 + i)
                        for i in range(1, 4)) + "</table>")

COOKIE_BODY = ("<p id='c'></p><script>document.getElementById('c').textContent="
               "'COOKIE='+(document.cookie||'NONE')</script>")

FP_BODY = ("<p id='f'></p><script>document.getElementById('f').textContent="
           "'WD='+navigator.webdriver+'|PL='+navigator.plugins.length+"
           "'|V='+window.innerWidth+'x'+window.innerHeight+'|UA='+navigator.userAgent.slice(0,26)"
           "</script>")

NEXT_LEVEL = {1: ["a", "b"], 2: ["c", "d"], 3: ["e"]}


def level_body(n):
    links = "".join("<a href='/lvl%d/%s'>%s</a>" % (n + 1, x, x)
                    for x in NEXT_LEVEL.get(n, []))
    return html("第%d层" % n, "<h1>LEVEL-%d</h1>%s" % (n, links))


class Handler(BaseHTTPRequestHandler):

    def reply(self, text, ctype="text/html; charset=utf-8", code=200, cookie=None):
        data = text.encode("utf-8")
        self.send_response(code)
        self.send_header("Content-Type", ctype)
        self.send_header("Content-Length", str(len(data)))
        if cookie:
            self.send_header("Set-Cookie", cookie)
        self.end_headers()
        self.wfile.write(data)

    def do_GET(self):
        import time
        path = self.path.split("?")[0]
        HITS[path] = HITS.get(path, 0) + 1
        if path == "/robots.txt":
            return self.reply("User-agent: *\nDisallow: /private/\n", "text/plain")
        if path.startswith("/private"):
            return self.reply(html("私密", "<h1>私密区</h1><p>PRIVATE-SECRET-BODY</p>"
                                         "<p>这段说明文字让页面不会被结构性判据当成空壳。</p>"))
        if path == "/count":
            return self.reply(json.dumps({"count": HITS["/count"]}), "application/json")
        if path == "/ua":
            return self.reply(html("UA", "<p>UA=%s</p>" % self.headers.get("User-Agent")))
        if path == "/list":
            return self.reply(html("列表", LIST_BODY))
        if path == "/lazy":
            return self.reply(html("懒加载", LAZY_BODY))
        if path == "/spa":
            return self.reply(html("SPA", SPA_BODY))
        if path == "/table":
            return self.reply(html("表格", TABLE_BODY))
        if path == "/cookie":
            return self.reply(html("Cookie", COOKIE_BODY))
        if path == "/setcookie":
            return self.reply(html("发cookie", COOKIE_BODY), cookie="probe_c4a=1; Path=/")
        if path == "/fp":
            return self.reply(html("指纹", FP_BODY))
        match = re.match(r"/lvl(\d)/([a-z])", path)
        if match:
            n = int(match.group(1))
            if n > 3:
                return self.reply(html("叶子", "NO-MORE-LEVEL"), code=404)
            return self.reply(level_body(n))
        match = re.match(r"/d/(\d+)", path)
        if match:
            i = int(match.group(1))
            return self.reply(html("片名%d" % i,
                                   "<h1 class='t'>片名%d</h1><span class='p'>%.1f</span>"
                                   % (i, i * 1.5)))
        if path == "/status500":
            return self.reply("boom", "text/plain", 500)
        if path.startswith("/slow"):
            time.sleep(2.5)
            return self.reply(html("慢", "SLOW-DONE"))
        return self.reply(html("首页", "<a href='/list'>去列表</a>"))

    def do_POST(self):
        size = int(self.headers.get("Content-Length", 0))
        body = self.rfile.read(size).decode("utf-8")
        HITS["/echo"] = HITS.get("/echo", 0) + 1
        return self.reply(html("回显", "<pre>POSTED %s</pre>" % body))

    def log_message(self, *args):
        pass


def serve():
    server = HTTPServer(("127.0.0.1", PORT), Handler)
    # 客户端超时断开时 socketserver 会往 stderr 打整段 traceback（里面是临时端口号），
    # 那不是结论只是噪声，接住。
    server.handle_error = lambda request, client_address: None
    threading.Thread(target=server.serve_forever, daemon=True).start()
    return server


srv = serve()


def quiet():
    """静音 crawl4ai 的计时行：[INIT] 那行归 BrowserConfig.verbose 管，
    [FETCH]/[SCRAPE]/[COMPLETE] 三行归 CrawlerRunConfig.verbose 管，两个都要关，
    输出里才不会留下每次不同的耗时。"""
    return BrowserConfig(verbose=False), CrawlerRunConfig(verbose=False)



import urllib.request

for path in ["/list", "/d/2", "/count", "/ua", "/robots.txt", "/private/", "/table", "/lvl1/a"]:
    with urllib.request.urlopen(BASE + path, timeout=10) as resp:
        print("%-12s -> %s %s" % (path, resp.status,
                                  resp.read().decode("utf-8")[:52].replace("\n", " ")))
req = urllib.request.Request(BASE + "/echo", data=b"name=%E7%89%87%E5%90%8D3")
print("POST /echo ->", urllib.request.urlopen(req, timeout=10).read()
      .decode("utf-8").split("<pre>")[1].split("</pre>")[0])
print("服务端命中 =", sorted(HITS.items()))
srv.shutdown()
1
2
3
4
5
6
7
8
9
10
/list        -> 200 <!doctype html><html><head><meta charset='utf-8'><ti
/d/2         -> 200 <!doctype html><html><head><meta charset='utf-8'><ti
/count       -> 200 {"count": 1}
/ua          -> 200 <!doctype html><html><head><meta charset='utf-8'><ti
/robots.txt  -> 200 User-agent: * Disallow: /private/ 
/private/    -> 200 <!doctype html><html><head><meta charset='utf-8'><ti
/table       -> 200 <!doctype html><html><head><meta charset='utf-8'><ti
/lvl1/a      -> 200 <!doctype html><html><head><meta charset='utf-8'><ti
POST /echo -> POSTED name=%E7%89%87%E5%90%8D3
服务端命中 = [('/count', 1), ('/d/2', 1), ('/echo', 1), ('/list', 1), ('/lvl1/a', 1), ('/private/', 1), ('/robots.txt', 1), ('/table', 1), ('/ua', 1)]

各端点的分工：

  • /list、/d/N：列表页与详情页，结构化抽取一节用它们的 class="name"、.price、<table>。
  • /count：服务端计数，缓存那一节就是靠它判「第二次到底读没读缓存」——不看耗时，看服务端被打了几次。URL 上带 ?mode=<模式名>，让五种 CacheMode 各用各的缓存键，免得前一种写完、后一种直接命中。
  • /robots.txt 与 /private/：robots.txt 写 Disallow: /private/，/private/ 里有 PRIVATE-SECRET-BODY 这串指纹，遵守与否一眼就能判。
  • /echo：do_POST 回显请求体，表单与 POST 那一节用它。
  • /lazy：必须滚动才追加 20 条目的页面；/spa：延时 200ms 的 JS 才渲染 8 行的页面。这两个是「等待类参数」的实验台。
  • /table：一张真表格，表格抽取用；/lvl1/a → /lvl2/* → /lvl3/* → /lvl4/e：链式四层，深度抓取用它数 max_depth。
  • /ua、/fp：把请求头里的 UA 和页面里的 navigator.webdriver、navigator.plugins.length、innerWidth×innerHeight 回显出来，UA/视口/反检测三节共用。
  • 判「请求真的到了」的口径：HITS 字典由靶站自己按路径计数，最后 sorted(HITS.items()) 打印；/count 的响应体是 JSON，小页面会被 crawl4ai 的结构性判据误判成「被拦截」（下一节专门讲），所以那些地方只把它当服务端计数源用，不看 success。
  • 每个块结尾都 srv.shutdown()，并且各块用不同端口（8060 起），不给下一块留残留监听——Windows 上 HTTPServer.allow_reuse_address=1 会让两个进程同时绑一个端口把请求偷走，这个亏已经吃过。

抓一页给什么

官方最小示例只有四行：async with AsyncWebCrawler() as crawler: result = await crawler.arun(url=...)。这段照抄官方写法，但把判读顺序按文档说的 success → status_code → error_message → markdown 排一遍，并且用两种引擎各抓一次 ssr1 与 spa1：

render_vs_not.py
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
# -*- coding: utf-8 -*-
"""同一个在线靶站，用浏览器抓与用 HTTP 策略抓，差别正好是「要不要开浏览器」的那笔账。"""

import asyncio
import re

from crawl4ai import AsyncWebCrawler, BrowserConfig, CrawlerRunConfig, HTTPCrawlerConfig
from crawl4ai.async_crawler_strategy import AsyncHTTPCrawlerStrategy

SSR = "https://ssr1.scrape.center/page/1"
SPA = "https://spa1.scrape.center"


def names(html):
    return re.findall(r'class="name">\s*<h2[^>]*>([^<]+)', html or "")


async def by_browser(url):
    async with AsyncWebCrawler(config=BrowserConfig(verbose=False)) as cr:
        return await cr.arun(url, config=CrawlerRunConfig(verbose=False, cache_mode="bypass"))


async def by_http(url):
    strategy = AsyncHTTPCrawlerStrategy(browser_config=HTTPCrawlerConfig())
    async with AsyncWebCrawler(config=BrowserConfig(verbose=False), crawler_strategy=strategy) as cr:
        return await cr.arun(url, config=CrawlerRunConfig(verbose=False, cache_mode="bypass"))


async def main():
    for label, url in [("服务端渲染 ssr1", SSR), ("前端渲染 spa1", SPA)]:
        browser = await by_browser(url)
        http = await by_http(url)
        print("%-16s 浏览器：状态 %s，抓到片名 %d 个，markdown 行数 %d"
              % (label, browser.status_code, len(names(browser.html)),
                 len((browser.markdown or "").splitlines())))
        print("%-16s HTTP  ：状态 %s，抓到片名 %d 个，markdown 行数 %d，success=%s"
              % ("", http.status_code, len(names(http.html)),
                 len((http.markdown or "").splitlines()), http.success))
    print("ssr1 浏览器抓到的前 3 个片名 =", (names((await by_browser(SSR)).html))[:3])

asyncio.run(main())
1
2
3
4
5
服务端渲染 ssr1       浏览器：状态 200，抓到片名 10 个，markdown 行数 74
                 HTTP  ：状态 200，抓到片名 10 个，markdown 行数 74，success=True
前端渲染 spa1        浏览器：状态 200，抓到片名 0 个，markdown 行数 3
                 HTTP  ：状态 200，抓到片名 0 个，markdown 行数 1，success=False
ssr1 浏览器抓到的前 3 个片名 = ['霸王别姬 - Farewell My Concubine', '这个杀手不太冷 - Léon', '肖申克的救赎 - The Shawshank Redemption']
  • ssr1（服务端渲染）两种引擎拿到的东西一模一样：浏览器抓是 74 行 markdown、10 个片名；不执行 JS 的 HTTP 策略也是 74 行、10 个片名。也就是说对服务端渲染的站点，开浏览器是纯浪费。
  • spa1（前端渲染）差别就在这儿：浏览器抓到 3 行 markdown（页面骨架 + 一点提示），HTTP 策略只有 1 行而且 success=False——它连骨架都没给过判据。片名两种都是 0 个，因为 spa1 的数据要靠接口（见《爬虫-Requests》里 limit/offset 那条线）。
  • 片名的取法要记一下：<a href="/detail/1" class="name"> 里套 <h2 class="m-b-sm">，class="name" 在 <a> 上、不在 <h2> 上，写 <h2 class="name"> 一个都取不到；能用的是 class="name">\s*<h2[^>]*>([^<]+)。
  • 判读顺序不能颠倒：result.success 是布尔、status_code 是 HTTP 状态、error_message 是文字。crawl4ai 有一类「状态 200 但 success=False」的情况（下一节），只看状态码会漏。

结果对象的字段

CrawlResult 是 Pydantic 模型（不是 dataclass），所以 dir() 出来的公开成员几乎全是 Pydantic 自带方法，真正该看的是 model_fields：

result_fields.py
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
# -*- coding: utf-8 -*-
"""结果对象 CrawlResult 的字段面：它是 Pydantic 模型，字段要看 model_fields，dir() 会骗人。"""

import asyncio
import dataclasses

from crawl4ai import AsyncWebCrawler, BrowserConfig, CrawlerRunConfig
from crawl4ai.models import CrawlResult

fields = sorted(CrawlResult.model_fields)
print("model_fields 总数 =", len(fields))
for i in range(0, len(fields), 9):
    print("   ", ", ".join(fields[i:i + 9]))
print("是 dataclass 吗 =", dataclasses.is_dataclass(CrawlResult),
      "| 有 model_fields 吗 =", hasattr(CrawlResult, "model_fields"))
print("dir(CrawlResult) 里公开成员数 =", len([n for n in dir(CrawlResult)
                                             if not n.startswith("_")]),
      "（多出来的是 Pydantic 自带方法，不是字段）")

r = CrawlResult(url="x", html="<p>y</p>", success=True, status_code=200)
print("只给三个必填之外多给 status_code 也能构造 =", r.url, r.status_code, r.success)
try:
    CrawlResult(url="x")
except Exception as exc:
    print("只给 url ->", type(exc).__name__, "| 报错里提到几个缺失字段 =",
          str(exc).count("Field required"))


async def main():
    async with AsyncWebCrawler(config=BrowserConfig(verbose=False)) as cr:
        got = await cr.arun("https://ssr1.scrape.center",
                            config=CrawlerRunConfig(verbose=False, cache_mode="bypass"))
    print("实抓一次之后：markdown 类型 =", type(got.markdown).__name__,
          "| cache_status =", got.cache_status, "| session_id 有没有 =", bool(got.session_id))
    try:
        print("老写法 got.meta =", got.meta)
    except Exception as exc:
        print("老写法 got.meta 抛 %s: %s" % (type(exc).__name__, str(exc)[:74]))
    print("正确写法 got.metadata =", got.metadata)


asyncio.run(main())
1
2
3
4
5
6
7
8
9
10
11
12
model_fields 总数 = 29
    cache_status, cached_at, cleaned_html, console_messages, crawl_stats, dispatch_result, downloaded_files, error_message, extracted_content
    fit_html, head_fingerprint, html, js_execution_result, links, media, metadata, mhtml, network_requests
    pdf, redirected_status_code, redirected_url, response_headers, screenshot, session_id, ssl_certificate, status_code, success
    tables, url
是 dataclass 吗 = False | 有 model_fields 吗 = True
dir(CrawlResult) 里公开成员数 = 31 （多出来的是 Pydantic 自带方法，不是字段）
只给三个必填之外多给 status_code 也能构造 = x 200 True
只给 url -> ValidationError | 报错里提到几个缺失字段 = 2
实抓一次之后：markdown 类型 = StringCompatibleMarkdown | cache_status = miss | session_id 有没有 = False
老写法 got.meta 抛 AttributeError: 'CrawlResult' object has no attribute 'meta'
正确写法 got.metadata = {'title': 'Scrape | Movie', 'description': None, 'keywords': None, 'author': None}
  • 29 个字段，按用途分四组：内容是 html、cleaned_html、fit_html、markdown、extracted_content、tables、media、links、mhtml；状态是 success、status_code、redirected_status_code、error_message、cache_status、cached_at；来源与元信息是 url、redirected_url、metadata、response_headers、ssl_certificate、head_fingerprint、session_id；附加产物是 screenshot、pdf、downloaded_files、console_messages、network_requests、js_execution_result、crawl_stats、dispatch_result。
  • 文档里若干页面写的 result.meta 在本机 0.9.5 会直接抛 AttributeError，正确字段是 metadata（实抓 ssr1 拿到 {'title': 'Scrape | Movie', 'description': None, 'keywords': None, 'author': None}）。报错信息还会友好地提示 Did you mean: 'media'?，别照着它抄。
  • 必填只有三个：url、html、success。用 CrawlResult(url=..., html=..., success=...) 能构造出来，只给 url 会抛 ValidationError（报错里数得出来缺 2 个字段）——这意味着你可以自己造 CrawlResult 喂给后续流程，做「先离线解析再入库」的解耦。
  • markdown 的类型默认是 str；传了 markdown_generator 之后是 StringCompatibleMarkdown（一个「字符串兼容」类，所以 str(...)、切片、in 都照旧能用），要看 fit_markdown 这些产物得直接调生成器，见「markdown 是怎么生成的」。
  • dir(obj) 与 model_fields 的差别（本机是 31 对 29）就是 Pydantic 方法名混进来了，以后查任何 Pydantic 模型的字段都走 model_fields。

不开浏览器的路径

不是所有站点都需要浏览器。crawl4ai 提供 AsyncHTTPCrawlerStrategy + HTTPCrawlerConfig，用 aiohttp 直接发请求，接口保持和浏览器路径一致（同样返回 CrawlResult）：

http_strategy.py
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
# -*- coding: utf-8 -*-
"""不想开浏览器时可以换 HTTP 策略：参数面、拿不到 JS 结果、以及它自己那套反爬判据。"""

import asyncio
import inspect
import json
import os
import re
import threading
from http.server import BaseHTTPRequestHandler, HTTPServer

# 云广告一天只打一次，打不打取决于家目录里的标记文件，会把输出变得不可复现，所以先静音
os.environ.setdefault("CRAWL4AI_NO_CLOUD_NOTICE", "1")

from crawl4ai import (AsyncWebCrawler, BrowserConfig, CacheMode, CrawlerRunConfig)
from crawl4ai.async_crawler_strategy import AsyncPlaywrightCrawlerStrategy


PORT = 8087
BASE = "http://127.0.0.1:8087"
HITS = {}


def html(title, body):
    return ("<!doctype html><html><head><meta charset='utf-8'><title>%s</title>"
            "<meta name='description' content='probe'><style>.nav{color:red}</style>"
            "</head><body>"
            "<div class='nav'>NAV 这是靶站的导航区，故意写长一点，"
            "免得页面被 crawl4ai 的结构性判据当成空壳</div>"
            "%s"
            "<div class='filler'>FILLER 这是靶站的页脚区，同样是有实际文字的标记</div>"
            "</body></html>" % (title, body))


LIST_BODY = ("<h1>列表页</h1><ul class='items'>"
             + "".join("<li><a href='/d/%d'>片名%d</a><span class='price'>%.1f</span></li>"
                       % (i, i, i * 1.5) for i in range(1, 6))
             + "</ul><a class='next' href='/list?page=2'>下一页</a>")

LAZY_BODY = ("<h1>懒加载列表</h1><div id='list'></div>"
             "<div style='height:4000px'></div>"
             "<script>window.addEventListener('scroll',function(){"
             "if(window.__on)return;window.__on=1;"
             "for(var i=0;i<20;i++){var d=document.createElement('div');"
             "d.className='item';d.textContent='滚动条目'+i;"
             "document.getElementById('list').appendChild(d);}});</script>")

SPA_BODY = ("<h1>前端渲染的列表</h1><div id='app'></div><script>setTimeout(function(){var s='';"
            "for(var i=0;i<8;i++){s+='<div class=row>渲染行'+i+'</div>'}"
            "document.getElementById('app').innerHTML=s;},200);</script>")

TABLE_BODY = ("<table class='movies'><tr><th>片名</th><th>年份</th></tr>"
              + "".join("<tr><td>片名%d</td><td>%d</td></tr>" % (i, 1990 + i)
                        for i in range(1, 4)) + "</table>")

COOKIE_BODY = ("<p id='c'></p><script>document.getElementById('c').textContent="
               "'COOKIE='+(document.cookie||'NONE')</script>")

FP_BODY = ("<p id='f'></p><script>document.getElementById('f').textContent="
           "'WD='+navigator.webdriver+'|PL='+navigator.plugins.length+"
           "'|V='+window.innerWidth+'x'+window.innerHeight+'|UA='+navigator.userAgent.slice(0,26)"
           "</script>")

NEXT_LEVEL = {1: ["a", "b"], 2: ["c", "d"], 3: ["e"]}


def level_body(n):
    links = "".join("<a href='/lvl%d/%s'>%s</a>" % (n + 1, x, x)
                    for x in NEXT_LEVEL.get(n, []))
    return html("第%d层" % n, "<h1>LEVEL-%d</h1>%s" % (n, links))


class Handler(BaseHTTPRequestHandler):

    def reply(self, text, ctype="text/html; charset=utf-8", code=200, cookie=None):
        data = text.encode("utf-8")
        self.send_response(code)
        self.send_header("Content-Type", ctype)
        self.send_header("Content-Length", str(len(data)))
        if cookie:
            self.send_header("Set-Cookie", cookie)
        self.end_headers()
        self.wfile.write(data)

    def do_GET(self):
        import time
        path = self.path.split("?")[0]
        HITS[path] = HITS.get(path, 0) + 1
        if path == "/robots.txt":
            return self.reply("User-agent: *\nDisallow: /private/\n", "text/plain")
        if path.startswith("/private"):
            return self.reply(html("私密", "<h1>私密区</h1><p>PRIVATE-SECRET-BODY</p>"
                                         "<p>这段说明文字让页面不会被结构性判据当成空壳。</p>"))
        if path == "/count":
            return self.reply(json.dumps({"count": HITS["/count"]}), "application/json")
        if path == "/ua":
            return self.reply(html("UA", "<p>UA=%s</p>" % self.headers.get("User-Agent")))
        if path == "/list":
            return self.reply(html("列表", LIST_BODY))
        if path == "/lazy":
            return self.reply(html("懒加载", LAZY_BODY))
        if path == "/spa":
            return self.reply(html("SPA", SPA_BODY))
        if path == "/table":
            return self.reply(html("表格", TABLE_BODY))
        if path == "/cookie":
            return self.reply(html("Cookie", COOKIE_BODY))
        if path == "/setcookie":
            return self.reply(html("发cookie", COOKIE_BODY), cookie="probe_c4a=1; Path=/")
        if path == "/fp":
            return self.reply(html("指纹", FP_BODY))
        match = re.match(r"/lvl(\d)/([a-z])", path)
        if match:
            n = int(match.group(1))
            if n > 3:
                return self.reply(html("叶子", "NO-MORE-LEVEL"), code=404)
            return self.reply(level_body(n))
        match = re.match(r"/d/(\d+)", path)
        if match:
            i = int(match.group(1))
            return self.reply(html("片名%d" % i,
                                   "<h1 class='t'>片名%d</h1><span class='p'>%.1f</span>"
                                   % (i, i * 1.5)))
        if path == "/status500":
            return self.reply("boom", "text/plain", 500)
        if path.startswith("/slow"):
            time.sleep(2.5)
            return self.reply(html("慢", "SLOW-DONE"))
        return self.reply(html("首页", "<a href='/list'>去列表</a>"))

    def do_POST(self):
        size = int(self.headers.get("Content-Length", 0))
        body = self.rfile.read(size).decode("utf-8")
        HITS["/echo"] = HITS.get("/echo", 0) + 1
        return self.reply(html("回显", "<pre>POSTED %s</pre>" % body))

    def log_message(self, *args):
        pass


def serve():
    server = HTTPServer(("127.0.0.1", PORT), Handler)
    # 客户端超时断开时 socketserver 会往 stderr 打整段 traceback（里面是临时端口号），
    # 那不是结论只是噪声，接住。
    server.handle_error = lambda request, client_address: None
    threading.Thread(target=server.serve_forever, daemon=True).start()
    return server


srv = serve()


def quiet():
    """静音 crawl4ai 的计时行：[INIT] 那行归 BrowserConfig.verbose 管，
    [FETCH]/[SCRAPE]/[COMPLETE] 三行归 CrawlerRunConfig.verbose 管，两个都要关，
    输出里才不会留下每次不同的耗时。"""
    return BrowserConfig(verbose=False), CrawlerRunConfig(verbose=False)



from crawl4ai import HTTPCrawlerConfig
from crawl4ai.async_crawler_strategy import AsyncHTTPCrawlerStrategy

hnames = list(inspect.signature(HTTPCrawlerConfig.__init__).parameters)[1:]
print("HTTPCrawlerConfig 的参数 %d 个 =" % len(hnames), ", ".join(hnames))
print("它没有 viewport 这一类参数 =", not hasattr(HTTPCrawlerConfig(), "viewport_width"))


async def grab(label, **http_kwargs):
    strategy = AsyncHTTPCrawlerStrategy(browser_config=HTTPCrawlerConfig(**http_kwargs))
    async with AsyncWebCrawler(config=BrowserConfig(verbose=False),
                               crawler_strategy=strategy) as cr:
        r = await cr.arun(BASE + "/spa", config=CrawlerRunConfig(verbose=False))
        print("%-30s success=%-5s 状态 %-5s 渲染行 %d | markdown 行数 %d | error=%r"
              % (label, r.success, r.status_code, (r.html or "").count("渲染行"),
                 len((r.markdown or "").splitlines()), (r.error_message or "")[:40]))
    return r


async def main():
    await grab("默认 GET")
    await grab("headers 里换 UA", headers={"User-Agent": "MyCrawler/1.0"})
    await grab("verify_ssl=False", verify_ssl=False)
    await grab("follow_redirects=False", follow_redirects=False)
    big = await grab("换成在线 ssr1（大页面）")
    async with AsyncWebCrawler(config=BrowserConfig(verbose=False),
                               crawler_strategy=AsyncHTTPCrawlerStrategy(
                                   browser_config=HTTPCrawlerConfig())) as cr:
        r = await cr.arun("https://ssr1.scrape.center", config=CrawlerRunConfig(verbose=False))
        print("ssr1 走 HTTP 策略：状态 %s，正文里片名个数 %d"
              % (r.status_code, (r.html or "").count('class="name"')))
        print("浏览器抓同一个页面得到的 markdown 行数 =",
              len((r.markdown or "").splitlines()))

asyncio.run(main())
1
2
3
4
5
6
7
8
9
HTTPCrawlerConfig 的参数 7 个 = method, headers, data, json, follow_redirects, verify_ssl, downloads_path
它没有 viewport 这一类参数 = True
默认 GET                         success=True  状态 200   渲染行 1 | markdown 行数 3 | error=''
headers 里换 UA                  success=True  状态 200   渲染行 1 | markdown 行数 3 | error=''
verify_ssl=False               success=True  状态 200   渲染行 1 | markdown 行数 3 | error=''
follow_redirects=False         success=True  状态 200   渲染行 1 | markdown 行数 3 | error=''
换成在线 ssr1（大页面）                 success=True  状态 200   渲染行 1 | markdown 行数 3 | error=''
ssr1 走 HTTP 策略：状态 200，正文里片名个数 10
浏览器抓同一个页面得到的 markdown 行数 = 74
  • HTTPCrawlerConfig 只有 7 个参数：method、headers、data、json、follow_redirects、verify_ssl、downloads_path。它没有 viewport_width 这类浏览器参数（hasattr 是 False），这是最容易误会的地方：换策略不是换配置对象的名字，而是换一整套参数面。
  • 抓 SPA 页面时 渲染行 1 | markdown 行数 3 的含义要说清：那 1 处是 JS 源码里的字符串字面量，渲染结果 0 行——HTTP 路径不执行 JS，setTimeout 那段永远不会跑。这也是为什么「不执行 JS」这一路只能验服务端渲染的站点。
  • 抓在线 ssr1 走 HTTP 策略：片名 10 个都在，而浏览器抓到的 markdown 也是 74 行，两边内容等价。所以「值不值得开浏览器」的判据就是这一条：正文已经在 HTML 里就别开。
  • headers/verify_ssl/follow_redirects 都实测「参数被接受且请求照发」（本机靶站没有 HTTPS，verify_ssl 的实际效果只能在自签证书上看，那条在《爬虫-基本入门》的四种 verify 里测过）。POST 表单的正路见「JS 注入与表单抓取」一节。

没被拦却报被拦

这一节解释上一节里那句「success=False 但状态码 200」。crawl4ai 在返回结果前会跑一个 antibot_detector.is_blocked(状态码, html)，本机 0.9.5 的判据是这样的：

antibot_heuristic.py
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
# -*- coding: utf-8 -*-
"""「没被拦却报被拦」：0.9.5 自带一个结构性的反爬判据，小页面会被误判。"""

import asyncio
import inspect
import json
import os
import re
import threading
from http.server import BaseHTTPRequestHandler, HTTPServer

# 云广告一天只打一次，打不打取决于家目录里的标记文件，会把输出变得不可复现，所以先静音
os.environ.setdefault("CRAWL4AI_NO_CLOUD_NOTICE", "1")

from crawl4ai import (AsyncWebCrawler, BrowserConfig, CacheMode, CrawlerRunConfig)
from crawl4ai.async_crawler_strategy import AsyncPlaywrightCrawlerStrategy


PORT = 8067
BASE = "http://127.0.0.1:8067"
HITS = {}


def html(title, body):
    return ("<!doctype html><html><head><meta charset='utf-8'><title>%s</title>"
            "<meta name='description' content='probe'><style>.nav{color:red}</style>"
            "</head><body>"
            "<div class='nav'>NAV 这是靶站的导航区，故意写长一点，"
            "免得页面被 crawl4ai 的结构性判据当成空壳</div>"
            "%s"
            "<div class='filler'>FILLER 这是靶站的页脚区，同样是有实际文字的标记</div>"
            "</body></html>" % (title, body))


LIST_BODY = ("<h1>列表页</h1><ul class='items'>"
             + "".join("<li><a href='/d/%d'>片名%d</a><span class='price'>%.1f</span></li>"
                       % (i, i, i * 1.5) for i in range(1, 6))
             + "</ul><a class='next' href='/list?page=2'>下一页</a>")

LAZY_BODY = ("<h1>懒加载列表</h1><div id='list'></div>"
             "<div style='height:4000px'></div>"
             "<script>window.addEventListener('scroll',function(){"
             "if(window.__on)return;window.__on=1;"
             "for(var i=0;i<20;i++){var d=document.createElement('div');"
             "d.className='item';d.textContent='滚动条目'+i;"
             "document.getElementById('list').appendChild(d);}});</script>")

SPA_BODY = ("<h1>前端渲染的列表</h1><div id='app'></div><script>setTimeout(function(){var s='';"
            "for(var i=0;i<8;i++){s+='<div class=row>渲染行'+i+'</div>'}"
            "document.getElementById('app').innerHTML=s;},200);</script>")

TABLE_BODY = ("<table class='movies'><tr><th>片名</th><th>年份</th></tr>"
              + "".join("<tr><td>片名%d</td><td>%d</td></tr>" % (i, 1990 + i)
                        for i in range(1, 4)) + "</table>")

COOKIE_BODY = ("<p id='c'></p><script>document.getElementById('c').textContent="
               "'COOKIE='+(document.cookie||'NONE')</script>")

FP_BODY = ("<p id='f'></p><script>document.getElementById('f').textContent="
           "'WD='+navigator.webdriver+'|PL='+navigator.plugins.length+"
           "'|V='+window.innerWidth+'x'+window.innerHeight+'|UA='+navigator.userAgent.slice(0,26)"
           "</script>")

NEXT_LEVEL = {1: ["a", "b"], 2: ["c", "d"], 3: ["e"]}


def level_body(n):
    links = "".join("<a href='/lvl%d/%s'>%s</a>" % (n + 1, x, x)
                    for x in NEXT_LEVEL.get(n, []))
    return html("第%d层" % n, "<h1>LEVEL-%d</h1>%s" % (n, links))


class Handler(BaseHTTPRequestHandler):

    def reply(self, text, ctype="text/html; charset=utf-8", code=200, cookie=None):
        data = text.encode("utf-8")
        self.send_response(code)
        self.send_header("Content-Type", ctype)
        self.send_header("Content-Length", str(len(data)))
        if cookie:
            self.send_header("Set-Cookie", cookie)
        self.end_headers()
        self.wfile.write(data)

    def do_GET(self):
        import time
        path = self.path.split("?")[0]
        HITS[path] = HITS.get(path, 0) + 1
        if path == "/robots.txt":
            return self.reply("User-agent: *\nDisallow: /private/\n", "text/plain")
        if path.startswith("/private"):
            return self.reply(html("私密", "<h1>私密区</h1><p>PRIVATE-SECRET-BODY</p>"
                                         "<p>这段说明文字让页面不会被结构性判据当成空壳。</p>"))
        if path == "/count":
            return self.reply(json.dumps({"count": HITS["/count"]}), "application/json")
        if path == "/ua":
            return self.reply(html("UA", "<p>UA=%s</p>" % self.headers.get("User-Agent")))
        if path == "/list":
            return self.reply(html("列表", LIST_BODY))
        if path == "/lazy":
            return self.reply(html("懒加载", LAZY_BODY))
        if path == "/spa":
            return self.reply(html("SPA", SPA_BODY))
        if path == "/table":
            return self.reply(html("表格", TABLE_BODY))
        if path == "/cookie":
            return self.reply(html("Cookie", COOKIE_BODY))
        if path == "/setcookie":
            return self.reply(html("发cookie", COOKIE_BODY), cookie="probe_c4a=1; Path=/")
        if path == "/fp":
            return self.reply(html("指纹", FP_BODY))
        match = re.match(r"/lvl(\d)/([a-z])", path)
        if match:
            n = int(match.group(1))
            if n > 3:
                return self.reply(html("叶子", "NO-MORE-LEVEL"), code=404)
            return self.reply(level_body(n))
        match = re.match(r"/d/(\d+)", path)
        if match:
            i = int(match.group(1))
            return self.reply(html("片名%d" % i,
                                   "<h1 class='t'>片名%d</h1><span class='p'>%.1f</span>"
                                   % (i, i * 1.5)))
        if path == "/status500":
            return self.reply("boom", "text/plain", 500)
        if path.startswith("/slow"):
            time.sleep(2.5)
            return self.reply(html("慢", "SLOW-DONE"))
        return self.reply(html("首页", "<a href='/list'>去列表</a>"))

    def do_POST(self):
        size = int(self.headers.get("Content-Length", 0))
        body = self.rfile.read(size).decode("utf-8")
        HITS["/echo"] = HITS.get("/echo", 0) + 1
        return self.reply(html("回显", "<pre>POSTED %s</pre>" % body))

    def log_message(self, *args):
        pass


def serve():
    server = HTTPServer(("127.0.0.1", PORT), Handler)
    # 客户端超时断开时 socketserver 会往 stderr 打整段 traceback（里面是临时端口号），
    # 那不是结论只是噪声，接住。
    server.handle_error = lambda request, client_address: None
    threading.Thread(target=server.serve_forever, daemon=True).start()
    return server


srv = serve()


def quiet():
    """静音 crawl4ai 的计时行：[INIT] 那行归 BrowserConfig.verbose 管，
    [FETCH]/[SCRAPE]/[COMPLETE] 三行归 CrawlerRunConfig.verbose 管，两个都要关，
    输出里才不会留下每次不同的耗时。"""
    return BrowserConfig(verbose=False), CrawlerRunConfig(verbose=False)



from crawl4ai.antibot_detector import is_blocked

SHELL = "<html><body><div id='app'></div><script>/* 一堆 JS */</script></body></html>"
TEXTY = "<html><body><p>" + "这段文字足够长，用来把可见文字从 50 字等几个信号里摘出去。" * 3 + "</p></body></html>"
BIG = "<html><body>" + "".join("<article>第%d段正文，用来把页面撑过 50000 字节的门槛。</article>" % i
                               for i in range(1200)) + "</body></html>"

for label, doc in [("光壳子（有 script 没内容）", SHELL), ("有正文的小页面", TEXTY), ("大页面", BIG)]:
    print("%-24s 长度 %6d is_blocked -> %s" % (label, len(doc), is_blocked(200, doc)))
print("状态码 403 加内容 ->", is_blocked(403, TEXTY))
print("状态码 429 加空正文 ->", is_blocked(429, ""))


async def main():
    bc, _ = quiet()
    url = BASE + "/private/"
    async with AsyncWebCrawler(config=bc) as cr:
        r = await cr.arun(url, config=CrawlerRunConfig(verbose=False))
        print("抓小页面 success=%s 状态 %s error=%r" % (r.success, r.status_code, r.error_message))
        print("   html 里确实有 PRIVATE-SECRET-BODY =", "PRIVATE-SECRET-BODY" in r.html)
    async with AsyncWebCrawler(config=bc) as cr:
        r2 = await cr.arun(BASE + "/list", config=CrawlerRunConfig(verbose=False))
        print("抓有内容的列表页 success=%s error=%r" % (r2.success, r2.error_message))

asyncio.run(main())
1
2
3
4
5
6
7
8
光壳子（有 script 没内容）        长度     74 is_blocked -> (True, 'Near-empty content (74 bytes) with HTTP 200')
有正文的小页面                  长度    126 is_blocked -> (False, '')
大页面                      长度  56516 is_blocked -> (False, '')
状态码 403 加内容 -> (True, 'HTTP 403 with HTML content (126 bytes)')
状态码 429 加空正文 -> (True, 'HTTP 429 Too Many Requests')
抓小页面 success=True 状态 200 error=''
   html 里确实有 PRIVATE-SECRET-BODY = True
抓有内容的列表页 success=True error=''
  • 判据是分层的：状态码 403/429 配上正文就判「被拦」；小于 50 KB 的 HTML 还要过一道结构性检查——没有 <body> 直接判死；否则看三个信号：可见文字少于 50 字（minimal_text）、没有任何语义内容标签（no_content_elements，判据是 <p|h1-h6|article|section|li|td|a|pre>）、有 script 但没内容（script_heavy_shell）。两个信号就判死；只有一个信号但页面小于 5000 字节也判死。
  • 上面的输出把这把尺子量出来了：74 字节的空壳判死、126 字节但有正文的小页不判死、56 KB 的大页完全不检查；403 + 有正文判死、429 + 空正文判死。
  • 于是就有了那条容易误读的实测：抓 /private/（一个 304 字节的说明页）时 success=False、error_message='Blocked by anti-bot protection: Structural: no_content_elements on small page (304 bytes, 91 chars visible)'，但 html 里 PRIVATE-SECRET-BODY 确实在——它不是被拦，是被自己的启发式误判。「抓到了却告诉你没抓到」比反过来更难查，所以判读顺序要加上一步：success=False 时先看 error_message 是不是 Blocked by anti-bot protection，是就回去核对 html 与 status_code。
  • 这条对写代码的直接影响：靶站/测试页要够大。本篇后来把靶站的导航区和页脚写成有实际文字的长句（见「本机靶站」一节的输出），否则每个块都会被这层判据污染，success 一路 False，测什么都像在测反爬。
  • 附带一条：判定成「被拦」时库会顺手打一行云广告（Crawl4AI Cloud is live …，一天一次、看家目录里的标记文件）。想让输出可复现，就在脚本开头设 CRAWL4AI_NO_CLOUD_NOTICE=1。

BrowserConfig 参数面

BrowserConfig 管的是「浏览器实例长什么样」，一共 44 个参数。下面这段把签名连默认值原样打出来，然后逐个用页面回读验证它到没到得了页面里：

browser_config_params.py
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
# -*- coding: utf-8 -*-
"""BrowserConfig 的参数面，以及 viewport / user_agent 到底改不改得动页面。"""

import asyncio
import inspect
import json
import os
import re
import threading
from http.server import BaseHTTPRequestHandler, HTTPServer

# 云广告一天只打一次，打不打取决于家目录里的标记文件，会把输出变得不可复现，所以先静音
os.environ.setdefault("CRAWL4AI_NO_CLOUD_NOTICE", "1")

from crawl4ai import (AsyncWebCrawler, BrowserConfig, CacheMode, CrawlerRunConfig)
from crawl4ai.async_crawler_strategy import AsyncPlaywrightCrawlerStrategy


PORT = 8081
BASE = "http://127.0.0.1:8081"
HITS = {}


def html(title, body):
    return ("<!doctype html><html><head><meta charset='utf-8'><title>%s</title>"
            "<meta name='description' content='probe'><style>.nav{color:red}</style>"
            "</head><body>"
            "<div class='nav'>NAV 这是靶站的导航区，故意写长一点，"
            "免得页面被 crawl4ai 的结构性判据当成空壳</div>"
            "%s"
            "<div class='filler'>FILLER 这是靶站的页脚区，同样是有实际文字的标记</div>"
            "</body></html>" % (title, body))


LIST_BODY = ("<h1>列表页</h1><ul class='items'>"
             + "".join("<li><a href='/d/%d'>片名%d</a><span class='price'>%.1f</span></li>"
                       % (i, i, i * 1.5) for i in range(1, 6))
             + "</ul><a class='next' href='/list?page=2'>下一页</a>")

LAZY_BODY = ("<h1>懒加载列表</h1><div id='list'></div>"
             "<div style='height:4000px'></div>"
             "<script>window.addEventListener('scroll',function(){"
             "if(window.__on)return;window.__on=1;"
             "for(var i=0;i<20;i++){var d=document.createElement('div');"
             "d.className='item';d.textContent='滚动条目'+i;"
             "document.getElementById('list').appendChild(d);}});</script>")

SPA_BODY = ("<h1>前端渲染的列表</h1><div id='app'></div><script>setTimeout(function(){var s='';"
            "for(var i=0;i<8;i++){s+='<div class=row>渲染行'+i+'</div>'}"
            "document.getElementById('app').innerHTML=s;},200);</script>")

TABLE_BODY = ("<table class='movies'><tr><th>片名</th><th>年份</th></tr>"
              + "".join("<tr><td>片名%d</td><td>%d</td></tr>" % (i, 1990 + i)
                        for i in range(1, 4)) + "</table>")

COOKIE_BODY = ("<p id='c'></p><script>document.getElementById('c').textContent="
               "'COOKIE='+(document.cookie||'NONE')</script>")

FP_BODY = ("<p id='f'></p><script>document.getElementById('f').textContent="
           "'WD='+navigator.webdriver+'|PL='+navigator.plugins.length+"
           "'|V='+window.innerWidth+'x'+window.innerHeight+'|UA='+navigator.userAgent.slice(0,26)"
           "</script>")

NEXT_LEVEL = {1: ["a", "b"], 2: ["c", "d"], 3: ["e"]}


def level_body(n):
    links = "".join("<a href='/lvl%d/%s'>%s</a>" % (n + 1, x, x)
                    for x in NEXT_LEVEL.get(n, []))
    return html("第%d层" % n, "<h1>LEVEL-%d</h1>%s" % (n, links))


class Handler(BaseHTTPRequestHandler):

    def reply(self, text, ctype="text/html; charset=utf-8", code=200, cookie=None):
        data = text.encode("utf-8")
        self.send_response(code)
        self.send_header("Content-Type", ctype)
        self.send_header("Content-Length", str(len(data)))
        if cookie:
            self.send_header("Set-Cookie", cookie)
        self.end_headers()
        self.wfile.write(data)

    def do_GET(self):
        import time
        path = self.path.split("?")[0]
        HITS[path] = HITS.get(path, 0) + 1
        if path == "/robots.txt":
            return self.reply("User-agent: *\nDisallow: /private/\n", "text/plain")
        if path.startswith("/private"):
            return self.reply(html("私密", "<h1>私密区</h1><p>PRIVATE-SECRET-BODY</p>"
                                         "<p>这段说明文字让页面不会被结构性判据当成空壳。</p>"))
        if path == "/count":
            return self.reply(json.dumps({"count": HITS["/count"]}), "application/json")
        if path == "/ua":
            return self.reply(html("UA", "<p>UA=%s</p>" % self.headers.get("User-Agent")))
        if path == "/list":
            return self.reply(html("列表", LIST_BODY))
        if path == "/lazy":
            return self.reply(html("懒加载", LAZY_BODY))
        if path == "/spa":
            return self.reply(html("SPA", SPA_BODY))
        if path == "/table":
            return self.reply(html("表格", TABLE_BODY))
        if path == "/cookie":
            return self.reply(html("Cookie", COOKIE_BODY))
        if path == "/setcookie":
            return self.reply(html("发cookie", COOKIE_BODY), cookie="probe_c4a=1; Path=/")
        if path == "/fp":
            return self.reply(html("指纹", FP_BODY))
        match = re.match(r"/lvl(\d)/([a-z])", path)
        if match:
            n = int(match.group(1))
            if n > 3:
                return self.reply(html("叶子", "NO-MORE-LEVEL"), code=404)
            return self.reply(level_body(n))
        match = re.match(r"/d/(\d+)", path)
        if match:
            i = int(match.group(1))
            return self.reply(html("片名%d" % i,
                                   "<h1 class='t'>片名%d</h1><span class='p'>%.1f</span>"
                                   % (i, i * 1.5)))
        if path == "/status500":
            return self.reply("boom", "text/plain", 500)
        if path.startswith("/slow"):
            time.sleep(2.5)
            return self.reply(html("慢", "SLOW-DONE"))
        return self.reply(html("首页", "<a href='/list'>去列表</a>"))

    def do_POST(self):
        size = int(self.headers.get("Content-Length", 0))
        body = self.rfile.read(size).decode("utf-8")
        HITS["/echo"] = HITS.get("/echo", 0) + 1
        return self.reply(html("回显", "<pre>POSTED %s</pre>" % body))

    def log_message(self, *args):
        pass


def serve():
    server = HTTPServer(("127.0.0.1", PORT), Handler)
    # 客户端超时断开时 socketserver 会往 stderr 打整段 traceback（里面是临时端口号），
    # 那不是结论只是噪声，接住。
    server.handle_error = lambda request, client_address: None
    threading.Thread(target=server.serve_forever, daemon=True).start()
    return server


srv = serve()


def quiet():
    """静音 crawl4ai 的计时行：[INIT] 那行归 BrowserConfig.verbose 管，
    [FETCH]/[SCRAPE]/[COMPLETE] 三行归 CrawlerRunConfig.verbose 管，两个都要关，
    输出里才不会留下每次不同的耗时。"""
    return BrowserConfig(verbose=False), CrawlerRunConfig(verbose=False)



import inspect

params = inspect.signature(BrowserConfig.__init__).parameters
bnames = list(params)
print("参数总数 =", len(bnames) - 1)
for i in range(0, len(bnames), 9):
    print("   ", ", ".join(bnames[i:i + 9]))
for key in ["browser_type", "headless", "browser_mode", "use_persistent_context",
            "viewport_width", "viewport_height", "viewport", "user_agent", "user_agent_mode",
            "enable_stealth", "verbose", "proxy", "proxy_config", "text_mode", "light_mode"]:
    print("   %-22s 默认 %r" % (key, params[key].default))
print("两个 verbose 的默认值 =", params["verbose"].default,
      "|", inspect.signature(CrawlerRunConfig.__init__).parameters["verbose"].default)


async def read(label, **kwargs):
    cr = AsyncWebCrawler(config=BrowserConfig(verbose=False, **kwargs))
    async with cr:
        r = await cr.arun(BASE + "/fp", config=CrawlerRunConfig(verbose=False))
        line = [l for l in r.markdown.splitlines() if "WD=" in l]
        print("%-40s -> %s" % (label, line[:1]))
    return r


async def main():
    await read("默认（UA 是库自带的 Chrome/116）")
    await read("user_agent 自定义", user_agent="MyCrawler/1.0")
    await read("viewport_width/height=480,320", viewport_width=480, viewport_height=320)
    await read("viewport=dict(320,240)（文档里那种写法）", viewport={"width": 320, "height": 240})
    await read("java_script_enabled=False", java_script_enabled=False)
    await read("headless=False（有头）", headless=False)

asyncio.run(main())
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
参数总数 = 44
    self, browser_type, headless, browser_mode, use_managed_browser, cdp_url, browser_context_id, target_id, cdp_cleanup_on_close
    cdp_close_delay, cache_cdp_connection, create_isolated_context, use_persistent_context, user_data_dir, chrome_channel, channel, proxy, proxy_config
    viewport_width, viewport_height, viewport, device_scale_factor, accept_downloads, downloads_path, storage_state, ignore_https_errors, java_script_enabled
    sleep_on_close, verbose, cookies, headers, user_agent, user_agent_mode, user_agent_generator_config, text_mode, light_mode
    extra_args, debugging_port, host, enable_stealth, avoid_ads, avoid_css, init_scripts, memory_saving_mode, max_pages_before_recycle
   browser_type           默认 'chromium'
   headless               默认 True
   browser_mode           默认 'dedicated'
   use_persistent_context 默认 False
   viewport_width         默认 1080
   viewport_height        默认 600
   viewport               默认 None
   user_agent             默认 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 Chrome/116.0.0.0 Safari/537.36'
   user_agent_mode        默认 ''
   enable_stealth         默认 False
   verbose                默认 True
   proxy                  默认 None
   proxy_config           默认 None
   text_mode              默认 False
   light_mode             默认 False
两个 verbose 的默认值 = True | True
默认（UA 是库自带的 Chrome/116）                  -> ['WD=false|PL=0|V=1080x600|UA=Mozilla/5.0 (X11; Linux x8']
user_agent 自定义                           -> ['WD=false|PL=0|V=1080x600|UA=MyCrawler/1.0']
viewport_width/height=480,320            -> ['WD=false|PL=0|V=480x320|UA=Mozilla/5.0 (X11; Linux x8']
viewport=dict(320,240)（文档里那种写法）          -> ['WD=false|PL=0|V=320x240|UA=Mozilla/5.0 (X11; Linux x8']
java_script_enabled=False                -> []
headless=False（有头）                       -> ['WD=false|PL=5|V=1080x600|UA=Mozilla/5.0 (X11; Linux x8']

默认值里几个要记住的：browser_type='chromium'、headless=True、browser_mode='dedicated'、use_persistent_context=False、viewport_width/height=1080×600、viewport=None、user_agent 是一个 Linux x86_64 的 Chrome/116 固定串、enable_stealth=False、verbose=True。

实测出来的四件事：

  • UA 是能被覆盖的：user_agent="MyCrawler/1.0" 之后页面里回读到的就是它。但别忘了 enable_stealth=False 时 UA 与真实浏览器的其它指纹位并不匹配，覆盖 UA 只是「看起来像」，这条在「反检测那三层」里量化。
  • 视口有两个入口，只有一条真生效路径：viewport_width/height=480,320 改得动（回读 V=480x320）；viewport={"width": 320, "height": 240} 这个 dict 也改得动（回读 V=320x240），所以两条都能用——这与本机 0.9.5 的实现有关，之前有资料写「dict 不生效」，实测证伪了。要判生效只能用页面里的 innerWidth×innerHeight，别看配置对象的属性。
  • java_script_enabled=False 时页面回读是空的：那行 -> [] 就是证据——JS 没跑，写 document.getElementById(...).textContent 的那段自然不会留下任何文字。这一条是「怎么确认自己关掉了 JS」的最省事办法。
  • headless=False（有头）能观察到插件：PL=5，而无头模式是 PL=0。这是无头浏览器最容易被识别的位之一，也是本篇「反检测」一节里唯一稳定可观察的差别。
  • 还有一条默认值很容易被忽略：BrowserConfig.verbose 与 CrawlerRunConfig.verbose 的默认值都是 True，两者管的是不同的日志行（见「本机靶站」那节末尾关于静音的说明）。要可复现的输出，两个都得关。

代理与 UA

代理这一路的参数名最近改过版，实测的报错原文最能说明问题：

proxy_and_ua.py
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
# -*- coding: utf-8 -*-
"""代理这一路：proxy 参数已废弃、proxy_config 才是正主，格式写错和端口没人听各报什么。"""

import asyncio
import inspect
import json
import os
import re
import threading
from http.server import BaseHTTPRequestHandler, HTTPServer

# 云广告一天只打一次，打不打取决于家目录里的标记文件，会把输出变得不可复现，所以先静音
os.environ.setdefault("CRAWL4AI_NO_CLOUD_NOTICE", "1")

from crawl4ai import (AsyncWebCrawler, BrowserConfig, CacheMode, CrawlerRunConfig)
from crawl4ai.async_crawler_strategy import AsyncPlaywrightCrawlerStrategy


PORT = 8082
BASE = "http://127.0.0.1:8082"
HITS = {}


def html(title, body):
    return ("<!doctype html><html><head><meta charset='utf-8'><title>%s</title>"
            "<meta name='description' content='probe'><style>.nav{color:red}</style>"
            "</head><body>"
            "<div class='nav'>NAV 这是靶站的导航区，故意写长一点，"
            "免得页面被 crawl4ai 的结构性判据当成空壳</div>"
            "%s"
            "<div class='filler'>FILLER 这是靶站的页脚区，同样是有实际文字的标记</div>"
            "</body></html>" % (title, body))


LIST_BODY = ("<h1>列表页</h1><ul class='items'>"
             + "".join("<li><a href='/d/%d'>片名%d</a><span class='price'>%.1f</span></li>"
                       % (i, i, i * 1.5) for i in range(1, 6))
             + "</ul><a class='next' href='/list?page=2'>下一页</a>")

LAZY_BODY = ("<h1>懒加载列表</h1><div id='list'></div>"
             "<div style='height:4000px'></div>"
             "<script>window.addEventListener('scroll',function(){"
             "if(window.__on)return;window.__on=1;"
             "for(var i=0;i<20;i++){var d=document.createElement('div');"
             "d.className='item';d.textContent='滚动条目'+i;"
             "document.getElementById('list').appendChild(d);}});</script>")

SPA_BODY = ("<h1>前端渲染的列表</h1><div id='app'></div><script>setTimeout(function(){var s='';"
            "for(var i=0;i<8;i++){s+='<div class=row>渲染行'+i+'</div>'}"
            "document.getElementById('app').innerHTML=s;},200);</script>")

TABLE_BODY = ("<table class='movies'><tr><th>片名</th><th>年份</th></tr>"
              + "".join("<tr><td>片名%d</td><td>%d</td></tr>" % (i, 1990 + i)
                        for i in range(1, 4)) + "</table>")

COOKIE_BODY = ("<p id='c'></p><script>document.getElementById('c').textContent="
               "'COOKIE='+(document.cookie||'NONE')</script>")

FP_BODY = ("<p id='f'></p><script>document.getElementById('f').textContent="
           "'WD='+navigator.webdriver+'|PL='+navigator.plugins.length+"
           "'|V='+window.innerWidth+'x'+window.innerHeight+'|UA='+navigator.userAgent.slice(0,26)"
           "</script>")

NEXT_LEVEL = {1: ["a", "b"], 2: ["c", "d"], 3: ["e"]}


def level_body(n):
    links = "".join("<a href='/lvl%d/%s'>%s</a>" % (n + 1, x, x)
                    for x in NEXT_LEVEL.get(n, []))
    return html("第%d层" % n, "<h1>LEVEL-%d</h1>%s" % (n, links))


class Handler(BaseHTTPRequestHandler):

    def reply(self, text, ctype="text/html; charset=utf-8", code=200, cookie=None):
        data = text.encode("utf-8")
        self.send_response(code)
        self.send_header("Content-Type", ctype)
        self.send_header("Content-Length", str(len(data)))
        if cookie:
            self.send_header("Set-Cookie", cookie)
        self.end_headers()
        self.wfile.write(data)

    def do_GET(self):
        import time
        path = self.path.split("?")[0]
        HITS[path] = HITS.get(path, 0) + 1
        if path == "/robots.txt":
            return self.reply("User-agent: *\nDisallow: /private/\n", "text/plain")
        if path.startswith("/private"):
            return self.reply(html("私密", "<h1>私密区</h1><p>PRIVATE-SECRET-BODY</p>"
                                         "<p>这段说明文字让页面不会被结构性判据当成空壳。</p>"))
        if path == "/count":
            return self.reply(json.dumps({"count": HITS["/count"]}), "application/json")
        if path == "/ua":
            return self.reply(html("UA", "<p>UA=%s</p>" % self.headers.get("User-Agent")))
        if path == "/list":
            return self.reply(html("列表", LIST_BODY))
        if path == "/lazy":
            return self.reply(html("懒加载", LAZY_BODY))
        if path == "/spa":
            return self.reply(html("SPA", SPA_BODY))
        if path == "/table":
            return self.reply(html("表格", TABLE_BODY))
        if path == "/cookie":
            return self.reply(html("Cookie", COOKIE_BODY))
        if path == "/setcookie":
            return self.reply(html("发cookie", COOKIE_BODY), cookie="probe_c4a=1; Path=/")
        if path == "/fp":
            return self.reply(html("指纹", FP_BODY))
        match = re.match(r"/lvl(\d)/([a-z])", path)
        if match:
            n = int(match.group(1))
            if n > 3:
                return self.reply(html("叶子", "NO-MORE-LEVEL"), code=404)
            return self.reply(level_body(n))
        match = re.match(r"/d/(\d+)", path)
        if match:
            i = int(match.group(1))
            return self.reply(html("片名%d" % i,
                                   "<h1 class='t'>片名%d</h1><span class='p'>%.1f</span>"
                                   % (i, i * 1.5)))
        if path == "/status500":
            return self.reply("boom", "text/plain", 500)
        if path.startswith("/slow"):
            time.sleep(2.5)
            return self.reply(html("慢", "SLOW-DONE"))
        return self.reply(html("首页", "<a href='/list'>去列表</a>"))

    def do_POST(self):
        size = int(self.headers.get("Content-Length", 0))
        body = self.rfile.read(size).decode("utf-8")
        HITS["/echo"] = HITS.get("/echo", 0) + 1
        return self.reply(html("回显", "<pre>POSTED %s</pre>" % body))

    def log_message(self, *args):
        pass


def serve():
    server = HTTPServer(("127.0.0.1", PORT), Handler)
    # 客户端超时断开时 socketserver 会往 stderr 打整段 traceback（里面是临时端口号），
    # 那不是结论只是噪声，接住。
    server.handle_error = lambda request, client_address: None
    threading.Thread(target=server.serve_forever, daemon=True).start()
    return server


srv = serve()


def quiet():
    """静音 crawl4ai 的计时行：[INIT] 那行归 BrowserConfig.verbose 管，
    [FETCH]/[SCRAPE]/[COMPLETE] 三行归 CrawlerRunConfig.verbose 管，两个都要关，
    输出里才不会留下每次不同的耗时。"""
    return BrowserConfig(verbose=False), CrawlerRunConfig(verbose=False)



import warnings

from crawl4ai import ProxyConfig

print("ProxyConfig 的参数 =", list(inspect.signature(ProxyConfig.__init__).parameters)[1:])
for kwargs in [{"proxy": "not-a-url"},
               {"proxy_cfg": "http://127.0.0.1:9"},
               {"proxy_config": ProxyConfig(server="http://127.0.0.1:9")}]:
    try:
        with warnings.catch_warnings(record=True) as caught:
            warnings.simplefilter("always")
            BrowserConfig(**kwargs)
        for w in caught:
            print("   警告:", str(w.message)[:78])
    except Exception as exc:
        print("%-40s -> %s: %s" % (str(kwargs)[:40], type(exc).__name__, str(exc)[:60]))


async def try_proxy(label, **browser_kwargs):
    try:
        async with AsyncWebCrawler(config=BrowserConfig(verbose=False, **browser_kwargs)) as cr:
            r = await cr.arun(BASE + "/ua", config=CrawlerRunConfig(verbose=False))
            print("%-34s success=%-5s 状态 %-5s error=%r"
                  % (label, r.success, r.status_code, (r.error_message or "")[:40]))
    except Exception as exc:
        print("%-34s 抛 %s: %s" % (label, type(exc).__name__, str(exc)[:56]))


async def main():
    await try_proxy("不带代理（对照）")
    await try_proxy("proxy=死端口（老参数）", proxy="http://127.0.0.1:9")
    await try_proxy("proxy_config=死端口", proxy_config=ProxyConfig(server="http://127.0.0.1:9"))

asyncio.run(main())
1
2
3
4
5
6
7
8
ProxyConfig 的参数 = ['server', 'username', 'password', 'ip']
{'proxy': 'not-a-url'}                   -> ValueError: Invalid proxy string format: not-a-url
{'proxy_cfg': 'http://127.0.0.1:9'}      -> TypeError: BrowserConfig.__init__() got an unexpected keyword argument 
不带代理（对照）                           success=True  状态 200   error=''
proxy=死端口（老参数）                     success=False 状态 None  error='Unexpected error in _crawl_web at line 7'
proxy_config=死端口                   success=False 状态 None  error='Unexpected error in _crawl_web at line 7'
E:\Desktop\Test\crawl4ai-env\Lib\site-packages\crawl4ai\async_configs.py:1113: UserWarning: The 'proxy' parameter is deprecated and will be removed in a future release. Use 'proxy_config' instead.
  warnings.warn("The 'proxy' parameter is deprecated and will be removed in a future release. Use 'proxy_config' instead.", UserWarning)
  • proxy 参数已经废弃但仍能用，每次用都打一句 UserWarning: The 'proxy' parameter is deprecated and will be removed in a future release. Use 'proxy_config' instead.。正主是 proxy_config，它接 ProxyConfig(server=..., username=..., password=..., ip=...)——这四个字段就是签名原文。
  • 名字写错不会被放过：proxy_cfg= 直接 TypeError: BrowserConfig.__init__() got an unexpected keyword argument 'proxy_cfg'. Did you mean 'proxy_config'?（这个 Did you mean 提示本身是对的，别照着网上抄 proxy_cfg）。
  • 格式错在构造期就报：proxy="not-a-url" → ValueError: Invalid proxy string format: not-a-url。
  • 指向一个没人听的端口时不抛异常，而是 success=False、状态是 None，错误信息是库内部那句以 Unexpected error in _crawl_web 开头、带它自己行号的话。也就是说代理挂了你要自己看 error_message，不能只靠捕获异常。
  • 对照项：不带代理时同一个 URL success=True、状态 200。

cookie 有三条入口，本段把三条都实测一遍，判据是靶站把 document.cookie 写回页面：

cookies_and_state.py
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
# -*- coding: utf-8 -*-
"""cookie 与登录态：Set-Cookie 在同一次会话里能不能留住、cookies 与 storage_state 怎么带进去。"""

import asyncio
import inspect
import json
import os
import re
import threading
from http.server import BaseHTTPRequestHandler, HTTPServer

# 云广告一天只打一次，打不打取决于家目录里的标记文件，会把输出变得不可复现，所以先静音
os.environ.setdefault("CRAWL4AI_NO_CLOUD_NOTICE", "1")

from crawl4ai import (AsyncWebCrawler, BrowserConfig, CacheMode, CrawlerRunConfig)
from crawl4ai.async_crawler_strategy import AsyncPlaywrightCrawlerStrategy


PORT = 8078
BASE = "http://127.0.0.1:8078"
HITS = {}


def html(title, body):
    return ("<!doctype html><html><head><meta charset='utf-8'><title>%s</title>"
            "<meta name='description' content='probe'><style>.nav{color:red}</style>"
            "</head><body>"
            "<div class='nav'>NAV 这是靶站的导航区，故意写长一点，"
            "免得页面被 crawl4ai 的结构性判据当成空壳</div>"
            "%s"
            "<div class='filler'>FILLER 这是靶站的页脚区，同样是有实际文字的标记</div>"
            "</body></html>" % (title, body))


LIST_BODY = ("<h1>列表页</h1><ul class='items'>"
             + "".join("<li><a href='/d/%d'>片名%d</a><span class='price'>%.1f</span></li>"
                       % (i, i, i * 1.5) for i in range(1, 6))
             + "</ul><a class='next' href='/list?page=2'>下一页</a>")

LAZY_BODY = ("<h1>懒加载列表</h1><div id='list'></div>"
             "<div style='height:4000px'></div>"
             "<script>window.addEventListener('scroll',function(){"
             "if(window.__on)return;window.__on=1;"
             "for(var i=0;i<20;i++){var d=document.createElement('div');"
             "d.className='item';d.textContent='滚动条目'+i;"
             "document.getElementById('list').appendChild(d);}});</script>")

SPA_BODY = ("<h1>前端渲染的列表</h1><div id='app'></div><script>setTimeout(function(){var s='';"
            "for(var i=0;i<8;i++){s+='<div class=row>渲染行'+i+'</div>'}"
            "document.getElementById('app').innerHTML=s;},200);</script>")

TABLE_BODY = ("<table class='movies'><tr><th>片名</th><th>年份</th></tr>"
              + "".join("<tr><td>片名%d</td><td>%d</td></tr>" % (i, 1990 + i)
                        for i in range(1, 4)) + "</table>")

COOKIE_BODY = ("<p id='c'></p><script>document.getElementById('c').textContent="
               "'COOKIE='+(document.cookie||'NONE')</script>")

FP_BODY = ("<p id='f'></p><script>document.getElementById('f').textContent="
           "'WD='+navigator.webdriver+'|PL='+navigator.plugins.length+"
           "'|V='+window.innerWidth+'x'+window.innerHeight+'|UA='+navigator.userAgent.slice(0,26)"
           "</script>")

NEXT_LEVEL = {1: ["a", "b"], 2: ["c", "d"], 3: ["e"]}


def level_body(n):
    links = "".join("<a href='/lvl%d/%s'>%s</a>" % (n + 1, x, x)
                    for x in NEXT_LEVEL.get(n, []))
    return html("第%d层" % n, "<h1>LEVEL-%d</h1>%s" % (n, links))


class Handler(BaseHTTPRequestHandler):

    def reply(self, text, ctype="text/html; charset=utf-8", code=200, cookie=None):
        data = text.encode("utf-8")
        self.send_response(code)
        self.send_header("Content-Type", ctype)
        self.send_header("Content-Length", str(len(data)))
        if cookie:
            self.send_header("Set-Cookie", cookie)
        self.end_headers()
        self.wfile.write(data)

    def do_GET(self):
        import time
        path = self.path.split("?")[0]
        HITS[path] = HITS.get(path, 0) + 1
        if path == "/robots.txt":
            return self.reply("User-agent: *\nDisallow: /private/\n", "text/plain")
        if path.startswith("/private"):
            return self.reply(html("私密", "<h1>私密区</h1><p>PRIVATE-SECRET-BODY</p>"
                                         "<p>这段说明文字让页面不会被结构性判据当成空壳。</p>"))
        if path == "/count":
            return self.reply(json.dumps({"count": HITS["/count"]}), "application/json")
        if path == "/ua":
            return self.reply(html("UA", "<p>UA=%s</p>" % self.headers.get("User-Agent")))
        if path == "/list":
            return self.reply(html("列表", LIST_BODY))
        if path == "/lazy":
            return self.reply(html("懒加载", LAZY_BODY))
        if path == "/spa":
            return self.reply(html("SPA", SPA_BODY))
        if path == "/table":
            return self.reply(html("表格", TABLE_BODY))
        if path == "/cookie":
            return self.reply(html("Cookie", COOKIE_BODY))
        if path == "/setcookie":
            return self.reply(html("发cookie", COOKIE_BODY), cookie="probe_c4a=1; Path=/")
        if path == "/fp":
            return self.reply(html("指纹", FP_BODY))
        match = re.match(r"/lvl(\d)/([a-z])", path)
        if match:
            n = int(match.group(1))
            if n > 3:
                return self.reply(html("叶子", "NO-MORE-LEVEL"), code=404)
            return self.reply(level_body(n))
        match = re.match(r"/d/(\d+)", path)
        if match:
            i = int(match.group(1))
            return self.reply(html("片名%d" % i,
                                   "<h1 class='t'>片名%d</h1><span class='p'>%.1f</span>"
                                   % (i, i * 1.5)))
        if path == "/status500":
            return self.reply("boom", "text/plain", 500)
        if path.startswith("/slow"):
            time.sleep(2.5)
            return self.reply(html("慢", "SLOW-DONE"))
        return self.reply(html("首页", "<a href='/list'>去列表</a>"))

    def do_POST(self):
        size = int(self.headers.get("Content-Length", 0))
        body = self.rfile.read(size).decode("utf-8")
        HITS["/echo"] = HITS.get("/echo", 0) + 1
        return self.reply(html("回显", "<pre>POSTED %s</pre>" % body))

    def log_message(self, *args):
        pass


def serve():
    server = HTTPServer(("127.0.0.1", PORT), Handler)
    # 客户端超时断开时 socketserver 会往 stderr 打整段 traceback（里面是临时端口号），
    # 那不是结论只是噪声，接住。
    server.handle_error = lambda request, client_address: None
    threading.Thread(target=server.serve_forever, daemon=True).start()
    return server


srv = serve()


def quiet():
    """静音 crawl4ai 的计时行：[INIT] 那行归 BrowserConfig.verbose 管，
    [FETCH]/[SCRAPE]/[COMPLETE] 三行归 CrawlerRunConfig.verbose 管，两个都要关，
    输出里才不会留下每次不同的耗时。"""
    return BrowserConfig(verbose=False), CrawlerRunConfig(verbose=False)



import inspect
import os
import tempfile

COOKIE_LINE = "COOKIE="


async def read_cookie(cr, label):
    r = await cr.arun(BASE + "/cookie", config=CrawlerRunConfig(verbose=False))
    line = [l for l in r.markdown.splitlines() if COOKIE_LINE in l]
    print("%-34s -> %s" % (label, line[:1]))


async def main():
    bc, _ = quiet()
    async with AsyncWebCrawler(config=bc) as cr:
        await cr.arun(BASE + "/setcookie", config=CrawlerRunConfig(verbose=False))
        await read_cookie(cr, "同一次会话：服务器 Set-Cookie 之后")
    async with AsyncWebCrawler(config=bc) as cr:
        await read_cookie(cr, "新开一次会话")

    state = os.path.join(tempfile.gettempdir(), "c4ai_state_probe.json")
    with open(state, "w", encoding="utf-8") as fp:
        json.dump({"cookies": [{"name": "from_state", "value": "1", "domain": "127.0.0.1",
                                "path": "/", "expires": -1, "httpOnly": False,
                                "secure": False, "sameSite": "Lax"}], "origins": []}, fp)
    for label, kwargs in [
            ("BrowserConfig(cookies=[…])", dict(cookies=[{"name": "from_config", "value": "1",
                                                          "url": BASE}])),
            ("BrowserConfig(storage_state=文件)", dict(storage_state=state)),
            ("都不配（对照）", dict())]:
        async with AsyncWebCrawler(config=BrowserConfig(verbose=False, **kwargs)) as cr:
            await read_cookie(cr, label)

    names = list(inspect.signature(CrawlerRunConfig.__init__).parameters)
    bnames = list(inspect.signature(BrowserConfig.__init__).parameters)
    print("CrawlerRunConfig 里 cookie 相关 =", [x for x in names if "cookie" in x.lower()])
    print("BrowserConfig 里 cookie/state 相关 =",
          [x for x in bnames if "cookie" in x.lower() or "state" in x.lower()])
    print("文档里那种 cookies_file 参数在不在 =", "cookies_file" in bnames)
    os.remove(state)

asyncio.run(main())
1
2
3
4
5
6
7
8
同一次会话：服务器 Set-Cookie 之后            -> ['COOKIE=cookiesEnabled=true; probe_c4a=1']
新开一次会话                             -> ['COOKIE=cookiesEnabled=true']
BrowserConfig(cookies=[…])         -> ['COOKIE=from_config=1; cookiesEnabled=true']
BrowserConfig(storage_state=文件)    -> ['COOKIE=from_state=1; cookiesEnabled=true']
都不配（对照）                            -> ['COOKIE=cookiesEnabled=true']
CrawlerRunConfig 里 cookie 相关 = []
BrowserConfig 里 cookie/state 相关 = ['storage_state', 'cookies']
文档里那种 cookies_file 参数在不在 = False
  • 服务端 Set-Cookie 之后，同一个爬虫实例（同一个浏览器上下文）里的后续请求会自动带上：COOKIE=cookiesEnabled=true; probe_c4a=1。新开一个实例就没了（只剩 cookiesEnabled=true，那是浏览器自己写的）。
  • 想把 cookie 带进新会话，两条正路：BrowserConfig(cookies=[{"name": ..., "value": ..., "url": ...}])，或者 BrowserConfig(storage_state=<playwright 状态文件>)。两种都被接受并真的带到了页面里（输出里两行分别回显 from_config=1 与 from_state=1）。
  • storage_state 的 JSON 就是 playwright 的标准格式：{"cookies": [{name, value, domain, path, expires, httpOnly, secure, sameSite}], "origins": []}，字段名不能省（expires 给 -1 表示会话 cookie，sameSite 只能是 Strict/Lax/None）。
  • 参数面上要记住两点：CrawlerRunConfig 里没有任何 cookie 参数（试探结果是空列表），cookie 只归 BrowserConfig 管；而文档里出现过的 cookies_file 参数在本机 0.9.5 不存在（"cookies_file" in bnames 是 False），BrowserConfig 里 cookie/state 相关的只有 storage_state 与 cookies 两个。

session_id 管到哪

session_id 常被当成「会话保持」来用，但它到底管到哪一层必须实测。这段把同实例/新实例/同 session_id/换 session_id/持久化目录/新进程六种情况摆在一起，判据仍然是靶站回显的 document.cookie：

session_modes.py
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
# -*- coding: utf-8 -*-
"""session_id 与 use_persistent_context 各管到哪一步：cookie 能不能跨实例、跨进程活下来。"""

import asyncio
import inspect
import json
import os
import re
import threading
from http.server import BaseHTTPRequestHandler, HTTPServer

# 云广告一天只打一次，打不打取决于家目录里的标记文件，会把输出变得不可复现，所以先静音
os.environ.setdefault("CRAWL4AI_NO_CLOUD_NOTICE", "1")

from crawl4ai import (AsyncWebCrawler, BrowserConfig, CacheMode, CrawlerRunConfig)
from crawl4ai.async_crawler_strategy import AsyncPlaywrightCrawlerStrategy
import shutil
import subprocess
import sys


PORT = 8090
BASE = "http://127.0.0.1:8090"
HITS = {}


def html(title, body):
    return ("<!doctype html><html><head><meta charset='utf-8'><title>%s</title>"
            "<meta name='description' content='probe'><style>.nav{color:red}</style>"
            "</head><body>"
            "<div class='nav'>NAV 这是靶站的导航区，故意写长一点，"
            "免得页面被 crawl4ai 的结构性判据当成空壳</div>"
            "%s"
            "<div class='filler'>FILLER 这是靶站的页脚区，同样是有实际文字的标记</div>"
            "</body></html>" % (title, body))


LIST_BODY = ("<h1>列表页</h1><ul class='items'>"
             + "".join("<li><a href='/d/%d'>片名%d</a><span class='price'>%.1f</span></li>"
                       % (i, i, i * 1.5) for i in range(1, 6))
             + "</ul><a class='next' href='/list?page=2'>下一页</a>")

LAZY_BODY = ("<h1>懒加载列表</h1><div id='list'></div>"
             "<div style='height:4000px'></div>"
             "<script>window.addEventListener('scroll',function(){"
             "if(window.__on)return;window.__on=1;"
             "for(var i=0;i<20;i++){var d=document.createElement('div');"
             "d.className='item';d.textContent='滚动条目'+i;"
             "document.getElementById('list').appendChild(d);}});</script>")

SPA_BODY = ("<h1>前端渲染的列表</h1><div id='app'></div><script>setTimeout(function(){var s='';"
            "for(var i=0;i<8;i++){s+='<div class=row>渲染行'+i+'</div>'}"
            "document.getElementById('app').innerHTML=s;},200);</script>")

TABLE_BODY = ("<table class='movies'><tr><th>片名</th><th>年份</th></tr>"
              + "".join("<tr><td>片名%d</td><td>%d</td></tr>" % (i, 1990 + i)
                        for i in range(1, 4)) + "</table>")

COOKIE_BODY = ("<p id='c'></p><script>document.getElementById('c').textContent="
               "'COOKIE='+(document.cookie||'NONE')</script>")

FP_BODY = ("<p id='f'></p><script>document.getElementById('f').textContent="
           "'WD='+navigator.webdriver+'|PL='+navigator.plugins.length+"
           "'|V='+window.innerWidth+'x'+window.innerHeight+'|UA='+navigator.userAgent.slice(0,26)"
           "</script>")

NEXT_LEVEL = {1: ["a", "b"], 2: ["c", "d"], 3: ["e"]}


def level_body(n):
    links = "".join("<a href='/lvl%d/%s'>%s</a>" % (n + 1, x, x)
                    for x in NEXT_LEVEL.get(n, []))
    return html("第%d层" % n, "<h1>LEVEL-%d</h1>%s" % (n, links))


class Handler(BaseHTTPRequestHandler):

    def reply(self, text, ctype="text/html; charset=utf-8", code=200, cookie=None):
        data = text.encode("utf-8")
        self.send_response(code)
        self.send_header("Content-Type", ctype)
        self.send_header("Content-Length", str(len(data)))
        if cookie:
            self.send_header("Set-Cookie", cookie)
        self.end_headers()
        self.wfile.write(data)

    def do_GET(self):
        import time
        path = self.path.split("?")[0]
        HITS[path] = HITS.get(path, 0) + 1
        if path == "/robots.txt":
            return self.reply("User-agent: *\nDisallow: /private/\n", "text/plain")
        if path.startswith("/private"):
            return self.reply(html("私密", "<h1>私密区</h1><p>PRIVATE-SECRET-BODY</p>"
                                         "<p>这段说明文字让页面不会被结构性判据当成空壳。</p>"))
        if path == "/count":
            return self.reply(json.dumps({"count": HITS["/count"]}), "application/json")
        if path == "/ua":
            return self.reply(html("UA", "<p>UA=%s</p>" % self.headers.get("User-Agent")))
        if path == "/list":
            return self.reply(html("列表", LIST_BODY))
        if path == "/lazy":
            return self.reply(html("懒加载", LAZY_BODY))
        if path == "/spa":
            return self.reply(html("SPA", SPA_BODY))
        if path == "/table":
            return self.reply(html("表格", TABLE_BODY))
        if path == "/cookie":
            return self.reply(html("Cookie", COOKIE_BODY))
        if path == "/setcookie":
            return self.reply(html("发cookie", COOKIE_BODY), cookie="probe_c4a=1; Path=/")
        if path == "/fp":
            return self.reply(html("指纹", FP_BODY))
        match = re.match(r"/lvl(\d)/([a-z])", path)
        if match:
            n = int(match.group(1))
            if n > 3:
                return self.reply(html("叶子", "NO-MORE-LEVEL"), code=404)
            return self.reply(level_body(n))
        match = re.match(r"/d/(\d+)", path)
        if match:
            i = int(match.group(1))
            return self.reply(html("片名%d" % i,
                                   "<h1 class='t'>片名%d</h1><span class='p'>%.1f</span>"
                                   % (i, i * 1.5)))
        if path == "/status500":
            return self.reply("boom", "text/plain", 500)
        if path.startswith("/slow"):
            time.sleep(2.5)
            return self.reply(html("慢", "SLOW-DONE"))
        return self.reply(html("首页", "<a href='/list'>去列表</a>"))

    def do_POST(self):
        size = int(self.headers.get("Content-Length", 0))
        body = self.rfile.read(size).decode("utf-8")
        HITS["/echo"] = HITS.get("/echo", 0) + 1
        return self.reply(html("回显", "<pre>POSTED %s</pre>" % body))

    def log_message(self, *args):
        pass


def serve():
    server = HTTPServer(("127.0.0.1", PORT), Handler)
    # 客户端超时断开时 socketserver 会往 stderr 打整段 traceback（里面是临时端口号），
    # 那不是结论只是噪声，接住。
    server.handle_error = lambda request, client_address: None
    threading.Thread(target=server.serve_forever, daemon=True).start()
    return server


srv = serve()


def quiet():
    """静音 crawl4ai 的计时行：[INIT] 那行归 BrowserConfig.verbose 管，
    [FETCH]/[SCRAPE]/[COMPLETE] 三行归 CrawlerRunConfig.verbose 管，两个都要关，
    输出里才不会留下每次不同的耗时。"""
    return BrowserConfig(verbose=False), CrawlerRunConfig(verbose=False)



def line_of(res):
    return [l for l in res.markdown.splitlines() if "COOKIE=" in l][:1]


async def probe(label, persistent=False, **run_kwargs):
    kwargs = {"verbose": False}
    if persistent:
        kwargs.update({"use_persistent_context": True, "user_data_dir": PROFILE})
    async with AsyncWebCrawler(config=BrowserConfig(**kwargs)) as cr:
        r = await cr.arun(BASE + "/cookie", config=CrawlerRunConfig(verbose=False, **run_kwargs))
        print("%-38s -> %s" % (label, line_of(r)))


# 子进程里跑的代码：只回显 document.cookie，不打印路径与耗时
CHILD = "\n".join([
    "import asyncio, sys",
    "from crawl4ai import AsyncWebCrawler, BrowserConfig, CrawlerRunConfig",
    "BASE, sid = sys.argv[1], (sys.argv[2] if len(sys.argv) > 2 else None)",
    "async def main():",
    "    kw = {'verbose': False}",
    "    if sid: kw['session_id'] = sid",
    "    async with AsyncWebCrawler(config=BrowserConfig(verbose=False)) as cr:",
    "        r = await cr.arun(BASE + '/cookie', config=CrawlerRunConfig(**kw))",
    "        print('CHILD', [l for l in r.markdown.splitlines() if 'COOKIE=' in l][:1])",
    "asyncio.run(main())",
])


def child(label, sid=None):
    args = [sys.executable, "-X", "utf8", "-c", CHILD, BASE]
    if sid:
        args.append(sid)
    done = subprocess.run(args, capture_output=True, text=True, encoding="utf-8",
                          errors="replace", timeout=300,
                          env=dict(os.environ, CRAWL4AI_NO_CLOUD_NOTICE="1"))
    got = [l for l in (done.stdout + done.stderr).splitlines() if l.startswith("CHILD ")]
    print("%-38s -> %s" % (label, got[:1]))


PROFILE = os.path.join(os.getcwd(), "c4ai-profile")


async def main():
    bc, _ = quiet()
    async with AsyncWebCrawler(config=bc) as cr:
        await cr.arun(BASE + "/setcookie", config=CrawlerRunConfig(verbose=False))
        print("① 同一个爬虫实例里再打一次 ->",
              line_of(await cr.arun(BASE + "/cookie", config=CrawlerRunConfig(verbose=False))))
    await probe("② 同进程、新实例、不给 session_id")
    await probe("③ 同进程、新实例、同 session_id", session_id="probe-s")
    await probe("④ 同进程、新实例、换 session_id", session_id="probe-t")
    # 关键一问：cookie 是不是绑在 session_id 上
    async with AsyncWebCrawler(config=bc) as cr:
        await cr.arun(BASE + "/setcookie", config=CrawlerRunConfig(verbose=False,
                                                                  session_id="bind-x"))
    await probe("⑤ 用 session_id=bind-x 存过，再用它读", session_id="bind-x")
    await probe("⑥ 用 session_id=bind-y 读（换个号）", session_id="bind-y")
    shutil.rmtree(PROFILE, ignore_errors=True)
    async with AsyncWebCrawler(config=BrowserConfig(
            verbose=False, use_persistent_context=True, user_data_dir=PROFILE)) as cr:
        await cr.arun(BASE + "/setcookie", config=CrawlerRunConfig(verbose=False))
    async with AsyncWebCrawler(config=BrowserConfig(
            verbose=False, use_persistent_context=True, user_data_dir=PROFILE)) as cr:
        print("⑦ 持久化目录里再开一个新实例 ->",
              line_of(await cr.arun(BASE + "/cookie", config=CrawlerRunConfig(verbose=False))))
    names = os.listdir(PROFILE)
    print("   持久化目录非空 =", bool(names), "| 有 Cookies 相关项 =",
          any("cookies" in n.lower() for n in names))
    child("⑧ 新进程、不给 session_id")
    child("⑨ 新进程、带 session_id=bind-x", "bind-x")
    shutil.rmtree(PROFILE, ignore_errors=True)

asyncio.run(main())
1
2
3
4
5
6
7
8
9
10
① 同一个爬虫实例里再打一次 -> ['COOKIE=cookiesEnabled=true; probe_c4a=1']
② 同进程、新实例、不给 session_id                -> ['COOKIE=cookiesEnabled=true']
③ 同进程、新实例、同 session_id                 -> ['COOKIE=cookiesEnabled=true']
④ 同进程、新实例、换 session_id                 -> ['COOKIE=cookiesEnabled=true']
⑤ 用 session_id=bind-x 存过，再用它读          -> ['COOKIE=cookiesEnabled=true']
⑥ 用 session_id=bind-y 读（换个号）           -> ['COOKIE=cookiesEnabled=true']
⑦ 持久化目录里再开一个新实例 -> ['COOKIE=NONE']
   持久化目录非空 = True | 有 Cookies 相关项 = False
⑧ 新进程、不给 session_id                    -> ["CHILD ['COOKIE=cookiesEnabled=true']"]
⑨ 新进程、带 session_id=bind-x              -> ["CHILD ['COOKIE=cookiesEnabled=true']"]

结论逐条对照输出：

  • cookie 只活在「同一个爬虫实例」里（① 有、② 没有）。
  • session_id 不会把 cookie 跨实例带过去：同一个 session_id 在新实例里读不到（③），换一个也读不到（④）；先带 session_id=bind-x 存一次、再用它读，还是没有（⑤），换个号也一样（⑥）。所以 session_id 不是「保持登录态」的开关，它是给缓存/调度侧做分组用的标识符（CrawlerRunConfig 里会话相关的参数只有 session_id 与三个 proxy_session_*）。
  • use_persistent_context=True + 同一个 user_data_dir 这次也没带过来（⑦ COOKIE=NONE），而且目录里非空但没有 Cookies 相关项——说明上下文没被真正落盘/复用。跨进程的两个探测（⑧⑨）同样读不到。
  • 于是要跨进程复用登录态，本篇实测可用的只有 storage_state（上一节），不是 session_id、也不是 use_persistent_context 的默认用法。
  • 顺带把 AsyncWebCrawler 的方法面钉住：公开方法只有 arun、arun_many、start、close、amap_domain、aprocess_html、aseed_urls、nullcontext 八个，文档里出现过的 crawl_pages 与 session() 在本机 0.9.5 都不存在。想「一组 URL 共享一个会话」，能做的就是给它们同一个 session_id（共享的是缓存/统计口径，不是浏览器上下文）。
session_and_hooks.py
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
# -*- coding: utf-8 -*-
"""会话与钩子：多标签的老方法没了，能用的只有 session_id 这一路。"""

import asyncio
import inspect
import json
import os
import re
import threading
from http.server import BaseHTTPRequestHandler, HTTPServer

# 云广告一天只打一次，打不打取决于家目录里的标记文件，会把输出变得不可复现，所以先静音
os.environ.setdefault("CRAWL4AI_NO_CLOUD_NOTICE", "1")

from crawl4ai import (AsyncWebCrawler, BrowserConfig, CacheMode, CrawlerRunConfig)
from crawl4ai.async_crawler_strategy import AsyncPlaywrightCrawlerStrategy


PORT = 8084
BASE = "http://127.0.0.1:8084"
HITS = {}


def html(title, body):
    return ("<!doctype html><html><head><meta charset='utf-8'><title>%s</title>"
            "<meta name='description' content='probe'><style>.nav{color:red}</style>"
            "</head><body>"
            "<div class='nav'>NAV 这是靶站的导航区，故意写长一点，"
            "免得页面被 crawl4ai 的结构性判据当成空壳</div>"
            "%s"
            "<div class='filler'>FILLER 这是靶站的页脚区，同样是有实际文字的标记</div>"
            "</body></html>" % (title, body))


LIST_BODY = ("<h1>列表页</h1><ul class='items'>"
             + "".join("<li><a href='/d/%d'>片名%d</a><span class='price'>%.1f</span></li>"
                       % (i, i, i * 1.5) for i in range(1, 6))
             + "</ul><a class='next' href='/list?page=2'>下一页</a>")

LAZY_BODY = ("<h1>懒加载列表</h1><div id='list'></div>"
             "<div style='height:4000px'></div>"
             "<script>window.addEventListener('scroll',function(){"
             "if(window.__on)return;window.__on=1;"
             "for(var i=0;i<20;i++){var d=document.createElement('div');"
             "d.className='item';d.textContent='滚动条目'+i;"
             "document.getElementById('list').appendChild(d);}});</script>")

SPA_BODY = ("<h1>前端渲染的列表</h1><div id='app'></div><script>setTimeout(function(){var s='';"
            "for(var i=0;i<8;i++){s+='<div class=row>渲染行'+i+'</div>'}"
            "document.getElementById('app').innerHTML=s;},200);</script>")

TABLE_BODY = ("<table class='movies'><tr><th>片名</th><th>年份</th></tr>"
              + "".join("<tr><td>片名%d</td><td>%d</td></tr>" % (i, 1990 + i)
                        for i in range(1, 4)) + "</table>")

COOKIE_BODY = ("<p id='c'></p><script>document.getElementById('c').textContent="
               "'COOKIE='+(document.cookie||'NONE')</script>")

FP_BODY = ("<p id='f'></p><script>document.getElementById('f').textContent="
           "'WD='+navigator.webdriver+'|PL='+navigator.plugins.length+"
           "'|V='+window.innerWidth+'x'+window.innerHeight+'|UA='+navigator.userAgent.slice(0,26)"
           "</script>")

NEXT_LEVEL = {1: ["a", "b"], 2: ["c", "d"], 3: ["e"]}


def level_body(n):
    links = "".join("<a href='/lvl%d/%s'>%s</a>" % (n + 1, x, x)
                    for x in NEXT_LEVEL.get(n, []))
    return html("第%d层" % n, "<h1>LEVEL-%d</h1>%s" % (n, links))


class Handler(BaseHTTPRequestHandler):

    def reply(self, text, ctype="text/html; charset=utf-8", code=200, cookie=None):
        data = text.encode("utf-8")
        self.send_response(code)
        self.send_header("Content-Type", ctype)
        self.send_header("Content-Length", str(len(data)))
        if cookie:
            self.send_header("Set-Cookie", cookie)
        self.end_headers()
        self.wfile.write(data)

    def do_GET(self):
        import time
        path = self.path.split("?")[0]
        HITS[path] = HITS.get(path, 0) + 1
        if path == "/robots.txt":
            return self.reply("User-agent: *\nDisallow: /private/\n", "text/plain")
        if path.startswith("/private"):
            return self.reply(html("私密", "<h1>私密区</h1><p>PRIVATE-SECRET-BODY</p>"
                                         "<p>这段说明文字让页面不会被结构性判据当成空壳。</p>"))
        if path == "/count":
            return self.reply(json.dumps({"count": HITS["/count"]}), "application/json")
        if path == "/ua":
            return self.reply(html("UA", "<p>UA=%s</p>" % self.headers.get("User-Agent")))
        if path == "/list":
            return self.reply(html("列表", LIST_BODY))
        if path == "/lazy":
            return self.reply(html("懒加载", LAZY_BODY))
        if path == "/spa":
            return self.reply(html("SPA", SPA_BODY))
        if path == "/table":
            return self.reply(html("表格", TABLE_BODY))
        if path == "/cookie":
            return self.reply(html("Cookie", COOKIE_BODY))
        if path == "/setcookie":
            return self.reply(html("发cookie", COOKIE_BODY), cookie="probe_c4a=1; Path=/")
        if path == "/fp":
            return self.reply(html("指纹", FP_BODY))
        match = re.match(r"/lvl(\d)/([a-z])", path)
        if match:
            n = int(match.group(1))
            if n > 3:
                return self.reply(html("叶子", "NO-MORE-LEVEL"), code=404)
            return self.reply(level_body(n))
        match = re.match(r"/d/(\d+)", path)
        if match:
            i = int(match.group(1))
            return self.reply(html("片名%d" % i,
                                   "<h1 class='t'>片名%d</h1><span class='p'>%.1f</span>"
                                   % (i, i * 1.5)))
        if path == "/status500":
            return self.reply("boom", "text/plain", 500)
        if path.startswith("/slow"):
            time.sleep(2.5)
            return self.reply(html("慢", "SLOW-DONE"))
        return self.reply(html("首页", "<a href='/list'>去列表</a>"))

    def do_POST(self):
        size = int(self.headers.get("Content-Length", 0))
        body = self.rfile.read(size).decode("utf-8")
        HITS["/echo"] = HITS.get("/echo", 0) + 1
        return self.reply(html("回显", "<pre>POSTED %s</pre>" % body))

    def log_message(self, *args):
        pass


def serve():
    server = HTTPServer(("127.0.0.1", PORT), Handler)
    # 客户端超时断开时 socketserver 会往 stderr 打整段 traceback（里面是临时端口号），
    # 那不是结论只是噪声，接住。
    server.handle_error = lambda request, client_address: None
    threading.Thread(target=server.serve_forever, daemon=True).start()
    return server


srv = serve()


def quiet():
    """静音 crawl4ai 的计时行：[INIT] 那行归 BrowserConfig.verbose 管，
    [FETCH]/[SCRAPE]/[COMPLETE] 三行归 CrawlerRunConfig.verbose 管，两个都要关，
    输出里才不会留下每次不同的耗时。"""
    return BrowserConfig(verbose=False), CrawlerRunConfig(verbose=False)



methods = [n for n in dir(AsyncWebCrawler) if not n.startswith("_")]
ms = sorted(methods)
print("AsyncWebCrawler 的公开方法 %d 个:" % len(ms))
for i in range(0, len(ms), 6):
    print("   ", ", ".join(ms[i:i + 6]))
print("文档里那种 crawl_pages 在不在 =", "crawl_pages" in methods,
      "| session 在不在 =", "session" in methods)
names = list(inspect.signature(CrawlerRunConfig.__init__).parameters)
print("hook 相关参数 =", [x for x in names if x.startswith("hook")])
print("会话相关参数 =", [x for x in names if "session" in x])


async def main():
    bc, _ = quiet()
    async with AsyncWebCrawler(config=bc) as cr:
        await cr.arun(BASE + "/setcookie", config=CrawlerRunConfig(verbose=False))
        r = await cr.arun(BASE + "/cookie", config=CrawlerRunConfig(verbose=False))
        print("同一个爬虫实例里 cookie 留住了 =",
              [l for l in r.markdown.splitlines() if "COOKIE=" in l][:1])
    for sid in ["probe-fix", "probe-fix"]:
        async with AsyncWebCrawler(config=bc) as cr:
            await cr.arun(BASE + "/setcookie",
                          config=CrawlerRunConfig(verbose=False, session_id=sid))
            r = await cr.arun(BASE + "/cookie",
                              config=CrawlerRunConfig(verbose=False, session_id=sid))
        print("新实例 + session_id=%s -> %s"
              % (sid, [l for l in r.markdown.splitlines() if "COOKIE=" in l][:1]))
    async with AsyncWebCrawler(config=bc) as cr:
        await cr.arun(BASE + "/setcookie",
                      config=CrawlerRunConfig(verbose=False, session_id="probe-a"))
        r = await cr.arun(BASE + "/cookie",
                          config=CrawlerRunConfig(verbose=False, session_id="probe-b"))
    print("换一个 session_id -> %s" % [l for l in r.markdown.splitlines() if "COOKIE=" in l][:1])

asyncio.run(main())
1
2
3
4
5
6
7
8
9
10
AsyncWebCrawler 的公开方法 8 个:
    amap_domain, aprocess_html, arun, arun_many, aseed_urls, close
    nullcontext, start
文档里那种 crawl_pages 在不在 = False | session 在不在 = False
hook 相关参数 = []
会话相关参数 = ['proxy_session_id', 'proxy_session_ttl', 'proxy_session_auto_release', 'session_id']
同一个爬虫实例里 cookie 留住了 = ['COOKIE=cookiesEnabled=true; probe_c4a=1']
新实例 + session_id=probe-fix -> ['COOKIE=cookiesEnabled=true; probe_c4a=1']
新实例 + session_id=probe-fix -> ['COOKIE=cookiesEnabled=true; probe_c4a=1']
换一个 session_id -> ['COOKIE=cookiesEnabled=true; probe_c4a=1']

这一段的输出把上面最后一条钉死了：公开方法八个、crawl_pages/session 都不在、hook_* 相关参数是空列表（本机 0.9.5 的 CrawlerRunConfig 没有 hook 系列参数，文档 Advanced 页里那些 hook_request_pre 之类要另行核对版本）。注意最后四行：同一个爬虫实例里先 /setcookie 再 /cookie，cookie 在；带同一个 session_id 也是同一实例，所以照样在——这正好说明「留得住」和「session_id」无关。

CrawlerRunConfig 参数面

CrawlerRunConfig 是这台机器上真正的中枢：100 个参数。这段把参数名全列出来（分行打印），再把常用的默认值逐个贴出，并验证两个已删的老参数：

run_config_params.py
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
# -*- coding: utf-8 -*-
"""CrawlerRunConfig 是这台机器上的中枢：100 个参数、5 种缓存模式、几个已删的老参数。"""

import asyncio
import inspect
import json
import os
import re
import threading
from http.server import BaseHTTPRequestHandler, HTTPServer

# 云广告一天只打一次，打不打取决于家目录里的标记文件，会把输出变得不可复现，所以先静音
os.environ.setdefault("CRAWL4AI_NO_CLOUD_NOTICE", "1")

from crawl4ai import (AsyncWebCrawler, BrowserConfig, CacheMode, CrawlerRunConfig)
from crawl4ai.async_crawler_strategy import AsyncPlaywrightCrawlerStrategy


PORT = 8080
BASE = "http://127.0.0.1:8080"
HITS = {}


def html(title, body):
    return ("<!doctype html><html><head><meta charset='utf-8'><title>%s</title>"
            "<meta name='description' content='probe'><style>.nav{color:red}</style>"
            "</head><body>"
            "<div class='nav'>NAV 这是靶站的导航区，故意写长一点，"
            "免得页面被 crawl4ai 的结构性判据当成空壳</div>"
            "%s"
            "<div class='filler'>FILLER 这是靶站的页脚区，同样是有实际文字的标记</div>"
            "</body></html>" % (title, body))


LIST_BODY = ("<h1>列表页</h1><ul class='items'>"
             + "".join("<li><a href='/d/%d'>片名%d</a><span class='price'>%.1f</span></li>"
                       % (i, i, i * 1.5) for i in range(1, 6))
             + "</ul><a class='next' href='/list?page=2'>下一页</a>")

LAZY_BODY = ("<h1>懒加载列表</h1><div id='list'></div>"
             "<div style='height:4000px'></div>"
             "<script>window.addEventListener('scroll',function(){"
             "if(window.__on)return;window.__on=1;"
             "for(var i=0;i<20;i++){var d=document.createElement('div');"
             "d.className='item';d.textContent='滚动条目'+i;"
             "document.getElementById('list').appendChild(d);}});</script>")

SPA_BODY = ("<h1>前端渲染的列表</h1><div id='app'></div><script>setTimeout(function(){var s='';"
            "for(var i=0;i<8;i++){s+='<div class=row>渲染行'+i+'</div>'}"
            "document.getElementById('app').innerHTML=s;},200);</script>")

TABLE_BODY = ("<table class='movies'><tr><th>片名</th><th>年份</th></tr>"
              + "".join("<tr><td>片名%d</td><td>%d</td></tr>" % (i, 1990 + i)
                        for i in range(1, 4)) + "</table>")

COOKIE_BODY = ("<p id='c'></p><script>document.getElementById('c').textContent="
               "'COOKIE='+(document.cookie||'NONE')</script>")

FP_BODY = ("<p id='f'></p><script>document.getElementById('f').textContent="
           "'WD='+navigator.webdriver+'|PL='+navigator.plugins.length+"
           "'|V='+window.innerWidth+'x'+window.innerHeight+'|UA='+navigator.userAgent.slice(0,26)"
           "</script>")

NEXT_LEVEL = {1: ["a", "b"], 2: ["c", "d"], 3: ["e"]}


def level_body(n):
    links = "".join("<a href='/lvl%d/%s'>%s</a>" % (n + 1, x, x)
                    for x in NEXT_LEVEL.get(n, []))
    return html("第%d层" % n, "<h1>LEVEL-%d</h1>%s" % (n, links))


class Handler(BaseHTTPRequestHandler):

    def reply(self, text, ctype="text/html; charset=utf-8", code=200, cookie=None):
        data = text.encode("utf-8")
        self.send_response(code)
        self.send_header("Content-Type", ctype)
        self.send_header("Content-Length", str(len(data)))
        if cookie:
            self.send_header("Set-Cookie", cookie)
        self.end_headers()
        self.wfile.write(data)

    def do_GET(self):
        import time
        path = self.path.split("?")[0]
        HITS[path] = HITS.get(path, 0) + 1
        if path == "/robots.txt":
            return self.reply("User-agent: *\nDisallow: /private/\n", "text/plain")
        if path.startswith("/private"):
            return self.reply(html("私密", "<h1>私密区</h1><p>PRIVATE-SECRET-BODY</p>"
                                         "<p>这段说明文字让页面不会被结构性判据当成空壳。</p>"))
        if path == "/count":
            return self.reply(json.dumps({"count": HITS["/count"]}), "application/json")
        if path == "/ua":
            return self.reply(html("UA", "<p>UA=%s</p>" % self.headers.get("User-Agent")))
        if path == "/list":
            return self.reply(html("列表", LIST_BODY))
        if path == "/lazy":
            return self.reply(html("懒加载", LAZY_BODY))
        if path == "/spa":
            return self.reply(html("SPA", SPA_BODY))
        if path == "/table":
            return self.reply(html("表格", TABLE_BODY))
        if path == "/cookie":
            return self.reply(html("Cookie", COOKIE_BODY))
        if path == "/setcookie":
            return self.reply(html("发cookie", COOKIE_BODY), cookie="probe_c4a=1; Path=/")
        if path == "/fp":
            return self.reply(html("指纹", FP_BODY))
        match = re.match(r"/lvl(\d)/([a-z])", path)
        if match:
            n = int(match.group(1))
            if n > 3:
                return self.reply(html("叶子", "NO-MORE-LEVEL"), code=404)
            return self.reply(level_body(n))
        match = re.match(r"/d/(\d+)", path)
        if match:
            i = int(match.group(1))
            return self.reply(html("片名%d" % i,
                                   "<h1 class='t'>片名%d</h1><span class='p'>%.1f</span>"
                                   % (i, i * 1.5)))
        if path == "/status500":
            return self.reply("boom", "text/plain", 500)
        if path.startswith("/slow"):
            time.sleep(2.5)
            return self.reply(html("慢", "SLOW-DONE"))
        return self.reply(html("首页", "<a href='/list'>去列表</a>"))

    def do_POST(self):
        size = int(self.headers.get("Content-Length", 0))
        body = self.rfile.read(size).decode("utf-8")
        HITS["/echo"] = HITS.get("/echo", 0) + 1
        return self.reply(html("回显", "<pre>POSTED %s</pre>" % body))

    def log_message(self, *args):
        pass


def serve():
    server = HTTPServer(("127.0.0.1", PORT), Handler)
    # 客户端超时断开时 socketserver 会往 stderr 打整段 traceback（里面是临时端口号），
    # 那不是结论只是噪声，接住。
    server.handle_error = lambda request, client_address: None
    threading.Thread(target=server.serve_forever, daemon=True).start()
    return server


srv = serve()


def quiet():
    """静音 crawl4ai 的计时行：[INIT] 那行归 BrowserConfig.verbose 管，
    [FETCH]/[SCRAPE]/[COMPLETE] 三行归 CrawlerRunConfig.verbose 管，两个都要关，
    输出里才不会留下每次不同的耗时。"""
    return BrowserConfig(verbose=False), CrawlerRunConfig(verbose=False)



import inspect

params = inspect.signature(CrawlerRunConfig.__init__).parameters
names = list(params)
print("参数总数 =", len(names) - 1)
for i in range(0, len(names), 9):
    print("   ", ", ".join(names[i:i + 9]))
print("CacheMode 的取值 =", [(m.name, m.value) for m in CacheMode])
for key in ["cache_mode", "word_count_threshold", "page_timeout", "wait_until", "wait_for",
            "magic", "scan_full_page", "check_robots_txt", "only_text", "verbose",
            "excluded_tags", "css_selector", "deep_crawl_strategy",
            "session_id", "screenshot", "pdf", "table_extraction"]:
    if key not in params:
        print("   %-22s 本机 0.9.5 没有这个参数" % key)
        continue
    print("   %-22s 默认 %r" % (key, params[key].default))
print("已删的老写法 keep_html -> ", end="")
try:
    CrawlerRunConfig(keep_html=True)
    print("居然还能用")
except Exception as exc:
    print(type(exc).__name__ + ": " + str(exc)[:66])
print("已删的老写法 bypass_cache -> ", end="")
try:
    print(repr(CrawlerRunConfig(bypass_cache=True).cache_mode))
except Exception as exc:
    print(type(exc).__name__ + ": " + str(exc)[:66])

async def check_html():
    bc, _ = quiet()
    async with AsyncWebCrawler(config=bc) as cr:
        r = await cr.arun(BASE + "/list", config=CrawlerRunConfig(verbose=False))
        print("keep_html 不存在时原始 html 仍然在结果里 =", r.html is not None,
              "| cleaned_html 也有 =", bool(r.cleaned_html))


asyncio.run(check_html())
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
参数总数 = 100
    self, word_count_threshold, extraction_strategy, chunking_strategy, markdown_generator, only_text, css_selector, target_elements, excluded_tags
    excluded_selector, keep_data_attributes, keep_attrs, remove_forms, prettiify, parser_type, scraping_strategy, proxy_config, proxy_rotation_strategy
    proxy_session_id, proxy_session_ttl, proxy_session_auto_release, locale, timezone_id, geolocation, fetch_ssl_certificate, cache_mode, session_id
    bypass_cache, disable_cache, no_cache_read, no_cache_write, shared_data, check_cache_freshness, cache_validation_timeout, wait_until, page_timeout
    wait_for, wait_for_timeout, wait_for_images, delay_before_return_html, mean_delay, max_range, semaphore_count, js_code, js_code_before_wait
    c4a_script, js_only, ignore_body_visibility, body_visibility_timeout, scan_full_page, scroll_delay, max_scroll_steps, process_iframes, flatten_shadow_dom
    remove_overlay_elements, remove_consent_popups, simulate_user, override_navigator, magic, adjust_viewport_to_content, screenshot, screenshot_wait_for, screenshot_height_threshold
    force_viewport_screenshot, pdf, capture_mhtml, image_description_min_word_threshold, image_score_threshold, table_score_threshold, table_extraction, exclude_external_images, exclude_all_images
    exclude_social_media_domains, exclude_external_links, exclude_social_media_links, exclude_domains, exclude_internal_links, score_links, preserve_https_for_internal_links, verbose, log_console
    capture_network_requests, capture_console_messages, method, stream, prefetch, process_in_browser, url, base_url, check_robots_txt
    user_agent, user_agent_mode, user_agent_generator_config, deep_crawl_strategy, link_preview_config, virtual_scroll_config, url_matcher, match_mode, experimental
    max_retries, fallback_fetch_function
CacheMode 的取值 = [('ENABLED', 'enabled'), ('DISABLED', 'disabled'), ('READ_ONLY', 'read_only'), ('WRITE_ONLY', 'write_only'), ('BYPASS', 'bypass')]
   cache_mode             默认 <CacheMode.BYPASS: 'bypass'>
   word_count_threshold   默认 1
   page_timeout           默认 60000
   wait_until             默认 'domcontentloaded'
   wait_for               默认 None
   magic                  默认 False
   scan_full_page         默认 False
   check_robots_txt       默认 False
   only_text              默认 False
   verbose                默认 True
   excluded_tags          默认 None
   css_selector           默认 None
   deep_crawl_strategy    默认 None
   session_id             默认 None
   screenshot             默认 False
   pdf                    默认 False
   table_extraction       默认 None
已删的老写法 keep_html -> TypeError: CrawlerRunConfig.__init__() got an unexpected keyword argument 'ke
已删的老写法 bypass_cache -> AttributeError: Setting 'bypass_cache' is deprecated. Instead, use cache_mode=Cach
keep_html 不存在时原始 html 仍然在结果里 = True | cleaned_html 也有 = True

默认值里这几条是「和文档/直觉不一样」的：

  • cache_mode 默认是 CacheMode.BYPASS，也就是默认不读缓存、抓完覆盖写。别按「默认会缓存」来理解第一次慢、第二次快的错觉。
  • word_count_threshold=1、page_timeout=60000、wait_until='domcontentloaded'、wait_for=None、magic=False、scan_full_page=False、check_robots_txt=False、only_text=False、verbose=True、excluded_tags=None、css_selector=None、deep_crawl_strategy=None、session_id=None、screenshot=False、pdf=False、table_extraction=None。
  • 参数面上有四个「看起来像开关但其实不是」的点：excluded_tags/css_selector/target_elements 是 None（不是空列表），所以传 [] 和省略不传是两回事；bypass_cache/disable_cache/no_cache_read/no_cache_write 这四个老参数名仍在签名里但已经不管用（下面这条实测）；dispatcher 不在这个类的参数里，它属于 arun_many（见批量抓取一节）。
  • 老写法实测：keep_html=True 直接 TypeError: ... got an unexpected keyword argument 'keep_html'；bypass_cache=True 更妙，它在赋值时就抛 AttributeError: Setting 'bypass_cache' is deprecated. Instead, use cache_mode=Cach...——这个类把废弃参数做成了 property setter，所以「传了不报错但没效果」的坑在这里反而不会发生。
  • keep_html 没了但原始 html 仍然在结果里（html 与 cleaned_html 都有），因为 crawl4ai 默认就把这几份产物挂在 CrawlResult 上，不需要开关。

缓存的五种模式

缓存这段的关键是别看耗时，看服务端被打了几次。靶站的 /count 每被请求一次就自增，URL 上带 ?mode=<模式名> 让五种模式各用各的缓存键：

cache_modes.py
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
# -*- coding: utf-8 -*-
"""五种 CacheMode 各自到底读不读缓存：判据是 cache_status + 服务端命中次数，不看耗时。

缓存落在 CRAWL4_AI_BASE_DIRECTORY 下的 SQLite 里，所以这一段先把家目录指到一个
临时目录，保证每次跑都是干净的第一次。"""

import os
import shutil
import tempfile

HOME = tempfile.mkdtemp(prefix="c4ai-home-")
os.environ["CRAWL4_AI_BASE_DIRECTORY"] = HOME

import asyncio
import inspect
import json
import os
import re
import threading
from http.server import BaseHTTPRequestHandler, HTTPServer

# 云广告一天只打一次，打不打取决于家目录里的标记文件，会把输出变得不可复现，所以先静音
os.environ.setdefault("CRAWL4AI_NO_CLOUD_NOTICE", "1")

from crawl4ai import (AsyncWebCrawler, BrowserConfig, CacheMode, CrawlerRunConfig)
from crawl4ai.async_crawler_strategy import AsyncPlaywrightCrawlerStrategy
import contextlib
import io
import os
import shutil
import tempfile


PORT = 8061
BASE = "http://127.0.0.1:8061"
HITS = {}


def html(title, body):
    return ("<!doctype html><html><head><meta charset='utf-8'><title>%s</title>"
            "<meta name='description' content='probe'><style>.nav{color:red}</style>"
            "</head><body>"
            "<div class='nav'>NAV 这是靶站的导航区，故意写长一点，"
            "免得页面被 crawl4ai 的结构性判据当成空壳</div>"
            "%s"
            "<div class='filler'>FILLER 这是靶站的页脚区，同样是有实际文字的标记</div>"
            "</body></html>" % (title, body))


LIST_BODY = ("<h1>列表页</h1><ul class='items'>"
             + "".join("<li><a href='/d/%d'>片名%d</a><span class='price'>%.1f</span></li>"
                       % (i, i, i * 1.5) for i in range(1, 6))
             + "</ul><a class='next' href='/list?page=2'>下一页</a>")

LAZY_BODY = ("<h1>懒加载列表</h1><div id='list'></div>"
             "<div style='height:4000px'></div>"
             "<script>window.addEventListener('scroll',function(){"
             "if(window.__on)return;window.__on=1;"
             "for(var i=0;i<20;i++){var d=document.createElement('div');"
             "d.className='item';d.textContent='滚动条目'+i;"
             "document.getElementById('list').appendChild(d);}});</script>")

SPA_BODY = ("<h1>前端渲染的列表</h1><div id='app'></div><script>setTimeout(function(){var s='';"
            "for(var i=0;i<8;i++){s+='<div class=row>渲染行'+i+'</div>'}"
            "document.getElementById('app').innerHTML=s;},200);</script>")

TABLE_BODY = ("<table class='movies'><tr><th>片名</th><th>年份</th></tr>"
              + "".join("<tr><td>片名%d</td><td>%d</td></tr>" % (i, 1990 + i)
                        for i in range(1, 4)) + "</table>")

COOKIE_BODY = ("<p id='c'></p><script>document.getElementById('c').textContent="
               "'COOKIE='+(document.cookie||'NONE')</script>")

FP_BODY = ("<p id='f'></p><script>document.getElementById('f').textContent="
           "'WD='+navigator.webdriver+'|PL='+navigator.plugins.length+"
           "'|V='+window.innerWidth+'x'+window.innerHeight+'|UA='+navigator.userAgent.slice(0,26)"
           "</script>")

NEXT_LEVEL = {1: ["a", "b"], 2: ["c", "d"], 3: ["e"]}


def level_body(n):
    links = "".join("<a href='/lvl%d/%s'>%s</a>" % (n + 1, x, x)
                    for x in NEXT_LEVEL.get(n, []))
    return html("第%d层" % n, "<h1>LEVEL-%d</h1>%s" % (n, links))


class Handler(BaseHTTPRequestHandler):

    def reply(self, text, ctype="text/html; charset=utf-8", code=200, cookie=None):
        data = text.encode("utf-8")
        self.send_response(code)
        self.send_header("Content-Type", ctype)
        self.send_header("Content-Length", str(len(data)))
        if cookie:
            self.send_header("Set-Cookie", cookie)
        self.end_headers()
        self.wfile.write(data)

    def do_GET(self):
        import time
        path = self.path.split("?")[0]
        HITS[path] = HITS.get(path, 0) + 1
        if path == "/robots.txt":
            return self.reply("User-agent: *\nDisallow: /private/\n", "text/plain")
        if path.startswith("/private"):
            return self.reply(html("私密", "<h1>私密区</h1><p>PRIVATE-SECRET-BODY</p>"
                                         "<p>这段说明文字让页面不会被结构性判据当成空壳。</p>"))
        if path == "/count":
            return self.reply(json.dumps({"count": HITS["/count"]}), "application/json")
        if path == "/ua":
            return self.reply(html("UA", "<p>UA=%s</p>" % self.headers.get("User-Agent")))
        if path == "/list":
            return self.reply(html("列表", LIST_BODY))
        if path == "/lazy":
            return self.reply(html("懒加载", LAZY_BODY))
        if path == "/spa":
            return self.reply(html("SPA", SPA_BODY))
        if path == "/table":
            return self.reply(html("表格", TABLE_BODY))
        if path == "/cookie":
            return self.reply(html("Cookie", COOKIE_BODY))
        if path == "/setcookie":
            return self.reply(html("发cookie", COOKIE_BODY), cookie="probe_c4a=1; Path=/")
        if path == "/fp":
            return self.reply(html("指纹", FP_BODY))
        match = re.match(r"/lvl(\d)/([a-z])", path)
        if match:
            n = int(match.group(1))
            if n > 3:
                return self.reply(html("叶子", "NO-MORE-LEVEL"), code=404)
            return self.reply(level_body(n))
        match = re.match(r"/d/(\d+)", path)
        if match:
            i = int(match.group(1))
            return self.reply(html("片名%d" % i,
                                   "<h1 class='t'>片名%d</h1><span class='p'>%.1f</span>"
                                   % (i, i * 1.5)))
        if path == "/status500":
            return self.reply("boom", "text/plain", 500)
        if path.startswith("/slow"):
            time.sleep(2.5)
            return self.reply(html("慢", "SLOW-DONE"))
        return self.reply(html("首页", "<a href='/list'>去列表</a>"))

    def do_POST(self):
        size = int(self.headers.get("Content-Length", 0))
        body = self.rfile.read(size).decode("utf-8")
        HITS["/echo"] = HITS.get("/echo", 0) + 1
        return self.reply(html("回显", "<pre>POSTED %s</pre>" % body))

    def log_message(self, *args):
        pass


def serve():
    server = HTTPServer(("127.0.0.1", PORT), Handler)
    # 客户端超时断开时 socketserver 会往 stderr 打整段 traceback（里面是临时端口号），
    # 那不是结论只是噪声，接住。
    server.handle_error = lambda request, client_address: None
    threading.Thread(target=server.serve_forever, daemon=True).start()
    return server


srv = serve()


def quiet():
    """静音 crawl4ai 的计时行：[INIT] 那行归 BrowserConfig.verbose 管，
    [FETCH]/[SCRAPE]/[COMPLETE] 三行归 CrawlerRunConfig.verbose 管，两个都要关，
    输出里才不会留下每次不同的耗时。"""
    return BrowserConfig(verbose=False), CrawlerRunConfig(verbose=False)



async def hit_twice(mode):
    bc, _ = quiet()
    url = BASE + "/count?mode=" + mode.name
    HITS.clear()
    async with AsyncWebCrawler(config=bc) as cr:
        first = await cr.arun(url, config=CrawlerRunConfig(verbose=False, cache_mode=mode))
        hit_after_first = HITS.get("/count", 0)
        second = await cr.arun(url, config=CrawlerRunConfig(verbose=False, cache_mode=mode))
    print("%-11s 第1次 %-6s（服务端被打 %d 次） 第2次 %-6s（服务端累计 %d 次）"
          % (mode.name, first.cache_status, hit_after_first,
             second.cache_status, HITS.get("/count", 0)))


async def main():
    for mode in CacheMode:
        await hit_twice(mode)
    print("缓存家目录里多了什么 =", sorted(os.listdir(HOME + "\\.crawl4ai")))


def noise(line):
    """家目录是新目录时，库会在启动时打「建备份 + 跑迁移」几行，文件名带时间戳，逐行丢掉。"""
    return any(key in line for key in ("Database backup created", "database migration",
                                       "Migration completed", "Database initialization",
                                       "post-installation", "Running post"))


out_buf, err_buf = io.StringIO(), io.StringIO()
with contextlib.redirect_stdout(out_buf), contextlib.redirect_stderr(err_buf):
    asyncio.run(main())
for stream in (out_buf, err_buf):
    for line in stream.getvalue().splitlines():
        if not noise(line):
            print(line)
shutil.rmtree(HOME, ignore_errors=True)
1
2
3
4
5
6
ENABLED     第1次 miss  （服务端被打 1 次） 第2次 hit   （服务端累计 1 次）
DISABLED    第1次 miss  （服务端被打 1 次） 第2次 miss  （服务端累计 2 次）
READ_ONLY   第1次 miss  （服务端被打 1 次） 第2次 miss  （服务端累计 2 次）
WRITE_ONLY  第1次 miss  （服务端被打 1 次） 第2次 miss  （服务端累计 2 次）
BYPASS      第1次 miss  （服务端被打 1 次） 第2次 miss  （服务端累计 2 次）
缓存家目录里多了什么 = ['.crawl4ai', 'cache', 'cleaned_html', 'crawl4ai.db', 'crawler.log', 'extracted_content', 'html_content', 'markdown_content', 'models', 'screenshots', 'version.txt']
  • 五种模式的实测语义：ENABLED 第一次 miss（服务端被打 1 次）、第二次 hit（服务端累计还是 1 次）——这是唯一一个「第二次不访问服务端」的模式；DISABLED、READ_ONLY、WRITE_ONLY、BYPASS 第二次都还是 miss、服务端累计 2 次。
  • 和文档表格对照，有两处要在意：文档说 READ_ONLY 是「只读缓存，未命中不抓」，但本机实跑第一次 miss、第二次仍 miss 且服务端被打了两次——在冷缓存条件下它与 BYPASS 的表现一样（这条只在本机这组条件下成立，别当成「READ_ONLY 无意义」）。WRITE_ONLY 的表现是每次都抓，与文档「只写不读」一致。
  • 默认 BYPASS（上一节）意味着：不加 cache_mode=CacheMode.ENABLED，重试同一 URL 就是真的重复打服务端。
  • 缓存落在哪：输出最后一行是临时家目录里的内容，crawl4ai.db 与 cache、html_content、markdown_content、extracted_content、cleaned_html、screenshots 等目录都在里面。把 CRAWL4_AI_BASE_DIRECTORY 指到一个临时目录，才能保证「每次跑都是干净的第一次」——注意库启动时会打四行「建备份 + 跑迁移」的日志，备份文件名带时间戳，这段按行把它们过滤掉了，否则输出无法复现。

markdown 怎么裁

excluded_tags、excluded_selector、css_selector、target_elements、only_text、word_count_threshold 都作用在「进 markdown 之前」的那一步。靶站页面里埋了 NAV 与 FILLER 两个标记，判据就是它们在不在：

content_tuning.py
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
# -*- coding: utf-8 -*-
"""markdown 的裁剪口径：word_count_threshold、excluded_tags、only_text、css 选择器。"""

import asyncio
import inspect
import json
import os
import re
import threading
from http.server import BaseHTTPRequestHandler, HTTPServer

# 云广告一天只打一次，打不打取决于家目录里的标记文件，会把输出变得不可复现，所以先静音
os.environ.setdefault("CRAWL4AI_NO_CLOUD_NOTICE", "1")

from crawl4ai import (AsyncWebCrawler, BrowserConfig, CacheMode, CrawlerRunConfig)
from crawl4ai.async_crawler_strategy import AsyncPlaywrightCrawlerStrategy


PORT = 8062
BASE = "http://127.0.0.1:8062"
HITS = {}


def html(title, body):
    return ("<!doctype html><html><head><meta charset='utf-8'><title>%s</title>"
            "<meta name='description' content='probe'><style>.nav{color:red}</style>"
            "</head><body>"
            "<div class='nav'>NAV 这是靶站的导航区，故意写长一点，"
            "免得页面被 crawl4ai 的结构性判据当成空壳</div>"
            "%s"
            "<div class='filler'>FILLER 这是靶站的页脚区，同样是有实际文字的标记</div>"
            "</body></html>" % (title, body))


LIST_BODY = ("<h1>列表页</h1><ul class='items'>"
             + "".join("<li><a href='/d/%d'>片名%d</a><span class='price'>%.1f</span></li>"
                       % (i, i, i * 1.5) for i in range(1, 6))
             + "</ul><a class='next' href='/list?page=2'>下一页</a>")

LAZY_BODY = ("<h1>懒加载列表</h1><div id='list'></div>"
             "<div style='height:4000px'></div>"
             "<script>window.addEventListener('scroll',function(){"
             "if(window.__on)return;window.__on=1;"
             "for(var i=0;i<20;i++){var d=document.createElement('div');"
             "d.className='item';d.textContent='滚动条目'+i;"
             "document.getElementById('list').appendChild(d);}});</script>")

SPA_BODY = ("<h1>前端渲染的列表</h1><div id='app'></div><script>setTimeout(function(){var s='';"
            "for(var i=0;i<8;i++){s+='<div class=row>渲染行'+i+'</div>'}"
            "document.getElementById('app').innerHTML=s;},200);</script>")

TABLE_BODY = ("<table class='movies'><tr><th>片名</th><th>年份</th></tr>"
              + "".join("<tr><td>片名%d</td><td>%d</td></tr>" % (i, 1990 + i)
                        for i in range(1, 4)) + "</table>")

COOKIE_BODY = ("<p id='c'></p><script>document.getElementById('c').textContent="
               "'COOKIE='+(document.cookie||'NONE')</script>")

FP_BODY = ("<p id='f'></p><script>document.getElementById('f').textContent="
           "'WD='+navigator.webdriver+'|PL='+navigator.plugins.length+"
           "'|V='+window.innerWidth+'x'+window.innerHeight+'|UA='+navigator.userAgent.slice(0,26)"
           "</script>")

NEXT_LEVEL = {1: ["a", "b"], 2: ["c", "d"], 3: ["e"]}


def level_body(n):
    links = "".join("<a href='/lvl%d/%s'>%s</a>" % (n + 1, x, x)
                    for x in NEXT_LEVEL.get(n, []))
    return html("第%d层" % n, "<h1>LEVEL-%d</h1>%s" % (n, links))


class Handler(BaseHTTPRequestHandler):

    def reply(self, text, ctype="text/html; charset=utf-8", code=200, cookie=None):
        data = text.encode("utf-8")
        self.send_response(code)
        self.send_header("Content-Type", ctype)
        self.send_header("Content-Length", str(len(data)))
        if cookie:
            self.send_header("Set-Cookie", cookie)
        self.end_headers()
        self.wfile.write(data)

    def do_GET(self):
        import time
        path = self.path.split("?")[0]
        HITS[path] = HITS.get(path, 0) + 1
        if path == "/robots.txt":
            return self.reply("User-agent: *\nDisallow: /private/\n", "text/plain")
        if path.startswith("/private"):
            return self.reply(html("私密", "<h1>私密区</h1><p>PRIVATE-SECRET-BODY</p>"
                                         "<p>这段说明文字让页面不会被结构性判据当成空壳。</p>"))
        if path == "/count":
            return self.reply(json.dumps({"count": HITS["/count"]}), "application/json")
        if path == "/ua":
            return self.reply(html("UA", "<p>UA=%s</p>" % self.headers.get("User-Agent")))
        if path == "/list":
            return self.reply(html("列表", LIST_BODY))
        if path == "/lazy":
            return self.reply(html("懒加载", LAZY_BODY))
        if path == "/spa":
            return self.reply(html("SPA", SPA_BODY))
        if path == "/table":
            return self.reply(html("表格", TABLE_BODY))
        if path == "/cookie":
            return self.reply(html("Cookie", COOKIE_BODY))
        if path == "/setcookie":
            return self.reply(html("发cookie", COOKIE_BODY), cookie="probe_c4a=1; Path=/")
        if path == "/fp":
            return self.reply(html("指纹", FP_BODY))
        match = re.match(r"/lvl(\d)/([a-z])", path)
        if match:
            n = int(match.group(1))
            if n > 3:
                return self.reply(html("叶子", "NO-MORE-LEVEL"), code=404)
            return self.reply(level_body(n))
        match = re.match(r"/d/(\d+)", path)
        if match:
            i = int(match.group(1))
            return self.reply(html("片名%d" % i,
                                   "<h1 class='t'>片名%d</h1><span class='p'>%.1f</span>"
                                   % (i, i * 1.5)))
        if path == "/status500":
            return self.reply("boom", "text/plain", 500)
        if path.startswith("/slow"):
            time.sleep(2.5)
            return self.reply(html("慢", "SLOW-DONE"))
        return self.reply(html("首页", "<a href='/list'>去列表</a>"))

    def do_POST(self):
        size = int(self.headers.get("Content-Length", 0))
        body = self.rfile.read(size).decode("utf-8")
        HITS["/echo"] = HITS.get("/echo", 0) + 1
        return self.reply(html("回显", "<pre>POSTED %s</pre>" % body))

    def log_message(self, *args):
        pass


def serve():
    server = HTTPServer(("127.0.0.1", PORT), Handler)
    # 客户端超时断开时 socketserver 会往 stderr 打整段 traceback（里面是临时端口号），
    # 那不是结论只是噪声，接住。
    server.handle_error = lambda request, client_address: None
    threading.Thread(target=server.serve_forever, daemon=True).start()
    return server


srv = serve()


def quiet():
    """静音 crawl4ai 的计时行：[INIT] 那行归 BrowserConfig.verbose 管，
    [FETCH]/[SCRAPE]/[COMPLETE] 三行归 CrawlerRunConfig.verbose 管，两个都要关，
    输出里才不会留下每次不同的耗时。"""
    return BrowserConfig(verbose=False), CrawlerRunConfig(verbose=False)



async def show(label, url=None, **kwargs):
    bc, _ = quiet()
    async with AsyncWebCrawler(config=bc) as cr:
        try:
            r = await cr.arun(url or BASE + "/list",
                              config=CrawlerRunConfig(verbose=False, **kwargs))
        except Exception as exc:
            print("%-34s 抛 %s: %s" % (label, type(exc).__name__, str(exc)[:60]))
            return ""
        md = r.markdown or ""
        print("%-34s 行数 %2d 字符 %5d | success=%-5s | NAV %s | FILLER %s | 片名 %d 个"
              % (label, len(md.splitlines()), len(md), r.success, "NAV" in md,
                 "FILLER" in md, md.count("片名")))
    return md


async def main():
    md = await show("默认")
    print("默认 markdown 开头 =", md.replace("\n", " ")[:74])
    await show("word_count_threshold=200", word_count_threshold=200)
    await show("only_text=True", only_text=True)
    await show("excluded_tags=['div']", excluded_tags=["div"])
    await show("excluded_selector='.filler'", excluded_selector=".filler")
    await show("css_selector='.items'", css_selector=".items")
    await show("target_elements=['.items']", target_elements=[".items"])
    print("靶站这页只有 300 多字符，阈值参数看不出差别，换在线的 ssr1：")
    for threshold in (0, 300, 3000):
        await show("ssr1 word_count_threshold=%d" % threshold,
                   url="https://ssr1.scrape.center", word_count_threshold=threshold)

asyncio.run(main())
1
2
3
4
5
6
7
8
9
10
11
12
默认                                 行数 10 字符   322 | success=True  | NAV True | FILLER True | 片名 5 个
默认 markdown 开头 = NAV 这是靶站的导航区，故意写长一点，免得页面被 crawl4ai 的结构性判据当成空壳 # 列表页   * [片名1](http://127.0
word_count_threshold=200           行数 10 字符   322 | success=True  | NAV True | FILLER True | 片名 5 个
only_text=True                     行数 10 字符   322 | success=True  | NAV True | FILLER True | 片名 5 个
excluded_tags=['div']              行数  8 字符   248 | success=True  | NAV False | FILLER False | 片名 5 个
excluded_selector='.filler'        行数  9 字符   294 | success=True  | NAV True | FILLER False | 片名 5 个
css_selector='.items'              行数  7 字符   202 | success=True  | NAV False | FILLER False | 片名 5 个
target_elements=['.items']         行数  7 字符   202 | success=True  | NAV False | FILLER False | 片名 5 个
靶站这页只有 300 多字符，阈值参数看不出差别，换在线的 ssr1：
ssr1 word_count_threshold=0        行数 74 字符  3015 | success=True  | NAV False | FILLER False | 片名 0 个
ssr1 word_count_threshold=300      行数 74 字符  3015 | success=True  | NAV False | FILLER False | 片名 0 个
ssr1 word_count_threshold=3000     行数 74 字符  3015 | success=True  | NAV False | FILLER False | 片名 0 个
  • excluded_tags=["div"] 把两个 div 都拿掉（NAV False | FILLER False，10 行变 8 行）；excluded_selector=".filler" 只拿掉页脚（9 行）；css_selector=".items" 与 target_elements=[".items"] 都只保留列表区，两者的行数与字符数完全一致（7 行、202 字符）——一个是 CSS 作用域、一个是「目标元素列表」，在这个用例里等价。
  • only_text=True 与 word_count_threshold=200 在靶站这页上看不出变化（只有 322 字符、词数很少），所以把同一参数换到在线的 ssr1 上试：word_count_threshold=0/300/3000 三档的 markdown 都是 74 行、3015 字符。这说明 word_count_threshold 不是「整页字数下限」，而是「文本块级别太短就丢掉」的阈值，用它来裁整页是没有效果的（要裁整页用 css_selector 或 markdown_generator(content_filter=...)）。
  • 默认 markdown 的开头也能读出结构：先是导航区那句长文字，然后 # 列表页，再是 * [片名1](…/d/1)1.5 这样的条目——标题、链接、正文价格粘连在一起。markdown 是给人和 LLM 读的，不是给解析用的，要结构化数据请看「结构化抽取四路」。

markdown 生成器

markdown_generator 传的是策略对象。这段把 DefaultMarkdownGenerator 的参数面、options 的三种开关、两种内容过滤器，以及「产物到底在哪」都跑了一遍：

markdown_gen.py
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
# -*- coding: utf-8 -*-
"""markdown 是怎么生成的：DefaultMarkdownGenerator 的开关与两种内容过滤器。"""

import asyncio
import inspect
import json
import os
import re
import threading
from http.server import BaseHTTPRequestHandler, HTTPServer

# 云广告一天只打一次，打不打取决于家目录里的标记文件，会把输出变得不可复现，所以先静音
os.environ.setdefault("CRAWL4AI_NO_CLOUD_NOTICE", "1")

from crawl4ai import (AsyncWebCrawler, BrowserConfig, CacheMode, CrawlerRunConfig)
from crawl4ai.async_crawler_strategy import AsyncPlaywrightCrawlerStrategy


PORT = 8070
BASE = "http://127.0.0.1:8070"
HITS = {}


def html(title, body):
    return ("<!doctype html><html><head><meta charset='utf-8'><title>%s</title>"
            "<meta name='description' content='probe'><style>.nav{color:red}</style>"
            "</head><body>"
            "<div class='nav'>NAV 这是靶站的导航区，故意写长一点，"
            "免得页面被 crawl4ai 的结构性判据当成空壳</div>"
            "%s"
            "<div class='filler'>FILLER 这是靶站的页脚区，同样是有实际文字的标记</div>"
            "</body></html>" % (title, body))


LIST_BODY = ("<h1>列表页</h1><ul class='items'>"
             + "".join("<li><a href='/d/%d'>片名%d</a><span class='price'>%.1f</span></li>"
                       % (i, i, i * 1.5) for i in range(1, 6))
             + "</ul><a class='next' href='/list?page=2'>下一页</a>")

LAZY_BODY = ("<h1>懒加载列表</h1><div id='list'></div>"
             "<div style='height:4000px'></div>"
             "<script>window.addEventListener('scroll',function(){"
             "if(window.__on)return;window.__on=1;"
             "for(var i=0;i<20;i++){var d=document.createElement('div');"
             "d.className='item';d.textContent='滚动条目'+i;"
             "document.getElementById('list').appendChild(d);}});</script>")

SPA_BODY = ("<h1>前端渲染的列表</h1><div id='app'></div><script>setTimeout(function(){var s='';"
            "for(var i=0;i<8;i++){s+='<div class=row>渲染行'+i+'</div>'}"
            "document.getElementById('app').innerHTML=s;},200);</script>")

TABLE_BODY = ("<table class='movies'><tr><th>片名</th><th>年份</th></tr>"
              + "".join("<tr><td>片名%d</td><td>%d</td></tr>" % (i, 1990 + i)
                        for i in range(1, 4)) + "</table>")

COOKIE_BODY = ("<p id='c'></p><script>document.getElementById('c').textContent="
               "'COOKIE='+(document.cookie||'NONE')</script>")

FP_BODY = ("<p id='f'></p><script>document.getElementById('f').textContent="
           "'WD='+navigator.webdriver+'|PL='+navigator.plugins.length+"
           "'|V='+window.innerWidth+'x'+window.innerHeight+'|UA='+navigator.userAgent.slice(0,26)"
           "</script>")

NEXT_LEVEL = {1: ["a", "b"], 2: ["c", "d"], 3: ["e"]}


def level_body(n):
    links = "".join("<a href='/lvl%d/%s'>%s</a>" % (n + 1, x, x)
                    for x in NEXT_LEVEL.get(n, []))
    return html("第%d层" % n, "<h1>LEVEL-%d</h1>%s" % (n, links))


class Handler(BaseHTTPRequestHandler):

    def reply(self, text, ctype="text/html; charset=utf-8", code=200, cookie=None):
        data = text.encode("utf-8")
        self.send_response(code)
        self.send_header("Content-Type", ctype)
        self.send_header("Content-Length", str(len(data)))
        if cookie:
            self.send_header("Set-Cookie", cookie)
        self.end_headers()
        self.wfile.write(data)

    def do_GET(self):
        import time
        path = self.path.split("?")[0]
        HITS[path] = HITS.get(path, 0) + 1
        if path == "/robots.txt":
            return self.reply("User-agent: *\nDisallow: /private/\n", "text/plain")
        if path.startswith("/private"):
            return self.reply(html("私密", "<h1>私密区</h1><p>PRIVATE-SECRET-BODY</p>"
                                         "<p>这段说明文字让页面不会被结构性判据当成空壳。</p>"))
        if path == "/count":
            return self.reply(json.dumps({"count": HITS["/count"]}), "application/json")
        if path == "/ua":
            return self.reply(html("UA", "<p>UA=%s</p>" % self.headers.get("User-Agent")))
        if path == "/list":
            return self.reply(html("列表", LIST_BODY))
        if path == "/lazy":
            return self.reply(html("懒加载", LAZY_BODY))
        if path == "/spa":
            return self.reply(html("SPA", SPA_BODY))
        if path == "/table":
            return self.reply(html("表格", TABLE_BODY))
        if path == "/cookie":
            return self.reply(html("Cookie", COOKIE_BODY))
        if path == "/setcookie":
            return self.reply(html("发cookie", COOKIE_BODY), cookie="probe_c4a=1; Path=/")
        if path == "/fp":
            return self.reply(html("指纹", FP_BODY))
        match = re.match(r"/lvl(\d)/([a-z])", path)
        if match:
            n = int(match.group(1))
            if n > 3:
                return self.reply(html("叶子", "NO-MORE-LEVEL"), code=404)
            return self.reply(level_body(n))
        match = re.match(r"/d/(\d+)", path)
        if match:
            i = int(match.group(1))
            return self.reply(html("片名%d" % i,
                                   "<h1 class='t'>片名%d</h1><span class='p'>%.1f</span>"
                                   % (i, i * 1.5)))
        if path == "/status500":
            return self.reply("boom", "text/plain", 500)
        if path.startswith("/slow"):
            time.sleep(2.5)
            return self.reply(html("慢", "SLOW-DONE"))
        return self.reply(html("首页", "<a href='/list'>去列表</a>"))

    def do_POST(self):
        size = int(self.headers.get("Content-Length", 0))
        body = self.rfile.read(size).decode("utf-8")
        HITS["/echo"] = HITS.get("/echo", 0) + 1
        return self.reply(html("回显", "<pre>POSTED %s</pre>" % body))

    def log_message(self, *args):
        pass


def serve():
    server = HTTPServer(("127.0.0.1", PORT), Handler)
    # 客户端超时断开时 socketserver 会往 stderr 打整段 traceback（里面是临时端口号），
    # 那不是结论只是噪声，接住。
    server.handle_error = lambda request, client_address: None
    threading.Thread(target=server.serve_forever, daemon=True).start()
    return server


srv = serve()


def quiet():
    """静音 crawl4ai 的计时行：[INIT] 那行归 BrowserConfig.verbose 管，
    [FETCH]/[SCRAPE]/[COMPLETE] 三行归 CrawlerRunConfig.verbose 管，两个都要关，
    输出里才不会留下每次不同的耗时。"""
    return BrowserConfig(verbose=False), CrawlerRunConfig(verbose=False)



import inspect
import warnings

import crawl4ai
from crawl4ai import BM25ContentFilter, DefaultMarkdownGenerator, PruningContentFilter

# 库自己会往 stderr 打 DeprecationWarning（带脚本绝对路径与行号，每次一样但属于噪声），
# 全局关掉；要看原文的地方用 catch_warnings 主动收集。
warnings.simplefilter("ignore")


async def gen(label, generator):
    bc, _ = quiet()
    async with AsyncWebCrawler(config=bc) as cr:
        r = await cr.arun(BASE + "/list",
                          config=CrawlerRunConfig(verbose=False, markdown_generator=generator))
        md = r.markdown
        if isinstance(md, str):
            print("%-52s str，行数 %2d 字符 %4d | 链接语法 %s | NAV %s"
                  % (label, len(md.splitlines()), len(md), "[片名1](" in md, "NAV" in md))
        else:
            raw = md.raw_markdown
            print("%-52s %s，raw 行数 %2d 字符 %4d | 链接语法 %s | references %d 条 | links %d 条"
                  % (label, type(md).__name__, len(raw.splitlines()), len(raw),
                     "[片名1](" in raw, len(md.references or ""), len(md.links or "")))


async def main():
    dnames = list(inspect.signature(DefaultMarkdownGenerator.__init__).parameters)[1:]
    print("DefaultMarkdownGenerator 的参数 %d 个 =" % len(dnames), ", ".join(dnames))
    print("顶层有没有 LLMarkdownGenerationStrategy =",
          hasattr(crawl4ai, "LLMarkdownGenerationStrategy"))
    print("顶层有没有 RequestCache =", hasattr(crawl4ai, "RequestCache"))
    await gen("不传 markdown_generator", None)
    await gen("DefaultMarkdownGenerator()", DefaultMarkdownGenerator())
    await gen("options={'include_links': False}",
              DefaultMarkdownGenerator(options={"include_links": False}))
    await gen("options={'include_images': True}",
              DefaultMarkdownGenerator(options={"include_images": True}))
    await gen("options={'citations': True}",
              DefaultMarkdownGenerator(options={"citations": True}))
    await gen("PruningContentFilter(threshold=0.48)",
              DefaultMarkdownGenerator(content_filter=PruningContentFilter(threshold=0.48)))
    await gen("BM25ContentFilter()", DefaultMarkdownGenerator(content_filter=BM25ContentFilter()))
    print("PruningContentFilter 这一路会不会告警 =", end=" ")
    with warnings.catch_warnings(record=True) as caught:
        warnings.simplefilter("always")
        DefaultMarkdownGenerator(content_filter=PruningContentFilter(threshold=0.48))
    print([str(w.message)[:56] for w in caught] or "没有告警")

    g = DefaultMarkdownGenerator()
    print("生成器实例上的字段 =", sorted(vars(g).keys()))
    bc, _ = quiet()
    async with AsyncWebCrawler(config=bc) as cr:
        r = await cr.arun(BASE + "/list",
                          config=CrawlerRunConfig(verbose=False, markdown_generator=g))
        print("传了生成器之后 result.markdown 的类型 =", type(r.markdown).__name__)
        print("字符串 markdown 上取 .fit_markdown ->",
              type(getattr(r, "fit_markdown", "没有这个属性")).__name__)
        made = g.generate_markdown(r.html)
        print("直接调 generate_markdown 得到的类型 =", type(made).__name__)
        print("   它的字段 =", sorted(type(made).model_fields))
        print("   raw_markdown 行数 =", len(made.raw_markdown.splitlines()),
              "| fit_markdown =", type(made.fit_markdown).__name__)
        cited = g.generate_markdown(r.html, citations=True)
        print("   citations=True 之后 markdown_with_citations 里有 [^ 标记 =",
              "[^" in (cited.markdown_with_citations or ""),
              "| references_markdown 行数 =",
              len((cited.references_markdown or "").splitlines()))

asyncio.run(main())
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
DefaultMarkdownGenerator 的参数 3 个 = content_filter, options, content_source
顶层有没有 LLMarkdownGenerationStrategy = False
顶层有没有 RequestCache = False
不传 markdown_generator                                str，行数 10 字符  322 | 链接语法 True | NAV True
DefaultMarkdownGenerator()                           str，行数 10 字符  322 | 链接语法 True | NAV True
options={'include_links': False}                     str，行数 10 字符  322 | 链接语法 True | NAV True
options={'include_images': True}                     str，行数 10 字符  322 | 链接语法 True | NAV True
options={'citations': True}                          str，行数 10 字符  322 | 链接语法 True | NAV True
PruningContentFilter(threshold=0.48)                 str，行数 10 字符  322 | 链接语法 True | NAV True
BM25ContentFilter()                                  str，行数 10 字符  322 | 链接语法 True | NAV True
PruningContentFilter 这一路会不会告警 = ['PruningContentFilter (BeautifulSoup-based) is deprecated']
生成器实例上的字段 = ['content_filter', 'content_source', 'options', 'verbose']
传了生成器之后 result.markdown 的类型 = StringCompatibleMarkdown
字符串 markdown 上取 .fit_markdown -> str
直接调 generate_markdown 得到的类型 = MarkdownGenerationResult
   它的字段 = ['fit_html', 'fit_markdown', 'markdown_with_citations', 'raw_markdown', 'references_markdown']
   raw_markdown 行数 = 10 | fit_markdown = str
   citations=True 之后 markdown_with_citations 里有 [^ 标记 = False | references_markdown 行数 = 10
  • DefaultMarkdownGenerator 只有 3 个参数：content_filter、options、content_source。include_links/include_images/citations 不是构造参数，要写进 options={...}——直接当关键字传会 TypeError。实例上的属性是 content_filter、content_source、options、verbose 四个。
  • 传了生成器之后 result.markdown 不再是 str，而是 StringCompatibleMarkdown（字符串兼容类：len()、切片、in 都还能用，但 .fit_markdown 取不到——本例打印出它的类型是 str，也就是取到的是那个兼容层给的同名属性而不是产物）。要拿 raw_markdown/fit_markdown/references_markdown 这些分开的产物，就直接调 generate_markdown(html)，它返回 MarkdownGenerationResult，字段是 raw_markdown、markdown_with_citations、references_markdown、fit_markdown、fit_html 五个。
  • 本例里 citations=True 之后 markdown_with_citations 中没有 [^ 标记、references_markdown 是 10 行——即引用表被生成但正文里没插引用号，因为这块靶站页面只有一个链接来源，不足触发引用。别把 citations 当「一定有引用」的开关用，要用就同时核对两个字段。
  • 两个内容过滤器都能构造并接进去（PruningContentFilter(threshold=0.48)、BM25ContentFilter()），但 PruningContentFilter 这条路会打一句 DeprecationWarning：BeautifulSoup 版的 PruningContentFilter 已废弃，官方改推 lxml 版（PruningContentFilterLXML，快约 10 倍、输出相同）。这是本篇实测到的唯一一个「文档还没改、库已经改」的迁移点，写新代码直接用 lxml 版。
  • LLMarkdownGenerationStrategy 这个名字在顶层与 async_configs 里都找不到（hasattr 是 False）——文档里那套「用 LLM 生成 markdown」的入口在本机 0.9.5 不叫这个；RequestCache 同样不在顶层。这两个名字别再照抄。

懒加载与等待

这一节把「等」这件事拆成四个参数，判据是靶站里两个专门设计的页面：/lazy 要滚动才追加 20 条，/spa 要等 200ms 的 JS 才渲染 8 行。

lazy_and_magic.py
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
# -*- coding: utf-8 -*-
"""懒加载与等待：默认不等、不等就什么都没有；wait_for 才是决定性的那一下。"""

import asyncio
import inspect
import json
import os
import re
import threading
from http.server import BaseHTTPRequestHandler, HTTPServer

# 云广告一天只打一次，打不打取决于家目录里的标记文件，会把输出变得不可复现，所以先静音
os.environ.setdefault("CRAWL4AI_NO_CLOUD_NOTICE", "1")

from crawl4ai import (AsyncWebCrawler, BrowserConfig, CacheMode, CrawlerRunConfig)
from crawl4ai.async_crawler_strategy import AsyncPlaywrightCrawlerStrategy
from crawl4ai import VirtualScrollConfig


PORT = 8065
BASE = "http://127.0.0.1:8065"
HITS = {}


def html(title, body):
    return ("<!doctype html><html><head><meta charset='utf-8'><title>%s</title>"
            "<meta name='description' content='probe'><style>.nav{color:red}</style>"
            "</head><body>"
            "<div class='nav'>NAV 这是靶站的导航区，故意写长一点，"
            "免得页面被 crawl4ai 的结构性判据当成空壳</div>"
            "%s"
            "<div class='filler'>FILLER 这是靶站的页脚区，同样是有实际文字的标记</div>"
            "</body></html>" % (title, body))


LIST_BODY = ("<h1>列表页</h1><ul class='items'>"
             + "".join("<li><a href='/d/%d'>片名%d</a><span class='price'>%.1f</span></li>"
                       % (i, i, i * 1.5) for i in range(1, 6))
             + "</ul><a class='next' href='/list?page=2'>下一页</a>")

LAZY_BODY = ("<h1>懒加载列表</h1><div id='list'></div>"
             "<div style='height:4000px'></div>"
             "<script>window.addEventListener('scroll',function(){"
             "if(window.__on)return;window.__on=1;"
             "for(var i=0;i<20;i++){var d=document.createElement('div');"
             "d.className='item';d.textContent='滚动条目'+i;"
             "document.getElementById('list').appendChild(d);}});</script>")

SPA_BODY = ("<h1>前端渲染的列表</h1><div id='app'></div><script>setTimeout(function(){var s='';"
            "for(var i=0;i<8;i++){s+='<div class=row>渲染行'+i+'</div>'}"
            "document.getElementById('app').innerHTML=s;},200);</script>")

TABLE_BODY = ("<table class='movies'><tr><th>片名</th><th>年份</th></tr>"
              + "".join("<tr><td>片名%d</td><td>%d</td></tr>" % (i, 1990 + i)
                        for i in range(1, 4)) + "</table>")

COOKIE_BODY = ("<p id='c'></p><script>document.getElementById('c').textContent="
               "'COOKIE='+(document.cookie||'NONE')</script>")

FP_BODY = ("<p id='f'></p><script>document.getElementById('f').textContent="
           "'WD='+navigator.webdriver+'|PL='+navigator.plugins.length+"
           "'|V='+window.innerWidth+'x'+window.innerHeight+'|UA='+navigator.userAgent.slice(0,26)"
           "</script>")

NEXT_LEVEL = {1: ["a", "b"], 2: ["c", "d"], 3: ["e"]}


def level_body(n):
    links = "".join("<a href='/lvl%d/%s'>%s</a>" % (n + 1, x, x)
                    for x in NEXT_LEVEL.get(n, []))
    return html("第%d层" % n, "<h1>LEVEL-%d</h1>%s" % (n, links))


class Handler(BaseHTTPRequestHandler):

    def reply(self, text, ctype="text/html; charset=utf-8", code=200, cookie=None):
        data = text.encode("utf-8")
        self.send_response(code)
        self.send_header("Content-Type", ctype)
        self.send_header("Content-Length", str(len(data)))
        if cookie:
            self.send_header("Set-Cookie", cookie)
        self.end_headers()
        self.wfile.write(data)

    def do_GET(self):
        import time
        path = self.path.split("?")[0]
        HITS[path] = HITS.get(path, 0) + 1
        if path == "/robots.txt":
            return self.reply("User-agent: *\nDisallow: /private/\n", "text/plain")
        if path.startswith("/private"):
            return self.reply(html("私密", "<h1>私密区</h1><p>PRIVATE-SECRET-BODY</p>"
                                         "<p>这段说明文字让页面不会被结构性判据当成空壳。</p>"))
        if path == "/count":
            return self.reply(json.dumps({"count": HITS["/count"]}), "application/json")
        if path == "/ua":
            return self.reply(html("UA", "<p>UA=%s</p>" % self.headers.get("User-Agent")))
        if path == "/list":
            return self.reply(html("列表", LIST_BODY))
        if path == "/lazy":
            return self.reply(html("懒加载", LAZY_BODY))
        if path == "/spa":
            return self.reply(html("SPA", SPA_BODY))
        if path == "/table":
            return self.reply(html("表格", TABLE_BODY))
        if path == "/cookie":
            return self.reply(html("Cookie", COOKIE_BODY))
        if path == "/setcookie":
            return self.reply(html("发cookie", COOKIE_BODY), cookie="probe_c4a=1; Path=/")
        if path == "/fp":
            return self.reply(html("指纹", FP_BODY))
        match = re.match(r"/lvl(\d)/([a-z])", path)
        if match:
            n = int(match.group(1))
            if n > 3:
                return self.reply(html("叶子", "NO-MORE-LEVEL"), code=404)
            return self.reply(level_body(n))
        match = re.match(r"/d/(\d+)", path)
        if match:
            i = int(match.group(1))
            return self.reply(html("片名%d" % i,
                                   "<h1 class='t'>片名%d</h1><span class='p'>%.1f</span>"
                                   % (i, i * 1.5)))
        if path == "/status500":
            return self.reply("boom", "text/plain", 500)
        if path.startswith("/slow"):
            time.sleep(2.5)
            return self.reply(html("慢", "SLOW-DONE"))
        return self.reply(html("首页", "<a href='/list'>去列表</a>"))

    def do_POST(self):
        size = int(self.headers.get("Content-Length", 0))
        body = self.rfile.read(size).decode("utf-8")
        HITS["/echo"] = HITS.get("/echo", 0) + 1
        return self.reply(html("回显", "<pre>POSTED %s</pre>" % body))

    def log_message(self, *args):
        pass


def serve():
    server = HTTPServer(("127.0.0.1", PORT), Handler)
    # 客户端超时断开时 socketserver 会往 stderr 打整段 traceback（里面是临时端口号），
    # 那不是结论只是噪声，接住。
    server.handle_error = lambda request, client_address: None
    threading.Thread(target=server.serve_forever, daemon=True).start()
    return server


srv = serve()


def quiet():
    """静音 crawl4ai 的计时行：[INIT] 那行归 BrowserConfig.verbose 管，
    [FETCH]/[SCRAPE]/[COMPLETE] 三行归 CrawlerRunConfig.verbose 管，两个都要关，
    输出里才不会留下每次不同的耗时。"""
    return BrowserConfig(verbose=False), CrawlerRunConfig(verbose=False)



async def count_items(label, url, marker, **kwargs):
    bc, _ = quiet()
    async with AsyncWebCrawler(config=bc) as cr:
        try:
            r = await cr.arun(url, config=CrawlerRunConfig(verbose=False, **kwargs))
            if r is None:
                print("%-40s 直接返回 None" % label)
                return
            body = r.html or ""
            print("%-40s %s %2d | markdown 行数 %2d | success=%s"
                  % (label, marker, body.count(marker),
                     len((r.markdown or "").splitlines()), r.success))
        except Exception as exc:
            print("%-40s 抛 %s: %s" % (label, type(exc).__name__, str(exc)[:58]))


async def main():
    lazy, spa = BASE + "/lazy", BASE + "/spa"
    await count_items("懒加载页 / 默认（不滚）", lazy, "滚动条目", magic=False)
    await count_items("懒加载页 / scan_full_page=True", lazy, "滚动条目", scan_full_page=True)
    print("VirtualScrollConfig 的参数 =",
          ", ".join(list(inspect.signature(VirtualScrollConfig.__init__).parameters)[1:]))
    await count_items("懒加载页 / 虚拟滚动 20 次", lazy, "滚动条目",
                      virtual_scroll_config=VirtualScrollConfig(
                          container_selector="body", scroll_count=20, scroll_by=800,
                          wait_after_scroll=0.1))
    await count_items("SPA 页 / 默认（200ms 的 JS 还没跑）", spa, "渲染行", magic=False)
    await count_items("SPA 页 / wait_for='.row'", spa, "渲染行", wait_for=".row")
    await count_items("SPA 页 / wait_for_timeout=800", spa, "渲染行", wait_for_timeout=800)
    await count_items("SPA 页 / wait_for 等一个不存在的元素", spa, "渲染行",
                      wait_for="#nope", page_timeout=1200)

asyncio.run(main())
1
2
3
4
5
6
7
8
懒加载页 / 默认（不滚）                            滚动条目  1 | markdown 行数  3 | success=True
懒加载页 / scan_full_page=True               滚动条目 21 | markdown 行数 23 | success=True
VirtualScrollConfig 的参数 = container_selector, scroll_count, scroll_by, wait_after_scroll
懒加载页 / 虚拟滚动 20 次                         滚动条目  1 | markdown 行数  3 | success=True
SPA 页 / 默认（200ms 的 JS 还没跑）               渲染行  1 | markdown 行数  3 | success=True
SPA 页 / wait_for='.row'                  渲染行  9 | markdown 行数 11 | success=True
SPA 页 / wait_for_timeout=800             渲染行  1 | markdown 行数  3 | success=True
SPA 页 / wait_for 等一个不存在的元素               渲染行  0 | markdown 行数  0 | success=False
  • 数值的含义先说清：滚动条目 1 是「只看到脚本源码里那个字符串」，也就是 JS 没跑出来任何条目；滚动条目 21 是 1 处源码 + 20 条真实渲染。渲染行 9 = 1 处源码 + 8 行渲染。
  • scan_full_page=True 是把 20 条滚出来的那个开关（21）；只传 virtual_scroll_config 不打开 scan_full_page 时什么也不会发生（还是 1）。VirtualScrollConfig 的真实参数是 container_selector, scroll_count, scroll_by, wait_after_scroll——网上流传的 step_factor/max_steps/wait_factor 在本机 0.9.5 会 TypeError。
  • SPA 页面这边：wait_for=".row" 是唯一决定性的那一下（9 行都出来了）；wait_for_timeout=800 反而没用（还是 1）——它不是「再等 800ms」的意思，别把它当 sleep 用。
  • wait_for 等一个永远不出现的元素时，success=False、markdown 空、渲染行 0，不会抛异常。所以 wait_for 的失败是靠 success 判的，写批量任务时要给这一类页面留重试。
  • magic 默认就是 True（CrawlerRunConfig.magic 的默认值在参数面那节打印过），所以「什么都不传」时 crawl4ai 已经在做模拟滚动/等待一类的事了；上面 magic=False 那两行才是裸行为。

JS 注入与表单

js_code 是在页面里执行一段 JS；表单抓取这一路 0.9.5 换了写法，必须实测才知道老参数已经全没了：

js_and_form.py
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
# -*- coding: utf-8 -*-
"""js_code 注入没问题；表单抓取这一路 0.9.5 换了写法：post_url/form_data 已经不在参数里了。"""

import asyncio
import inspect
import json
import os
import re
import threading
from http.server import BaseHTTPRequestHandler, HTTPServer

# 云广告一天只打一次，打不打取决于家目录里的标记文件，会把输出变得不可复现，所以先静音
os.environ.setdefault("CRAWL4AI_NO_CLOUD_NOTICE", "1")

from crawl4ai import (AsyncWebCrawler, BrowserConfig, CacheMode, CrawlerRunConfig)
from crawl4ai.async_crawler_strategy import AsyncPlaywrightCrawlerStrategy


PORT = 8064
BASE = "http://127.0.0.1:8064"
HITS = {}


def html(title, body):
    return ("<!doctype html><html><head><meta charset='utf-8'><title>%s</title>"
            "<meta name='description' content='probe'><style>.nav{color:red}</style>"
            "</head><body>"
            "<div class='nav'>NAV 这是靶站的导航区，故意写长一点，"
            "免得页面被 crawl4ai 的结构性判据当成空壳</div>"
            "%s"
            "<div class='filler'>FILLER 这是靶站的页脚区，同样是有实际文字的标记</div>"
            "</body></html>" % (title, body))


LIST_BODY = ("<h1>列表页</h1><ul class='items'>"
             + "".join("<li><a href='/d/%d'>片名%d</a><span class='price'>%.1f</span></li>"
                       % (i, i, i * 1.5) for i in range(1, 6))
             + "</ul><a class='next' href='/list?page=2'>下一页</a>")

LAZY_BODY = ("<h1>懒加载列表</h1><div id='list'></div>"
             "<div style='height:4000px'></div>"
             "<script>window.addEventListener('scroll',function(){"
             "if(window.__on)return;window.__on=1;"
             "for(var i=0;i<20;i++){var d=document.createElement('div');"
             "d.className='item';d.textContent='滚动条目'+i;"
             "document.getElementById('list').appendChild(d);}});</script>")

SPA_BODY = ("<h1>前端渲染的列表</h1><div id='app'></div><script>setTimeout(function(){var s='';"
            "for(var i=0;i<8;i++){s+='<div class=row>渲染行'+i+'</div>'}"
            "document.getElementById('app').innerHTML=s;},200);</script>")

TABLE_BODY = ("<table class='movies'><tr><th>片名</th><th>年份</th></tr>"
              + "".join("<tr><td>片名%d</td><td>%d</td></tr>" % (i, 1990 + i)
                        for i in range(1, 4)) + "</table>")

COOKIE_BODY = ("<p id='c'></p><script>document.getElementById('c').textContent="
               "'COOKIE='+(document.cookie||'NONE')</script>")

FP_BODY = ("<p id='f'></p><script>document.getElementById('f').textContent="
           "'WD='+navigator.webdriver+'|PL='+navigator.plugins.length+"
           "'|V='+window.innerWidth+'x'+window.innerHeight+'|UA='+navigator.userAgent.slice(0,26)"
           "</script>")

NEXT_LEVEL = {1: ["a", "b"], 2: ["c", "d"], 3: ["e"]}


def level_body(n):
    links = "".join("<a href='/lvl%d/%s'>%s</a>" % (n + 1, x, x)
                    for x in NEXT_LEVEL.get(n, []))
    return html("第%d层" % n, "<h1>LEVEL-%d</h1>%s" % (n, links))


class Handler(BaseHTTPRequestHandler):

    def reply(self, text, ctype="text/html; charset=utf-8", code=200, cookie=None):
        data = text.encode("utf-8")
        self.send_response(code)
        self.send_header("Content-Type", ctype)
        self.send_header("Content-Length", str(len(data)))
        if cookie:
            self.send_header("Set-Cookie", cookie)
        self.end_headers()
        self.wfile.write(data)

    def do_GET(self):
        import time
        path = self.path.split("?")[0]
        HITS[path] = HITS.get(path, 0) + 1
        if path == "/robots.txt":
            return self.reply("User-agent: *\nDisallow: /private/\n", "text/plain")
        if path.startswith("/private"):
            return self.reply(html("私密", "<h1>私密区</h1><p>PRIVATE-SECRET-BODY</p>"
                                         "<p>这段说明文字让页面不会被结构性判据当成空壳。</p>"))
        if path == "/count":
            return self.reply(json.dumps({"count": HITS["/count"]}), "application/json")
        if path == "/ua":
            return self.reply(html("UA", "<p>UA=%s</p>" % self.headers.get("User-Agent")))
        if path == "/list":
            return self.reply(html("列表", LIST_BODY))
        if path == "/lazy":
            return self.reply(html("懒加载", LAZY_BODY))
        if path == "/spa":
            return self.reply(html("SPA", SPA_BODY))
        if path == "/table":
            return self.reply(html("表格", TABLE_BODY))
        if path == "/cookie":
            return self.reply(html("Cookie", COOKIE_BODY))
        if path == "/setcookie":
            return self.reply(html("发cookie", COOKIE_BODY), cookie="probe_c4a=1; Path=/")
        if path == "/fp":
            return self.reply(html("指纹", FP_BODY))
        match = re.match(r"/lvl(\d)/([a-z])", path)
        if match:
            n = int(match.group(1))
            if n > 3:
                return self.reply(html("叶子", "NO-MORE-LEVEL"), code=404)
            return self.reply(level_body(n))
        match = re.match(r"/d/(\d+)", path)
        if match:
            i = int(match.group(1))
            return self.reply(html("片名%d" % i,
                                   "<h1 class='t'>片名%d</h1><span class='p'>%.1f</span>"
                                   % (i, i * 1.5)))
        if path == "/status500":
            return self.reply("boom", "text/plain", 500)
        if path.startswith("/slow"):
            time.sleep(2.5)
            return self.reply(html("慢", "SLOW-DONE"))
        return self.reply(html("首页", "<a href='/list'>去列表</a>"))

    def do_POST(self):
        size = int(self.headers.get("Content-Length", 0))
        body = self.rfile.read(size).decode("utf-8")
        HITS["/echo"] = HITS.get("/echo", 0) + 1
        return self.reply(html("回显", "<pre>POSTED %s</pre>" % body))

    def log_message(self, *args):
        pass


def serve():
    server = HTTPServer(("127.0.0.1", PORT), Handler)
    # 客户端超时断开时 socketserver 会往 stderr 打整段 traceback（里面是临时端口号），
    # 那不是结论只是噪声，接住。
    server.handle_error = lambda request, client_address: None
    threading.Thread(target=server.serve_forever, daemon=True).start()
    return server


srv = serve()


def quiet():
    """静音 crawl4ai 的计时行：[INIT] 那行归 BrowserConfig.verbose 管，
    [FETCH]/[SCRAPE]/[COMPLETE] 三行归 CrawlerRunConfig.verbose 管，两个都要关，
    输出里才不会留下每次不同的耗时。"""
    return BrowserConfig(verbose=False), CrawlerRunConfig(verbose=False)



async def main():
    bc, _ = quiet()
    async with AsyncWebCrawler(config=bc) as cr:
        r = await cr.arun(BASE + "/list", config=CrawlerRunConfig(
            verbose=False, js_code="document.title = '被JS改过';"))
        print("js_code 注入之后 metadata.title =", r.metadata["title"])
        r2 = await cr.arun(BASE + "/list", config=CrawlerRunConfig(verbose=False))
        print("不注入时 metadata.title         =", r2.metadata["title"])

    async with AsyncWebCrawler(config=bc) as cr:
        for label, cfg in [
                ("method=POST + data=bytes", dict(method="POST", data=b"name=probe")),
                ("method=POST + data=dict", dict(method="POST", data={"name": "probe"})),
                ("method=POST + form=dict", dict(method="POST", form={"name": "probe"})),
                ("老写法 post_url=…", dict(post_url=BASE + "/echo", form_data={"a": "1"}))]:
            try:
                res = await cr.arun(BASE + "/echo", config=CrawlerRunConfig(verbose=False, **cfg))
                hits = [l for l in (res.markdown or "").splitlines() if "POSTED" in l]
                print("%-26s success=%-5s 状态 %-5s 回显 %r error=%r"
                      % (label, res.success, res.status_code, hits[:1], (res.error_message or "")[:36]))
            except Exception as exc:
                print("%-26s 抛 %s: %s" % (label, type(exc).__name__, str(exc)[:70]))

    from crawl4ai import HTTPCrawlerConfig
    from crawl4ai.async_crawler_strategy import AsyncHTTPCrawlerStrategy
    for label, kw in [("HTTP 策略 method=POST data=bytes", dict(data=b"name=probe")),
                      ("HTTP 策略 method=POST data=dict", dict(data={"name": "probe"})),
                      ("HTTP 策略 json=", dict(json={"name": "probe"}))]:
        strategy = AsyncHTTPCrawlerStrategy(browser_config=HTTPCrawlerConfig(method="POST", **kw))
        async with AsyncWebCrawler(config=bc, crawler_strategy=strategy) as cr:
            res = await cr.arun(BASE + "/echo", config=CrawlerRunConfig(verbose=False))
            hits = [l for l in (res.markdown or "").splitlines() if "POSTED" in l]
            print("%-30s success=%-5s 回显 %s" % (label, res.success, hits[:1]))

asyncio.run(main())
1
2
3
4
5
6
7
8
9
js_code 注入之后 metadata.title = 被JS改过
不注入时 metadata.title         = 列表
method=POST + data=bytes   抛 TypeError: CrawlerRunConfig.__init__() got an unexpected keyword argument 'data'
method=POST + data=dict    抛 TypeError: CrawlerRunConfig.__init__() got an unexpected keyword argument 'data'
method=POST + form=dict    抛 TypeError: CrawlerRunConfig.__init__() got an unexpected keyword argument 'form'
老写法 post_url=…             抛 TypeError: CrawlerRunConfig.__init__() got an unexpected keyword argument 'post_u
HTTP 策略 method=POST data=bytes success=True  回显 ['POSTED name=probe']
HTTP 策略 method=POST data=dict  success=True  回显 ['POSTED name=probe']
HTTP 策略 json=                  success=True  回显 ['POSTED {"name": "probe"}']
  • js_code="document.title = '被JS改过';" 的效果直接体现在 metadata.title 上（被JS改过 vs 不注入时的 列表）——这是验证「JS 到底跑没跑」最省事的手法，比读 html 靠谱。
  • 表单/POST 这一层的实测结论比较硬：CrawlerRunConfig 里没有 data、form、post_url、form_data 任何一个，传进去全是 TypeError: ... got an unexpected keyword argument ...。文档里那种 post_url + form_data 的写法在 0.9.5 已经退役。
  • 真正能发 POST 的是上一节那条轻量 HTTP 策略：HTTPCrawlerConfig(method="POST", data=...)。三种体写法都跑通了：data=b"name=probe" 与 data={"name": "probe"} 靶站都回显 POSTED name=probe（表单编码），json={"name": "probe"} 回显 POSTED {"name": "probe"}（JSON 原文）。
  • 浏览器那条路要 POST，实测的可行做法是把逻辑写进 js_code（在页面里 fetch）或者干脆用 API 接口——后者见《爬虫-Requests》《爬虫-多线程爬虫》的接口线。

robots 默认不遵守

这一点必须单独一节，因为它是 crawl4ai 与 Scrapy/Crawlee 的口径差别所在。靶站的 robots.txt 写着 Disallow: /private/，而 /private/ 里有 PRIVATE-SECRET-BODY 这串指纹：

robots_default.py
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
# -*- coding: utf-8 -*-
"""robots.txt：默认完全不遵守，开了 check_robots_txt 才拒绝。"""

import asyncio
import inspect
import json
import os
import re
import threading
from http.server import BaseHTTPRequestHandler, HTTPServer

# 云广告一天只打一次，打不打取决于家目录里的标记文件，会把输出变得不可复现，所以先静音
os.environ.setdefault("CRAWL4AI_NO_CLOUD_NOTICE", "1")

from crawl4ai import (AsyncWebCrawler, BrowserConfig, CacheMode, CrawlerRunConfig)
from crawl4ai.async_crawler_strategy import AsyncPlaywrightCrawlerStrategy


PORT = 8063
BASE = "http://127.0.0.1:8063"
HITS = {}


def html(title, body):
    return ("<!doctype html><html><head><meta charset='utf-8'><title>%s</title>"
            "<meta name='description' content='probe'><style>.nav{color:red}</style>"
            "</head><body>"
            "<div class='nav'>NAV 这是靶站的导航区，故意写长一点，"
            "免得页面被 crawl4ai 的结构性判据当成空壳</div>"
            "%s"
            "<div class='filler'>FILLER 这是靶站的页脚区，同样是有实际文字的标记</div>"
            "</body></html>" % (title, body))


LIST_BODY = ("<h1>列表页</h1><ul class='items'>"
             + "".join("<li><a href='/d/%d'>片名%d</a><span class='price'>%.1f</span></li>"
                       % (i, i, i * 1.5) for i in range(1, 6))
             + "</ul><a class='next' href='/list?page=2'>下一页</a>")

LAZY_BODY = ("<h1>懒加载列表</h1><div id='list'></div>"
             "<div style='height:4000px'></div>"
             "<script>window.addEventListener('scroll',function(){"
             "if(window.__on)return;window.__on=1;"
             "for(var i=0;i<20;i++){var d=document.createElement('div');"
             "d.className='item';d.textContent='滚动条目'+i;"
             "document.getElementById('list').appendChild(d);}});</script>")

SPA_BODY = ("<h1>前端渲染的列表</h1><div id='app'></div><script>setTimeout(function(){var s='';"
            "for(var i=0;i<8;i++){s+='<div class=row>渲染行'+i+'</div>'}"
            "document.getElementById('app').innerHTML=s;},200);</script>")

TABLE_BODY = ("<table class='movies'><tr><th>片名</th><th>年份</th></tr>"
              + "".join("<tr><td>片名%d</td><td>%d</td></tr>" % (i, 1990 + i)
                        for i in range(1, 4)) + "</table>")

COOKIE_BODY = ("<p id='c'></p><script>document.getElementById('c').textContent="
               "'COOKIE='+(document.cookie||'NONE')</script>")

FP_BODY = ("<p id='f'></p><script>document.getElementById('f').textContent="
           "'WD='+navigator.webdriver+'|PL='+navigator.plugins.length+"
           "'|V='+window.innerWidth+'x'+window.innerHeight+'|UA='+navigator.userAgent.slice(0,26)"
           "</script>")

NEXT_LEVEL = {1: ["a", "b"], 2: ["c", "d"], 3: ["e"]}


def level_body(n):
    links = "".join("<a href='/lvl%d/%s'>%s</a>" % (n + 1, x, x)
                    for x in NEXT_LEVEL.get(n, []))
    return html("第%d层" % n, "<h1>LEVEL-%d</h1>%s" % (n, links))


class Handler(BaseHTTPRequestHandler):

    def reply(self, text, ctype="text/html; charset=utf-8", code=200, cookie=None):
        data = text.encode("utf-8")
        self.send_response(code)
        self.send_header("Content-Type", ctype)
        self.send_header("Content-Length", str(len(data)))
        if cookie:
            self.send_header("Set-Cookie", cookie)
        self.end_headers()
        self.wfile.write(data)

    def do_GET(self):
        import time
        path = self.path.split("?")[0]
        HITS[path] = HITS.get(path, 0) + 1
        if path == "/robots.txt":
            return self.reply("User-agent: *\nDisallow: /private/\n", "text/plain")
        if path.startswith("/private"):
            return self.reply(html("私密", "<h1>私密区</h1><p>PRIVATE-SECRET-BODY</p>"
                                         "<p>这段说明文字让页面不会被结构性判据当成空壳。</p>"))
        if path == "/count":
            return self.reply(json.dumps({"count": HITS["/count"]}), "application/json")
        if path == "/ua":
            return self.reply(html("UA", "<p>UA=%s</p>" % self.headers.get("User-Agent")))
        if path == "/list":
            return self.reply(html("列表", LIST_BODY))
        if path == "/lazy":
            return self.reply(html("懒加载", LAZY_BODY))
        if path == "/spa":
            return self.reply(html("SPA", SPA_BODY))
        if path == "/table":
            return self.reply(html("表格", TABLE_BODY))
        if path == "/cookie":
            return self.reply(html("Cookie", COOKIE_BODY))
        if path == "/setcookie":
            return self.reply(html("发cookie", COOKIE_BODY), cookie="probe_c4a=1; Path=/")
        if path == "/fp":
            return self.reply(html("指纹", FP_BODY))
        match = re.match(r"/lvl(\d)/([a-z])", path)
        if match:
            n = int(match.group(1))
            if n > 3:
                return self.reply(html("叶子", "NO-MORE-LEVEL"), code=404)
            return self.reply(level_body(n))
        match = re.match(r"/d/(\d+)", path)
        if match:
            i = int(match.group(1))
            return self.reply(html("片名%d" % i,
                                   "<h1 class='t'>片名%d</h1><span class='p'>%.1f</span>"
                                   % (i, i * 1.5)))
        if path == "/status500":
            return self.reply("boom", "text/plain", 500)
        if path.startswith("/slow"):
            time.sleep(2.5)
            return self.reply(html("慢", "SLOW-DONE"))
        return self.reply(html("首页", "<a href='/list'>去列表</a>"))

    def do_POST(self):
        size = int(self.headers.get("Content-Length", 0))
        body = self.rfile.read(size).decode("utf-8")
        HITS["/echo"] = HITS.get("/echo", 0) + 1
        return self.reply(html("回显", "<pre>POSTED %s</pre>" % body))

    def log_message(self, *args):
        pass


def serve():
    server = HTTPServer(("127.0.0.1", PORT), Handler)
    # 客户端超时断开时 socketserver 会往 stderr 打整段 traceback（里面是临时端口号），
    # 那不是结论只是噪声，接住。
    server.handle_error = lambda request, client_address: None
    threading.Thread(target=server.serve_forever, daemon=True).start()
    return server


srv = serve()


def quiet():
    """静音 crawl4ai 的计时行：[INIT] 那行归 BrowserConfig.verbose 管，
    [FETCH]/[SCRAPE]/[COMPLETE] 三行归 CrawlerRunConfig.verbose 管，两个都要关，
    输出里才不会留下每次不同的耗时。"""
    return BrowserConfig(verbose=False), CrawlerRunConfig(verbose=False)



async def grab(label, **kwargs):
    bc, _ = quiet()
    async with AsyncWebCrawler(config=bc) as cr:
        r = await cr.arun(BASE + "/private/", config=CrawlerRunConfig(verbose=False, **kwargs))
        print("%-26s success=%-5s 状态 %-5s error=%r 抓到 PRIVATE-SECRET-BODY=%s"
              % (label, r.success, r.status_code, r.error_message[:40],
                 "PRIVATE-SECRET-BODY" in (r.html or "")))


async def main():
    import urllib.request
    print("靶站自报的 robots.txt =", repr(urllib.request.urlopen(BASE + "/robots.txt", timeout=5)
                                          .read().decode()))
    await grab("默认（不查 robots）")
    await grab("check_robots_txt=True", check_robots_txt=True)

asyncio.run(main())
1
2
3
靶站自报的 robots.txt = 'User-agent: *\nDisallow: /private/\n'
默认（不查 robots）              success=True  状态 200   error='' 抓到 PRIVATE-SECRET-BODY=True
check_robots_txt=True      success=False 状态 403   error='Access denied by robots.txt' 抓到 PRIVATE-SECRET-BODY=False
  • 默认配置照样把 /private/ 抓回来了（success=True、抓到 PRIVATE-SECRET-BODY=True），check_robots_txt 的默认值就是 False（参数面那节已验证）。
  • 显式 check_robots_txt=True 之后是 success=False、status_code=403、error_message='Access denied by robots.txt'，而且请求根本没到 /private/（靶站那一侧的命中数在批量块里可以复算）。
  • 对照口径：Scrapy 是 ROBOTSTXT_OBEY（默认也是 False，但生态里常开），Crawlee 是 robots.txt 参与调度。crawl4ai 的差别在于它把这件事做成了一次「抓取前置检查」，命中就直接给 403 而不是跳过该链接，所以判 error_message 就能区分「被 robots 挡」和「被反爬判据误伤」。
  • 一句风险提示（这条不在实测范围内，但必须写在笔记里）：默认不遵守不等于可以随便抓，robots.txt 之外还有站点条款与频率问题，《爬虫-基本入门》那篇「法律与道德边界」一节讲过边界。

失败长什么样

四类失败各打一次，输出把 crawl4ai 的报错风格说清楚了：大多数失败不抛异常，而是把 success=False 和一句内部错误文本还给你。

timeout_and_errors.py
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
# -*- coding: utf-8 -*-
"""四类失败各长什么样：超时、500、协议名写错、端口没人听。"""

import asyncio
import inspect
import json
import os
import re
import threading
from http.server import BaseHTTPRequestHandler, HTTPServer

# 云广告一天只打一次，打不打取决于家目录里的标记文件，会把输出变得不可复现，所以先静音
os.environ.setdefault("CRAWL4AI_NO_CLOUD_NOTICE", "1")

from crawl4ai import (AsyncWebCrawler, BrowserConfig, CacheMode, CrawlerRunConfig)
from crawl4ai.async_crawler_strategy import AsyncPlaywrightCrawlerStrategy


PORT = 8066
BASE = "http://127.0.0.1:8066"
HITS = {}


def html(title, body):
    return ("<!doctype html><html><head><meta charset='utf-8'><title>%s</title>"
            "<meta name='description' content='probe'><style>.nav{color:red}</style>"
            "</head><body>"
            "<div class='nav'>NAV 这是靶站的导航区，故意写长一点，"
            "免得页面被 crawl4ai 的结构性判据当成空壳</div>"
            "%s"
            "<div class='filler'>FILLER 这是靶站的页脚区，同样是有实际文字的标记</div>"
            "</body></html>" % (title, body))


LIST_BODY = ("<h1>列表页</h1><ul class='items'>"
             + "".join("<li><a href='/d/%d'>片名%d</a><span class='price'>%.1f</span></li>"
                       % (i, i, i * 1.5) for i in range(1, 6))
             + "</ul><a class='next' href='/list?page=2'>下一页</a>")

LAZY_BODY = ("<h1>懒加载列表</h1><div id='list'></div>"
             "<div style='height:4000px'></div>"
             "<script>window.addEventListener('scroll',function(){"
             "if(window.__on)return;window.__on=1;"
             "for(var i=0;i<20;i++){var d=document.createElement('div');"
             "d.className='item';d.textContent='滚动条目'+i;"
             "document.getElementById('list').appendChild(d);}});</script>")

SPA_BODY = ("<h1>前端渲染的列表</h1><div id='app'></div><script>setTimeout(function(){var s='';"
            "for(var i=0;i<8;i++){s+='<div class=row>渲染行'+i+'</div>'}"
            "document.getElementById('app').innerHTML=s;},200);</script>")

TABLE_BODY = ("<table class='movies'><tr><th>片名</th><th>年份</th></tr>"
              + "".join("<tr><td>片名%d</td><td>%d</td></tr>" % (i, 1990 + i)
                        for i in range(1, 4)) + "</table>")

COOKIE_BODY = ("<p id='c'></p><script>document.getElementById('c').textContent="
               "'COOKIE='+(document.cookie||'NONE')</script>")

FP_BODY = ("<p id='f'></p><script>document.getElementById('f').textContent="
           "'WD='+navigator.webdriver+'|PL='+navigator.plugins.length+"
           "'|V='+window.innerWidth+'x'+window.innerHeight+'|UA='+navigator.userAgent.slice(0,26)"
           "</script>")

NEXT_LEVEL = {1: ["a", "b"], 2: ["c", "d"], 3: ["e"]}


def level_body(n):
    links = "".join("<a href='/lvl%d/%s'>%s</a>" % (n + 1, x, x)
                    for x in NEXT_LEVEL.get(n, []))
    return html("第%d层" % n, "<h1>LEVEL-%d</h1>%s" % (n, links))


class Handler(BaseHTTPRequestHandler):

    def reply(self, text, ctype="text/html; charset=utf-8", code=200, cookie=None):
        data = text.encode("utf-8")
        self.send_response(code)
        self.send_header("Content-Type", ctype)
        self.send_header("Content-Length", str(len(data)))
        if cookie:
            self.send_header("Set-Cookie", cookie)
        self.end_headers()
        self.wfile.write(data)

    def do_GET(self):
        import time
        path = self.path.split("?")[0]
        HITS[path] = HITS.get(path, 0) + 1
        if path == "/robots.txt":
            return self.reply("User-agent: *\nDisallow: /private/\n", "text/plain")
        if path.startswith("/private"):
            return self.reply(html("私密", "<h1>私密区</h1><p>PRIVATE-SECRET-BODY</p>"
                                         "<p>这段说明文字让页面不会被结构性判据当成空壳。</p>"))
        if path == "/count":
            return self.reply(json.dumps({"count": HITS["/count"]}), "application/json")
        if path == "/ua":
            return self.reply(html("UA", "<p>UA=%s</p>" % self.headers.get("User-Agent")))
        if path == "/list":
            return self.reply(html("列表", LIST_BODY))
        if path == "/lazy":
            return self.reply(html("懒加载", LAZY_BODY))
        if path == "/spa":
            return self.reply(html("SPA", SPA_BODY))
        if path == "/table":
            return self.reply(html("表格", TABLE_BODY))
        if path == "/cookie":
            return self.reply(html("Cookie", COOKIE_BODY))
        if path == "/setcookie":
            return self.reply(html("发cookie", COOKIE_BODY), cookie="probe_c4a=1; Path=/")
        if path == "/fp":
            return self.reply(html("指纹", FP_BODY))
        match = re.match(r"/lvl(\d)/([a-z])", path)
        if match:
            n = int(match.group(1))
            if n > 3:
                return self.reply(html("叶子", "NO-MORE-LEVEL"), code=404)
            return self.reply(level_body(n))
        match = re.match(r"/d/(\d+)", path)
        if match:
            i = int(match.group(1))
            return self.reply(html("片名%d" % i,
                                   "<h1 class='t'>片名%d</h1><span class='p'>%.1f</span>"
                                   % (i, i * 1.5)))
        if path == "/status500":
            return self.reply("boom", "text/plain", 500)
        if path.startswith("/slow"):
            time.sleep(2.5)
            return self.reply(html("慢", "SLOW-DONE"))
        return self.reply(html("首页", "<a href='/list'>去列表</a>"))

    def do_POST(self):
        size = int(self.headers.get("Content-Length", 0))
        body = self.rfile.read(size).decode("utf-8")
        HITS["/echo"] = HITS.get("/echo", 0) + 1
        return self.reply(html("回显", "<pre>POSTED %s</pre>" % body))

    def log_message(self, *args):
        pass


def serve():
    server = HTTPServer(("127.0.0.1", PORT), Handler)
    # 客户端超时断开时 socketserver 会往 stderr 打整段 traceback（里面是临时端口号），
    # 那不是结论只是噪声，接住。
    server.handle_error = lambda request, client_address: None
    threading.Thread(target=server.serve_forever, daemon=True).start()
    return server


srv = serve()


def quiet():
    """静音 crawl4ai 的计时行：[INIT] 那行归 BrowserConfig.verbose 管，
    [FETCH]/[SCRAPE]/[COMPLETE] 三行归 CrawlerRunConfig.verbose 管，两个都要关，
    输出里才不会留下每次不同的耗时。"""
    return BrowserConfig(verbose=False), CrawlerRunConfig(verbose=False)



async def show(label, url, **kwargs):
    bc, _ = quiet()
    async with AsyncWebCrawler(config=bc) as cr:
        try:
            r = await cr.arun(url, config=CrawlerRunConfig(verbose=False, **kwargs))
            print("%-24s success=%-5s 状态 %-5s error=%r"
                  % (label, r.success, r.status_code, (r.error_message or "")[:56]))
        except Exception as exc:
            print("%-24s 抛 %s: %s" % (label, type(exc).__name__, str(exc)[:56]))


async def main():
    await show("page_timeout=300", BASE + "/slow", page_timeout=300)
    await show("page_timeout=60000", BASE + "/slow")
    await show("500 页面", BASE + "/status500")
    await show("协议名写错", "htp://127.0.0.1:%d/list" % PORT)
    await show("端口没人听", "http://127.0.0.1:9/list")
    await show("空字符串 url", "")

asyncio.run(main())
1
2
3
4
5
6
page_timeout=300         success=False 状态 None  error='Unexpected error in _crawl_web at line 779 in _crawl_web'
page_timeout=60000       success=False 状态 200   error='Blocked by anti-bot protection: Structural: no_content_e'
500 页面                   success=False 状态 500   error='Blocked by anti-bot protection: Structural: minimal_text'
协议名写错                    success=False 状态 None  error='Unexpected error in _crawl_web at line 511 in crawl (E:\\'
端口没人听                    success=False 状态 None  error='Unexpected error in _crawl_web at line 779 in _crawl_web'
空字符串 url                 抛 ValueError: Invalid URL, make sure the URL is a non-empty string
  • page_timeout=300 打一个要 2.5 秒的端点：success=False、status_code=None、错误是 Unexpected error in _crawl_web at line 779——行号是库自己的行号，随版本变，别拿它做判据；能用的判据是「状态为空 + 超时」。
  • 同一端点用默认 page_timeout=60000：抓到了（状态 200），但 success=False，原因是「没被拦却报被拦」那节的结构性判据（这个端点的页面只有 300 字节）。这就是把两类失败混在一个字段里的代价：判读时必须先看 status_code 有没有、再看 error_message 是不是以 Blocked by anti-bot protection 开头。
  • 500 页面：状态 500 + Blocked by anti-bot protection: Structural: minimal_text…，即服务端错误也走同一层判定，不会给你「HTTP 500」这种直白的话。想要直白的状态码，判 r.status_code == 500 而不是读 error_message。
  • 协议名写错（htp://…）与端口没人听：都是 success=False、status_code=None，错误文本里带库内部行号或 crawl (E:\… 的路径片段。这两种情况不抛 Python 异常，只有空字符串 URL 才抛 ValueError: Invalid URL, make sure the URL is a non-empty string——也就是说 try/except 只能兜住空串这一种。
  • 顺带把「为什么这一节的输出只有 6 行」交代掉：靶站的 handle_error 被关掉了，客户端超时断开时 socketserver 会打整段 traceback（里面是临时端口号），那不是结论。

批量与调度

arun_many 是批量入口，调度器（dispatcher）是它的第三个参数，不是 CrawlerRunConfig 的参数——这一点最容易记错：

arun_many_dispatch.py
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
# -*- coding: utf-8 -*-
"""arun_many 与三种调度器：并发跑 20 个 URL，看结果顺序与失败处理。"""

import asyncio
import inspect
import json
import os
import re
import threading
from http.server import BaseHTTPRequestHandler, HTTPServer

# 云广告一天只打一次，打不打取决于家目录里的标记文件，会把输出变得不可复现，所以先静音
os.environ.setdefault("CRAWL4AI_NO_CLOUD_NOTICE", "1")

from crawl4ai import (AsyncWebCrawler, BrowserConfig, CacheMode, CrawlerRunConfig)
from crawl4ai.async_crawler_strategy import AsyncPlaywrightCrawlerStrategy


PORT = 8068
BASE = "http://127.0.0.1:8068"
HITS = {}


def html(title, body):
    return ("<!doctype html><html><head><meta charset='utf-8'><title>%s</title>"
            "<meta name='description' content='probe'><style>.nav{color:red}</style>"
            "</head><body>"
            "<div class='nav'>NAV 这是靶站的导航区，故意写长一点，"
            "免得页面被 crawl4ai 的结构性判据当成空壳</div>"
            "%s"
            "<div class='filler'>FILLER 这是靶站的页脚区，同样是有实际文字的标记</div>"
            "</body></html>" % (title, body))


LIST_BODY = ("<h1>列表页</h1><ul class='items'>"
             + "".join("<li><a href='/d/%d'>片名%d</a><span class='price'>%.1f</span></li>"
                       % (i, i, i * 1.5) for i in range(1, 6))
             + "</ul><a class='next' href='/list?page=2'>下一页</a>")

LAZY_BODY = ("<h1>懒加载列表</h1><div id='list'></div>"
             "<div style='height:4000px'></div>"
             "<script>window.addEventListener('scroll',function(){"
             "if(window.__on)return;window.__on=1;"
             "for(var i=0;i<20;i++){var d=document.createElement('div');"
             "d.className='item';d.textContent='滚动条目'+i;"
             "document.getElementById('list').appendChild(d);}});</script>")

SPA_BODY = ("<h1>前端渲染的列表</h1><div id='app'></div><script>setTimeout(function(){var s='';"
            "for(var i=0;i<8;i++){s+='<div class=row>渲染行'+i+'</div>'}"
            "document.getElementById('app').innerHTML=s;},200);</script>")

TABLE_BODY = ("<table class='movies'><tr><th>片名</th><th>年份</th></tr>"
              + "".join("<tr><td>片名%d</td><td>%d</td></tr>" % (i, 1990 + i)
                        for i in range(1, 4)) + "</table>")

COOKIE_BODY = ("<p id='c'></p><script>document.getElementById('c').textContent="
               "'COOKIE='+(document.cookie||'NONE')</script>")

FP_BODY = ("<p id='f'></p><script>document.getElementById('f').textContent="
           "'WD='+navigator.webdriver+'|PL='+navigator.plugins.length+"
           "'|V='+window.innerWidth+'x'+window.innerHeight+'|UA='+navigator.userAgent.slice(0,26)"
           "</script>")

NEXT_LEVEL = {1: ["a", "b"], 2: ["c", "d"], 3: ["e"]}


def level_body(n):
    links = "".join("<a href='/lvl%d/%s'>%s</a>" % (n + 1, x, x)
                    for x in NEXT_LEVEL.get(n, []))
    return html("第%d层" % n, "<h1>LEVEL-%d</h1>%s" % (n, links))


class Handler(BaseHTTPRequestHandler):

    def reply(self, text, ctype="text/html; charset=utf-8", code=200, cookie=None):
        data = text.encode("utf-8")
        self.send_response(code)
        self.send_header("Content-Type", ctype)
        self.send_header("Content-Length", str(len(data)))
        if cookie:
            self.send_header("Set-Cookie", cookie)
        self.end_headers()
        self.wfile.write(data)

    def do_GET(self):
        import time
        path = self.path.split("?")[0]
        HITS[path] = HITS.get(path, 0) + 1
        if path == "/robots.txt":
            return self.reply("User-agent: *\nDisallow: /private/\n", "text/plain")
        if path.startswith("/private"):
            return self.reply(html("私密", "<h1>私密区</h1><p>PRIVATE-SECRET-BODY</p>"
                                         "<p>这段说明文字让页面不会被结构性判据当成空壳。</p>"))
        if path == "/count":
            return self.reply(json.dumps({"count": HITS["/count"]}), "application/json")
        if path == "/ua":
            return self.reply(html("UA", "<p>UA=%s</p>" % self.headers.get("User-Agent")))
        if path == "/list":
            return self.reply(html("列表", LIST_BODY))
        if path == "/lazy":
            return self.reply(html("懒加载", LAZY_BODY))
        if path == "/spa":
            return self.reply(html("SPA", SPA_BODY))
        if path == "/table":
            return self.reply(html("表格", TABLE_BODY))
        if path == "/cookie":
            return self.reply(html("Cookie", COOKIE_BODY))
        if path == "/setcookie":
            return self.reply(html("发cookie", COOKIE_BODY), cookie="probe_c4a=1; Path=/")
        if path == "/fp":
            return self.reply(html("指纹", FP_BODY))
        match = re.match(r"/lvl(\d)/([a-z])", path)
        if match:
            n = int(match.group(1))
            if n > 3:
                return self.reply(html("叶子", "NO-MORE-LEVEL"), code=404)
            return self.reply(level_body(n))
        match = re.match(r"/d/(\d+)", path)
        if match:
            i = int(match.group(1))
            return self.reply(html("片名%d" % i,
                                   "<h1 class='t'>片名%d</h1><span class='p'>%.1f</span>"
                                   % (i, i * 1.5)))
        if path == "/status500":
            return self.reply("boom", "text/plain", 500)
        if path.startswith("/slow"):
            time.sleep(2.5)
            return self.reply(html("慢", "SLOW-DONE"))
        return self.reply(html("首页", "<a href='/list'>去列表</a>"))

    def do_POST(self):
        size = int(self.headers.get("Content-Length", 0))
        body = self.rfile.read(size).decode("utf-8")
        HITS["/echo"] = HITS.get("/echo", 0) + 1
        return self.reply(html("回显", "<pre>POSTED %s</pre>" % body))

    def log_message(self, *args):
        pass


def serve():
    server = HTTPServer(("127.0.0.1", PORT), Handler)
    # 客户端超时断开时 socketserver 会往 stderr 打整段 traceback（里面是临时端口号），
    # 那不是结论只是噪声，接住。
    server.handle_error = lambda request, client_address: None
    threading.Thread(target=server.serve_forever, daemon=True).start()
    return server


srv = serve()


def quiet():
    """静音 crawl4ai 的计时行：[INIT] 那行归 BrowserConfig.verbose 管，
    [FETCH]/[SCRAPE]/[COMPLETE] 三行归 CrawlerRunConfig.verbose 管，两个都要关，
    输出里才不会留下每次不同的耗时。"""
    return BrowserConfig(verbose=False), CrawlerRunConfig(verbose=False)



import inspect

import crawl4ai

urls = [BASE + "/d/%d" % i for i in range(1, 21)]


async def main():
    bc, _ = quiet()
    async with AsyncWebCrawler(config=bc) as cr:
        rs = await cr.arun_many(urls, config=CrawlerRunConfig(verbose=False))
        print("默认调度：返回 %d 条，成功 %d 条，覆盖 %d 个 URL"
              % (len(rs), sum(1 for x in rs if x.success), len({x.url for x in rs})))
        print("   服务端命中过的 /d/ 路径数 =", len([k for k in HITS if k.startswith("/d/")]))

    print("顶层有没有 DispatcherType =", hasattr(crawl4ai, "DispatcherType"))
    import crawl4ai.async_dispatcher as D
    print("async_dispatcher 里的调度器 =", [n for n in dir(D) if n.endswith("Dispatcher")])
    print("SemaphoreDispatcher 的参数 =",
          list(inspect.signature(D.SemaphoreDispatcher.__init__).parameters)[1:])
    print("MemoryAdaptiveDispatcher 的参数 =",
          list(inspect.signature(D.MemoryAdaptiveDispatcher.__init__).parameters)[1:6])
    for label, disp in [("SemaphoreDispatcher(2)", D.SemaphoreDispatcher(semaphore_count=2)),
                        ("MemoryAdaptiveDispatcher()", D.MemoryAdaptiveDispatcher())]:
        try:
            async with AsyncWebCrawler(config=bc) as cr:
                rs = await cr.arun_many(urls, config=CrawlerRunConfig(verbose=False),
                                        dispatcher=disp)
            print("%-24s 返回 %d 条，成功 %d 条" % (label, len(rs), sum(1 for x in rs if x.success)))
        except Exception as exc:
            print("%-24s 抛 %s: %s" % (label, type(exc).__name__, str(exc)[:70]))
    print("dispatcher 是 CrawlerRunConfig 的参数吗 =",
          "dispatcher" in list(inspect.signature(CrawlerRunConfig.__init__).parameters))
    print("dispatcher 是 arun_many 的参数吗 =",
          "dispatcher" in list(inspect.signature(AsyncWebCrawler.arun_many).parameters))

    rc, _ = quiet()
    async with AsyncWebCrawler(config=bc) as cr:
        rs = await cr.arun_many(urls[:3] + ["http://127.0.0.1:9/none"],
                                config=CrawlerRunConfig(verbose=False))
    print("混一个没人听的端口：", [(x.url.split("/")[-2], x.success) for x in rs])

asyncio.run(main())
1
2
3
4
5
6
7
8
9
10
11
默认调度：返回 20 条，成功 20 条，覆盖 20 个 URL
   服务端命中过的 /d/ 路径数 = 20
顶层有没有 DispatcherType = False
async_dispatcher 里的调度器 = ['BaseDispatcher', 'MemoryAdaptiveDispatcher', 'SemaphoreDispatcher']
SemaphoreDispatcher 的参数 = ['semaphore_count', 'max_session_permit', 'rate_limiter', 'monitor']
MemoryAdaptiveDispatcher 的参数 = ['memory_threshold_percent', 'critical_threshold_percent', 'recovery_threshold_percent', 'check_interval', 'max_session_permit']
SemaphoreDispatcher(2)   返回 20 条，成功 20 条
MemoryAdaptiveDispatcher() 返回 20 条，成功 20 条
dispatcher 是 CrawlerRunConfig 的参数吗 = False
dispatcher 是 arun_many 的参数吗 = True
混一个没人听的端口： [('127.0.0.1:9', False), ('d', True), ('d', True), ('d', True)]
  • 20 个 URL 默认调度：返回 20 条、成功 20 条、覆盖 20 个 URL，靶站的 /d/ 路径正好被命中 20 次，没有丢也没有重复。
  • 顶层没有 DispatcherType 枚举（文档里那种 dispatcher=DispatcherType.MEMORY_ADAPTIVE 的写法本机不存在），可用调度器只有 BaseDispatcher / MemoryAdaptiveDispatcher / SemaphoreDispatcher 三个。构造参数实测分别是：SemaphoreDispatcher(semaphore_count, max_session_permit, rate_limiter, monitor)、MemoryAdaptiveDispatcher(memory_threshold_percent, critical_threshold_percent, recovery_threshold_percent, check_interval, max_session_permit)。传实例给 arun_many(urls, config=..., dispatcher=...)，两种都能跑（各返回 20 条、成功 20 条）。
  • 最后那行是判读顺序的示范：一批里混一个没人听的端口，返回结果里那条是 False、其余三条 True——arun_many 不会因为个别 URL 失败而整体抛错，必须逐条看 success。
  • 打印里刻意没有「返回顺序」这一项：实测发现 arun_many 的返回顺序与请求顺序不一致（本机出现过 1, 13, 12, 11, 10, 14 这种），而每次调度结果都可能不同，所以想要顺序只能自己按 result.url 重排，也不能把这个数字写进可复现的输出里。

深度抓取

深度抓取现在的写法是「传一个策略对象给 deep_crawl_strategy」。靶站是链式的四层：/lvl1/a → /lvl2/{a,b} → /lvl3/{c,d} → /lvl4/e：

deep_crawl.py
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
# -*- coding: utf-8 -*-
"""深度抓取：BFS/DFS 的 max_depth 与过滤器链，抓完能看到走完了哪几层。"""

import asyncio
import inspect
import json
import os
import re
import threading
from http.server import BaseHTTPRequestHandler, HTTPServer

# 云广告一天只打一次，打不打取决于家目录里的标记文件，会把输出变得不可复现，所以先静音
os.environ.setdefault("CRAWL4AI_NO_CLOUD_NOTICE", "1")

from crawl4ai import (AsyncWebCrawler, BrowserConfig, CacheMode, CrawlerRunConfig)
from crawl4ai.async_crawler_strategy import AsyncPlaywrightCrawlerStrategy


PORT = 8069
BASE = "http://127.0.0.1:8069"
HITS = {}


def html(title, body):
    return ("<!doctype html><html><head><meta charset='utf-8'><title>%s</title>"
            "<meta name='description' content='probe'><style>.nav{color:red}</style>"
            "</head><body>"
            "<div class='nav'>NAV 这是靶站的导航区，故意写长一点，"
            "免得页面被 crawl4ai 的结构性判据当成空壳</div>"
            "%s"
            "<div class='filler'>FILLER 这是靶站的页脚区，同样是有实际文字的标记</div>"
            "</body></html>" % (title, body))


LIST_BODY = ("<h1>列表页</h1><ul class='items'>"
             + "".join("<li><a href='/d/%d'>片名%d</a><span class='price'>%.1f</span></li>"
                       % (i, i, i * 1.5) for i in range(1, 6))
             + "</ul><a class='next' href='/list?page=2'>下一页</a>")

LAZY_BODY = ("<h1>懒加载列表</h1><div id='list'></div>"
             "<div style='height:4000px'></div>"
             "<script>window.addEventListener('scroll',function(){"
             "if(window.__on)return;window.__on=1;"
             "for(var i=0;i<20;i++){var d=document.createElement('div');"
             "d.className='item';d.textContent='滚动条目'+i;"
             "document.getElementById('list').appendChild(d);}});</script>")

SPA_BODY = ("<h1>前端渲染的列表</h1><div id='app'></div><script>setTimeout(function(){var s='';"
            "for(var i=0;i<8;i++){s+='<div class=row>渲染行'+i+'</div>'}"
            "document.getElementById('app').innerHTML=s;},200);</script>")

TABLE_BODY = ("<table class='movies'><tr><th>片名</th><th>年份</th></tr>"
              + "".join("<tr><td>片名%d</td><td>%d</td></tr>" % (i, 1990 + i)
                        for i in range(1, 4)) + "</table>")

COOKIE_BODY = ("<p id='c'></p><script>document.getElementById('c').textContent="
               "'COOKIE='+(document.cookie||'NONE')</script>")

FP_BODY = ("<p id='f'></p><script>document.getElementById('f').textContent="
           "'WD='+navigator.webdriver+'|PL='+navigator.plugins.length+"
           "'|V='+window.innerWidth+'x'+window.innerHeight+'|UA='+navigator.userAgent.slice(0,26)"
           "</script>")

NEXT_LEVEL = {1: ["a", "b"], 2: ["c", "d"], 3: ["e"]}


def level_body(n):
    links = "".join("<a href='/lvl%d/%s'>%s</a>" % (n + 1, x, x)
                    for x in NEXT_LEVEL.get(n, []))
    return html("第%d层" % n, "<h1>LEVEL-%d</h1>%s" % (n, links))


class Handler(BaseHTTPRequestHandler):

    def reply(self, text, ctype="text/html; charset=utf-8", code=200, cookie=None):
        data = text.encode("utf-8")
        self.send_response(code)
        self.send_header("Content-Type", ctype)
        self.send_header("Content-Length", str(len(data)))
        if cookie:
            self.send_header("Set-Cookie", cookie)
        self.end_headers()
        self.wfile.write(data)

    def do_GET(self):
        import time
        path = self.path.split("?")[0]
        HITS[path] = HITS.get(path, 0) + 1
        if path == "/robots.txt":
            return self.reply("User-agent: *\nDisallow: /private/\n", "text/plain")
        if path.startswith("/private"):
            return self.reply(html("私密", "<h1>私密区</h1><p>PRIVATE-SECRET-BODY</p>"
                                         "<p>这段说明文字让页面不会被结构性判据当成空壳。</p>"))
        if path == "/count":
            return self.reply(json.dumps({"count": HITS["/count"]}), "application/json")
        if path == "/ua":
            return self.reply(html("UA", "<p>UA=%s</p>" % self.headers.get("User-Agent")))
        if path == "/list":
            return self.reply(html("列表", LIST_BODY))
        if path == "/lazy":
            return self.reply(html("懒加载", LAZY_BODY))
        if path == "/spa":
            return self.reply(html("SPA", SPA_BODY))
        if path == "/table":
            return self.reply(html("表格", TABLE_BODY))
        if path == "/cookie":
            return self.reply(html("Cookie", COOKIE_BODY))
        if path == "/setcookie":
            return self.reply(html("发cookie", COOKIE_BODY), cookie="probe_c4a=1; Path=/")
        if path == "/fp":
            return self.reply(html("指纹", FP_BODY))
        match = re.match(r"/lvl(\d)/([a-z])", path)
        if match:
            n = int(match.group(1))
            if n > 3:
                return self.reply(html("叶子", "NO-MORE-LEVEL"), code=404)
            return self.reply(level_body(n))
        match = re.match(r"/d/(\d+)", path)
        if match:
            i = int(match.group(1))
            return self.reply(html("片名%d" % i,
                                   "<h1 class='t'>片名%d</h1><span class='p'>%.1f</span>"
                                   % (i, i * 1.5)))
        if path == "/status500":
            return self.reply("boom", "text/plain", 500)
        if path.startswith("/slow"):
            time.sleep(2.5)
            return self.reply(html("慢", "SLOW-DONE"))
        return self.reply(html("首页", "<a href='/list'>去列表</a>"))

    def do_POST(self):
        size = int(self.headers.get("Content-Length", 0))
        body = self.rfile.read(size).decode("utf-8")
        HITS["/echo"] = HITS.get("/echo", 0) + 1
        return self.reply(html("回显", "<pre>POSTED %s</pre>" % body))

    def log_message(self, *args):
        pass


def serve():
    server = HTTPServer(("127.0.0.1", PORT), Handler)
    # 客户端超时断开时 socketserver 会往 stderr 打整段 traceback（里面是临时端口号），
    # 那不是结论只是噪声，接住。
    server.handle_error = lambda request, client_address: None
    threading.Thread(target=server.serve_forever, daemon=True).start()
    return server


srv = serve()


def quiet():
    """静音 crawl4ai 的计时行：[INIT] 那行归 BrowserConfig.verbose 管，
    [FETCH]/[SCRAPE]/[COMPLETE] 三行归 CrawlerRunConfig.verbose 管，两个都要关，
    输出里才不会留下每次不同的耗时。"""
    return BrowserConfig(verbose=False), CrawlerRunConfig(verbose=False)



from crawl4ai import BFSDeepCrawlStrategy, DFSDeepCrawlStrategy

RC = CrawlerRunConfig


async def walk(label, strategy):
    bc, _ = quiet()
    async with AsyncWebCrawler(config=bc) as cr:
        rs = await cr.arun(BASE + "/lvl1/a", config=CrawlerRunConfig(
            verbose=False, deep_crawl_strategy=strategy))
        urls = sorted({x.url.replace(BASE, "") for x in rs})
        print("%-22s 抓到 %2d 条 -> %s" % (label, len(rs), urls[:9]))


async def main():
    await walk("BFS max_depth=1", BFSDeepCrawlStrategy(max_depth=1))
    await walk("BFS max_depth=2", BFSDeepCrawlStrategy(max_depth=2))
    await walk("DFS max_depth=2", DFSDeepCrawlStrategy(max_depth=2))
    await walk("BFS max_depth=3", BFSDeepCrawlStrategy(max_depth=3))
    import inspect
    import crawl4ai
    names = list(inspect.signature(CrawlerRunConfig.__init__).parameters)
    print("CrawlerRunConfig 里 deep 相关的参数 =", [x for x in names if "deep" in x])
    print("文档里那种 deep_walk_strategy 还在吗 =", "deep_walk_strategy" in names)
    print("顶层有没有 FilterChain =", hasattr(crawl4ai, "FilterChain"),
          "| DomainFilter =", hasattr(crawl4ai, "DomainFilter"),
          "| URLPatternFilter =", hasattr(crawl4ai, "URLPatternFilter"))
    try:
        await walk("BFS + URLPatternFilter", BFSDeepCrawlStrategy(
            max_depth=2, url_filter=crawl4ai.URLPatternFilter(patterns=["*/lvl2/*"])))
    except Exception as exc:
        print("带过滤器的写法抛 %s: %s" % (type(exc).__name__, str(exc)[:70]))
    print("URLPatternFilter 的参数 =",
          ", ".join(list(inspect.signature(crawl4ai.URLPatternFilter.__init__).parameters)[1:]))
    print("FilterChain 的参数 =",
          ", ".join(list(inspect.signature(crawl4ai.FilterChain.__init__).parameters)[1:]))
    try:
        await walk("BFS + FilterChain(只放 lvl2/a)", BFSDeepCrawlStrategy(
            max_depth=2, filter_chain=crawl4ai.FilterChain([
                crawl4ai.URLPatternFilter(patterns=["*/lvl2/a"])])))
    except Exception as exc:
        print("FilterChain 写法抛 %s: %s" % (type(exc).__name__, str(exc)[:70]))

asyncio.run(main())
1
2
3
4
5
6
7
8
9
10
11
BFS max_depth=1        抓到  3 条 -> ['/lvl1/a', '/lvl2/a', '/lvl2/b']
BFS max_depth=2        抓到  5 条 -> ['/lvl1/a', '/lvl2/a', '/lvl2/b', '/lvl3/c', '/lvl3/d']
DFS max_depth=2        抓到  5 条 -> ['/lvl1/a', '/lvl2/a', '/lvl2/b', '/lvl3/c', '/lvl3/d']
BFS max_depth=3        抓到  6 条 -> ['/lvl1/a', '/lvl2/a', '/lvl2/b', '/lvl3/c', '/lvl3/d', '/lvl4/e']
CrawlerRunConfig 里 deep 相关的参数 = ['deep_crawl_strategy']
文档里那种 deep_walk_strategy 还在吗 = False
顶层有没有 FilterChain = True | DomainFilter = True | URLPatternFilter = True
带过滤器的写法抛 TypeError: BFSDeepCrawlStrategy.__init__() got an unexpected keyword argument 'ur
URLPatternFilter 的参数 = patterns, use_glob, reverse
FilterChain 的参数 = filters
BFS + FilterChain(只放 lvl2/a) 抓到  2 条 -> ['/lvl1/a', '/lvl2/a']
  • max_depth 的语义被这组数字钉死了：BFS max_depth=1 抓到 3 条（起点 + 第二层两个），=2 抓到 5 条（再加第三层），=3 抓到 6 条（到第四层那个叶子）。max_depth 数的是「往下几层」，不含起点自身那一层。
  • DFS 与 BFS 在同样 max_depth=2 下抓到的集合完全相同（5 条），只是访问顺序不同；所以选型看的是内存与早停需求，不是覆盖率。
  • 参数名这条要划重点：CrawlerRunConfig 里 deep 相关的参数只有 deep_crawl_strategy，文档里那种 deep_walk_strategy 在本机不存在（传进去 TypeError: ... unexpected keyword argument 'deep_walk_strategy'）。
  • 过滤器链是能用的：FilterChain 与 URLPatternFilter/DomainFilter 都在顶层存在；但 URLPatternFilter 的参数是 patterns, use_glob, reverse（不是 url_pattern，写错直接 TypeError）。把 FilterChain([URLPatternFilter(patterns=["*/lvl2/a"])]) 传给 BFSDeepCrawlStrategy(filter_chain=...) 之后，深度 2 的结果从 5 条缩到 2 条（只放行了 /lvl2/a）——过滤器是在入队前生效的，被滤掉的链接根本不会被抓。
  • 老参数 deep_crawl、bypass_cache、disable_cache 都不再是策略参数（bypass_cache 现在会给一句 AttributeError: Setting 'bypass_cache' is deprecated，见缓存那节）。

结构化抽取四路

markdown 给人读，结构化数据要另走一条路。这一段把四种抽取方式打在同一个靶站上（/list 有 5 个条目、/table 有一张三行表格）：

extract_strategies.py
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
# -*- coding: utf-8 -*-
"""结构化抽取四路：JsonCss / JsonXPath / Regex / Table，全都跑在同一个靶站上。"""

import asyncio
import inspect
import json
import os
import re
import threading
from http.server import BaseHTTPRequestHandler, HTTPServer

# 云广告一天只打一次，打不打取决于家目录里的标记文件，会把输出变得不可复现，所以先静音
os.environ.setdefault("CRAWL4AI_NO_CLOUD_NOTICE", "1")

from crawl4ai import (AsyncWebCrawler, BrowserConfig, CacheMode, CrawlerRunConfig)
from crawl4ai.async_crawler_strategy import AsyncPlaywrightCrawlerStrategy


PORT = 8071
BASE = "http://127.0.0.1:8071"
HITS = {}


def html(title, body):
    return ("<!doctype html><html><head><meta charset='utf-8'><title>%s</title>"
            "<meta name='description' content='probe'><style>.nav{color:red}</style>"
            "</head><body>"
            "<div class='nav'>NAV 这是靶站的导航区，故意写长一点，"
            "免得页面被 crawl4ai 的结构性判据当成空壳</div>"
            "%s"
            "<div class='filler'>FILLER 这是靶站的页脚区，同样是有实际文字的标记</div>"
            "</body></html>" % (title, body))


LIST_BODY = ("<h1>列表页</h1><ul class='items'>"
             + "".join("<li><a href='/d/%d'>片名%d</a><span class='price'>%.1f</span></li>"
                       % (i, i, i * 1.5) for i in range(1, 6))
             + "</ul><a class='next' href='/list?page=2'>下一页</a>")

LAZY_BODY = ("<h1>懒加载列表</h1><div id='list'></div>"
             "<div style='height:4000px'></div>"
             "<script>window.addEventListener('scroll',function(){"
             "if(window.__on)return;window.__on=1;"
             "for(var i=0;i<20;i++){var d=document.createElement('div');"
             "d.className='item';d.textContent='滚动条目'+i;"
             "document.getElementById('list').appendChild(d);}});</script>")

SPA_BODY = ("<h1>前端渲染的列表</h1><div id='app'></div><script>setTimeout(function(){var s='';"
            "for(var i=0;i<8;i++){s+='<div class=row>渲染行'+i+'</div>'}"
            "document.getElementById('app').innerHTML=s;},200);</script>")

TABLE_BODY = ("<table class='movies'><tr><th>片名</th><th>年份</th></tr>"
              + "".join("<tr><td>片名%d</td><td>%d</td></tr>" % (i, 1990 + i)
                        for i in range(1, 4)) + "</table>")

COOKIE_BODY = ("<p id='c'></p><script>document.getElementById('c').textContent="
               "'COOKIE='+(document.cookie||'NONE')</script>")

FP_BODY = ("<p id='f'></p><script>document.getElementById('f').textContent="
           "'WD='+navigator.webdriver+'|PL='+navigator.plugins.length+"
           "'|V='+window.innerWidth+'x'+window.innerHeight+'|UA='+navigator.userAgent.slice(0,26)"
           "</script>")

NEXT_LEVEL = {1: ["a", "b"], 2: ["c", "d"], 3: ["e"]}


def level_body(n):
    links = "".join("<a href='/lvl%d/%s'>%s</a>" % (n + 1, x, x)
                    for x in NEXT_LEVEL.get(n, []))
    return html("第%d层" % n, "<h1>LEVEL-%d</h1>%s" % (n, links))


class Handler(BaseHTTPRequestHandler):

    def reply(self, text, ctype="text/html; charset=utf-8", code=200, cookie=None):
        data = text.encode("utf-8")
        self.send_response(code)
        self.send_header("Content-Type", ctype)
        self.send_header("Content-Length", str(len(data)))
        if cookie:
            self.send_header("Set-Cookie", cookie)
        self.end_headers()
        self.wfile.write(data)

    def do_GET(self):
        import time
        path = self.path.split("?")[0]
        HITS[path] = HITS.get(path, 0) + 1
        if path == "/robots.txt":
            return self.reply("User-agent: *\nDisallow: /private/\n", "text/plain")
        if path.startswith("/private"):
            return self.reply(html("私密", "<h1>私密区</h1><p>PRIVATE-SECRET-BODY</p>"
                                         "<p>这段说明文字让页面不会被结构性判据当成空壳。</p>"))
        if path == "/count":
            return self.reply(json.dumps({"count": HITS["/count"]}), "application/json")
        if path == "/ua":
            return self.reply(html("UA", "<p>UA=%s</p>" % self.headers.get("User-Agent")))
        if path == "/list":
            return self.reply(html("列表", LIST_BODY))
        if path == "/lazy":
            return self.reply(html("懒加载", LAZY_BODY))
        if path == "/spa":
            return self.reply(html("SPA", SPA_BODY))
        if path == "/table":
            return self.reply(html("表格", TABLE_BODY))
        if path == "/cookie":
            return self.reply(html("Cookie", COOKIE_BODY))
        if path == "/setcookie":
            return self.reply(html("发cookie", COOKIE_BODY), cookie="probe_c4a=1; Path=/")
        if path == "/fp":
            return self.reply(html("指纹", FP_BODY))
        match = re.match(r"/lvl(\d)/([a-z])", path)
        if match:
            n = int(match.group(1))
            if n > 3:
                return self.reply(html("叶子", "NO-MORE-LEVEL"), code=404)
            return self.reply(level_body(n))
        match = re.match(r"/d/(\d+)", path)
        if match:
            i = int(match.group(1))
            return self.reply(html("片名%d" % i,
                                   "<h1 class='t'>片名%d</h1><span class='p'>%.1f</span>"
                                   % (i, i * 1.5)))
        if path == "/status500":
            return self.reply("boom", "text/plain", 500)
        if path.startswith("/slow"):
            time.sleep(2.5)
            return self.reply(html("慢", "SLOW-DONE"))
        return self.reply(html("首页", "<a href='/list'>去列表</a>"))

    def do_POST(self):
        size = int(self.headers.get("Content-Length", 0))
        body = self.rfile.read(size).decode("utf-8")
        HITS["/echo"] = HITS.get("/echo", 0) + 1
        return self.reply(html("回显", "<pre>POSTED %s</pre>" % body))

    def log_message(self, *args):
        pass


def serve():
    server = HTTPServer(("127.0.0.1", PORT), Handler)
    # 客户端超时断开时 socketserver 会往 stderr 打整段 traceback（里面是临时端口号），
    # 那不是结论只是噪声，接住。
    server.handle_error = lambda request, client_address: None
    threading.Thread(target=server.serve_forever, daemon=True).start()
    return server


srv = serve()


def quiet():
    """静音 crawl4ai 的计时行：[INIT] 那行归 BrowserConfig.verbose 管，
    [FETCH]/[SCRAPE]/[COMPLETE] 三行归 CrawlerRunConfig.verbose 管，两个都要关，
    输出里才不会留下每次不同的耗时。"""
    return BrowserConfig(verbose=False), CrawlerRunConfig(verbose=False)



import json

from crawl4ai import (JsonCssExtractionStrategy, JsonXPathExtractionStrategy,
                      RegexExtractionStrategy, TableExtractionStrategy)

SCHEMA = {"name": "movies", "baseSelector": "li",
          "fields": [{"name": "title", "selector": "a", "type": "text"},
                      {"name": "href", "selector": "a", "type": "attribute", "attribute": "href"},
                      {"name": "price", "selector": ".price", "type": "text"}]}

XPATH_SCHEMA = {"name": "movies", "baseSelector": "//li",
                "fields": [{"name": "title", "selector": "./a", "type": "text"}]}


async def run(label, strategy, url=None):
    bc, _ = quiet()
    async with AsyncWebCrawler(config=bc) as cr:
        r = await cr.arun(url or BASE + "/list", config=CrawlerRunConfig(
            verbose=False, extraction_strategy=strategy))
        text = r.extracted_content or ""
        try:
            data = json.loads(text)
            print("%-24s 条数 %d 第一条 %s" % (label, len(data), str(data[0])[:78]))
        except Exception:
            print("%-24s 不是 JSON：%s" % (label, text[:70].replace("\n", " ")))


async def main():
    await run("JsonCss 三个字段", JsonCssExtractionStrategy(schema=SCHEMA))
    await run("JsonCss 只取属性", JsonCssExtractionStrategy(schema={
        "name": "links", "baseSelector": "li a",
        "fields": [{"name": "href", "type": "attribute", "attribute": "href"}]}))
    await run("JsonXPath", JsonXPathExtractionStrategy(schema=XPATH_SCHEMA))
    print("RegexExtractionStrategy 的 pattern 是位标志枚举，成员有：")
    bits = [(m.name, m.value) for m in RegexExtractionStrategy._B]
    for i in range(0, len(bits), 6):
        print("   ", ", ".join("%s=%d" % kv for kv in bits[i:i + 6]))
    await run("Regex 内置 URL+EMAIL", RegexExtractionStrategy(
        pattern=RegexExtractionStrategy._B.URL | RegexExtractionStrategy._B.EMAIL))
    await run("Regex 自定义 dict", RegexExtractionStrategy(
        pattern=RegexExtractionStrategy._B.NOTHING,
        custom={"movie": r"片名(\d+)"}))
    print("（pattern 传字符串会怎样）", end=" ")
    try:
        RegexExtractionStrategy(pattern="all-caps")
        print("居然能用")
    except Exception as exc:
        print("%s: %s" % (type(exc).__name__, str(exc)[:58]))
    try:
        await run("TableExtractionStrategy()", TableExtractionStrategy(), BASE + "/table")
    except Exception as exc:
        print("Table 策略直接实例化 ->", type(exc).__name__)
        print("   原文 =", str(exc)[:88])
    bc, _ = quiet()
    async with AsyncWebCrawler(config=bc) as cr:
        r = await cr.arun(BASE + "/table", config=CrawlerRunConfig(verbose=False))
        print("不配策略时 result.tables =", json.dumps(r.tables, ensure_ascii=False)[:120])
        r2 = await cr.arun(BASE + "/table", config=CrawlerRunConfig(
            verbose=False, table_extraction=True))
        print("table_extraction=True 的 tables =", json.dumps(r2.tables, ensure_ascii=False)[:120])

asyncio.run(main())
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
JsonCss 三个字段             条数 5 第一条 {'title': '片名1', 'href': '/d/1', 'price': '1.5'}
JsonCss 只取属性             条数 5 第一条 {'href': '/d/1'}
JsonXPath                条数 5 第一条 {'title': '片名1'}
RegexExtractionStrategy 的 pattern 是位标志枚举，成员有：
    EMAIL=1, PHONE_INTL=2, PHONE_US=4, URL=8, IPV4=16, IPV6=32
    UUID=64, CURRENCY=128, PERCENTAGE=256, NUMBER=512, DATE_ISO=1024, DATE_US=2048
    TIME_24H=4096, POSTAL_US=8192, POSTAL_UK=16384, HTML_COLOR_HEX=32768, TWITTER_HANDLE=65536, HASHTAG=131072
    MAC_ADDR=262144, IBAN=524288, CREDIT_CARD=1048576, NOTHING=2097152
Regex 内置 URL+EMAIL       不是 JSON：[]
Regex 自定义 dict           条数 5 第一条 {'url': 'http://127.0.0.1:8071/list', 'label': 'movie', 'value': '片名1', 'span'
（pattern 传字符串会怎样） TypeError: unsupported operand type(s) for &: 'int' and 'str'
Table 策略直接实例化 -> TypeError
   原文 = Can't instantiate abstract class TableExtractionStrategy without an implementation for a
不配策略时 result.tables = [{"headers": ["片名", "年份"], "rows": [["片名1", "1991"], ["片名2", "1992"], ["片名3", "1993"]], "caption": "", "summary": "", "m
table_extraction=True 的 tables = []
  • JsonCssExtractionStrategy 的 schema 就是「baseSelector + fields」，字段可以是 type="text" 或 type="attribute"（配 attribute="href"）：5 条都拿到了，第一条是 {'title': '片名1', 'href': '/d/1', 'price': '1.5'}。这是最推荐的一条路，schema 是纯字典，可以直接存配置文件。
  • JsonXPathExtractionStrategy 用 XPath 版 schema（baseSelector="//li"、selector="./a"），同样 5 条，只是表达式换成 XPath。两条路的字段命名规则完全一致，选哪个取决于你熟哪套选择器（见《爬虫-XPath》与《爬虫-BeautifulSoup》）。
  • RegexExtractionStrategy 有个大坑：pattern 不是字符串，是位标志枚举（成员是 EMAIL=1、URL=8、IPV4=16…NOTHING=2097152 这一套内置片段）。传字符串会 TypeError: unsupported operand type(s) for &: 'int' and 'str'；传 pattern=_B.URL | _B.EMAIL 能跑但本机靶站正文里没有邮箱和绝对 URL，结果是 []；自定义正则要写进 custom={"movie": r"片名(\d+)"}，这样 5 条都出来了，且每条带 url/label/value/span 四个键（span 是命中位置，做实体定位时很有用）。
  • TableExtractionStrategy 是抽象类，直接实例化会 TypeError: Can't instantiate abstract class ...——但表格抽取本身是自动做的：不配任何策略时 result.tables 已经是 [{"headers": ["片名", "年份"], "rows": [["片名1", "1991"], …], "caption": "", "summary": "", …}]。反过来显式传 table_extraction=True 之后 tables 变成空列表——这个开关在本机 0.9.5 是「关掉自动抽取」的效果，别按字面理解成「打开」。
  • 语义路线（CosineStrategy、LLMExtractionStrategy）本篇没有实测抓取效果：前者要下载 HuggingFace 文本分类模型（见命令行那节的离线失败），后者要 API 密钥。这里如实标注，不给结论。

结果里的附加产物

截图、PDF、链接、媒体、控制台、网络请求这几个字段都是「默认不开、开了给你」的东西：

screenshot_pdf_media.py
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
# -*- coding: utf-8 -*-
"""结果对象上那些「额外产物」：截图、PDF、链接、媒体、控制台输出、网络请求。"""

import asyncio
import inspect
import json
import os
import re
import threading
from http.server import BaseHTTPRequestHandler, HTTPServer

# 云广告一天只打一次，打不打取决于家目录里的标记文件，会把输出变得不可复现，所以先静音
os.environ.setdefault("CRAWL4AI_NO_CLOUD_NOTICE", "1")

from crawl4ai import (AsyncWebCrawler, BrowserConfig, CacheMode, CrawlerRunConfig)
from crawl4ai.async_crawler_strategy import AsyncPlaywrightCrawlerStrategy


PORT = 8089
BASE = "http://127.0.0.1:8089"
HITS = {}


def html(title, body):
    return ("<!doctype html><html><head><meta charset='utf-8'><title>%s</title>"
            "<meta name='description' content='probe'><style>.nav{color:red}</style>"
            "</head><body>"
            "<div class='nav'>NAV 这是靶站的导航区，故意写长一点，"
            "免得页面被 crawl4ai 的结构性判据当成空壳</div>"
            "%s"
            "<div class='filler'>FILLER 这是靶站的页脚区，同样是有实际文字的标记</div>"
            "</body></html>" % (title, body))


LIST_BODY = ("<h1>列表页</h1><ul class='items'>"
             + "".join("<li><a href='/d/%d'>片名%d</a><span class='price'>%.1f</span></li>"
                       % (i, i, i * 1.5) for i in range(1, 6))
             + "</ul><a class='next' href='/list?page=2'>下一页</a>")

LAZY_BODY = ("<h1>懒加载列表</h1><div id='list'></div>"
             "<div style='height:4000px'></div>"
             "<script>window.addEventListener('scroll',function(){"
             "if(window.__on)return;window.__on=1;"
             "for(var i=0;i<20;i++){var d=document.createElement('div');"
             "d.className='item';d.textContent='滚动条目'+i;"
             "document.getElementById('list').appendChild(d);}});</script>")

SPA_BODY = ("<h1>前端渲染的列表</h1><div id='app'></div><script>setTimeout(function(){var s='';"
            "for(var i=0;i<8;i++){s+='<div class=row>渲染行'+i+'</div>'}"
            "document.getElementById('app').innerHTML=s;},200);</script>")

TABLE_BODY = ("<table class='movies'><tr><th>片名</th><th>年份</th></tr>"
              + "".join("<tr><td>片名%d</td><td>%d</td></tr>" % (i, 1990 + i)
                        for i in range(1, 4)) + "</table>")

COOKIE_BODY = ("<p id='c'></p><script>document.getElementById('c').textContent="
               "'COOKIE='+(document.cookie||'NONE')</script>")

FP_BODY = ("<p id='f'></p><script>document.getElementById('f').textContent="
           "'WD='+navigator.webdriver+'|PL='+navigator.plugins.length+"
           "'|V='+window.innerWidth+'x'+window.innerHeight+'|UA='+navigator.userAgent.slice(0,26)"
           "</script>")

NEXT_LEVEL = {1: ["a", "b"], 2: ["c", "d"], 3: ["e"]}


def level_body(n):
    links = "".join("<a href='/lvl%d/%s'>%s</a>" % (n + 1, x, x)
                    for x in NEXT_LEVEL.get(n, []))
    return html("第%d层" % n, "<h1>LEVEL-%d</h1>%s" % (n, links))


class Handler(BaseHTTPRequestHandler):

    def reply(self, text, ctype="text/html; charset=utf-8", code=200, cookie=None):
        data = text.encode("utf-8")
        self.send_response(code)
        self.send_header("Content-Type", ctype)
        self.send_header("Content-Length", str(len(data)))
        if cookie:
            self.send_header("Set-Cookie", cookie)
        self.end_headers()
        self.wfile.write(data)

    def do_GET(self):
        import time
        path = self.path.split("?")[0]
        HITS[path] = HITS.get(path, 0) + 1
        if path == "/robots.txt":
            return self.reply("User-agent: *\nDisallow: /private/\n", "text/plain")
        if path.startswith("/private"):
            return self.reply(html("私密", "<h1>私密区</h1><p>PRIVATE-SECRET-BODY</p>"
                                         "<p>这段说明文字让页面不会被结构性判据当成空壳。</p>"))
        if path == "/count":
            return self.reply(json.dumps({"count": HITS["/count"]}), "application/json")
        if path == "/ua":
            return self.reply(html("UA", "<p>UA=%s</p>" % self.headers.get("User-Agent")))
        if path == "/list":
            return self.reply(html("列表", LIST_BODY))
        if path == "/lazy":
            return self.reply(html("懒加载", LAZY_BODY))
        if path == "/spa":
            return self.reply(html("SPA", SPA_BODY))
        if path == "/table":
            return self.reply(html("表格", TABLE_BODY))
        if path == "/cookie":
            return self.reply(html("Cookie", COOKIE_BODY))
        if path == "/setcookie":
            return self.reply(html("发cookie", COOKIE_BODY), cookie="probe_c4a=1; Path=/")
        if path == "/fp":
            return self.reply(html("指纹", FP_BODY))
        match = re.match(r"/lvl(\d)/([a-z])", path)
        if match:
            n = int(match.group(1))
            if n > 3:
                return self.reply(html("叶子", "NO-MORE-LEVEL"), code=404)
            return self.reply(level_body(n))
        match = re.match(r"/d/(\d+)", path)
        if match:
            i = int(match.group(1))
            return self.reply(html("片名%d" % i,
                                   "<h1 class='t'>片名%d</h1><span class='p'>%.1f</span>"
                                   % (i, i * 1.5)))
        if path == "/status500":
            return self.reply("boom", "text/plain", 500)
        if path.startswith("/slow"):
            time.sleep(2.5)
            return self.reply(html("慢", "SLOW-DONE"))
        return self.reply(html("首页", "<a href='/list'>去列表</a>"))

    def do_POST(self):
        size = int(self.headers.get("Content-Length", 0))
        body = self.rfile.read(size).decode("utf-8")
        HITS["/echo"] = HITS.get("/echo", 0) + 1
        return self.reply(html("回显", "<pre>POSTED %s</pre>" % body))

    def log_message(self, *args):
        pass


def serve():
    server = HTTPServer(("127.0.0.1", PORT), Handler)
    # 客户端超时断开时 socketserver 会往 stderr 打整段 traceback（里面是临时端口号），
    # 那不是结论只是噪声，接住。
    server.handle_error = lambda request, client_address: None
    threading.Thread(target=server.serve_forever, daemon=True).start()
    return server


srv = serve()


def quiet():
    """静音 crawl4ai 的计时行：[INIT] 那行归 BrowserConfig.verbose 管，
    [FETCH]/[SCRAPE]/[COMPLETE] 三行归 CrawlerRunConfig.verbose 管，两个都要关，
    输出里才不会留下每次不同的耗时。"""
    return BrowserConfig(verbose=False), CrawlerRunConfig(verbose=False)



async def main():
    bc, _ = quiet()
    async with AsyncWebCrawler(config=bc) as cr:
        r = await cr.arun(BASE + "/table", config=CrawlerRunConfig(
            verbose=False, screenshot=True, pdf=True))
        print("screenshot 类型 =", type(r.screenshot).__name__,
              "长度 =", len(r.screenshot or ""))
        print("pdf 类型 =", type(r.pdf).__name__, "长度 =", len(r.pdf or ""))
        print("links =", {k: len(v) for k, v in (r.links or {}).items()}
              if isinstance(r.links, dict) else type(r.links).__name__)
        print("media 的键 =", sorted((r.media or {}).keys()))
        print("console_messages 条数 =", len(r.console_messages or []))
        print("network_requests 条数 =", len(r.network_requests or []))
        print("js_execution_result =", r.js_execution_result)
        print("response_headers 的键数 =", len(r.response_headers or {}))
        print("redirected_url =", r.redirected_url, "| status_code =", r.status_code)
        r2 = await cr.arun(BASE + "/list", config=CrawlerRunConfig(
            verbose=False, js_code="console.log('从页面里打的一条'); window.__x = 7;"))
        print("注入 JS 之后 console_messages =", r2.console_messages)
    async with AsyncWebCrawler(config=bc) as cr:
        r = await cr.arun(BASE + "/list", config=CrawlerRunConfig(verbose=False, screenshot=False))
        print("不开截图时 screenshot =", r.screenshot, "| pdf =", r.pdf)

asyncio.run(main())
1
2
3
4
5
6
7
8
9
10
11
screenshot 类型 = str 长度 = 18732
pdf 类型 = bytes 长度 = 37898
links = {'internal': 0, 'external': 0}
media 的键 = ['audios', 'images', 'videos']
console_messages 条数 = 0
network_requests 条数 = 0
js_execution_result = None
response_headers 的键数 = 4
redirected_url = http://127.0.0.1:8089/table | status_code = 200
注入 JS 之后 console_messages = None
不开截图时 screenshot = None | pdf = None
  • screenshot=True 给的是 base64 字符串（本机这一页 18732 字符，以 data:image/… 形式内嵌，所以不能按图片二进制处理）；pdf=True 给的是 bytes（37898 字节）。两者不开时都是 None。
  • links 是 {'internal': [...], 'external': [...]} 这个形状的字典（本例这张表格里没有链接，所以计数都是 0）；media 的键固定是 audios/images/videos 三个。
  • console_messages 与 network_requests 默认都是 0 条：要它们得分别开 capture_console_messages=True、capture_network_requests=True。输出里那行「注入 JS 之后 console_messages = None」就是证据——js_code 里的 console.log 不会自动被收集，js_execution_result 同样是 None（那是给「带返回值执行 JS」用的字段，需要配合专门的参数）。
  • redirected_url 在没有重定向时等于原 URL，response_headers 只有 4 个键（本机靶站只发 Content-Type/Content-Length/Date/Server）。
  • 一句话总结这一节：crawl4ai 把「附加产物」都放在同一个结果对象上，但每一项都要显式打开；写排查脚本时先确认开关，再怀疑抓不到。

反检测那三层

文档给了三层反检测：magic、enable_stealth、以及 patchright 那条 undetected 路线。本机用一个回显 navigator.webdriver|plugins|innerWidth|UA 的页面把它们量出来：

stealth_adapters.py
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
# -*- coding: utf-8 -*-
"""反检测这一层：navigator.webdriver 默认就是 false，enable_stealth 改的是别的位。"""

import asyncio
import inspect
import json
import os
import re
import threading
from http.server import BaseHTTPRequestHandler, HTTPServer

# 云广告一天只打一次，打不打取决于家目录里的标记文件，会把输出变得不可复现，所以先静音
os.environ.setdefault("CRAWL4AI_NO_CLOUD_NOTICE", "1")

from crawl4ai import (AsyncWebCrawler, BrowserConfig, CacheMode, CrawlerRunConfig)
from crawl4ai.async_crawler_strategy import AsyncPlaywrightCrawlerStrategy


PORT = 8085
BASE = "http://127.0.0.1:8085"
HITS = {}


def html(title, body):
    return ("<!doctype html><html><head><meta charset='utf-8'><title>%s</title>"
            "<meta name='description' content='probe'><style>.nav{color:red}</style>"
            "</head><body>"
            "<div class='nav'>NAV 这是靶站的导航区，故意写长一点，"
            "免得页面被 crawl4ai 的结构性判据当成空壳</div>"
            "%s"
            "<div class='filler'>FILLER 这是靶站的页脚区，同样是有实际文字的标记</div>"
            "</body></html>" % (title, body))


LIST_BODY = ("<h1>列表页</h1><ul class='items'>"
             + "".join("<li><a href='/d/%d'>片名%d</a><span class='price'>%.1f</span></li>"
                       % (i, i, i * 1.5) for i in range(1, 6))
             + "</ul><a class='next' href='/list?page=2'>下一页</a>")

LAZY_BODY = ("<h1>懒加载列表</h1><div id='list'></div>"
             "<div style='height:4000px'></div>"
             "<script>window.addEventListener('scroll',function(){"
             "if(window.__on)return;window.__on=1;"
             "for(var i=0;i<20;i++){var d=document.createElement('div');"
             "d.className='item';d.textContent='滚动条目'+i;"
             "document.getElementById('list').appendChild(d);}});</script>")

SPA_BODY = ("<h1>前端渲染的列表</h1><div id='app'></div><script>setTimeout(function(){var s='';"
            "for(var i=0;i<8;i++){s+='<div class=row>渲染行'+i+'</div>'}"
            "document.getElementById('app').innerHTML=s;},200);</script>")

TABLE_BODY = ("<table class='movies'><tr><th>片名</th><th>年份</th></tr>"
              + "".join("<tr><td>片名%d</td><td>%d</td></tr>" % (i, 1990 + i)
                        for i in range(1, 4)) + "</table>")

COOKIE_BODY = ("<p id='c'></p><script>document.getElementById('c').textContent="
               "'COOKIE='+(document.cookie||'NONE')</script>")

FP_BODY = ("<p id='f'></p><script>document.getElementById('f').textContent="
           "'WD='+navigator.webdriver+'|PL='+navigator.plugins.length+"
           "'|V='+window.innerWidth+'x'+window.innerHeight+'|UA='+navigator.userAgent.slice(0,26)"
           "</script>")

NEXT_LEVEL = {1: ["a", "b"], 2: ["c", "d"], 3: ["e"]}


def level_body(n):
    links = "".join("<a href='/lvl%d/%s'>%s</a>" % (n + 1, x, x)
                    for x in NEXT_LEVEL.get(n, []))
    return html("第%d层" % n, "<h1>LEVEL-%d</h1>%s" % (n, links))


class Handler(BaseHTTPRequestHandler):

    def reply(self, text, ctype="text/html; charset=utf-8", code=200, cookie=None):
        data = text.encode("utf-8")
        self.send_response(code)
        self.send_header("Content-Type", ctype)
        self.send_header("Content-Length", str(len(data)))
        if cookie:
            self.send_header("Set-Cookie", cookie)
        self.end_headers()
        self.wfile.write(data)

    def do_GET(self):
        import time
        path = self.path.split("?")[0]
        HITS[path] = HITS.get(path, 0) + 1
        if path == "/robots.txt":
            return self.reply("User-agent: *\nDisallow: /private/\n", "text/plain")
        if path.startswith("/private"):
            return self.reply(html("私密", "<h1>私密区</h1><p>PRIVATE-SECRET-BODY</p>"
                                         "<p>这段说明文字让页面不会被结构性判据当成空壳。</p>"))
        if path == "/count":
            return self.reply(json.dumps({"count": HITS["/count"]}), "application/json")
        if path == "/ua":
            return self.reply(html("UA", "<p>UA=%s</p>" % self.headers.get("User-Agent")))
        if path == "/list":
            return self.reply(html("列表", LIST_BODY))
        if path == "/lazy":
            return self.reply(html("懒加载", LAZY_BODY))
        if path == "/spa":
            return self.reply(html("SPA", SPA_BODY))
        if path == "/table":
            return self.reply(html("表格", TABLE_BODY))
        if path == "/cookie":
            return self.reply(html("Cookie", COOKIE_BODY))
        if path == "/setcookie":
            return self.reply(html("发cookie", COOKIE_BODY), cookie="probe_c4a=1; Path=/")
        if path == "/fp":
            return self.reply(html("指纹", FP_BODY))
        match = re.match(r"/lvl(\d)/([a-z])", path)
        if match:
            n = int(match.group(1))
            if n > 3:
                return self.reply(html("叶子", "NO-MORE-LEVEL"), code=404)
            return self.reply(level_body(n))
        match = re.match(r"/d/(\d+)", path)
        if match:
            i = int(match.group(1))
            return self.reply(html("片名%d" % i,
                                   "<h1 class='t'>片名%d</h1><span class='p'>%.1f</span>"
                                   % (i, i * 1.5)))
        if path == "/status500":
            return self.reply("boom", "text/plain", 500)
        if path.startswith("/slow"):
            time.sleep(2.5)
            return self.reply(html("慢", "SLOW-DONE"))
        return self.reply(html("首页", "<a href='/list'>去列表</a>"))

    def do_POST(self):
        size = int(self.headers.get("Content-Length", 0))
        body = self.rfile.read(size).decode("utf-8")
        HITS["/echo"] = HITS.get("/echo", 0) + 1
        return self.reply(html("回显", "<pre>POSTED %s</pre>" % body))

    def log_message(self, *args):
        pass


def serve():
    server = HTTPServer(("127.0.0.1", PORT), Handler)
    # 客户端超时断开时 socketserver 会往 stderr 打整段 traceback（里面是临时端口号），
    # 那不是结论只是噪声，接住。
    server.handle_error = lambda request, client_address: None
    threading.Thread(target=server.serve_forever, daemon=True).start()
    return server


srv = serve()


def quiet():
    """静音 crawl4ai 的计时行：[INIT] 那行归 BrowserConfig.verbose 管，
    [FETCH]/[SCRAPE]/[COMPLETE] 三行归 CrawlerRunConfig.verbose 管，两个都要关，
    输出里才不会留下每次不同的耗时。"""
    return BrowserConfig(verbose=False), CrawlerRunConfig(verbose=False)



import crawl4ai.browser_adapter as BA

RC = CrawlerRunConfig


async def read(label, browser_kwargs=None, adapter=None):
    kwargs = dict(browser_kwargs or {})
    strategy = None
    if adapter is not None:
        strategy = AsyncPlaywrightCrawlerStrategy(
            browser_config=BrowserConfig(verbose=False, **kwargs), browser_adapter=adapter)
    async with AsyncWebCrawler(config=BrowserConfig(verbose=False, **kwargs),
                               crawler_strategy=strategy) as cr:
        r = await cr.arun(BASE + "/fp", config=RC(verbose=False))
        line = [l for l in r.markdown.splitlines() if "WD=" in l]
        print("%-38s -> %s" % (label, line[:1]))


async def main():
    await read("默认（PlaywrightAdapter）")
    await read("enable_stealth=True", {"enable_stealth": True})
    await read("StealthAdapter()", None, BA.StealthAdapter())
    await read("UndetectedAdapter()（走 patchright）", None, BA.UndetectedAdapter())
    await read("user_agent 自己写", {"user_agent": "MyCrawler/1.0"})
    print("StealthAdapter 的构造参数 =",
          list(inspect.signature(BA.StealthAdapter.__init__).parameters)[1:])
    print("UndetectedAdapter 的构造参数 =",
          list(inspect.signature(BA.UndetectedAdapter.__init__).parameters)[1:])

asyncio.run(main())
1
2
3
4
5
6
7
默认（PlaywrightAdapter）                  -> ['WD=false|PL=0|V=1080x600|UA=Mozilla/5.0 (X11; Linux x8']
enable_stealth=True                    -> ['WD=false|PL=3|V=1080x600|UA=Mozilla/5.0 (X11; Linux x8']
StealthAdapter()                       -> ['WD=false|PL=0|V=1080x600|UA=Mozilla/5.0 (X11; Linux x8']
UndetectedAdapter()（走 patchright）      -> ['WD=false|PL=0|V=1080x600|UA=Mozilla/5.0 (X11; Linux x8']
user_agent 自己写                         -> ['WD=false|PL=0|V=1080x600|UA=MyCrawler/1.0']
StealthAdapter 的构造参数 = []
UndetectedAdapter 的构造参数 = []
  • navigator.webdriver 四种情况下都是 false，包括默认的 PlaywrightAdapter。现代 Chromium 在无头模式下已经不写这个位了，所以「检测 webdriver」这一招对 crawl4ai 无效，它不是区分三种配置的指标。
  • 真正有差别的是 navigator.plugins.length：默认无头是 0，enable_stealth=True 变成 3，而换 StealthAdapter()/UndetectedAdapter() 仍是 0（这两个适配器在本机走的是别的补丁路径）。也就是说 enable_stealth 改的是插件表这类"环境位"，不是 webdriver。
  • 顺带解释了为什么「有头模式」PL=5（上一节的输出）：真实 Chrome 自带 5 个插件，无头是 0，这是最朴素的差别。
  • StealthAdapter 与 UndetectedAdapter 的构造参数都是空（不接受任何参数），换适配器靠的是 AsyncPlaywrightCrawlerStrategy(browser_adapter=...)，而这个类不在顶层导出里，要从 crawl4ai.async_crawler_strategy 导入——这也是本篇实测到的一处文档漂移：Advanced 页写的顶层导入路径本机拿不到。
  • 最后要提醒的是判据本身：本机靶站是自己写的页面，只能证明「指纹位变了没有」，不能证明「某个真实站点会不会放行」。真实反爬的验证要靠多次采样，不适合写进可复现笔记。

落盘与 crwl

crawl4ai 会把东西写进用户目录的 .crawl4ai，还有一个 crwl 命令行。这段列出目录内容并真跑一次命令行：

persist_and_cli.py
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
# -*- coding: utf-8 -*-
"""落在磁盘上的东西与 crwl 命令行：家目录里有哪些文件、命令行抓一次是什么样子。"""

import asyncio
import inspect
import json
import os
import re
import threading
from http.server import BaseHTTPRequestHandler, HTTPServer

# 云广告一天只打一次，打不打取决于家目录里的标记文件，会把输出变得不可复现，所以先静音
os.environ.setdefault("CRAWL4AI_NO_CLOUD_NOTICE", "1")

from crawl4ai import (AsyncWebCrawler, BrowserConfig, CacheMode, CrawlerRunConfig)
from crawl4ai.async_crawler_strategy import AsyncPlaywrightCrawlerStrategy


PORT = 8086
BASE = "http://127.0.0.1:8086"
HITS = {}


def html(title, body):
    return ("<!doctype html><html><head><meta charset='utf-8'><title>%s</title>"
            "<meta name='description' content='probe'><style>.nav{color:red}</style>"
            "</head><body>"
            "<div class='nav'>NAV 这是靶站的导航区，故意写长一点，"
            "免得页面被 crawl4ai 的结构性判据当成空壳</div>"
            "%s"
            "<div class='filler'>FILLER 这是靶站的页脚区，同样是有实际文字的标记</div>"
            "</body></html>" % (title, body))


LIST_BODY = ("<h1>列表页</h1><ul class='items'>"
             + "".join("<li><a href='/d/%d'>片名%d</a><span class='price'>%.1f</span></li>"
                       % (i, i, i * 1.5) for i in range(1, 6))
             + "</ul><a class='next' href='/list?page=2'>下一页</a>")

LAZY_BODY = ("<h1>懒加载列表</h1><div id='list'></div>"
             "<div style='height:4000px'></div>"
             "<script>window.addEventListener('scroll',function(){"
             "if(window.__on)return;window.__on=1;"
             "for(var i=0;i<20;i++){var d=document.createElement('div');"
             "d.className='item';d.textContent='滚动条目'+i;"
             "document.getElementById('list').appendChild(d);}});</script>")

SPA_BODY = ("<h1>前端渲染的列表</h1><div id='app'></div><script>setTimeout(function(){var s='';"
            "for(var i=0;i<8;i++){s+='<div class=row>渲染行'+i+'</div>'}"
            "document.getElementById('app').innerHTML=s;},200);</script>")

TABLE_BODY = ("<table class='movies'><tr><th>片名</th><th>年份</th></tr>"
              + "".join("<tr><td>片名%d</td><td>%d</td></tr>" % (i, 1990 + i)
                        for i in range(1, 4)) + "</table>")

COOKIE_BODY = ("<p id='c'></p><script>document.getElementById('c').textContent="
               "'COOKIE='+(document.cookie||'NONE')</script>")

FP_BODY = ("<p id='f'></p><script>document.getElementById('f').textContent="
           "'WD='+navigator.webdriver+'|PL='+navigator.plugins.length+"
           "'|V='+window.innerWidth+'x'+window.innerHeight+'|UA='+navigator.userAgent.slice(0,26)"
           "</script>")

NEXT_LEVEL = {1: ["a", "b"], 2: ["c", "d"], 3: ["e"]}


def level_body(n):
    links = "".join("<a href='/lvl%d/%s'>%s</a>" % (n + 1, x, x)
                    for x in NEXT_LEVEL.get(n, []))
    return html("第%d层" % n, "<h1>LEVEL-%d</h1>%s" % (n, links))


class Handler(BaseHTTPRequestHandler):

    def reply(self, text, ctype="text/html; charset=utf-8", code=200, cookie=None):
        data = text.encode("utf-8")
        self.send_response(code)
        self.send_header("Content-Type", ctype)
        self.send_header("Content-Length", str(len(data)))
        if cookie:
            self.send_header("Set-Cookie", cookie)
        self.end_headers()
        self.wfile.write(data)

    def do_GET(self):
        import time
        path = self.path.split("?")[0]
        HITS[path] = HITS.get(path, 0) + 1
        if path == "/robots.txt":
            return self.reply("User-agent: *\nDisallow: /private/\n", "text/plain")
        if path.startswith("/private"):
            return self.reply(html("私密", "<h1>私密区</h1><p>PRIVATE-SECRET-BODY</p>"
                                         "<p>这段说明文字让页面不会被结构性判据当成空壳。</p>"))
        if path == "/count":
            return self.reply(json.dumps({"count": HITS["/count"]}), "application/json")
        if path == "/ua":
            return self.reply(html("UA", "<p>UA=%s</p>" % self.headers.get("User-Agent")))
        if path == "/list":
            return self.reply(html("列表", LIST_BODY))
        if path == "/lazy":
            return self.reply(html("懒加载", LAZY_BODY))
        if path == "/spa":
            return self.reply(html("SPA", SPA_BODY))
        if path == "/table":
            return self.reply(html("表格", TABLE_BODY))
        if path == "/cookie":
            return self.reply(html("Cookie", COOKIE_BODY))
        if path == "/setcookie":
            return self.reply(html("发cookie", COOKIE_BODY), cookie="probe_c4a=1; Path=/")
        if path == "/fp":
            return self.reply(html("指纹", FP_BODY))
        match = re.match(r"/lvl(\d)/([a-z])", path)
        if match:
            n = int(match.group(1))
            if n > 3:
                return self.reply(html("叶子", "NO-MORE-LEVEL"), code=404)
            return self.reply(level_body(n))
        match = re.match(r"/d/(\d+)", path)
        if match:
            i = int(match.group(1))
            return self.reply(html("片名%d" % i,
                                   "<h1 class='t'>片名%d</h1><span class='p'>%.1f</span>"
                                   % (i, i * 1.5)))
        if path == "/status500":
            return self.reply("boom", "text/plain", 500)
        if path.startswith("/slow"):
            time.sleep(2.5)
            return self.reply(html("慢", "SLOW-DONE"))
        return self.reply(html("首页", "<a href='/list'>去列表</a>"))

    def do_POST(self):
        size = int(self.headers.get("Content-Length", 0))
        body = self.rfile.read(size).decode("utf-8")
        HITS["/echo"] = HITS.get("/echo", 0) + 1
        return self.reply(html("回显", "<pre>POSTED %s</pre>" % body))

    def log_message(self, *args):
        pass


def serve():
    server = HTTPServer(("127.0.0.1", PORT), Handler)
    # 客户端超时断开时 socketserver 会往 stderr 打整段 traceback（里面是临时端口号），
    # 那不是结论只是噪声，接住。
    server.handle_error = lambda request, client_address: None
    threading.Thread(target=server.serve_forever, daemon=True).start()
    return server


srv = serve()


def quiet():
    """静音 crawl4ai 的计时行：[INIT] 那行归 BrowserConfig.verbose 管，
    [FETCH]/[SCRAPE]/[COMPLETE] 三行归 CrawlerRunConfig.verbose 管，两个都要关，
    输出里才不会留下每次不同的耗时。"""
    return BrowserConfig(verbose=False), CrawlerRunConfig(verbose=False)



import os
import subprocess
import sys

HOME = os.path.join(os.environ["USERPROFILE"], ".crawl4ai")
raw = sorted(os.listdir(HOME))
backups = [e for e in raw if re.match(r"crawl4ai\.db\.backup_\d{8}_\d{6}$", e)]
entries = [e for e in raw if e not in backups]
print("~/.crawl4ai 里的固定项 %d 个：" % len(entries))
for i in range(0, len(entries), 6):
    print("   ", ", ".join(entries[i:i + 6]))
print("带时间戳的数据库备份存在 =", bool(backups), "（每跑一次迁移多一个，所以只报有没有）")
db = os.path.join(HOME, "crawl4ai.db")
print("crawl4ai.db 存在 =", os.path.exists(db))

exe = os.path.join(os.path.dirname(sys.executable), "crwl.exe")
CLI_ENV = dict(os.environ, CRAWL4AI_NO_CLOUD_NOTICE="1", LITELLM_LOCAL_MODEL_COST_MAP="True")
help_text = subprocess.run([exe, "--help"], capture_output=True, text=True,
                           encoding="utf-8", errors="replace", env=CLI_ENV).stdout
keys = [l.strip().split()[0] for l in help_text.splitlines() if l.strip().startswith("-")]
print("crwl --help 里的选项（前 12 个）=", keys[:12])
run = subprocess.run([exe, BASE + "/list", "--verbose", "false"], capture_output=True,
                     text=True, encoding="utf-8", errors="replace", env=CLI_ENV, timeout=300)
# 过滤：空行、带本机用户目录的绝对路径、litellm 那条带时间戳的告警
NOISE = ("LiteLLM", "Please fix", "post-installation", "Database backup")


def keep(line):
    if not line.strip() or os.path.expanduser("~") in line:
        return False
    return not any(key in line for key in NOISE)


lines = [l for l in (run.stdout + run.stderr).splitlines() if keep(l)]
print("crwl 抓本机靶站：退出码 %d，过滤后输出行数 %d" % (run.returncode, len(lines)))
for l in lines[:12]:
    print("   |", l[:110])
1
2
3
4
5
6
7
8
9
10
11
~/.crawl4ai 里的固定项 13 个：
    .cloud-notice-blocked, .cloud-notice-shown, .crawl4ai, cache, cleaned_html, crawl4ai.db
    crawler.log, extracted_content, html_content, markdown_content, models, screenshots
    version.txt
带时间戳的数据库备份存在 = True （每跑一次迁移多一个，所以只报有没有）
crawl4ai.db 存在 = True
crwl --help 里的选项（前 12 个）= ['-B,', '-C,', '-f,', '-e,', '-j,', '-s,', '-b,', '-c,', '-o,', '-O,', '-bc,', '-q,']
crwl 抓本机靶站：退出码 2，过滤后输出行数 3
   | Usage: crwl crawl [OPTIONS] URL
   | Try 'crwl crawl --help' for help.
   | Error: Got unexpected extra argument (false)
  • 家目录里的固定项是 13 个：crawl4ai.db（缓存与元数据的 SQLite）、cache、html_content、markdown_content、extracted_content、cleaned_html、screenshots、models、crawler.log、version.txt，外加两个云广告标记 .cloud-notice-shown/.cloud-notice-blocked 和一层 .crawl4ai 子目录（CRAWL4_AI_BASE_DIRECTORY 指过去时生成的）。
  • crawl4ai.db.backup_<时间戳> 每跑一次迁移就多一个，所以这里只报「有没有」，不报个数——不然这段输出每次都不一样。
  • crwl --help 里的选项是一串短横（-B,、-C,、-f,、-e,、-j,、-s,、-b,、-c,、-o,、-O,、-bc,、-q,），真正的子命令是 crwl crawl URL。
  • 本段那次 crwl <url> --verbose false 的退出码是 2，报错是 Error: Got unexpected extra argument (false)——--verbose 是布尔开关，不接受 false 这个值。想安静就得去掉它，或者用库里的 verbose=False（那两个 verbose 的分工见「本机靶站」一节）。
  • LiteLLM 那条 Failed to fetch remote model cost map 告警被过滤掉了（它带时间戳且每次相同前缀不同后缀）；本篇所有块的开头都设了 CRAWL4AI_NO_CLOUD_NOTICE=1，这也是过滤规则的一部分。

与 requests 抓同一页

最后一个对照：同一个在线靶站、同一个解析规则，requests 与 crawl4ai（不开浏览器的那条 HTTP 策略）各抓一次。requests 侧基线：

same_page_requests.py
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
# -*- coding: utf-8 -*-
"""同一个在线靶站在 requests 下的基线：状态码、字节数、能数出几个片名。"""

import re

import requests

URL = "https://ssr1.scrape.center/page/1"
resp = requests.get(URL, timeout=30, headers={"User-Agent": "Mozilla/5.0"})
html = resp.text
names = re.findall(r'class="name">\s*<h2[^>]*>([^<]+)', html)
print("requests 状态", resp.status_code)
print("字节数", len(resp.content))
print("编码", resp.encoding, "| Content-Type", resp.headers.get("Content-Type"))
print("片名个数", len(names))
print("前 3 个", names[:3])
1
2
3
4
5
requests 状态 200
字节数 41667
编码 utf-8 | Content-Type text/html; charset=utf-8
片名个数 10
前 3 个 ['霸王别姬 - Farewell My Concubine', '这个杀手不太冷 - Léon', '肖申克的救赎 - The Shawshank Redemption']

crawl4ai 侧，把取法与打印口径写成完全一样：

same_page_crawl4ai.py
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
# -*- coding: utf-8 -*-
"""同一个在线靶站，用 crawl4ai 的不开浏览器路径抓一次，口径与 requests 基线保持一致。"""

import re

import asyncio

from crawl4ai import AsyncWebCrawler, BrowserConfig, CrawlerRunConfig, HTTPCrawlerConfig
from crawl4ai.async_crawler_strategy import AsyncHTTPCrawlerStrategy

URL = "https://ssr1.scrape.center/page/1"


async def main():
    strategy = AsyncHTTPCrawlerStrategy(browser_config=HTTPCrawlerConfig())
    async with AsyncWebCrawler(config=BrowserConfig(verbose=False),
                              crawler_strategy=strategy) as cr:
        r = await cr.arun(URL, config=CrawlerRunConfig(verbose=False, cache_mode="bypass"))
    html = r.html or ""
    names = re.findall(r'class="name">\s*<h2[^>]*>([^<]+)', html)
    ctype = (r.response_headers or {}).get("Content-Type")
    print("crawl4ai 状态", r.status_code)
    print("字节数", len(html.encode("utf-8")))
    print("Content-Type", ctype, "| 从里面读到的编码", ctype.split("charset=")[-1])
    print("片名个数", len(names))
    print("前 3 个", names[:3])

asyncio.run(main())
1
2
3
4
5
crawl4ai 状态 200
字节数 41667
Content-Type text/html; charset=utf-8 | 从里面读到的编码 utf-8
片名个数 10
前 3 个 ['霸王别姬 - Farewell My Concubine', '这个杀手不太冷 - Léon', '肖申克的救赎 - The Shawshank Redemption']

两份输出除了第一行的库名，其余逐字相同：状态 200、41667 字节、Content-Type: text/html; charset=utf-8、片名 10 个、前三个是「霸王别姬 / 这个杀手不太冷 / 肖申克的救赎」。注意 crawl4ai 这边没有 resp.content，字节数是把 result.html 按 UTF-8 编码之后量的，Content-Type 取自 response_headers。

这条对照的用意是把选型说清楚：只要拿正文，requests 就够了，crawl4ai 的 HTTP 路径不会给你更多内容。它的价值在后面的加工——把 HTML 变成 markdown（「抓一页给什么」）、把表格与字段抽成结构化 JSON（「结构化抽取四路」）、以及在必须执行 JS 时把浏览器那一层的成本花得值（同一节里 spa1 只有 3 行）。

三篇新框架的分工（与《爬虫-DrissionPage》《爬虫-curl_cffi》同口径）：

需求 该用谁
只要 HTTP 客户端、要浏览器级 TLS 指纹 curl_cffi（见《爬虫-curl_cffi》）
要操控浏览器、像人一样点与滚、还要 requests 风格的 API DrissionPage（见《爬虫-DrissionPage》）
要「抓完直接给 markdown / 结构化 JSON」，批量喂给 LLM 或索引 crawl4ai（本篇）
要工程化的队列、重试、代理轮换、续爬 Crawlee（见《爬虫-Crawlee》）、Scrapy（见《爬虫-Scrapy》）

常见坑速查

把本篇各节里最容易踩的收在一张表上。

现象 真正的原因 出处
crawl4ai.__version__ 打出来不是版本号 它是个子模块对象 版本号怎么取
第二次抓同一页还是打了服务端 cache_mode 默认 BYPASS 缓存的五种模式
success=False 但内容明明抓到了 结构性反爬判据误判小页面 没被拦却报被拦
输出里混进云广告横幅 判据命中时会打广告，一天一次 没被拦却报被拦
BrowserConfig(proxy_cfg=...) 报 TypeError 正名是 proxy_config，proxy 已废弃并打警告 代理与 UA
post_url=/form_data= 全部 TypeError 0.9.5 已删，POST 要走 HTTP 策略的 method/data/json JS 注入与表单
deep_walk_strategy= 报错 参数名是 deep_crawl_strategy 深度抓取
result.meta 抛 AttributeError 字段名是 metadata 结果对象的字段
include_links= 传给生成器报错 这些开关要写进 options={} markdown 生成器
表格抽取「打开了反而没了」 table_extraction=True 实为关掉自动抽取 结构化抽取四路
wait_for_timeout=800 等不到 SPA 内容 它不等元素，要用 wait_for 懒加载与等待
虚拟滚动没滚 必须同时开 scan_full_page=True 懒加载与等待
console.log 没被收集 要 capture_console_messages=True 结果里的附加产物
session_id 换实例就丢 cookie session_id 不共享浏览器上下文 session_id 管到哪
browser_type="safari" 竟然成功 不校验取值，悄悄按 chromium 跑 浏览器这一层
dispatcher= 传给 RunConfig 报错 dispatcher 是 arun_many 的参数 批量与调度
crwl … --verbose false 退出码 2 --verbose 是布尔开关，不带值 落盘与 crwl
每次跑输出都不同的四样东西 耗时、内存地址、随机 id、带时间戳的备份名 各节的归一化处理

参考