爬虫-Requests
这篇讲什么
- requests 的面一次过写完:默认头、五个动词、参数、Response、Session、重试、代理、证书、上传下载、异常体系,最后是验证码识别。
- 抓数据的示例都跑在公开练习站上(服务端渲染 ssr1、接口 spa1、登录用的 quotes.toscrape.com);编码判定、上传、代理、证书、超时与重试这些协议级的仍在可自控的本机靶站上跑,输出逐字贴出。实测环境 Python 3.14.10 + requests 2.34.2 + urllib3 2.8.0。
- 老笔记里的百度、人人网这类站点要么改版要么加了签名,代码留不住,这一版全部换成公开练习站或本机靶站可复现的等价写法;打码平台和百度文字识别要账号与 key,本机没法验,只留用法说明并标注未实测。
安装与版本
-
pip install requests带进四个依赖:charset_normalizer、idna、urllib3、certifi。要 SOCKS 代理写pip install "requests[socks]",多带一个 PySocks。requests 底层是 urllib3,连接池与 keep-alive 全自动。1
pip install "requests[socks]"
1
2
3
4
5
6Requirement already satisfied: requests[socks] in D:\ProgramData\anaconda3\envs\spider\Lib\site-packages (2.34.2)
Requirement already satisfied: charset_normalizer<4,>=2 in D:\ProgramData\anaconda3\envs\spider\Lib\site-packages (from requests[socks]) (3.5.2)
Requirement already satisfied: idna<4,>=2.5 in D:\ProgramData\anaconda3\envs\spider\Lib\site-packages (from requests[socks]) (3.20)
Requirement already satisfied: urllib3<3,>=1.26 in D:\ProgramData\anaconda3\envs\spider\Lib\site-packages (from requests[socks]) (2.8.0)
Requirement already satisfied: certifi>=2023.5.7 in D:\ProgramData\anaconda3\envs\spider\Lib\site-packages (from requests[socks]) (2026.10.23)
Requirement already satisfied: PySocks!=1.5.7,>=1.5.6 in D:\ProgramData\anaconda3\envs\spider\Lib\site-packages (from requests[socks]) (1.7.1) -
版本号别猜,直接打印(
sys.version、requests.__version__、urllib3.__version__、certifi.where())。certifi.where()是 requests 默认信任的 CA 包位置(不是系统证书库),verify=想指别的证书,指的就是这一类文件的路径。req_versions.py 1
2
3
4
5
6
7
8
9
10import sys
import certifi
import requests
import urllib3
print("python : %s" % sys.version.split()[0])
print("requests : %s" % requests.__version__)
print("urllib3 : %s" % urllib3.__version__)
print("certifi.where() : %s" % certifi.where())1
2
3
4python : 3.14.10
requests : 2.34.2
urllib3 : 2.8.0
certifi.where() : D:\ProgramData\anaconda3\envs\spider\Lib\site-packages\certifi\cacert.pem
默认请求头
-
靶站的
/whoami把头原样回显,requests 与 urllib 的差别一眼看完。三点值得记住:requests 默认声明支持gzip, deflate, br, zstd并自动解压(/gzip线上 63 字节,落地len(content)=660 len(text)=300),urllib 只报identity;requests 发keep-alive,urllib 发close;默认 UA 是python-requests/2.34.2,对反爬站点等于自报家门,必须覆盖。req_default_headers.py 1
2
3
4
5
6
7
8
9import json
import urllib.request
import requests
url = "http://127.0.0.1:8010/whoami"
print("requests ->")
print(json.dumps(requests.get(url).json()["headers"], ensure_ascii=False, indent=2))
with urllib.request.urlopen(url) as resp:
print("urllib ->")
print(json.dumps(json.loads(resp.read())["headers"], ensure_ascii=False, indent=2))1
2
3
4
5
6
7
8
9
10
11
12
13
14
15requests ->
{
"Host": "127.0.0.1:8010",
"User-Agent": "python-requests/2.34.2",
"Accept-Encoding": "gzip, deflate, br, zstd",
"Accept": "*/*",
"Connection": "keep-alive"
}
urllib ->
{
"Accept-Encoding": "identity",
"Host": "127.0.0.1:8010",
"User-Agent": "Python-urllib/3.14",
"Connection": "close"
} -
UA 不改的代价用
/ua演示:UA 里没有Mozilla就 403。req_ua.py 1
2
3
4
5import requests
url = "http://127.0.0.1:8010/ua"
print("默认 UA ->", requests.get(url).status_code, requests.get(url).json())
good = requests.get(url, headers={"User-Agent": "Mozilla/5.0 (Windows NT 10.0) Chrome/156.0"})
print("换 UA ->", good.status_code, good.json())1
2默认 UA -> 403 {'ok': False, 'ua': 'python-requests/2.34.2', '判定': '非浏览器 UA,拒绝'}
换 UA -> 200 {'ok': True, 'ua': 'Mozilla/5.0 (Windows NT 10.0) Chrome/156.0', '判定': '放行'} -
请求头是大小写无关的字典,值只接受
str或bytes:给列表抛InvalidHeader,键名带中文抛UnicodeEncodeError,值带换行抛InvalidHeader: Invalid leading whitespace, reserved character(s), or return character(s) in header value。同一个键要发多个值只能自己拼成"a, b"一个字符串。 -
反爬站认 UA,但公开练习站 ssr1 不看 UA:默认 UA 和 fake-useragent 随机 UA 都 200、正文逐字相同。这类站拦不住你,换 UA 只是习惯、不是护身符。
req_fake_ua.py 1
2
3
4
5
6
7
8import requests
from fake_useragent import UserAgent
ssr = "https://ssr1.scrape.center/"
d = requests.get(ssr, timeout=20)
u = requests.get(ssr, headers={"User-Agent": UserAgent().random}, timeout=20)
print("默认 UA:", d.status_code, len(d.text), "字节")
print("随机 UA:", u.status_code, len(u.text), "字节")
print("两次正文逐字相同:", d.text == u.text)1
2
3默认 UA: 200 41316 字节
随机 UA: 200 41316 字节
两次正文逐字相同: True
五个动词
-
getpostputpatchdelete五个模块级函数签名一致、参数一样,只是方法名不同;方法名是变量时用requests.request(method, url, **kwargs),与五个函数完全等价。head的正文永远是空的,状态码与头正常。req_verbs.py 1
2
3
4
5
6
7import requests
url = "http://127.0.0.1:8010/echo"
for verb in ("get", "put", "patch", "delete"):
r = getattr(requests, verb)(url)
print("%-6s -> %s 服务端收到方法=%s" % (verb, r.status_code, r.json()["method"]))
print("post ->", requests.post("http://127.0.0.1:8010/api/books", json={"title": "新书"}).json())
print("head 正文长度", len(requests.head(url).content), "options Allow:", requests.options(url).headers.get("Allow"))1
2
3
4
5
6get -> 200 服务端收到方法=GET
put -> 200 服务端收到方法=PUT
patch -> 200 服务端收到方法=PATCH
delete -> 200 服务端收到方法=DELETE
post -> {'created': {'title': '新书'}, 'kind': 'json'}
head 正文长度 0 options Allow: GET, HEAD, POST, PUT, PATCH, DELETE, OPTIONS
常用参数
-
params负责查询串,中文、列表、None、数字各有各的结果,别手拼;URL 里已有 query 时再传params,两边会合并。嵌套 dict 是坑:被urlencode当成键序列展开,只剩一个键名,值全丢。req_params.py 1
2
3
4
5
6
7
8
9
10
11import requests
base = "http://127.0.0.1:8010"
for label, params in (
("中文值", {"q": "三国", "field": "title"}),
("列表值", {"tag": ["a", "b"]}),
("None 值", {"a": 1, "b": None}),
("数字与布尔", {"f": 1.5, "n": 3, "ok": True}),
("嵌套 dict", {"d": {"a": 1}}),
):
q = requests.get(base + "/whoami", params=params).url.split("?", 1)[1]
print("%-12s %s" % (label, q))1
2
3
4
5中文值 q=%E4%B8%89%E5%9B%BD&field=title
列表值 tag=a&tag=b
None 值 a=1
数字与布尔 f=1.5&n=3&ok=True
嵌套 dict d=a -
data有四副面孔,差别全在Content-Type上;bytes和str都不补Content-Type。生成器或文件对象当data走分块传输,Transfer-Encoding='chunked'、没有Content-Length,服务端要会拆 chunk 才读得到内容。传 JSON 要用json=,它顺手把Content-Type设成application/json,中文按 UTF-8 发出去;json=与data=同时给时data赢,json被静默忽略。req_data_json.py 1
2
3
4
5
6
7
8
9
10
11import requests
echo = "http://127.0.0.1:8014/upload"
api = "http://127.0.0.1:8010/api/books"
for label, body in (("dict", {"q": "三国", "name": "lilei"}), ("bytes", b"raw-bytes-hello"),
("str", "中文 str"), ("二元组列表", [("k", "v1"), ("k", "v2")])):
r = requests.post(echo, data=body)
print("%-10s Content-Type=%-32r body=%r" % (
label, r.request.headers.get("Content-Type"), r.json()["body_preview"]))
r = requests.post(api, json={"title": "新书", "价": 19.9})
print("json= ->", r.status_code, repr(r.request.headers["Content-Type"]), r.json())
print("json= 和 data= 同时给 ->", requests.post(api, json={"a": 1}, data={"b": 2}).json())1
2
3
4
5
6dict Content-Type='application/x-www-form-urlencoded' body='q=%E4%B8%89%E5%9B%BD&name=lilei'
bytes Content-Type=None body='raw-bytes-hello'
str Content-Type=None body='中文 str'
二元组列表 Content-Type='application/x-www-form-urlencoded' body='k=v1&k=v2'
json= -> 201 'application/json' {'created': {'title': '新书', '价': 19.9}, 'kind': 'json'}
json= 和 data= 同时给 -> {'created': {'b': '2'}, 'kind': 'form'} -
cookies=收字典也收CookieJar,CookieJar按域名和路径过滤,字典不会。req_cookies.py 1
2
3
4
5
6
7import requests
jar = requests.cookies.RequestsCookieJar()
jar.set("sid", "12345678", domain="127.0.0.1", path="/")
jar.set("other", "x", domain="example.com")
for label, cookies in (("字典", {"sid": "abcd1234", "lang": "zh"}), ("CookieJar", jar)):
r = requests.get("http://127.0.0.1:8010/readcookie", cookies=cookies)
print("%-10s 服务端看到的 Cookie=%r" % (label, r.json()["cookie"]))1
2字典 服务端看到的 Cookie='sid=abcd1234; lang=zh'
CookieJar 服务端看到的 Cookie='sid=12345678' -
auth=只配 Basic,等价于HTTPBasicAuth,密码从环境变量取别写死。摘要认证要用requests-toolbelt,本机没装,未实测。req_auth.py 1
2
3
4
5import requests
url = "http://127.0.0.1:8014/basic"
print("不带 auth ->", requests.get(url).status_code)
r = requests.get(url, auth=("lilei", "password"))
print("带 auth ->", r.status_code, "Authorization=%r" % r.request.headers["Authorization"])1
2不带 auth -> 401
带 auth -> 200 Authorization='Basic bGlsZWk6cGFzc3dvcmQ='
超时
-
timeout给一个数字不等于「这次请求的总时长」,它同时是连接超时和读超时,两个阶段各自计时。不给timeout就等于交给操作系统,Windows 上可能挂几十秒;读超时按次计,流式下载时每读一块重新计时。连不上时抛ConnectTimeout,报错里直接写着connect timeout=2,本机打不通的 IP 实测 2.01s 抛出。在线练习站快且抖动小,很难稳定触发超时,连/读两段各自计时也用可控靶站/slow演示更准。req_timeout.py 1
2
3
4
5
6
7
8
9
10
11
12
13
14import time
import requests
base = "http://127.0.0.1:8010"
t0 = time.perf_counter()
requests.get(base + "/slow?ms=200", timeout=5)
print("慢 0.2 秒、timeout=5 -> 完成,耗时 %.2fs" % (time.perf_counter() - t0))
t0 = time.perf_counter()
try:
requests.get(base + "/slow?ms=2000", timeout=0.5)
except Exception as e:
print("慢 2 秒、timeout=0.5 -> 等了 %.2fs,%s" % (time.perf_counter() - t0, e))
t0 = time.perf_counter()
requests.get(base + "/slow?ms=1000", timeout=(1, 5))
print("元组 (1, 5) 打慢 1 秒的接口 -> 通过,总耗时 %.2fs" % (time.perf_counter() - t0))1
2
3慢 0.2 秒、timeout=5 -> 完成,耗时 0.20s
慢 2 秒、timeout=0.5 -> 等了 0.52s,Read timed out. (read timeout=0.5)
元组 (1, 5) 打慢 1 秒的接口 -> 通过,总耗时 1.02s
响应对象
-
一次拿全:状态码、头、正文、最终 URL、走过的跳数
history。状态码不是异常,ok只看小于 400,302 也算 ok;要让它抛得调raise_for_status(),只对 4xx/5xx 抛HTTPError,不抛时返回None,别写成r = r.raise_for_status()。响应头是大小写无关的CaseInsensitiveDict,取不到get给默认值不抛KeyError。req_status.py 1
2
3
4
5
6
7
8
9
10
11
12
13import requests
base = "http://127.0.0.1:8010"
r = requests.get(base + "/detail/1")
print("status_code=%s ok=%s history=%s" % (r.status_code, r.ok, r.history))
print("headers 类型=%s len(content)=%d len(text)=%d" % (
type(r.headers).__name__, len(r.content), len(r.text)))
for code in (302, 404, 451, 500):
rr = requests.get("%s/status/%d" % (base, code))
print("/status/%-3d -> %s ok=%s" % (code, rr.status_code, rr.ok))
try:
requests.get(base + "/no-such-page").raise_for_status()
except Exception as e:
print(type(e).__name__, e)1
2
3
4
5
6
7status_code=200 ok=True history=[]
headers 类型=CaseInsensitiveDict len(content)=804 len(text)=690
/status/302 -> 302 ok=True
/status/404 -> 404 ok=False
/status/451 -> 451 ok=False
/status/500 -> 500 ok=False
HTTPError 404 Client Error: Not Found for url: http://127.0.0.1:8010/no-such-page -
allow_redirects默认 True,自动跟到底并把每一跳塞进history;关掉就停在 302,Location交给你自己处理。超过 30 跳成环直接抛TooManyRedirects。req_redirect.py 1
2
3
4
5
6
7
8
9
10import requests
base = "http://127.0.0.1:8010"
r = requests.get(base + "/redirect")
print("默认跟随 -> %s 最终 url=%s history=%s" % (r.status_code, r.url, [h.status_code for h in r.history]))
r = requests.get(base + "/redirect", allow_redirects=False)
print("关闭跟随 -> %s Location=%r history=%s" % (r.status_code, r.headers.get("Location"), r.history))
try:
requests.get(base + "/redirect-loop")
except Exception as e:
print("成环 ->", type(e).__name__, e)1
2
3默认跟随 -> 200 最终 url=http://127.0.0.1:8010/detail/1 history=[302]
关闭跟随 -> 302 Location='/detail/1' history=[]
成环 -> TooManyRedirects Exceeded 30 redirects. -
json()内容不对时抛requests.exceptions.JSONDecodeError,它同时是RequestException和ValueError的子类。反爬最常见的伪装是接口 200 但返回 HTML,所以json()前一句raise_for_status()、后一句try。req_json_error.py 1
2
3
4
5
6
7
8
9
10
11
12import requests
api = requests.get("https://spa1.scrape.center/api/movie", params={"limit": 3}, timeout=20)
api.raise_for_status()
print("接口 .json() 顶层键:", list(api.json()), "count =", api.json()["count"])
try:
requests.get("https://ssr1.scrape.center/page/11", timeout=20).raise_for_status()
except requests.exceptions.HTTPError as e:
print("对 500 页面 raise_for_status ->", e)
try:
requests.get("https://ssr1.scrape.center/", timeout=20).json()
except requests.exceptions.JSONDecodeError as e:
print("对 HTML 调 .json() ->", type(e).__name__)1
2
3接口 .json() 顶层键: ['count', 'results'] count = 104
对 500 页面 raise_for_status -> 500 Server Error: Internal Error for url: https://ssr1.scrape.center/page/11
对 HTML 调 .json() -> JSONDecodeError -
raw是 urllib3 的响应对象,动过它之后再取.content不报错、只会少字节(静默截断),已经流完的响应再取.content才真的报错。二进制响应别取.text:PNG 的encoding是None。req_raw.py 1
2
3
4
5
6
7
8
9
10
11
12
13import requests
r = requests.get("http://127.0.0.1:8010/video", stream=True)
r.raw.read(10)
print("先动 raw 再取 content ->", len(r.content), "字节(完整应为 1048576)")
r2 = requests.get("http://127.0.0.1:8010/video", stream=True)
sum(len(c) for c in r2.iter_content(64 * 1024))
try:
r2.content
except Exception as e:
print(type(e).__name__, e)
img = requests.get("http://127.0.0.1:8010/img/1.png",
headers={"Referer": "http://127.0.0.1:8010/lazy"})
print("PNG: encoding=%r content 前 8 字节=%r" % (img.encoding, img.content[:8]))1
2
3先动 raw 再取 content -> 1048566 字节(完整应为 1048576)
RuntimeError The content for this response was already consumed
PNG: encoding=None content 前 8 字节=b'\x89PNG\r\n\x1a\n'
编码与中文
-
响应头里不写
charset时,requests 对text/*兜底给ISO-8859-1,中文必乱。两条修法:知道编码就r.encoding = "utf-8",不知道就r.encoding = r.apparent_encoding(靠charset_normalizer猜)。反过来也成立:服务端声明 GBK,硬设 utf-8 只会把对的改成错的。在线站 ssr1 的Content-Type: text/html; charset=utf-8写得很规矩,r.encoding直接是utf-8、不用猜也不乱码;靶站这两个端点是专门抹掉、写错 charset 来演示判定的。req_nocharset.py 1
2
3
4
5
6
7
8
9
10import requests
r = requests.get("http://127.0.0.1:8014/nocharset")
print("Content-Type=%r" % r.headers.get("Content-Type"))
print("encoding=%r apparent_encoding=%r" % (r.encoding, r.apparent_encoding))
print("默认 text=%r" % r.text)
r.encoding = "utf-8"
print("手设 utf-8 -> %r" % r.text)
g = requests.get("http://127.0.0.1:8014/gbk")
g.encoding = "utf-8"
print("GBK 硬设 utf-8 -> %r" % g.text)1
2
3
4
5Content-Type='text/plain'
encoding='ISO-8859-1' apparent_encoding='utf-8'
默认 text='ä¸\xadæ\x96\x87æ\xad£æ\x96\x87ï¼\x9aä¸\x89å\x9b½æ¼\x94ä¹\x89\n'
手设 utf-8 -> '中文正文:三国演义\n'
GBK 硬设 utf-8 -> '??????????' -
URL 里的中文交给 requests 会自动百分号编码(
params={"q": "三国演义"}得到?q=%E4%B8%89%E5%9B%BD%E6%BC%94%E4%B9%89);手拼给 urllib 就直接抛UnicodeEncodeError: 'ascii' codec can't encode characters in position 14-17: ordinal not in range(128)。
Session 会话
-
Session干三件事:cookie 自动收自动带、headers 配一次全程复用(update合入,单次请求的头不污染 Session)、连接复用。公开站quotes.toscrape.com一 GET/login就下发sessioncookie,同一个 Session 后面自动带上;另起一个就丢了。req_session_cookie.py 1
2
3
4
5
6
7
8
9
10
11
12
13import re
import requests
base = "https://quotes.toscrape.com"
sess = requests.Session()
sess.headers["User-Agent"] = "Mozilla/5.0 (Windows NT 10.0; Win64; x64) Chrome/156.0"
r = sess.get(base + "/login", timeout=20)
print("GET /login ->", r.status_code, "收到 cookie:", list(sess.cookies.get_dict()))
token = re.search(r'name="csrf_token" value="([^"]+)"', r.text).group(1)
bad = sess.post(base + "/login", data={"username": "admin", "password": "password"}, timeout=20)
print("不带 token ->", bad.status_code, "落在", bad.url.split("://", 1)[1], "已登录?", 'href="/logout"' in bad.text)
fresh = re.search(r'name="csrf_token" value="([^"]+)"', sess.get(base + "/login", timeout=20).text).group(1)
ok = sess.post(base + "/login", data={"username": "admin", "password": "password", "csrf_token": fresh}, timeout=20)
print("带 token ->", ok.status_code, "落在", ok.url.split("://", 1)[1], "已登录?", 'href="/logout"' in ok.text)1
2
3GET /login -> 200 收到 cookie: ['session']
不带 token -> 200 落在 quotes.toscrape.com/login 已登录? False
带 token -> 200 落在 quotes.toscrape.com/ 已登录? True -
「GET 拿 token、POST 提交」里 cookie 和 token 是一对:
quotes.toscrape.com把csrf_token藏进sessioncookie,提交时对不上不报错、只 200 退回登录页;token 要现从 HTML 抠、不能写死。本机靶站的/form+/submit(token 不对直接 403)是这套逻辑的可控版本。 -
连接复用:同一个 Session 连续两次请求拿到的是同一条底层连接对象,翻页抓取时每条都省一次 TLS 握手(不是省服务端的事);用完记得
close()或with requests.Session() as s:。下面用同一个 Session 按offset把 spa1 接口翻到底,靠results为空才停。req_crawl.py 1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20import requests
API = "https://spa1.scrape.center/api/movie"
sess = requests.Session()
sess.headers["User-Agent"] = "Mozilla/5.0 (Windows NT 10.0; Win64; x64) Chrome/156.0"
a = sess.get("https://ssr1.scrape.center/page/3", timeout=20)
b = sess.get("https://ssr1.scrape.center/page/3", timeout=20)
print("同一个 Session 两次请求同一条连接:", a.raw._connection is b.raw._connection)
offset, got = 0, []
while True:
r = sess.get(API, params={"limit": 20, "offset": offset}, timeout=20)
r.raise_for_status()
batch = r.json()["results"]
if not batch:
break
got += batch
offset += 20
print("用同一个 Session 翻完接口共 %d 条,count=%s" % (len(got), r.json()["count"]))
print("最后 3 条 id:", [m["id"] for m in got[-3:]])
p2 = sess.get(API, params={"page": 2, "limit": 10}, timeout=20).json()["results"]
print("接口只认 limit/offset,带 page=2 仍返回 id", [m["id"] for m in p2[:3]])1
2
3
4同一个 Session 两次请求同一条连接: True
用同一个 Session 翻完接口共 104 条,count=104
最后 3 条 id: [106, 107, 108]
接口只认 limit/offset,带 page=2 仍返回 id [1, 2, 3] -
想给某个前缀挂专用配置用
mount,get_adapter按最长前缀命中;HTTPAdapter默认max_retries=0,也就是默认不重试,下一节详谈。
自动重试
-
重试不是 requests 的功能,是把 urllib3 的
Retry挂到HTTPAdapter上再mount进 Session。靶站/flaky前两次固定回 503、第三次才 200;不带Retry时第一次 503 就原样返回,服务端只被打了 1 次。在线练习站的 500(ssr1/page/11)是稳定的,重试多少次都还是 500,要演示「前几次坏、后几次好」只能靠可控靶站。req_retry.py 1
2
3
4
5
6
7
8
9
10import time
import requests
from requests.adapters import HTTPAdapter
from urllib3.util.retry import Retry
s = requests.Session()
s.mount("http://", HTTPAdapter(max_retries=Retry(
total=3, status_forcelist=[500, 502, 503], allowed_methods=["GET"])))
t0 = time.perf_counter()
r = s.get("http://127.0.0.1:8014/flaky")
print("重试后 %s %s 耗时 %.2fs" % (r.status_code, r.json(), time.perf_counter() - t0))1
重试后 200 {'ok': True, 'attempts': 3} 耗时 0.00s
-
Retry的关键参数:参数 默认 作用 total10 总重试次数上限 connect/readNone 单独限制连接期、读取期重试 status_forcelistNone 哪些状态码算该重试 allowed_methods不含 POST 与 PATCH 哪些方法允许重试 backoff_factor0 退避系数,决定两次重试之间等多久 respect_retry_after_headerTrue 优先听服务端的 Retry-After -
退避第一次重试不等待,之后按
backoff_factor * 2 ** n增长。靶站一直回 503、backoff_factor=0.5,total从 1 加到 4 的实测耗时与服务端被打次数:req_backoff.py 1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22import time
import requests
from requests.adapters import HTTPAdapter
from urllib3.util.retry import Retry
help_url = "http://127.0.0.1:8014"
for total in (1, 2, 3, 4):
requests.get(help_url + "/reset")
session = requests.Session()
session.mount("http://", HTTPAdapter(max_retries=Retry(
total=total, status_forcelist=[503], backoff_factor=0.5,
allowed_methods=["GET"], respect_retry_after_header=False)))
t0 = time.perf_counter()
try:
session.get(help_url + "/status/503")
except Exception as exc:
hits = requests.get(help_url + "/hits").json()["hits"]
print("total=%d 耗时 %.2fs 次数=%d -> %s" % (
total, time.perf_counter() - t0, hits, type(exc).__name__))1
2
3
4total=1 耗时 0.00s 次数=2 -> RetryError
total=2 耗时 1.00s 次数=3 -> RetryError
total=3 耗时 3.01s 次数=4 -> RetryError
total=4 耗时 7.01s 次数=5 -> RetryError等待序列正好是 0 + 1.0 + 2.0 + 4.0。系数换成 0.1,形状不变、幅度等比缩小:0.00s、0.21s、0.61s。
-
allowed_methods默认不含 POST,怕的是重复下单,想重试 POST 必须显式写进去。次数用尽不静默返回,而是抛RetryError,它挂在RequestException下面但不属于ConnectionError。429 带Retry-After: 1默认会老实等,关掉立刻重试完。规则化的重试交给Retry,要换 UA、换代理的智能重试只能自己写循环。req_retry_exhaust.py 1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47import time
import requests
from requests.adapters import HTTPAdapter
from urllib3.util.retry import Retry
help_url = "http://127.0.0.1:8014"
def make_session(total, **kwargs):
requests.get(help_url + "/reset")
session = requests.Session()
session.mount("http://", HTTPAdapter(max_retries=Retry(total=total, **kwargs)))
return session
def hits():
return requests.get(help_url + "/hits").json()["hits"]
session = make_session(3, status_forcelist=[503])
t0 = time.perf_counter()
resp = session.post(help_url + "/status/503")
print("POST 全 503(不在 allowed_methods) %s 耗时 %.2fs 次数 %d" % (
resp.status_code, time.perf_counter() - t0, hits()))
session = make_session(3, status_forcelist=[503], allowed_methods=["GET"])
try:
session.get(help_url + "/status/503")
except Exception as exc:
err = exc
print("GET 同配置 次数 %d" % hits())
session = make_session(2, status_forcelist=[429], backoff_factor=0)
t0 = time.perf_counter()
try:
session.get(help_url + "/ratelimited")
except Exception as exc:
print("GET 429(默认,total=2) 等了 %.2fs -> %s" % (
time.perf_counter() - t0, type(exc).__name__))
session = make_session(2, status_forcelist=[429], backoff_factor=0,
respect_retry_after_header=False)
t0 = time.perf_counter()
try:
session.get(help_url + "/ratelimited")
except Exception as exc:
print("GET 429(关掉 Retry-After) 等了 %.2fs -> %s" % (
time.perf_counter() - t0, type(exc).__name__))
print("%s: %s" % (type(err).__name__, err))1
2
3
4
5POST 全 503(不在 allowed_methods) 503 耗时 0.00s 次数 1
GET 同配置 次数 4
GET 429(默认,total=2) 等了 2.01s -> RetryError
GET 429(关掉 Retry-After) 等了 0.00s -> RetryError
RetryError: HTTPConnectionPool(host='127.0.0.1', port=8014): Max retries exceeded with url: /status/503 (Caused by ResponseError('too many 503 error responses'))
代理
-
proxies的键只有http和https两个,值可以不同。本机 HTTP 正向代理会在转发的请求上补Via与X-Forwarded-For,走没走一测就知道;SOCKS5 只建隧道、不改 HTTP 头,所以Via是空的,它的证据在代理端日志(SOCKS5 CONNECT 127.0.0.1:8010)。req_proxy.py 1
2
3
4
5
6
7
8
9import requests
url = "http://127.0.0.1:8010/whoami"
for label, proxies in (
("不走代理", {}),
("HTTP 代理", {"http": "http://127.0.0.1:8011"}),
("SOCKS5", {"http": "socks5://127.0.0.1:8012"}),
):
h = requests.get(url, proxies=proxies).json()["headers"]
print("%-10s Via=%-18r X-Forwarded-For=%r" % (label, h.get("Via"), h.get("X-Forwarded-For")))1
2
3不走代理 Via=None X-Forwarded-For=None
HTTP 代理 Via='1.1 spider-proxy' X-Forwarded-For='127.0.0.1'
SOCKS5 Via=None X-Forwarded-For=None -
socks5://与socks5h://的差别是谁来解析域名:打一个解析不了的域名,前者在本地 DNS 就失败,后者是域名交给代理才被打回,要做代理端解析必须写socks5h。HTTPS 走 HTTP 代理时先CONNECT再 TLS,代理看不见明文,正文它看不到。req_socks_dns.py 1
2
3
4
5
6
7
8
9
10import requests
bad_url = "http://bad-host-for-dns.invalid/"
for label, scheme in (("socks5://", "socks5"), ("socks5h://", "socks5h")):
try:
requests.get(bad_url, proxies={"http": "%s://127.0.0.1:8012" % scheme}, timeout=10)
except Exception as exc:
mark = "Failed to establish a new connection: "
tail = str(exc).split(mark)[-1].rstrip(chr(34)+chr(41))
print("%-10s %s: ... %s%s" % (label, type(exc).__name__, mark, tail))1
2socks5:// ConnectionError: ... Failed to establish a new connection: [Errno 11001] getaddrinfo failed
socks5h:// ConnectionError: ... Failed to establish a new connection: 0x05: Connection refused -
账号密码写进代理 URL 就行,requests 直接造出
Proxy-Authorization,不等服务端来问;本机代理不校验凭据,407 挑战那条路未实测。req_proxy_https_auth.py 1
2
3
4
5
6
7from requests.adapters import HTTPAdapter
from requests.utils import get_auth_from_url
import requests
r = requests.get("https://127.0.0.1:8443/", verify=False, proxies={"https": "http://127.0.0.1:8011"})
print("HTTPS 走代理 ->", r.status_code, r.text.strip())
proxy = "http://lilei:password@127.0.0.1:8011"
print("拆出的凭据:", get_auth_from_url(proxy), "造出的头:", HTTPAdapter().proxy_headers(proxy))1
2HTTPS 走代理 -> 200 self-signed ok
拆出的凭据: ('lilei', 'password') 造出的头: {'Proxy-Authorization': 'Basic bGlsZWk6cGFzc3dvcmQ='} -
不写
proxies时 requests 会读环境变量HTTP_PROXY、HTTPS_PROXY、NO_PROXY,这套判断可用trust_env = False整体关掉。这里有个大坑:NO_PROXY写 CIDR 时 requests 认、urllib 不认,同一份配置两个库结论相反。判断依据是requests.utils.should_bypass_proxies();本机注册表代理开关ProxyEnable=0,那份例外列表对两个库都不生效,以本机实测为准。显式写proxies=时不看NO_PROXY,一定走代理。代理本身挂了抛ProxyError,它是ConnectionError的子类。req_proxy_env.py 1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41import json
import os
import urllib.request
import requests
proxy = "http://127.0.0.1:8011"
targets = ("http://127.0.0.1:8010/whoami", "http://localhost:8010/whoami")
def req_via(url):
return requests.get(url).json()["headers"].get("Via")
def urllib_via(url):
with urllib.request.urlopen(url, timeout=10) as resp:
return json.loads(resp.read())["headers"].get("Via")
print("--- NO_PROXY 只写 CIDR")
os.environ["HTTP_PROXY"] = proxy
os.environ["NO_PROXY"] = "127.0.0.0/8"
for url in targets:
print(" 目标 %s" % url[:-len("/whoami")])
print(" requests Via=%-18r urllib Via=%r" % (req_via(url), urllib_via(url)))
print("\n环境 NO_PROXY='127.0.0.1,localhost' HTTP_PROXY='%s'" % proxy)
os.environ["NO_PROXY"] = "127.0.0.1,localhost"
print(" 只靠环境变量 Via=%r" % req_via(targets[0]))
print(" 显式 proxies= Via=%r" % requests.get(
targets[0], proxies={"http": proxy}).json()["headers"].get("Via"))
session = requests.Session()
session.trust_env = False
print(" trust_env=False+显式 Via=%r" % session.get(
targets[0], proxies={"http": proxy}).json()["headers"].get("Via"))
try:
requests.get(targets[0], proxies={"http": "http://127.0.0.1:8099"})
except Exception as exc:
head = str(exc).split(" with url:")[0]
print("%s: %s ... (Caused by ProxyError('Unable to connect to proxy', NewConnectionError(...)))" % (
type(exc).__name__, head))1
2
3
4
5
6
7
8
9
10
11--- NO_PROXY 只写 CIDR
目标 http://127.0.0.1:8010
requests Via=None urllib Via='1.1 spider-proxy'
目标 http://localhost:8010
requests Via='1.1 spider-proxy' urllib Via='1.1 spider-proxy'
环境 NO_PROXY='127.0.0.1,localhost' HTTP_PROXY='http://127.0.0.1:8011'
只靠环境变量 Via=None
显式 proxies= Via='1.1 spider-proxy'
trust_env=False+显式 Via='1.1 spider-proxy'
ProxyError: HTTPConnectionPool(host='127.0.0.1', port=8099): Max retries exceeded ... (Caused by ProxyError('Unable to connect to proxy', NewConnectionError(...)))
证书校验
-
本机 8443 跑着一个自签证书的 HTTPS 服务。默认严格校验,自签证书直接拒绝;
verify=指到那张自签证书就认了,指到 certifi 的 CA 包还是不认——certifi 是 requests 自带的 CA 文件,跟系统证书库不是一回事。verify=路径写错报的是OSError不是SSLError。req_verify.py 1
2
3
4
5
6
7
8
9
10
11import certifi
import requests
try:
requests.get("https://127.0.0.1:8443/")
except Exception as e:
print("默认:", type(e).__name__, str(e)[:80])
print("指到自签证书:", requests.get("https://127.0.0.1:8443/", verify="selfcert.pem").status_code)
try:
requests.get("https://127.0.0.1:8443/", verify=certifi.where())
except Exception as e:
print("指到 certifi:", type(e).__name__)1
2
3
4默认: SSLError HTTPSConnectionPool(host='127.0.0.1', port=8443): Max retries exceeded with u
指到自签证书: 200
指到 certifi: SSLError
OSError: Could not find a suitable TLS CA certificate bundle, invalid path: ...\missing.pem -
verify=False能过但每次都告警,消音不改问题只是让日志干净。全局改校验有两处:Session.verify,以及环境变量REQUESTS_CA_BUNDLE(CURL_CA_BUNDLE也认,本机实测都返回 200)。req_insecure_warning.py 1
2
3
4
5
6
7
8
9
10import warnings
import requests
url = "https://127.0.0.1:8443/"
with warnings.catch_warnings(record=True) as caught:
warnings.simplefilter("always")
requests.get(url, verify=False)
for item in caught:
print("%s: %s" % (item.category.__name__, item.message))1
InsecureRequestWarning: Unverified HTTPS request is being made to host '127.0.0.1'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings
req_silence_warning.py 1
2
3
4
5
6
7import urllib3
import requests
urllib3.disable_warnings(urllib3.exceptions.InsecureRequestWarning)
print("verify=False:", requests.get("https://127.0.0.1:8443/", verify=False).status_code)
s = requests.Session()
s.verify = "selfcert.pem"
print("Session.verify=CA:", s.get("https://127.0.0.1:8443/").status_code)1
2verify=False: 200
Session.verify=CA: 200 -
cert=是客户端证书,用于双向 TLS,写法是cert=("client.pem", "client.key");本机 8443 没开双向校验,这条未实测。协议头不对时报错在连接层之前:req_url_error.py 1
2
3
4
5
6
7import requests
for bad in ("127.0.0.1:8010/whoami", ""):
try:
requests.get(bad)
except Exception as exc:
print("%s: %s" % (type(exc).__name__, exc))1
2InvalidSchema: No connection adapters were found for '127.0.0.1:8010/whoami'
MissingSchema: Invalid URL '': No scheme supplied. Perhaps you meant https://?
上传与下载
-
files=就是 multipart 上传。Request(...).prepare()不用发请求就能看它拼成什么样;三元组是(文件名, 内容, Content-Type),boundary 每次随机;data里的普通字段和文件同包混发,下面是desc字段与file文件同包的样子。真正的文件用文件对象传,别read()整个进内存。req_multipart.py 1
2
3
4
5
6
7import requests
req = requests.Request(
"POST", "http://127.0.0.1:8010/api/books",
files={"file": ("books.csv", b"name,price\n1,2\n", "text/csv")},
data={"desc": "带一个普通字段"}).prepare()
print(req.headers["Content-Type"])
print(req.body.decode("utf-8"))1
2
3
4
5
6
7
8
9
10
11
12
13multipart/form-data; boundary=341fc0723937e1ad9b27c8d10979dbd3
--341fc0723937e1ad9b27c8d10979dbd3
Content-Disposition: form-data; name="desc"
带一个普通字段
--341fc0723937e1ad9b27c8d10979dbd3
Content-Disposition: form-data; name="file"; filename="books.csv"
Content-Type: text/csv
name,price
1,2
--341fc0723937e1ad9b27c8d10979dbd3--文件对象上传实测:本地 136 字节,服务端收到 281 字节,多出来的是 boundary 和
Content-Disposition的封包开销,不是内容变了。 -
下载要用
stream=True配iter_content,否则.content会把整包先堆进内存。chunk_size=8192是每次读多少字节,与服务端怎么发无关,靶站按 64KB 写、这里按 8KB 收,落盘字节数一样。req_download.py 1
2
3
4
5
6
7
8
9
10
11
12
13import time
import requests
url = "http://127.0.0.1:8010/video"
t0 = time.perf_counter()
with requests.get(url, stream=True, timeout=30) as resp:
resp.raise_for_status()
total = int(resp.headers["Content-Length"])
got = 0
with open("video.bin", "wb") as fp:
for chunk in resp.iter_content(chunk_size=8192):
fp.write(chunk)
got += len(chunk)
print("写完 %d 字节,与 Content-Length 一致: %s,耗时 %.2fs" % (got, got == total, time.perf_counter() - t0))1
写完 1048576 字节,与 Content-Length 一致: True,耗时 0.09s
-
断点续传要服务端配合。先探
Accept-Ranges,再发Range:206 是听明白了,200 就是当你没说过。靶站/video不支持 Range,换到支持的服务上同一个请求就是 206;正确写法是先看本地已有多少字节,再从那个偏移请求、剩下的追加写,拼出来的文件与整包逐字节相同。req_range.py 1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18import requests
base = "http://127.0.0.1:8010"
resp = requests.get(base + "/video")
print("靶站 /video Accept-Ranges=%s" % resp.headers.get("Accept-Ranges"))
resp = requests.get(base + "/video", headers={"Range": "bytes=0-99"})
print("带 Range 请求 -> %s,Content-Range=%s,实收 %d 字节" % (
resp.status_code, resp.headers.get("Content-Range"), len(resp.content)))
print("\n支持 Range 的服务:")
url = "http://127.0.0.1:8014/file"
resp = requests.get(url, headers={"Range": "bytes=0-"})
print("请求 Range: bytes=0- %s Content-Range=%r 收到 %d 字节" % (
resp.status_code, resp.headers.get("Content-Range"), len(resp.content)))
resp = requests.get(url, headers={"Range": "bytes=10-19"})
print("Range: bytes=10-19 %s %r" % (resp.status_code, resp.content))
resp = requests.get(url, headers={"Range": "bytes=100000-200000"})
print("越界 Range %s %s" % (resp.status_code, resp.text.strip()))1
2
3
4
5
6
7靶站 /video Accept-Ranges=None
带 Range 请求 -> 200,Content-Range=None,实收 1048576 字节
支持 Range 的服务:
请求 Range: bytes=0- 206 Content-Range='bytes 0-99999/100000' 收到 100000 字节
Range: bytes=10-19 206 b'\x82\x8f\x9c\xa9\xb6\xc3\xd0\xdd\xea\xf7'
越界 Range 416 范围越界 -
防盗链靠
Referer判断,不带就 403。requests 不会像浏览器那样自动补Referer,Session 里也不会,必须自己写。req_referer.py 1
2
3
4
5
6
7
8import requests
base = "http://127.0.0.1:8010"
print("不带 Referer ->", requests.get(base + "/img/1.png").status_code)
s = requests.Session()
s.get(base + "/lazy")
print("Session 取过 /lazy 再取图 ->", s.get(base + "/img/1.png").status_code)
r = s.get(base + "/img/1.png", headers={"Referer": base + "/lazy"})
print("手动补 Referer ->", r.status_code, r.headers["Content-Type"], len(r.content), "字节")1
2
3不带 Referer -> 403
Session 取过 /lazy 再取图 -> 403
手动补 Referer -> 200 image/png 162 字节 -
Transfer-Encoding: chunked的响应 requests 拆好给你(第 0 块到第 3 块共 4 块),但iter_lines()交出的是bytes,要中文就chunk.decode("utf-8")。慢速大响应配stream=True时,读超时是每块单独计时。
异常体系
-
requests 的异常全部从
RequestException派生,而RequestException本身继承OSError,实测__mro__前四层是(<class 'requests.exceptions.RequestException'>, <class 'OSError'>, <class 'Exception'>, <class 'BaseException'>)。常用几类的从属关系,是实测issubclass的结果:异常 父类 什么时候抛 ConnectionErrorRequestException DNS 解析失败、连接被拒、代理挂了 ProxyErrorConnectionError 连不上代理 SSLErrorConnectionError 证书不过 TimeoutRequestException 超时的父类 ConnectTimeoutTimeout 且 ConnectionError 连接阶段超时 ReadTimeoutTimeout 等响应体超时 TooManyRedirectsRequestException 超过 30 跳 HTTPErrorRequestException raise_for_status()抛的就是它RetryErrorRequestException Retry次数用尽MissingSchema/InvalidSchemaRequestException URL 没写协议、协议不支持 JSONDecodeErrorRequestException 且 ValueError .json()解析失败ChunkedEncodingErrorRequestException 分块流中途断 RetryError不在ConnectionError下面,except requests.ConnectionError接不住它,这是最容易踩的一条。半路断流抛ChunkedEncodingError: Response ended prematurely,假 gzip 抛ContentDecodingError: ('Received response with content-encoding: gzip, but failed to decode it.', ...),本机都触发过。 -
except必须从具体写到一般,写反了后面的分支永远进不去。ConnectTimeout同时是ConnectionError和Timeout的子类,落到哪个分支取决于书写顺序。req_except_order.py 1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17import requests
def right_order(url):
try:
resp = requests.get(url, timeout=0.5)
resp.raise_for_status()
return resp.status_code
except requests.exceptions.HTTPError as e:
return "HTTPError(状态码不对):%s" % e
except requests.exceptions.Timeout:
return "Timeout(超时)"
except requests.exceptions.ConnectionError:
return "ConnectionError(连不上 / DNS / 代理)"
except requests.exceptions.RequestException as e:
return "兜底:%s" % type(e).__name__
for u in ("http://127.0.0.1:8010/status/500", "http://nonexistent-probe-test.invalid/",
"http://127.0.0.1:8010/slow?ms=5000", "http://127.0.0.1:8010/whoami"):
print("%-45s -> %s" % (u, right_order(u)))1
2
3
4http://127.0.0.1:8010/status/500 -> HTTPError(状态码不对):500 Server Error ...
http://nonexistent-probe-test.invalid/ -> ConnectionError(连不上 / DNS / 代理)
http://127.0.0.1:8010/slow?ms=5000 -> Timeout(超时)
http://127.0.0.1:8010/whoami -> 200
验证码识别
-
靶站
/login有一张 4 位数字验证码,/captcha出 PNG,页面里还把当前答案打出来,正好核对。取图就是普通请求,但必须用同一个 Session,它认的是 cookie 里的sid,换个 Session 图和答案就对不上。本机实测:答案8100,图片尺寸 (102, 42) RGB,PNG 179 字节。换到在线的captcha1.scrape.center就取不到图:首页只有 1082 字节的单页外壳,/captcha、/verify_code、/code这些路径 requests 拿回来全是text/html、没有一张image/png,验证码得靠浏览器执行前端脚本才画得出来,所以取图识别这套仍用能对照答案的本机/captcha。 -
Tesseract 是外部程序,Python 这边只有
pytesseract这层壳(pip install pytesseract pillow,再设pytesseract.pytesseract.tesseract_cmd指向 exe)。本机语言包只有eng和osd,没有chi_sim,识别中文必须另外下载chi_sim.traineddata放进tessdata目录,否则一模就报Failed loading language 'chi_sim'。最有用的参数是--psm(分割模式)和tessedit_char_whitelist(限定字符集):数字验证码不给白名单时整张返回空,给--psm 7 -c tessedit_char_whitelist=0123456789后8100就认对了。req_tesseract_captcha.py 1
2
3
4
5
6
7
8
9
10
11
12
13
14import io
import pytesseract
import requests
from PIL import Image
pytesseract.pytesseract.tesseract_cmd = r"C:\Program Files\Tesseract-OCR\tesseract.exe"
img = Image.open(io.BytesIO(requests.get(
"http://127.0.0.1:8010/img/8100.png",
headers={"Referer": "http://127.0.0.1:8010/lazy"}).content))
wl = "--psm 7 -c tessedit_char_whitelist=0123456789"
print("%s %s" % (pytesseract.get_tesseract_version(), pytesseract.get_languages(config="")))
print(repr(pytesseract.image_to_string(img).strip()))
print(repr(pytesseract.image_to_string(img, config=wl).strip()))1
2
35.4.0.20240606 ['eng', 'osd']
''
'8100' -
一张说明不了问题,准确率要用一批图统计:四种配置(原图 psm7、放大 4 倍二值化后 psm7、预处理后 psm13、ddddocr),同一批 20 张验证码跑两轮(换一批图重跑同一段脚本):
req_captcha_batch.py 1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53import io
import random
import time
import ddddocr
import pytesseract
import requests
from PIL import Image
pytesseract.pytesseract.tesseract_cmd = r"C:\Program Files\Tesseract-OCR\tesseract.exe"
base = "http://127.0.0.1:8010"
ref = {"Referer": base + "/lazy"}
wl7 = "--psm 7 -c tessedit_char_whitelist=0123456789"
wl13 = "--psm 13 -c tessedit_char_whitelist=0123456789"
def code_png(code):
return requests.get("%s/img/%04d.png" % (base, code), headers=ref).content
def upsample(png):
small = Image.open(io.BytesIO(png))
big = small.resize((small.width * 4, small.height * 4), Image.LANCZOS)
return big.convert("L").point(lambda p: 255 if p > 128 else 0)
ocr = ddddocr.DdddOcr(show_ad=False)
rnd = random.Random(42)
batches = [[rnd.randint(1111, 9999) for _ in range(20)] for _ in range(2)]
configs = [
("Tesseract 原图 psm7", lambda p: pytesseract.image_to_string(
Image.open(io.BytesIO(p)), config=wl7).strip()),
("Tesseract 预处理后 psm7", lambda p: pytesseract.image_to_string(
upsample(p), config=wl7).strip()),
("Tesseract 预处理后 psm13", lambda p: pytesseract.image_to_string(
upsample(p), config=wl13).strip()),
("ddddocr", ocr.classification),
]
for index, codes in enumerate(batches, start=1):
print("%s批:" % ("第一" if index == 1 else "第二"))
answers = ["%04d" % c for c in codes]
pngs = [code_png(c) for c in codes]
for label, recognize in configs:
hits, spent = 0, 0.0
for png, ans in zip(pngs, answers):
t0 = time.perf_counter()
hits += recognize(png) == ans
spent += time.perf_counter() - t0
print("%-26s 对 %2d/%d = %5.1f%% 平均 %.4fs/张" % (
label, hits, len(answers), hits / len(answers) * 100, spent / len(answers)))
if index == 1:
print()1
2
3
4
5
6
7
8
9
10
11第一批:
Tesseract 原图 psm7 对 5/20 = 25.0% 平均 0.1070s/张
Tesseract 预处理后 psm7 对 7/20 = 35.0% 平均 0.1094s/张
Tesseract 预处理后 psm13 对 4/20 = 20.0% 平均 0.1139s/张
ddddocr 对 9/20 = 45.0% 平均 0.0047s/张
第二批:
Tesseract 原图 psm7 对 2/20 = 10.0% 平均 0.1013s/张
Tesseract 预处理后 psm7 对 1/20 = 5.0% 平均 0.1147s/张
Tesseract 预处理后 psm13 对 5/20 = 25.0% 平均 0.1084s/张
ddddocr 对 6/20 = 30.0% 平均 0.0050s/张结论:Tesseract 在 5% 到 35% 之间飘,预处理有时帮倒忙——靶站的数字是 3×5 像素块放大出来的,笔画没有真实字体的形状,Tesseract 的字形模型对不上;ddddocr 稳定在 30% 到 45%,而且快一个数量级。调参的收益远小于换引擎。共同的失败模式是丢位和串位,ddddocr 还会把
8认成日、把0认成D。 -
ddddocr 是本地推理,
pip install ddddocr装完就能跑,不用网络。首次构造加载模型约 0.1 秒,之后单张 5 到 7 毫秒。show_ad=False不是可选项:默认构造会往 stdout 打四行项目宣传(欢迎使用ddddocr……之类),混进你的结果里。ocr_ddddocr.py 1
2
3
4
5
6
7import time
import ddddocr
t0 = time.perf_counter()
ocr = ddddocr.DdddOcr(show_ad=False)
print("模型加载 %.2fs" % (time.perf_counter() - t0))
with open("captcha.png", "rb") as fp:
print(ocr.classification(fp.read()))1
2模型加载 0.09s
日1Dd同一张
captcha.png(答案 8100)Tesseract 加白名单能认对,ddddocr 反而错了:8认成日、两个0认成D和d,两家在同样的图上各错各的。实测体积:模型
common.onnx51.58 MB,整个包 85.33 MB,连带onnxruntime45.8 MB、opencv-python112.4 MB、numpy31.4 MB。DdddOcr在 1.6.1 的签名比老笔记多了ocr/det/old/beta等参数;use_gpu与device_id本机没 CUDA,未实测;old=True与beta=True在一批 20 张上分别是 30% 与 25%,与默认模型没有实质差别。classification(probability=True)返回含text、confidence的 dict,置信度低时可先看confidence再决定换不换图。 -
端到端闭环:取图、识别、填回表单提交。验证码错时靶站回 401,对的时候 200;识别错就换一张重认,靶站验证码一次有效,重试必须重新取图。用户名和密码是靶站占位值,真实项目要从环境变量取。
captcha_login.py 1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19import re
import ddddocr
import requests
base = "http://127.0.0.1:8010"
ocr = ddddocr.DdddOcr(show_ad=False)
session = requests.Session()
session.headers["User-Agent"] = "Mozilla/5.0 (Windows NT 10.0) Chrome/156.0"
def one_round():
session.cookies.clear()
session.get(base + "/login")
guess = ocr.classification(session.get(base + "/captcha").content)
resp = session.post(base + "/login", data={"username": "lilei", "password": "password", "captcha": guess})
msg = re.search(r"<h1>([^<]+)</h1>", resp.text).group(1)
return guess, resp.status_code, msg
for attempt in range(1, 12):
guess, code, msg = one_round()
print("第 %2d 次:识别 %-8r -> %s %s" % (attempt, guess, code, msg))
if code == 200:
break1
2第 1 次:识别 '4367' -> 401 登录失败
第 2 次:识别 '2959' -> 200 登录成功 -
最后一层现实:老笔记里那套「识别验证码就能登录」的写法,只适用于把验证码当唯一关卡的站点;密码填错、验证码照抄页面正确答案也照样 401,风控、签名、短信、滑块任何一关都在 OCR 之外。
打码平台与云 OCR
- 这两类都是把图传出去、别人帮你认,需要账号、密钥、余额,本机没法验,以下内容未实测,只留判断依据。打码平台:注册拿到 appid 与 appkey,把验证码字节传上去,人工或模型给出文本,按次计费,适合识别量小、验证码种类杂、又不想自己养模型的场景;老笔记里的云打码这类服务,接口文档和域名如今是否还在,本次没有核实。
- 云厂商 OCR:几家大厂都有通用文字识别接口,要创建应用拿 AppID、API Key、Secret Key,SDK 走
pip install baidu-aip一类;本机python -m pip show baidu-aip返回WARNING: Package(s) not found: baidu-aip,没装这个包也没申请 key。和 ddddocr 怎么选看三条判断:量级上本地模型按张毫秒级、零成本,云 OCR 有 QPS 限制和费用;数据上验证码属于敏感流量,愿不愿意出站是个决策点;精度上通用文字识别、尤其中文和复杂背景是云服务的强项,ddddocr 的通用模型只覆盖常见字符集。老笔记里那段编造的调用代码和返回样例,本次全部删掉,没跑过的东西不该写成能跑的样子。
踩过的坑
- 默认 UA 是
python-requests/2.34.2,正经站点第一步就拦你,Session 里headers.update一次改好;timeout一个数字不等于总时长,它同时是连接和读两个阶段各自计时,不给就等于交给操作系统。 - 状态码不是异常,
raise_for_status()才是;r.ok把 302 也算成功。服务端不写charset时.text必乱,先试apparent_encoding;二进制响应别碰.text。 - 嵌套 dict 传给
params会被展开成只剩键名;json=与data=同时给,data赢,json被静默忽略。 RetryError不是ConnectionError的子类,except requests.ConnectionError接不住它;except要从具体写到一般。- 显式
proxies=不受NO_PROXY影响;不设时环境变量判定优先,而 requests 认 CIDR、urllib 不认。verify=False每次都刷InsecureRequestWarning,disable_warnings只是消音;requests 不会自动补Referer,防盗链图片必须自己带。 - 流式响应先动了
raw再取.content不报错只会少字节,这种静默截断比抛异常难查;批量跑本地靶站时别每次新建 Session,Windows 临时端口会先耗尽报WinError 10048,复用连接、只清 cookie 就行。
参考
- Requests 官方文档,本机实测时可访问,返回 200。
- 公开练习站 ssr1/scrape.center、spa1.scrape.center 与 quotes.toscrape.com,本篇抓数据的示例都打的是这几个在线站。
- PyPI 上的 ddddocr,元数据写着
requires_python >= 3.10,本机 3.14.10 实测可跑;老笔记里「3.8 到 3.10」两头都过时了。 - Tesseract OCR 的 Windows 安装包:本机
github.com连不上报ConnectTimeout,安装位置与版本以本机tesseract --version输出为准。