爬虫-Selenium和PhantomJS

这篇讲什么

  • 主线是 Selenium 4.52.0 + 无头 Chrome(--headless=new)在 Python 3.14.10 上的用法:驱动从哪来、页面怎么开、元素怎么定位、等待怎么配、表单怎么提交、JS 怎么执行,最后用公开练习站 spa1 的前端渲染页跑一遍「等卡片出现再取数」。
  • scrape.center 是《Python3 网络爬虫开发实战》作者搭的公开练习站,专门拿来练渲染和反爬;本篇用的是它的 spa1 子站,页面数据由前端 JS 现取现渲染,天然适合练等待。
  • 懒加载、上传、iframe、alert、多窗口、代理这些机制类演示仍留在本机靶站(http://127.0.0.1:8010,另有一个本机辅助页面服务):在线练习站不会给你「第 1.2 秒才弹出的 alert」这种可控条件,也没有 data-src 懒加载。
  • PhantomJS 只留一小节给「它已经装不上了」的实测判据;所有数字和报错都逐字来自本机输出,没跑过的一律标「未实测」,只当线索不当结论。

装与驱动

1
2
pip install selenium
python -c "import selenium; print(selenium.__version__)"
  • 本机这一轮是 Python 3.14.10 + selenium 4.52.0,4.x 和 3.x 的 API 差别大到旧教程基本抄不动,所以第一件事是把版本打出来。

冷启动与热启动

  • 第一次跑(缓存目录被临时挪走)要联网下载驱动,之后每次启动都省掉这一步。
driver_cache_cold.py
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
import os
import shutil
import tempfile
import time

from selenium import webdriver
from selenium.webdriver.chrome.options import Options

BASE = "http://127.0.0.1:8010/list"
DEFAULT_CACHE = os.path.expandvars(r"%USERPROFILE%\.cache\selenium")
opts = Options()
opts.add_argument("--headless=new")

tmp = tempfile.mkdtemp(prefix="selcache-")
os.environ["SE_CACHE_PATH"] = tmp
try:
print("缓存目录已挪走:", os.environ["SE_CACHE_PATH"] != DEFAULT_CACHE)
t0 = time.time()
driver = webdriver.Chrome(options=opts)
print("冷启动建 driver 耗时: %.2fs" % (time.time() - t0))
driver.get(BASE)
print("title:", driver.title)
print("driver 本地服务端口:", driver.service.service_url.rsplit(":", 1)[-1])
driver.quit()
print("重启后目录:", sorted(os.listdir(os.path.join(tmp, "chromedriver", "win64"))))
finally:
os.environ.pop("SE_CACHE_PATH", None)
shutil.rmtree(tmp, ignore_errors=True)

print("热启动对照:")
t0 = time.time()
driver = webdriver.Chrome(options=opts)
print("第二次建 driver 耗时: %.2fs" % (time.time() - t0))
driver.get(BASE)
print("title:", driver.title)
driver.quit()
1
2
3
4
5
6
7
8
缓存目录已挪走: True
冷启动建 driver 耗时: 12.85s
title: 书单列表
driver 本地服务端口: 13462
重启后目录: ['156.0.8078.80']
热启动对照:
第二次建 driver 耗时: 1.05s
title: 书单列表
  • 冷启动 12.85 秒、热启动 1.05 秒,差的是下载环节;缓存结构是 chromedriver/win64/<版本号>/chromedriver.exe,可以并存多个版本。握手时 chromedriver 会起一个本地 HTTP 服务,实测那次端口是 13462。

三种建 driver 的写法

chrome_first.py
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
from selenium import webdriver
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.chrome.service import Service

URL = "http://127.0.0.1:8010/list"
opts = Options()
opts.add_argument("--headless=new")

# 写法一:什么都不指定,驱动交给 Selenium Manager 自己找、自己下
driver = webdriver.Chrome(options=opts)
try:
driver.get(URL)
print("1 默认写法:", driver.title)
finally:
driver.quit()

# 写法二:显式给 Service,把驱动文件路径钉死
path = r"C:\Users\Loong\.cache\selenium\chromedriver\win64\156.0.8078.80\chromedriver.exe"
driver = webdriver.Chrome(service=Service(executable_path=path), options=opts)
try:
driver.get(URL)
print("2 Service 写法:", driver.title)
finally:
driver.quit()

# 写法三:with 语句块,离开代码块自动 quit()
with webdriver.Chrome(options=opts) as driver:
driver.get(URL)
print("3 with 写法跑通, current_url =", driver.current_url)
try:
print(" 退出后还能读 title:", driver.title)
except Exception as exc:
print(" with 退出后 driver 已不可用: %s" % type(exc).__name__)
1
2
3
4
1 默认写法: 书单列表
2 Service 写法: 书单列表
3 with 写法跑通, current_url = http://127.0.0.1:8010/list
with 退出后 driver 已不可用: MaxRetryError
  • 三种写法实测都能跑通,区别只在「驱动从哪来、谁负责关」:
    • 写法一(默认):最省事,Selenium Manager 按浏览器版本自己找驱动,找不到就联网下载并缓存到 C:\Users\<用户>\.cache\selenium\;日常和教学都用这个。
    • 写法二(显式 Service):把驱动文件路径钉死,适合离线机器、CI 里固定版本,或者你想用手动下载的驱动;代价是浏览器升级后路径里的版本号要对得上。
    • 写法三(with):省掉 finally: driver.quit(),但退出代码块后 driver 立刻不可用——上面那句读 title 实测抛 MaxRetryError,所以取数据要在块内取完。

换别的浏览器

  • Chrome、Edge、Firefox 本机都跑通了,代价是启动慢不少。
edge_firefox_cold.py
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
import os
import shutil
import time

from selenium import webdriver
from selenium.webdriver.edge.options import Options as EdgeOptions
from selenium.webdriver.firefox.options import Options as FirefoxOptions

BASE = "http://127.0.0.1:8010/list"
tmp = os.path.join(os.environ["TEMP"], "selcold2")
shutil.rmtree(tmp, ignore_errors=True)
os.makedirs(tmp)
os.environ["SE_CACHE_PATH"] = tmp
try:
e = EdgeOptions()
e.add_argument("--headless=new")
t0 = time.time()
d = webdriver.Edge(options=e)
print("Edge 启动耗时 %.2fs" % (time.time() - t0))
try:
d.get(BASE)
print("Edge title: %s | 版本: %s" % (d.title, d.capabilities["browserVersion"]))
finally:
d.quit()

f = FirefoxOptions()
f.add_argument("--headless")
t0 = time.time()
d = webdriver.Firefox(options=f)
print("Firefox 启动耗时 %.2fs" % (time.time() - t0))
try:
d.get(BASE)
print("Firefox title: %s | 版本: %s"
% (d.title, d.capabilities["browserVersion"]))
print("Firefox 驱动:", d.capabilities.get("moz:geckodriverVersion"))
finally:
d.quit()
finally:
os.environ.pop("SE_CACHE_PATH", None)
shutil.rmtree(tmp, ignore_errors=True)
1
2
3
4
5
Edge 启动耗时 6.11s
Edge title: 书单列表 | 版本: 156.0.4314.58
Firefox 启动耗时 8.57s
Firefox title: 书单列表 | 版本: 159.0.1
Firefox 驱动: 0.37.1
  • Chrome 热启动 1 秒左右,Edge 6.11 秒、Firefox 8.57 秒:频繁起停浏览器的任务用 Chrome 更划算。
  • Safari / Ie / WebKitGTK / WPEWebKit / Remote 这几个对象在本机能导入,未实测启动(Windows 装不了 Safari,Remote 要另起 Grid 节点)。

PhantomJS 已死

老写法、包目录和那几个包

phantomjs_dead.py
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
import json
import os
import urllib.request

import selenium
from selenium import webdriver
from selenium.webdriver.chrome.options import Options

print("selenium:", selenium.__version__)

# 老写法的类名、模块名、参数名都用字符串拼出来,免得这份文档里再出现死写法本身
CLS = "Phantom" + "JS"
PKG = "phantom" + "js"
MOD = "selenium.webdriver." + PKG + ".webdriver"
KW = "desired" + "_" + "capabilities"

print("-- 1. 三种老写法 --")
try:
# __import__ 的 fromlist 不会替你做「取名字」那一步,所以按老语句原样编译执行
exec("from selenium.webdriver import " + CLS)
print("from selenium.webdriver import %s -> 成功" % CLS)
except Exception as exc:
print("from selenium.webdriver import %s -> %s" % (CLS, type(exc).__name__))
print(" %s" % exc)

try:
__import__(MOD)
print("老模块还能导入")
except Exception as exc:
print("from %s import WebDriver -> %s" % (MOD, type(exc).__name__))
print(" %s" % exc)

try:
getattr(webdriver, CLS)()
except Exception as exc:
print("webdriver.%s() -> %s" % (CLS, type(exc).__name__))
print(" %s" % exc)

print("\n-- 2. selenium.webdriver 包目录下还剩哪些浏览器 --")
pkg = os.path.dirname(webdriver.__file__)
entries = sorted(x for x in os.listdir(pkg)
if os.path.isdir(os.path.join(pkg, x)) and not x.startswith("_"))
print(entries)
print("有没有 %s 目录:" % PKG, PKG in entries)
print("webdriver.%s 有吗: %s | webdriver.Opera 有吗: %s"
% (CLS, hasattr(webdriver, CLS), hasattr(webdriver, "Opera")))

print("\n-- 3. Options/%s 这些老参数 --" % KW)
print("Chrome Options 还有 %s 属性吗:" % KW, hasattr(Options(), KW))
try:
webdriver.Chrome(**{KW: {}})
print("带 %s 建 driver -> 成功" % KW)
except Exception as exc:
print("webdriver.Chrome(%s={}) -> %s | %s"
% (KW, type(exc).__name__, str(exc).splitlines()[0]))
try:
getattr(Options(), KW)
except Exception as exc:
print("options.%s -> %s | %s" % (KW, type(exc).__name__, str(exc).splitlines()[0]))

print("\n-- 4. PyPI 上那几个 %s 包的元数据 --" % PKG)
for name in (PKG, "py" + PKG, "selenium-" + PKG + "-driver"):
try:
with urllib.request.urlopen("https://pypi.org/pypi/%s/json" % name) as r:
d = json.loads(r.read().decode("utf-8"))
ver = d["info"]["version"]
rel = d["releases"].get(ver, [{}])
print("%-26s 最新版 %-10s 上传于 %s requires_python=%r yanked=%s 共 %d 个版本"
% (name, ver, rel[0].get("upload_time", "?"),
d["info"].get("requires_python", ""), rel[0].get("yanked", False),
len(d["releases"])))
except Exception as exc:
code = getattr(exc, "code", type(exc).__name__)
print("%-26s HTTP %s" % (name, code))
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
selenium: 4.52.0
-- 1. 三种老写法 --
from selenium.webdriver import PhantomJS -> ImportError
cannot import name 'PhantomJS' from 'selenium.webdriver' (D:\ProgramData\anaconda3\envs\spider\Lib\site-packages\selenium\webdriver\__init__.py)
from selenium.webdriver.phantomjs.webdriver import WebDriver -> ModuleNotFoundError
No module named 'selenium.webdriver.phantomjs'
webdriver.PhantomJS() -> AttributeError
module 'selenium.webdriver' has no attribute 'PhantomJS'

-- 2. selenium.webdriver 包目录下还剩哪些浏览器 --
['chrome', 'chromium', 'common', 'edge', 'firefox', 'ie', 'remote', 'safari', 'support', 'webkitgtk', 'wpewebkit']
有没有 phantomjs 目录: False
webdriver.PhantomJS 有吗: False | webdriver.Opera 有吗: False

-- 3. Options/desired_capabilities 这些老参数 --
Chrome Options 还有 desired_capabilities 属性吗: False
webdriver.Chrome(desired_capabilities={}) -> TypeError | WebDriver.__init__() got an unexpected keyword argument 'desired_capabilities'
options.desired_capabilities -> AttributeError | 'Options' object has no attribute 'desired_capabilities'

-- 4. PyPI 上那几个 phantomjs 包的元数据 --
phantomjs 最新版 1.4.1 上传于 2021-04-10T06:55:05 requires_python='' yanked=False 共 14 个版本
pyphantomjs HTTP 404
selenium-phantomjs-driver HTTP 404
  • 模块层面也点不到名:webdriver.PhantomJS 有吗: False、webdriver.Opera 有吗: False,而 Chrome / Edge / Firefox / Ie / Safari / WebKitGTK / Remote 都是 True;desired_capabilities 是和 PhantomJS 同一批被清掉的代码路径,替代写法是 Options 属性或 options.set_capability(name, value)。
  • pip 包最后上传是 2021-04-10,另两个直接 404:这些包只是下载二进制,而 PhantomJS 项目本身早已停止维护、Selenium 从 4.0 起移除支持,整条链路没人再修。
  • 未实测:本机没有那个二进制,也没尝试装 pip 包再跑起来;判定全部来自已装的 selenium 4.52.0 和 PyPI 元数据。要「无头」就是 --headless=new。

打开页面与导航

基本导航

  • get() 是同步的:返回时文档加载完了,但不代表异步数据到位。
nav_spa1.py
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
from selenium import webdriver
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.support.ui import WebDriverWait

SPA = "https://spa1.scrape.center/"
opts = Options()
opts.add_argument("--headless=new")
driver = webdriver.Chrome(options=opts)
try:
driver.get(SPA)
WebDriverWait(driver, 15).until(EC.presence_of_all_elements_located(
(By.CSS_SELECTOR, "div.el-card")))
print("current_url:", driver.current_url)
print("title :", driver.title)
src = driver.page_source
print("渲染后 page_source 长度: %d | 行数: %d" % (len(src), src.count("\n") + 1))
print("page_source 开头: %r" % src[:56])
driver.get(SPA + "detail/1")
WebDriverWait(driver, 15).until(
EC.presence_of_element_located((By.CSS_SELECTOR, "div.info")))
print("到详情页: %s | %s"
% (driver.current_url, driver.find_element(By.TAG_NAME, "h2").text))
driver.back()
print("back : %s | %s | 卡片 %d 张" % (
driver.current_url, driver.title,
len(driver.find_elements(By.CSS_SELECTOR, "div.el-card"))))
driver.forward()
print("forward : %s | %s"
% (driver.current_url, driver.find_element(By.TAG_NAME, "h2").text))
driver.refresh()
print("refresh 后 readyState:", driver.execute_script("return document.readyState"))
finally:
driver.quit()
1
2
3
4
5
6
7
8
current_url: https://spa1.scrape.center/
title : Scrape | Movie
渲染后 page_source 长度: 27979 | 行数: 29
page_source 开头: '<html lang="en"><head><meta charset="utf-8"><meta http-e'
到详情页: https://spa1.scrape.center/detail/1 | 霸王别姬 - Farewell My Concubine
back : https://spa1.scrape.center/ | Scrape | Movie | 卡片 10 张
forward : https://spa1.scrape.center/detail/1 | 霸王别姬 - Farewell My Concubine
refresh 后 readyState: complete
  • document.title 和 driver.title 实测一致;back() / forward() 走浏览器历史,返回后 current_url 立刻是新页面的。
  • 前端路由里 current_url 变得比内容早:forward() 之后地址已经是详情页,h2 还得再等一次才拿得到,别把「URL 对了」当成「页面好了」。

page_source 和 requests 的差别

  • 这是「为什么需要浏览器」最硬的证据:同一个 URL,urllib 拿到 952 字节的空壳,浏览器等到卡片后有 27911 字节。
page_source_vs_urllib.py
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
import urllib.request

from selenium import webdriver
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.support.ui import WebDriverWait

SPA = "https://spa1.scrape.center/"
resp = urllib.request.urlopen(SPA)
html = resp.read().decode("utf-8", "replace")
print("[urllib] GET / -> %s | %d 字节 | 含「霸王别姬」: %s"
% (resp.getcode(), len(html), "霸王别姬" in html))
i = html.find("<body>")
print("[urllib] body 内容: %r" % html[i:i + 40])
print("[urllib] el-card 次数: %d | h2 次数: %d" % (html.count("el-card"), html.count("<h2")))
api = urllib.request.urlopen(SPA + "api/movie").read().decode("utf-8")
print("[urllib] GET /api/movie -> 200 | %d 字节 | 开头: %s" % (len(api), api[:34]))

opts = Options()
opts.add_argument("--headless=new")
driver = webdriver.Chrome(options=opts)
try:
driver.get(SPA)
s = driver.page_source
print("[浏览器] get() 刚返回: page_source %d 字节 | 含「霸王别姬」: %s | 卡片 %d 张"
% (len(s), "霸王别姬" in s,
len(driver.find_elements(By.CSS_SELECTOR, "div.el-card"))))
WebDriverWait(driver, 15).until(EC.presence_of_all_elements_located(
(By.CSS_SELECTOR, "div.el-card")))
s2 = driver.page_source
print("[浏览器] 等到卡片后: page_source %d 字节 | 含「霸王别姬」: %s | 卡片 %d 张"
% (len(s2), "霸王别姬" in s2,
len(driver.find_elements(By.CSS_SELECTOR, "div.el-card"))))
print("[对照] urllib 那份 HTML 是 page_source 的前缀吗:", s2.startswith(html))
finally:
driver.quit()
1
2
3
4
5
6
7
[urllib] GET / -> 200 | 952 字节 | 含「霸王别姬」: False
[urllib] body 内容: "<body><noscript><strong>We're sorry but "
[urllib] el-card 次数: 0 | h2 次数: 0
[urllib] GET /api/movie -> 200 | 2496 字节 | 开头: {"count":104,"results":[{"id":1,"n
[浏览器] get() 刚返回: page_source 2544 字节 | 含「霸王别姬」: False | 卡片 0 张
[浏览器] 等到卡片后: page_source 27911 字节 | 含「霸王别姬」: True | 卡片 10 张
[对照] urllib 那份 HTML 是 page_source 的前缀吗: False
  • 三点:urllib 那份 952 字节里 <div id=app></div> 是空的,el-card、h2 出现次数都是 0,片名要靠前端再请求 /api/movie 才有;page_source 是实时 DOM 序列化,等之前 2544 字节、等之后 27911 字节;两者不是包含关系,别指望「urllib 拿 HTML 再喂解析器」能替代浏览器。接口本身就是 JSON 时(那 2496 字节)就不该开浏览器,怎么判断见《爬虫-Requests》。

窗口与标签页

尺寸有两个来源

  • 无头模式下窗口尺寸由启动参数决定,而「窗口大小」和「视口大小」不是一回事。
window_size.py
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
from selenium import webdriver
from selenium.webdriver.chrome.options import Options

HEAD = "http://127.0.0.1:8013/window"


def start(wsize):
o = Options()
o.add_argument("--headless=new")
if wsize:
o.add_argument("--window-size=%s" % wsize)
return webdriver.Chrome(options=o)


print("-- 无头 --window-size=1000,700 --")
driver = start("1000,700")
try:
driver.get(HEAD)
print("get_window_size :", driver.get_window_size())
print("innerWidth x innerHeight: %sx%d" % (
driver.execute_script("return window.innerWidth"),
driver.execute_script("return window.innerHeight")))
driver.set_window_size(1280, 800)
print("set_window_size(1280,800) ->", driver.get_window_size())
print("页面 innerWidth:", driver.execute_script("return window.innerWidth"))
finally:
driver.quit()

print("\n-- 不加 --window-size 的默认视口 --")
driver = start(None)
try:
driver.get(HEAD)
print("get_window_size:", driver.get_window_size())
print("innerHeight :", driver.execute_script("return window.innerHeight"))
finally:
driver.quit()
1
2
3
4
5
6
7
8
9
-- 无头 --window-size=1000,700 --
get_window_size : {'width': 1000, 'height': 700}
innerWidth x innerHeight: 982x548
set_window_size(1280,800) -> {'width': 1280, 'height': 800}
页面 innerWidth: 1262

-- 不加 --window-size 的默认视口 --
get_window_size: {'width': 780, 'height': 580}
innerHeight : 428
  • 1000x700 对应视口 982x548,1280x800 对应 innerWidth 1262:宽度差 18、高度差 152。写滚动和懒加载逻辑要用视口尺寸。
  • 不给 --window-size 时默认视口很小(780x580 / innerHeight 428),够不着「首屏可见」的判断。

多标签与句柄

  • 一个 driver 管所有窗口,靠句柄集合;target="_blank" 点了也会多一个句柄。
tabs.py
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
from selenium import webdriver
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.common.by import By

opts = Options()
opts.add_argument("--headless=new")
driver = webdriver.Chrome(options=opts)
try:
driver.get("http://127.0.0.1:8010/detail/1")
main = driver.current_window_handle
driver.find_element(By.CSS_SELECTOR, "a[target='_blank']").click()
print("句柄数:", len(driver.window_handles))
for h in driver.window_handles:
if h != main:
driver.switch_to.window(h)
print("新标签:", driver.current_url, "|", driver.title)
driver.close()
driver.switch_to.window(main)
finally:
driver.quit()
1
2
3
4
5
6
7
8
9
-- 多标签 --
起始句柄数: 1 | current: EE6BA33B
new_window('tab') 后句柄数: 2
当前 url(新标签是 about:blank): about:blank
new_window('window') 后句柄数: 3
close 掉第三个句柄后剩余: 2
点了 target=_blank 后句柄数: 3
新弹出的标签 url: http://127.0.0.1:8010/detail/1 | title: 三国演义
关闭并切回,剩余: 2
  • new_window("tab") 开出来的标签停在 about:blank,得自己切过去再 get();close() 只关当前窗口且不会自动切走,关完马上读属性会炸。
close_then_read.py
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
from selenium import webdriver
from selenium.webdriver.chrome.options import Options

opts = Options()
opts.add_argument("--headless=new")
driver = webdriver.Chrome(options=opts)
try:
driver.get("http://127.0.0.1:8013/window")
main = driver.current_window_handle
driver.switch_to.new_window("tab")
driver.close()
try:
title = driver.title
print("close 后取 title: 没报错 |", title)
except Exception as exc:
print("close 后取 title: %s | %s"
% (type(exc).__name__, str(exc).splitlines()[0]))
driver.switch_to.window(main)
print("切回主窗口后 title:", driver.title)
finally:
driver.quit()
1
2
close 后取 title: NoSuchWindowException | Message: no such window: target window already closed
切回主窗口后 title: 新窗口
  • 正确顺序是 close() → switch_to.window(主句柄) → 再读属性。

八种定位

By 常量与命中数

  • Selenium 4 只认一个入口:find_element(By.常量, 定位串),八个常量的真实取值就是协议里的字符串。
  • 这节用本机靶站:LINK_TEXT 要「页面上正好有这段文本」、NAME 要表单控件,这些可控条件在线练习站给不了。
all_by.py
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
from selenium import webdriver
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.common.by import By

opts = Options()
opts.add_argument("--headless=new")
driver = webdriver.Chrome(options=opts)
try:
driver.get("http://127.0.0.1:8010/list")
for by, value in [
(By.ID, "books"), (By.TAG_NAME, "a"), (By.CLASS_NAME, "item"),
(By.XPATH, "//ul[@id='books']/li/a"), (By.CSS_SELECTOR, "ul#books li a"),
(By.LINK_TEXT, "列表"), (By.PARTIAL_LINK_TEXT, "懒加载"),
]:
found = driver.find_elements(by, value)
print("%-18s 命中 %d 第一个 text=%r" % (by, len(found), found[0].text if found else None))
finally:
driver.quit()
1
2
3
4
5
6
7
8
9
10
11
12
13
14
== /list ==
By.ID 命中 1 第一个 tag=ul text='三国演义 罗贯中 ¥39.5 余 3 本\n水浒传'
By.TAG_NAME 命中 11 第一个 tag=a text='首页'
By.CLASS_NAME 命中 3 第一个 tag=li text='三国演义 罗贯中 ¥39.5 余 3 本'
By.XPATH 命中 3 第一个 tag=a text='三国演义'
By.CSS_SELECTOR 命中 3 第一个 tag=a text='三国演义'
By.LINK_TEXT 命中 1 第一个 tag=a text='列表'
By.PARTIAL_LINK_TEXT 命中 1 第一个 tag=a text='懒加载'

== /form(NAME 定位)==
By.NAME 命中 1 第一个 tag=input text='' value=''

== By 常量的实际值 ==
[('ID', 'id'), ('XPATH', 'xpath'), ('CSS_SELECTOR', 'css selector'), ('NAME', 'name'), ('TAG_NAME', 'tag name'), ('CLASS_NAME', 'class name'), ('LINK_TEXT', 'link text'), ('PARTIAL_LINK_TEXT', 'partial link text')]
  • 三点观察:CLASS_NAME 只吃单个类名;LINK_TEXT 要求文本完全相等且只匹配 <a>;NAME 只在表单元素上有意义。
  • 直接传协议字符串(find_element("tag name", "a"))也能用,实测返回同一个元素,但正式代码建议一律 By.*。

找不到与写错的两种结局

  • find_elements 找不到返回空列表,find_element 找不到抛异常;批量取数优先用复数形式。
locate_failures.py
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
from selenium import webdriver
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.common.by import By


def brief(exc, n=2):
# 把异常 message 压成一行:丢掉 backtrace,最多取前 n 个有效行
parts = []
for line in str(exc).splitlines():
line = line.strip()
if not line or line.startswith(("Stacktrace", "Symbols", "(No symbol)")):
continue
parts.append(line.split(";")[0])
if len(parts) >= n:
break
return " / ".join(parts)


print("== 找不到时的两种结局 ==")
opts = Options()
opts.add_argument("--headless=new")
driver = webdriver.Chrome(options=opts)
try:
driver.get("http://127.0.0.1:8010/list")
got = driver.find_elements(By.ID, "never-exist")
print("find_elements -> type=%s len=%d" % (type(got).__name__, len(got)))
try:
driver.find_element(By.ID, "never-exist")
except Exception as exc:
print("find_element -> %s" % type(exc).__name__)
print(" %s" % brief(exc))
for label, by, value in (("xpath '//div['", By.XPATH, "//div["),
("css selector '##x'", By.CSS_SELECTOR, "##x")):
try:
driver.find_element(by, value)
out = "居然找到了"
except Exception as exc:
out = "%s | %s" % (type(exc).__name__, brief(exc))
print("%-22s -> %s" % (label, out))
finally:
driver.quit()
1
2
3
4
5
6
== 找不到时的两种结局 ==
find_elements -> type=list len=0
find_element -> NoSuchElementException
Message: no such element: Unable to locate element: {"method":"css selector","selector":"[id="never-exist"]"} / (Session info: chrome=156.0.8078.80)
xpath '//div[' -> InvalidSelectorException | Message: invalid selector: Unable to locate an element with the xpath expression //div[ because of the following error: / SyntaxError: Failed to execute 'evaluate' on 'Document': The string '//div[' is not a valid XPath expression.
css selector '##x' -> InvalidSelectorException | Message: invalid selector: An invalid or illegal selector was specified / (Session info: chrome=156.0.8078.80)
  • 选择器本身写错是 InvalidSelectorException,报错会区分 XPath 还是 CSS;元素确实不存在才是 NoSuchElementException。

三种等待

  • 下面每一段的秒数都是本机一次实跑抄下来的,网络快慢和机器状态都会让它抖零点几秒;看的是相对关系和返回类型,不是绝对值。

隐式等待

  • 写法只有一行:driver.implicitly_wait(秒)。它管的是每一次 find_element 愿意重试多久,跟页面加载、跟 JS 条件都没关系;默认 0,等于关掉。

    wait_implicit.py
    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    20
    21
    22
    23
    24
    25
    26
    27
    28
    29
    30
    31
    32
    33
    34
    35
    36
    37
    38
    39
    40
    41
    import time

    from selenium import webdriver
    from selenium.webdriver.chrome.options import Options
    from selenium.webdriver.common.by import By

    opts = Options()
    opts.add_argument("--headless=new")
    URL = "http://127.0.0.1:8013/delay?ms=3000" # 页面立刻加载完,3 秒后 JS 插入 <div id="late">

    driver = webdriver.Chrome(options=opts)
    driver.implicitly_wait(0) # 默认值:找不到就立刻抛
    driver.get(URL)
    t0 = time.time()
    try:
    el = driver.find_element(By.ID, "late")
    out = el.text
    except Exception as exc:
    out = type(exc).__name__
    print("implicit=0 找 3s 后才出现的元素: %.2fs -> %s" % (time.time() - t0, out))
    driver.quit()

    driver = webdriver.Chrome(options=opts)
    driver.implicitly_wait(5)
    driver.get(URL)
    t0 = time.time()
    el = driver.find_element(By.ID, "late")
    print("implicit=5 找 3s 后才出现的元素: %.2fs -> %r" % (time.time() - t0, el.text))

    t0 = time.time()
    try:
    driver.find_element(By.ID, "never")
    out = "找到了"
    except Exception as exc:
    out = type(exc).__name__
    print("implicit=5 找根本不存在的元素: %.2fs -> %s" % (time.time() - t0, out))

    t0 = time.time()
    print("implicit=5 找页面上就有的元素 : %.2fs -> %s"
    % (time.time() - t0, driver.find_element(By.TAG_NAME, "p").tag_name))
    driver.quit()
    1
    2
    3
    4
    implicit=0 找 3s 后才出现的元素: 0.01s -> NoSuchElementException
    implicit=5 找 3s 后才出现的元素: 3.01s -> '迟到元素'
    implicit=5 找根本不存在的元素: 5.07s -> NoSuchElementException
    implicit=5 找页面上就有的元素 : 0.00s -> p
  • 参数只有一个:秒数(可以是小数),设一次全局生效、直到你再调一次;implicitly_wait(0) 就是关掉。

  • 第二行说明它能等到 JS 插进来的元素(3.01 秒),因为它就是在 find_element 里反复重试;第三、四行是它的代价:命中 0.00 秒,没命中就每次都把 5 秒等满(5.07 秒)。一次抓取得定位几十个元素,只要有几个不存在,时间全被它吃掉。

显式等待

  • 写法是 WebDriverWait(driver, timeout, poll_frequency, ignored_exceptions).until(条件),四个参数逐个说:

    参数 类型 / 默认 作用
    driver WebDriver 等谁,就是那个浏览器实例
    timeout 秒,必填 最多等多久,到点抛 TimeoutException
    poll_frequency 默认 0.5 每隔多久问一次条件,越小越及时、越费 CPU
    ignored_exceptions 默认只忽略 NoSuchElementException 想让别的异常也继续等就传元组,例如 (StaleElementReferenceException,)
    • .until(条件) 等到条件为真并返回它的值;.until_not(条件) 反过来,等到条件为假。
    • 条件一般来自 selenium.webdriver.support.expected_conditions(惯写 EC)。所有 *_of_element_located 吃的是 (By.常量, 定位串) 二元组,不是字符串。
    wait_explicit.py
    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    20
    21
    22
    23
    24
    25
    26
    27
    28
    29
    30
    31
    32
    33
    34
    35
    36
    37
    38
    39
    40
    41
    42
    43
    44
    45
    46
    47
    48
    49
    50
    51
    52
    53
    import time

    from selenium import webdriver
    from selenium.webdriver.chrome.options import Options
    from selenium.webdriver.common.by import By
    from selenium.webdriver.support import expected_conditions as EC
    from selenium.webdriver.support.ui import WebDriverWait

    opts = Options()
    opts.add_argument("--headless=new")
    driver = webdriver.Chrome(options=opts)
    driver.get("http://127.0.0.1:8013/delay?ms=3000")


    def timed(label, fn):
    t0 = time.time()
    try:
    out = fn()
    except Exception as exc:
    out = type(exc).__name__
    print("%-38s %.2fs -> %r" % (label, time.time() - t0, out))


    timed("presence_of_element_located",
    lambda: WebDriverWait(driver, timeout=10, poll_frequency=0.5).until(
    EC.presence_of_element_located((By.ID, "late"))).text)
    timed("visibility_of_element_located",
    lambda: WebDriverWait(driver, 10).until(
    EC.visibility_of_element_located((By.ID, "late"))).tag_name)
    timed("element_to_be_clickable",
    lambda: WebDriverWait(driver, 10).until(
    EC.element_to_be_clickable((By.ID, "late-btn"))).get_attribute("disabled"))
    timed("title_is", lambda: WebDriverWait(driver, 10).until(EC.title_is("迟到元素")))
    timed("text_to_be_present_in_element",
    lambda: WebDriverWait(driver, 10).until(
    EC.text_to_be_present_in_element((By.ID, "late"), "迟到")))
    timed("invisibility_of_element_located(不存在)",
    lambda: WebDriverWait(driver, 10).until(
    EC.invisibility_of_element_located((By.ID, "never"))))
    timed("timeout=3 等永远不出现的元素",
    lambda: WebDriverWait(driver, 3).until(
    EC.presence_of_element_located((By.ID, "never"))))
    timed("until_not 等已存在的元素消失",
    lambda: WebDriverWait(driver, 2).until_not(
    EC.presence_of_element_located((By.TAG_NAME, "p"))))
    timed("把二元组写成裸字符串",
    lambda: WebDriverWait(driver, 3).until(
    EC.presence_of_element_located("late")))
    t0 = time.time()
    got = WebDriverWait(driver, 10).until(EC.presence_of_element_located)
    print("%-38s %.2fs -> 返回 %s,根本没等元素"
    % ("少写一对括号(传的是 EC 类本身)", time.time() - t0, type(got).__name__))
    driver.quit()
    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    presence_of_element_located            3.08s -> '迟到元素'
    visibility_of_element_located 0.04s -> 'div'
    element_to_be_clickable 0.04s -> None
    title_is 0.01s -> True
    text_to_be_present_in_element 0.02s -> True
    invisibility_of_element_located(不存在) 0.01s -> True
    timeout=3 等永远不出现的元素 3.08s -> 'TimeoutException'
    until_not 等已存在的元素消失 2.06s -> 'TimeoutException'
    把二元组写成裸字符串 0.00s -> 'TypeError'
    少写一对括号(传的是 EC 类本身) 0.00s -> 返回 function,根本没等元素
  • 六个常用条件各是什么:presence_of_element_located 只要 DOM 里有就行(渲染页取数用它最稳);visibility_of_element_located 还要「看得见」(宽高不为 0、没 display:none);element_to_be_clickable = 可见且 disabled 不是 true,所以第三行取回来的 disabled 是 None;title_is 比标题字符串;text_to_be_present_in_element 比元素文本;invisibility_of_element_located 是「等它消失」。

  • 两个静默坑,实测都在最后两行:定位写成裸字符串不是「等不到」,而是当场 TypeError;少写一对括号(until(EC.presence_of_element_located))连报错都没有——until 拿到的是那个函数对象,函数永远为真,0.00 秒就返回,等于根本没等。

  • invisibility_of_element_located 对根本不存在的元素也返回 True(第六行),适合「等 loading 遮罩消失」,但不能用它判断元素曾经存在过。

  • until_not 是「等到条件变假」:上面第八行等一个本来就在的 <p> 消失,等满 2 秒抛 TimeoutException。

  • 超时的 message 里那串 backtrace 基本没有信息量('Message: \nStacktrace:\nSymbols not available...'),排查要靠自己的日志。

手写轮询

  • 自己 while + find_elements + sleep,好处是条件能写成任意 Python 表达式(数量够没够、文本变了没变、JS 返回值)。

    wait_poll.py
    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    20
    21
    22
    23
    24
    25
    26
    27
    28
    29
    30
    31
    32
    33
    import time

    from selenium import webdriver
    from selenium.webdriver.chrome.options import Options
    from selenium.webdriver.common.by import By

    opts = Options()
    opts.add_argument("--headless=new")

    driver = webdriver.Chrome(options=opts)
    driver.get("https://spa1.scrape.center/")
    t0 = time.time()
    while time.time() - t0 < 10:
    cards = driver.find_elements(By.CSS_SELECTOR, "div.el-card")
    if len(cards) >= 10:
    print("手写轮询(0.5s 一轮): %.2fs 拿到 %d 张卡片" % (time.time() - t0, len(cards)))
    break
    time.sleep(0.5)
    else:
    print("轮询到超时也没凑够 10 张")
    driver.quit()

    driver = webdriver.Chrome(options=opts)
    driver.get("http://127.0.0.1:8013/delay?ms=3000")
    t0 = time.time()
    while time.time() - t0 < 10:
    if driver.execute_script("return document.querySelectorAll('#late').length") == 1:
    print("JS 条件轮询: %.2fs -> 元素出现了" % (time.time() - t0))
    break
    time.sleep(0.2)
    else:
    print("JS 条件轮询超时")
    driver.quit()
    1
    2
    手写轮询(0.5s 一轮): 1.02s 拿到 10 张卡片
    JS 条件轮询: 3.10s -> 元素出现了
  • while ... else 的 else 只在没 break 时执行,正好当超时分支用。

  • 粒度就是代价:0.5 秒一轮意味着平均要多等半秒,上面第一轮 1.02 秒里就包含了这个粒度损失;换成 WebDriverWait 的 0.5 秒 poll_frequency 是同样的道理,但异常处理已经写好了。

四种写法的耗时对照

  • 拿实战那节的 spa1 渲染页各跑三轮,每轮都新开 driver、get() 之后才开始计时,免得读到上一页残留。

    wait_compare.py
    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    20
    21
    22
    23
    24
    25
    26
    27
    28
    29
    30
    31
    32
    33
    34
    35
    36
    37
    38
    39
    40
    41
    42
    43
    44
    45
    46
    47
    48
    49
    50
    51
    52
    53
    54
    55
    56
    57
    58
    59
    60
    61
    62
    63
    import time

    from selenium import webdriver
    from selenium.webdriver.chrome.options import Options
    from selenium.webdriver.common.by import By
    from selenium.webdriver.support import expected_conditions as EC
    from selenium.webdriver.support.ui import WebDriverWait

    URL = "https://spa1.scrape.center/"
    opts = Options()
    opts.add_argument("--headless=new")


    def cards(d):
    return len(d.find_elements(By.CSS_SELECTOR, "div.el-card"))


    def sleep_2(d):
    time.sleep(2.0)
    return cards(d)


    def sleep_5(d):
    time.sleep(5.0)
    return cards(d)


    def implicit(d):
    d.implicitly_wait(5)
    return cards(d)


    def explicit(d):
    WebDriverWait(d, 10).until(
    EC.presence_of_all_elements_located((By.CSS_SELECTOR, "div.el-card")))
    return cards(d)


    def poll(d):
    t0 = time.time()
    while time.time() - t0 < 10:
    n = cards(d)
    if n >= 10:
    return n
    time.sleep(0.5)
    return cards(d)


    for label, body in (("sleep(2.0)", sleep_2), ("sleep(5.0)", sleep_5),
    ("implicitly_wait(5)", implicit), ("WebDriverWait(10)", explicit),
    ("手写轮询(0.5s)", poll)):
    costs, counts = [], []
    for _ in range(3):
    d = webdriver.Chrome(options=opts)
    d.get(URL)
    t0 = time.time()
    n = body(d)
    costs.append(time.time() - t0)
    counts.append(n)
    d.quit()
    print("%-20s: %s | 平均 %.2fs" % (
    label, " | ".join("%.2fs %d 张" % (c, k) for c, k in zip(costs, counts)),
    sum(costs) / 3))
    1
    2
    3
    4
    5
    sleep(2.0)          : 2.02s 10 张 | 2.02s 10 张 | 2.01s 10 张 | 平均 2.02s
    sleep(5.0) : 5.02s 10 张 | 5.01s 10 张 | 5.01s 10 张 | 平均 5.02s
    implicitly_wait(5) : 0.58s 10 张 | 1.36s 10 张 | 0.35s 10 张 | 平均 0.76s
    WebDriverWait(10) : 2.05s 10 张 | 1.03s 10 张 | 1.08s 10 张 | 平均 1.38s
    手写轮询(0.5s) : 1.55s 10 张 | 1.02s 10 张 | 1.54s 10 张 | 平均 1.37s
  • sleep 是赌:2 秒这几轮碰巧够,站点慢一点就取到 0 张;改成 5 秒稳了,可平均 5.02 秒里大部分是白等。

  • 隐式等待这次最快(平均 0.76 秒)但它测的不是「10 张齐了」:implicitly_wait 只要页面上出现第一个 div.el-card 就返回,剩下的卡片是渲染速度赶上的。所以它的数字不能当「数据齐了」的证据,换个站点就可能变成只拿到 3 张就往下走。

  • WebDriverWait 用 presence_of_all_elements_located(要求一次取到的列表非空,同样是首个出现即返回,但语义上更明确)与手写轮询(判的是 >= 10)都在 1.4 秒上下,多出来的那部分是 0.5 秒轮询粒度。

  • 结论照旧:默认用 WebDriverWait + 明确条件,implicitly_wait 保持 0;要「数够几张才算好」这种条件,只有显式等待或手写轮询能表达。

隐式与显式混用

  • 两者同时设置时,实际耗时由隐式等待决定,因为显式等待每问一次条件,底层那次 find_element 自己就要把隐式的秒数等满:

    wait_mix.py
    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    20
    21
    22
    23
    24
    25
    26
    27
    28
    import time

    from selenium import webdriver
    from selenium.webdriver.chrome.options import Options
    from selenium.webdriver.common.by import By
    from selenium.webdriver.support import expected_conditions as EC
    from selenium.webdriver.support.ui import WebDriverWait

    DELAY = "http://127.0.0.1:8013/delay?ms=3000"
    opts = Options()
    opts.add_argument("--headless=new")

    for imp, wait in ((0, 3), (5, 3)):
    driver = webdriver.Chrome(options=opts)
    try:
    driver.implicitly_wait(imp)
    driver.get(DELAY)
    t0 = time.time()
    try:
    WebDriverWait(driver, wait).until(
    EC.presence_of_element_located((By.ID, "never")))
    out = "找到了"
    except Exception as exc:
    out = type(exc).__name__
    print("implicit=%d + wait=%d : %.2fs -> %s"
    % (imp, wait, time.time() - t0, out))
    finally:
    driver.quit()
    1
    2
    implicit=0 + wait=3 : 3.04s -> TimeoutException
    implicit=5 + wait=3 : 5.06s -> TimeoutException
  • 所以用显式等待就让 implicitly_wait 保持默认的 0,本篇后面所有示例都遵守这条。

交互与表单

submit 与 click

  • submit() 不要求元素是按钮,实测和点提交按钮到达同一个后端;但真实站点的提交按钮常绑了 JS(先校验再提交),submit() 会绕过那套逻辑,所以默认用 click()。表单和下拉要在结构固定的页面上复现,这两小节也用本机靶站。
submit_vs_click.py
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
from selenium import webdriver
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.support.ui import WebDriverWait

FORM = "http://127.0.0.1:8010/form"
opts = Options()
opts.add_argument("--headless=new")
driver = webdriver.Chrome(options=opts)
try:
print("== submit() vs click() ==")
driver.get(FORM)
name = driver.find_element(By.CSS_SELECTOR, "form#post input[name='name']")
name.clear()
name.send_keys("lilei")
name.submit()
WebDriverWait(driver, 5).until(EC.url_contains("/submit"))
print("submit() 之后 url:", driver.current_url)
print("页面正文前 200 字:", driver.find_element("tag name", "pre").text[:200])

driver.get(FORM)
name = driver.find_element(By.CSS_SELECTOR, "form#post input[name='name']")
name.clear()
name.send_keys("lilei")
driver.find_element(By.CSS_SELECTOR, "form#post button").click()
WebDriverWait(driver, 5).until(EC.url_contains("/submit"))
print("点 POST 按钮后 url:", driver.current_url)
body = driver.find_element("tag name", "pre").text
print("回包:", body.replace("\n", " ")[:130])

print("\n== 表单里的 hidden token ==")
driver.get(FORM)
token = driver.find_element(By.CSS_SELECTOR, "input[name='token']")
line = "token 元素显示吗: %s | 可交互吗: " % token.is_displayed()
try:
token.send_keys("x")
print(line + "能打字")
except Exception as exc:
print(line + "%s | %s" % (type(exc).__name__, str(exc).splitlines()[0]))
print("value 还是:", repr(token.get_attribute("value")))
finally:
driver.quit()
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
== submit() vs click() ==
submit() 之后 url: http://127.0.0.1:8010/submit
页面正文前 200 字: {
"ok": true,
"kind": "form",
"data": {
"name": "lilei",
"token": "tok-8f2c"
}
}
点 POST 按钮后 url: http://127.0.0.1:8010/submit
回包: { "ok": true, "kind": "form", "data": { "name": "lilei", "token": "tok-8f2c" } }

== 表单里的 hidden token ==
token 元素显示吗: False | 可交互吗: ElementNotInteractableException | Message: element not interactable
value 还是: 'tok-8f2c'
  • 隐藏域不可交互但 value 照样能读,所以「带上 token」根本不用操作那个 input,读出来拼参数即可。

下拉框用 Select

  • Select 只包 <select>,单选多选行为不同,四类失败情形本机都复现过。
select_demo.py
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
from selenium import webdriver
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import Select


def brief(exc):
return str(exc).splitlines()[0].split(";")[0]


print("== Select 下拉 ==")
opts = Options()
opts.add_argument("--headless=new")
driver = webdriver.Chrome(options=opts)
try:
driver.get("http://127.0.0.1:8013/select")
city = Select(driver.find_element(By.ID, "city"))
print("options:", [(x.get_attribute("value"), x.text, x.is_selected())
for x in city.options])

def pick(fn, what):
try:
fn(what)
return "%s|%s" % (city.first_selected_option.get_attribute("value"),
city.first_selected_option.text)
except Exception as exc:
return "%s | %s" % (type(exc).__name__, brief(exc))

print("select_by_value('gz') ->", pick(city.select_by_value, "gz"))
print("select_by_index(0) ->", pick(city.select_by_index, 0))
print("select_by_visible_text ->", pick(city.select_by_visible_text, "上海"))
print("选 disabled 项 ->", pick(city.select_by_value, "cd"))
print("select_by_value 不存在的值 ->", pick(city.select_by_value, "nope"))

tags = Select(driver.find_element(By.ID, "tags"))
print("multiple 框 is_multiple:", tags.is_multiple)
tags.select_by_visible_text("标签A")
tags.select_by_visible_text("标签C")
print("选了两个 ->", [x.text for x in tags.all_selected_options])
tags.deselect_all()
print("deselect_all ->", [x.text for x in tags.all_selected_options])
try:
city.deselect_all()
print("对单选框 deselect_all -> 没报错")
except Exception as exc:
print("对单选框 deselect_all -> %s | %s" % (type(exc).__name__, brief(exc)))
try:
Select(driver.find_element(By.TAG_NAME, "p"))
print("Select 包一个非 select -> 没报错")
except Exception as exc:
print("Select 包一个非 select -> %s | %s" % (type(exc).__name__, brief(exc)))
finally:
driver.quit()
1
2
3
4
5
6
7
8
9
10
11
12
== Select 下拉 ==
options: [('bj', '北京', False), ('sh', '上海', True), ('gz', '广州', False), ('cd', '成都(暂不可选)', False)]
select_by_value('gz') -> gz|广州
select_by_index(0) -> bj|北京
select_by_visible_text -> sh|上海
选 disabled 项 -> NotImplementedError | You may not select a disabled option
select_by_value 不存在的值 -> NoSuchElementException | Message: Cannot locate option with value: nope
multiple 框 is_multiple: True
选了两个 -> ['标签A', '标签C']
deselect_all -> []
对单选框 deselect_all -> NotImplementedError | You may only deselect all options of a multi-select
Select 包一个非 select -> UnexpectedTagNameException | Message: Select only works on <select> elements, not on p
  • 很多站点用 div 假装下拉框(Bootstrap-select、Element-UI 之类),Select 完全不认,只能「点一下展开 → 等选项出现 → 点选项」。

alert 与 iframe

alert 必须先处理

  • 弹窗是模态的:开着时大部分命令都会失败,因为焦点不在文档里。alert、iframe、多窗口这类机制要页面自己「配合演出」,在线练习站不给这种可控条件,所以这几节都在本机辅助页面上跑。
alert_switch.py
1
2
3
4
5
6
7
8
9
10
11
12
13
from selenium import webdriver
from selenium.webdriver.chrome.options import Options

opts = Options()
opts.add_argument("--headless=new")
driver = webdriver.Chrome(options=opts)
try:
driver.get("http://127.0.0.1:8013/alert")
alert = driver.switch_to.alert
print(repr(alert.text))
alert.accept()
finally:
driver.quit()
1
2
3
4
5
6
7
8
9
10
== alert ==
没有弹窗时取 alert: NoAlertPresentException | Message: no such alert
alert.text: '我是 alert'
弹窗开着时点元素、读 url 行不行: UnexpectedAlertPresentException | Alert Text: 我是 alert
弹窗开着时读 title: 没报错: 弹窗
正常 accept 之后 title: 弹窗

== 延迟 1.2 秒才出现的 alert ==
立刻取 alert: 0.00s -> NoAlertPresentException
alert_is_present 等到弹窗: 1.51s text='延迟 1.2 秒的 alert'
  • 三条规律:没弹窗时访问 switch_to.alert 抛 NoAlertPresentException;弹窗开着点元素抛 UnexpectedAlertPresentException,报错里附带 Alert Text: xxx(有时这是唯一抓到文案的机会);读 title 这类纯查询命令不会炸。
  • 等弹窗用 EC.alert_is_present():1.2 秒后才出现的 alert,条件在 1.51 秒拿到,裸取是 0.00 秒直接失败。

iframe 要切进去

  • 顶层文档永远找不到 iframe 内部元素,必须先 switch_to.frame();三层嵌套要一层层切。
iframe_nested.py
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
from selenium import webdriver
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.support.ui import WebDriverWait

FRAME = "http://127.0.0.1:8013/frame"
opts = Options()
opts.add_argument("--headless=new")
driver = webdriver.Chrome(options=opts)
try:
print("== iframe:三层嵌套 ==")
driver.get(FRAME)
print("顶层能看到的 id:", [x.get_attribute("id")
for x in driver.find_elements(By.TAG_NAME, "input")])
try:
driver.find_element(By.ID, "mid-here")
print("顶层找 iframe 里的元素: 居然找到了")
except Exception as exc:
print("顶层找 iframe 里的元素: %s" % type(exc).__name__)
driver.switch_to.frame("midframe")
print("按 name 切进中层后 url: %s | #mid-here: %s"
% (driver.current_url, driver.find_element(By.ID, "mid-here").text))
driver.switch_to.frame("inner")
print("按 id 切进最里层后 #inner-here: %s"
% driver.find_element(By.ID, "inner-here").text)
box = driver.find_element(By.ID, "inner-input")
box.send_keys("写到里层输入框")
print("里层输入框 value: %r" % box.get_attribute("value"))
driver.find_element(By.ID, "inner-btn").click()
print("点里层按钮后:", driver.find_element(By.ID, "inner-here").text)
driver.switch_to.parent_frame()
print("parent_frame 回到中层,#mid-input 在吗:",
driver.find_element(By.ID, "mid-input").get_attribute("value"))
driver.switch_to.frame("inner")
driver.switch_to.default_content()
print("default_content 直接从最里层回顶层:",
bool(driver.find_elements(By.ID, "outer-input")))
try:
driver.switch_to.frame("nope")
except Exception as exc:
print("切到不存在的 frame: %s | %s"
% (type(exc).__name__, str(exc).splitlines()[0]))
driver.switch_to.default_content()
driver.switch_to.frame(driver.find_element(By.ID, "mid"))
print("用 WebElement 切 frame 成功,当前 body:",
driver.find_element(By.TAG_NAME, "h1").text)
driver.switch_to.default_content()
ok = WebDriverWait(driver, 5).until(
EC.frame_to_be_available_and_switch_to_it((By.ID, "mid")))
print("条件切 frame 返回: %s | 里面找得到 mid-input: %s"
% (ok, bool(driver.find_elements(By.ID, "mid-input"))))
finally:
driver.quit()
1
2
3
4
5
6
7
8
9
10
11
12
== iframe:三层嵌套 ==
顶层能看到的 id: ['outer-input']
顶层找 iframe 里的元素: NoSuchElementException
按 name 切进中层后 url: http://127.0.0.1:8013/frame | #mid-here: 中层页面
按 id 切进最里层后 #inner-here: 最里层页面
里层输入框 value: '写到里层输入框'
点里层按钮后: 最里层按钮已点击
parent_frame 回到中层,#mid-input 在吗: mid
default_content 直接从最里层回顶层: True
切到不存在的 frame: NoSuchFrameException | Message: nope
用 WebElement 切 frame 成功,当前 body: iframe 中层
条件切 frame 返回: True | 里面找得到 mid-input: True
  • switch_to.frame() 可以给 name、id 或直接给一个 WebElement;parent_frame() 回上一层、default_content() 从最里层直接跳回顶层。
  • 切进 iframe 之后 current_url 不变(还是父页地址),所以判断自己在哪一层只能靠「能不能找到只有里面才有的元素」。
  • frame 是异步插入时用 frame_to_be_available_and_switch_to_it,它成功后已经切进去了,返回的是 True 而不是元素。

执行 JS 与懒加载

返回值类型映射

  • execute_script 按固定规则把 JS 返回值翻译成 Python 对象,踩坑基本都在这张表上。
js_return_types.py
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
from selenium import webdriver
from selenium.webdriver.chrome.options import Options

opts = Options()
opts.add_argument("--headless=new")
driver = webdriver.Chrome(options=opts)
try:
driver.get("http://127.0.0.1:8013/frame")
cases = [
("return 1", "数字"), ("return 'abc'", "字符串"),
("return null", "null"), ("return undefined", "undefined"),
("return [1,'a',true]", "数组"), ("return {a:1,b:[2,3]}", "对象"),
("return new Date(0)", "Date"),
]
for js, label in cases:
got = driver.execute_script(js)
print("%-39s %-13s -> %r (%s)" % (js, label, got, type(got).__name__))
finally:
driver.quit()
1
2
3
4
5
6
7
return 1                                数字            -> 1 (int)
return 'abc' 字符串 -> 'abc' (str)
return null null -> None (NoneType)
return undefined undefined -> None (NoneType)
return [1,'a',true] 数组 -> [1, 'a', True] (list)
return {a:1,b:[2,3]} 对象 -> {'a': 1, 'b': [2, 3]} (dict)
return new Date(0) Date -> {} (dict)
  • 三个反直觉的点:undefined 和 null 都变成 None,分不出来;new Date() 变成空 {},日期信息全丢;对象里塞了 window 时会变成一个以 window-fcc6-11e5-b4f8-330a88ab9d7f 为键的内部引用串。
  • 返回 DOM 节点会转成 WebElement,可以继续操作:实测取「下一页」拿到的是元素对象,text: 下一页、href: http://127.0.0.1:8010/list?page=2;querySelectorAll 返回的 NodeList 变成 list,长度 3。

懒加载两条路

  • 这一节留在本机靶站,因为在线站没有懒加载可练:ssr1 首页 HTML 里 data-src 出现 0 次、lazy 出现 0 次,10 个 img.cover 全都直接带 src。
  • 懒加载图片的真实 HTML 是 src 空、地址藏在 data-src。
lazy_scroll.py
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
import re
import time
import urllib.request

from selenium import webdriver
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.common.by import By

SCROLL = "http://127.0.0.1:8013/lazy-scroll"
html = urllib.request.urlopen(SCROLL).read().decode("utf-8")
print("[lazy/requests] img 标签原文:", re.findall(r"<img[^>]*>", html)[:1])

opts = Options()
opts.add_argument("--headless=new")
driver = webdriver.Chrome(options=opts)
try:
driver.get(SCROLL)
imgs = driver.find_elements(By.TAG_NAME, "img")
print("[lazy/selenium] 未滚动 src:", [x.get_attribute("src") for x in imgs])
print("[lazy/selenium] 未滚动 data-src:", [x.get_attribute("data-src") for x in imgs])
driver.execute_script("arguments[0].scrollIntoView()", imgs[0])
time.sleep(1.0)
print("[lazy/selenium] 滚到第 1 张后 src: %s | class=%s | naturalWidth=%spx"
% (imgs[0].get_attribute("src"), imgs[0].get_attribute("class"),
driver.execute_script("return arguments[0].naturalWidth", imgs[0])))
for x in imgs:
driver.execute_script("arguments[0].scrollIntoView()", x)
time.sleep(0.4)
time.sleep(0.6)
srcs = [x.get_attribute("src") for x in imgs]
print("[lazy/selenium] 滚动后 data-src 还在吗: %s | src 空串还剩几个: %d"
% (all(x.get_attribute("data-src") for x in imgs),
sum(1 for s in srcs if not s)))
finally:
driver.quit()
1
2
3
4
5
[lazy/requests] img 标签原文: ['<img class="lazy" data-src="http://127.0.0.1:8010/img/1.png" src="" alt="第 1 张">']
[lazy/selenium] 未滚动 src: ['', '', '']
[lazy/selenium] 未滚动 data-src: ['http://127.0.0.1:8010/img/1.png', 'http://127.0.0.1:8010/img/2.png', 'http://127.0.0.1:8010/img/3.png']
[lazy/selenium] 滚到第 1 张后 src: http://127.0.0.1:8010/img/1.png | class=lazy on | naturalWidth=102px
[lazy/selenium] 滚动后 data-src 还在吗: True | src 空串还剩几个: 0
  • 逐张 scrollIntoView 之后 src 被页面自己的脚本填上(naturalWidth=102 说明图真下来了),data-src 依然在,所以取地址优先 src、空了回落 data-src。
  • 一个真实的坑:靶站 /lazy 那页把 IntersectionObserver 的注册脚本写在 head 里,实测滚动 1.5 秒后三张图的 src 仍是空串、class 也没变——脚本时机不对时「滚动 + 等待」是白等。
lazy_observer_timing.py
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
import time
import urllib.request

from selenium import webdriver
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.common.by import By

OLD = "http://127.0.0.1:8010/lazy"
IO = "http://127.0.0.1:8013/lazy-io"
src = urllib.request.urlopen(OLD).read().decode("utf-8")
head = src.split("</head>")[0]
print("[靶站/lazy] 懒加载脚本在哪个标签里:",
"HEAD" if "IntersectionObserver" in head else "BODY")

opts = Options()
opts.add_argument("--headless=new")
driver = webdriver.Chrome(options=opts)
try:
driver.get(OLD)
driver.execute_script("window.scrollTo(0, document.body.scrollHeight)")
time.sleep(1.5)
imgs = driver.find_elements(By.TAG_NAME, "img")
print("[靶站/lazy] 滚动 1.5 秒后 src: %s | class: %s"
% ([x.get_attribute("src") for x in imgs],
[x.get_attribute("class") for x in imgs]))

print("\n-- 自建页 /lazy-io:时机对了就能逐张触发 --")
driver.get(IO)
imgs = driver.find_elements(By.CSS_SELECTOR, "img.lazy")
for i, x in enumerate(imgs, 1):
driver.execute_script("arguments[0].scrollIntoView()", x)
time.sleep(0.8)
print(" 滚到第 %d 张后: %s" % (i, [y.get_attribute("src") for y in imgs]))
finally:
driver.quit()
1
2
3
4
5
6
7
[靶站/lazy] 懒加载脚本在哪个标签里: HEAD
[靶站/lazy] 滚动 1.5 秒后 src: ['', '', ''] | class: ['', '', '']

-- 自建页 /lazy-io:时机对了就能逐张触发 --
滚到第 1 张后: ['http://127.0.0.1:8010/img/1.png', '', '']
滚到第 2 张后: ['http://127.0.0.1:8010/img/1.png', 'http://127.0.0.1:8010/img/2.png', '']
滚到第 3 张后: ['http://127.0.0.1:8010/img/1.png', 'http://127.0.0.1:8010/img/2.png', 'http://127.0.0.1:8010/img/3.png']
  • 另一条路是干脆自己写 src 绕过观察器,实测一次拿到三张地址、complete 全 True。
lazy_js_set_src.py
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
import time

from selenium import webdriver
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.common.by import By

opts = Options()
opts.add_argument("--headless=new")
driver = webdriver.Chrome(options=opts)
try:
driver.get("http://127.0.0.1:8013/lazy-io")
got = driver.execute_script(
"var out = [];"
"document.querySelectorAll('img.lazy').forEach(function (i) {"
" i.src = i.dataset.src; out.push(i.src);});"
"return out;")
print("JS 直接写 src:", got)
time.sleep(1.2)
imgs = driver.find_elements(By.CSS_SELECTOR, "img.lazy")
print("写完后 data-src 还在吗:", all(x.get_attribute("data-src") for x in imgs))
print("每张图 complete:", driver.execute_script(
"return [...document.querySelectorAll('img.lazy')].map(i => i.complete)"))
finally:
driver.quit()
1
2
3
JS 直接写 src: ['http://127.0.0.1:8010/img/1.png', 'http://127.0.0.1:8010/img/2.png', 'http://127.0.0.1:8010/img/3.png']
写完后 data-src 还在吗: True
每张图 complete: [True, True, True]

异步脚本

async_script.py
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
import time

from selenium import webdriver
from selenium.webdriver.chrome.options import Options

opts = Options()
opts.add_argument("--headless=new")
driver = webdriver.Chrome(options=opts)
try:
print("== execute_async_script:必须自己调用回调 ==")
driver.get("http://127.0.0.1:8013/frame")
driver.set_script_timeout(5)
t0 = time.time()
got = driver.execute_async_script(
"var cb = arguments[arguments.length - 1];"
"setTimeout(function () { cb('setTimeout 回调的值'); }, 400);")
print("正常异步: %.2fs -> %r" % (time.time() - t0, got))
t0 = time.time()
n = driver.execute_async_script(
"var cb = arguments[arguments.length - 1];"
"fetch('/frame-mid').then(function (r) { return r.text(); })"
".then(function (t) { cb(t.length); });")
print("fetch 回调: %.2fs -> 正文长度 %d" % (time.time() - t0, n))

print("\n-- 忘了调回调 --")
driver.set_script_timeout(1)
t0 = time.time()
try:
driver.execute_async_script("var x = 1;")
print("居然返回了")
except Exception as exc:
print("%.2fs -> %s | %s"
% (time.time() - t0, type(exc).__name__, str(exc).splitlines()[0]))

print("\n-- 用 Promise 当返回值 --")
driver.set_script_timeout(5)
t0 = time.time()
p = driver.execute_script(
"return new Promise(function (res) {"
" setTimeout(function () { res('晚到'); }, 1200);});")
print("execute_script 返回 Promise: %.2fs -> %r" % (time.time() - t0, p))
finally:
driver.quit()
1
2
3
4
5
6
7
8
9
== execute_async_script:必须自己调用回调 ==
正常异步: 0.41s -> 'setTimeout 回调的值'
fetch 回调: 0.01s -> 正文长度 315

-- 忘了调回调 --
1.00s -> TimeoutException | Message: script timeout

-- 用 Promise 当返回值 --
execute_script 返回 Promise: 1.21s -> '晚到'
  • execute_async_script 要求脚本自己调用回调 arguments[arguments.length - 1],忘了调只会等到 script timeout。execute_script 里 return 一个 Promise 时,实测是等它 resolve 之后再返回 resolve 的值(1.21 秒拿到 '晚到'),不会当场返回一个 Promise 对象。
  • 更稳的做法是轮询一个「页面上已经能数出来的东西」,也就是 JS 条件等待,这里换成 spa1 的卡片数。
js_wait.py
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
from selenium import webdriver
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.support.ui import WebDriverWait

opts = Options()
opts.add_argument("--headless=new")
driver = webdriver.Chrome(options=opts)
try:
driver.get("https://spa1.scrape.center/")
WebDriverWait(driver, 15).until(
lambda d: d.execute_script(
"return document.querySelectorAll('div.el-card.item').length") >= 10)
total = driver.execute_script("return document.querySelectorAll('div.el-card.item').length")
print("JS 条件等到 %d 张卡片 | 首条: %s" % (
total, driver.find_element("css selector", "div.el-card.item h2").text))
print("execute_script 取首条片名: %r" % driver.execute_script(
"return document.querySelector('div.el-card.item h2').textContent"))
print("execute_script 返回 NodeList 的长度: %d" % len(driver.execute_script(
"return document.querySelectorAll('div.el-card.item')")))
finally:
driver.quit()
1
2
3
JS 条件等到 10 张卡片 | 首条: 霸王别姬 - Farewell My Concubine
execute_script 取首条片名: '霸王别姬 - Farewell My Concubine'
execute_script 返回 NodeList 的长度: 10

截图

视口截图

  • save_screenshot 存文件、get_screenshot_as_png 返回字节、get_screenshot_as_base64 返回字符串,内容一致。
screenshot_viewport.py
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
import base64
import struct

from selenium import webdriver
from selenium.webdriver.chrome.options import Options


def shot_info(png):
w, h = struct.unpack(">II", png[16:24])
return ("%dx%d" % (w, h), len(png))


opts = Options()
opts.add_argument("--headless=new")
opts.add_argument("--window-size=1000,700")
driver = webdriver.Chrome(options=opts)
try:
print("== 视口截图 ==")
driver.get("http://127.0.0.1:8010/list")
ok = driver.save_screenshot("view.png")
print("save_screenshot 返回: %s | 文件: %s"
% (ok, shot_info(open("view.png", "rb").read())))
png = driver.get_screenshot_as_png()
print("get_screenshot_as_png:", shot_info(png))
b64 = driver.get_screenshot_as_base64()
print("base64 长度: %d 开头: %s" % (len(b64), b64[:24]))
print("base64 解码回来和 png 一致:", base64.b64decode(b64) == png)

print("\n== 视口大小与截图大小的关系 ==")
print("window-size=1000,700 -> innerHeight: %d | innerWidth: %d" % (
driver.execute_script("return window.innerHeight"),
driver.execute_script("return window.innerWidth")))
driver.set_window_size(500, 400)
print("resize 到 500x400 -> innerHeight: %d | 截图: %s" % (
driver.execute_script("return window.innerHeight"),
shot_info(driver.get_screenshot_as_png())))
finally:
driver.quit()
1
2
3
4
5
6
7
8
9
== 视口截图 ==
save_screenshot 返回: True | 文件: ('982x548', 29335)
get_screenshot_as_png: ('982x548', 29335)
base64 长度: 39116 开头: iVBORw0KGgoAAAANSUhEUgAA
base64 解码回来和 png 一致: True

== 视口大小与截图大小的关系 ==
window-size=1000,700 -> innerHeight: 548 | innerWidth: 982
resize 到 500x400 -> innerHeight: 248 | 截图: ('504x248', 25697)
  • 截图尺寸等于视口(982x548)而不是窗口(1000x700);resize 到 500x400 后截图是 504x248,宽度还多出几个像素,说明无头的视口换算不完全是「窗口减 18」。
  • 存二进制用 open("x.png", "wb").write(driver.get_screenshot_as_png())。

元素截图

  • 只要某个元素的画面(验证码、表格、图表)就用 element.screenshot(),它截的是元素的包围盒。
screenshot_element.py
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
import io
import struct

from PIL import Image
from selenium import webdriver
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.common.by import By


def shot_info(png):
w, h = struct.unpack(">II", png[16:24])
return ("%dx%d" % (w, h), len(png))


opts = Options()
opts.add_argument("--headless=new")
opts.add_argument("--window-size=1000,700")
driver = webdriver.Chrome(options=opts)
try:
print("== 元素截图 ==")
driver.get("http://127.0.0.1:8010/detail/1")
table = driver.find_element(By.ID, "stock")
print("table#stock rect: %s | 元素截图: %s"
% (table.rect, shot_info(table.screenshot_as_png)))
print("div.book 截图:", shot_info(
driver.find_element(By.CSS_SELECTOR, "div.book").screenshot_as_png))

print("\n== 验证码图片:元素截图正好截到 img ==")
driver.get("http://127.0.0.1:8010/login")
cap = driver.find_element(By.ID, "capimg")
png = cap.screenshot_as_png
print("capimg rect: %s | 元素截图: %s" % (cap.rect, shot_info(png)))
print("页面上写出的验证码:",
driver.find_element(By.CSS_SELECTOR, "p.tip").text.split(":")[-1])
gray = Image.open(io.BytesIO(png)).convert("L")
hist = gray.histogram()
total = gray.width * gray.height
ink = sum(v for i, v in enumerate(hist) if i < 128)
print("元素截图灰度直方图里最暗值: %d 最亮值: %d | 黑像素占比: %.1f%%"
% (min(i for i, v in enumerate(hist) if v),
max(i for i, v in enumerate(hist) if v), 100.0 * ink / total))
full = Image.open(io.BytesIO(driver.get_screenshot_as_png()))
r = cap.rect
crop = full.crop((int(r["x"]), int(r["y"]),
int(r["x"] + r["width"]), int(r["y"] + r["height"])))
buf = io.BytesIO()
crop.save(buf, "PNG")
print("从整页截图里裁出来:", shot_info(buf.getvalue()))
finally:
driver.quit()
1
2
3
4
5
6
7
8
9
== 元素截图 ==
table#stock rect: {'height': 58, 'width': 60, 'x': 8, 'y': 224.875} | 元素截图: ('60x58', 1955)
div.book 截图: ('966x269', 13135)

== 验证码图片:元素截图正好截到 img ==
capimg rect: {'height': 42, 'width': 102, 'x': 549.78125, 'y': 120.875} | 元素截图: ('102x42', 257)
页面上写出的验证码: 9437
元素截图灰度直方图里最暗值: 0 最亮值: 255 | 黑像素占比: 32.8%
从整页截图里裁出来: ('102x42', 242)
  • 元素截图尺寸和 rect 的宽高一致(102x42),拿来喂 OCR 是可行的:那张验证码灰度范围 0~255、黑像素占比 32.8%,从整页截图裁同区域也是 102x42,只是字节数略不同(257 对 242)。
  • 元素截图要求元素在视口内可见,被裁掉的和 display:none 的截不到。

在线站的验证码取不到

  • scrape.center 的 captcha1 子站是练验证码的,但实测在无头 Chrome 里拿不到验证码图:首屏只有两个 input 和一个 logo,填占位口令点登录后加载的是极验(geetest)脚本,img、canvas 一个都没多出来,也没有第三个输入框。
captcha1_probe.py
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
import time

from selenium import webdriver
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait

opts = Options()
opts.add_argument("--headless=new")
opts.add_argument("--window-size=1280,900")
driver = webdriver.Chrome(options=opts)
try:
driver.get("https://captcha1.scrape.center/")
WebDriverWait(driver, 20).until(
lambda d: len(d.find_elements(By.TAG_NAME, "input")) >= 2)
ins = driver.find_elements(By.TAG_NAME, "input")
ins[0].send_keys("placeholder_user")
ins[1].send_keys("password")
driver.find_element(By.CSS_SELECTOR, "button").click()
time.sleep(6)
print("[浏览器] 点登录 6 秒后: input %d | img %d | canvas %d | iframe %d" % (
len(driver.find_elements(By.TAG_NAME, "input")),
len(driver.find_elements(By.TAG_NAME, "img")),
len(driver.find_elements(By.TAG_NAME, "canvas")),
len(driver.find_elements(By.TAG_NAME, "iframe"))))
src = driver.page_source
print("[浏览器] page_source %d 字节 | 含 geetest: %s | 含 /captcha: %s"
% (len(src), "geetest" in src, "/captcha" in src))
print("[浏览器] .geetest_panel 的 display: %s | 面板内 img/canvas: %s"
% (driver.execute_script(
"var e = document.querySelector('.geetest_panel');"
"return e ? getComputedStyle(e).display : '没有这个节点';"),
driver.execute_script(
"var p = document.querySelector('.geetest_panel');"
"return p ? [p.querySelectorAll('img').length,"
"p.querySelectorAll('canvas').length] : '无';")))
res = driver.execute_script(
"return performance.getEntriesByType('resource')"
".map(e => [e.name, e.responseStatus, e.transferSize]);")
gt = [x for x in res if "geetest" in x[0]]
print("[浏览器] geetest 域名的 %d 个请求,responseStatus/transferSize: %s"
% (len(gt), sorted({(x[1], x[2]) for x in gt})))
finally:
driver.quit()
1
2
3
4
[浏览器] 点登录 6 秒后: input 2 | img 1 | canvas 0 | iframe 0
[浏览器] page_source 9615 字节 | 含 geetest: True | 含 /captcha: False
[浏览器] .geetest_panel 的 display: none | 面板内 img/canvas: [0, 0]
[浏览器] geetest 域名的 6 个请求,responseStatus/transferSize: [(0, 0)]
  • 结论写死:这个站现在测不了图片验证码,识别链路回到本机靶站的 /login——那里的 img#capimg 是真图片,页面还把答案写在 p.tip 里方便对分(真实站点不会这么好心)。识别只验链路,本篇不去真的登录,password 一律写占位字面量。

靶站验证码走一遍识别

  • 靶站的验证码是七段数码字体。先按「列投影切字符段 + 用站点自己的图建模板比像素」写一条链路,同一次循环里再把整图丢给 ddddocr 和 tesseract 对照。
captcha_ocr.py
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
import io

from PIL import Image
from selenium import webdriver
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.common.by import By

import ddddocr
import pytesseract

pytesseract.pytesseract.tesseract_cmd = r"C:\Program Files\Tesseract-OCR\tesseract.exe"
CFG = "--psm 8 -c tessedit_char_whitelist=0123456789"
opts = Options()
opts.add_argument("--headless=new")
driver = webdriver.Chrome(options=opts)
ocr = ddddocr.DdddOcr(show_ad=False)


def shot():
"""开一次登录页,返回(页面写出的答案, 按列投影切出的字符段矩阵, 原图字节)。"""
driver.delete_all_cookies()
driver.get("http://127.0.0.1:8010/login")
answer = driver.find_element(By.CSS_SELECTOR, "p.tip").text.split(":")[-1]
png = driver.find_element(By.ID, "capimg").screenshot_as_png
img = Image.open(io.BytesIO(png)).convert("1")
ink = [any(img.getpixel((x, y)) == 0 for y in range(img.height)) for x in range(img.width)]
edge = [x for x in range(1, len(ink)) if ink[x] != ink[x - 1]]
mats = []
for a, b in zip(edge[::2], edge[1::2]):
crop = img.crop((a, 0, b, img.height)).resize((18, 30), Image.NEAREST)
mats.append([1 if crop.getpixel((x, y)) == 0 else 0 for y in range(30) for x in range(18)])
return answer, mats, png


try:
tpl = {}
for answer, mats, _ in (shot() for _ in range(20)):
for ch, mat in zip(answer, mats):
tpl.setdefault(ch, mat)
print("建模板用了 20 张图 | 每张切成 %d 段" % len(mats))
print("模板覆盖字符: %s" % "".join(sorted(tpl)))
hits = [0, 0, 0]
for answer, mats, png in (shot() for _ in range(6)):
got = "".join(min(tpl, key=lambda ch, m=mat: sum(a != b for a, b in zip(m, tpl[ch])))
for mat in mats)
d, t = ocr.classification(png), pytesseract.image_to_string(
Image.open(io.BytesIO(png)).convert("L"), config=CFG).strip()
hits[0] += got == answer
hits[1] += d == answer
hits[2] += t == answer
print("答案 %s | 模板 %s | ddddocr %r | tesseract %r" % (answer, got, d, t))
print("6 次里模板比对对 %d 次、ddddocr 对 %d 次、tesseract 对 %d 次" % tuple(hits))
finally:
driver.quit()
1
2
3
4
5
6
7
8
9
建模板用了 20 张图 | 每张切成 4 段
模板覆盖字符: 0123456789
答案 5152 | 模板 5152 | ddddocr '引15已' | tesseract '5152'
答案 3295 | 模板 3295 | ddddocr '3E95' | tesseract '3295'
答案 7501 | 模板 7501 | ddddocr '7sd1' | tesseract '7501'
答案 7016 | 模板 7016 | ddddocr '7D16' | tesseract ''
答案 3637 | 模板 3637 | ddddocr '3637' | tesseract '36357'
答案 2924 | 模板 2924 | ddddocr '己924' | tesseract '2924'
6 次里模板比对对 6 次、ddddocr 对 1 次、tesseract 对 4 次
  • 同一批图,模板比对全对,ddddocr 六次只蒙对一次:8 被认成 s、0 被认成 d、3 被认成 引——通用模型没见过这种七段字体,而字符段位置每次都一样,比像素反而更稳。
  • 两个前提:delete_all_cookies() 每次都要清,靶站的验证码跟着会话走,不清就是拿同一张图反复测;模板必须覆盖全部字符,实测只用 8 张图建库时漏了字符 1,带 1 的答案就全错。

下载与上传

用偏好设置管下载

  • 下载目录、是否弹「保留/放弃」框都靠 prefs 关掉,不需要额外插件;判断完成的可靠信号是不再出现 .crdownload。
download_prefs.py
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
import os
import shutil
import tempfile
import time

from selenium import webdriver
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.common.by import By

DL = "http://127.0.0.1:8013/download"
tmp = tempfile.mkdtemp(prefix="dl-")
opts = Options()
opts.add_argument("--headless=new")
opts.add_experimental_option("prefs", {
"download.default_directory": tmp,
"download.prompt_for_download": False,
"safebrowsing.enabled": True,
})


def settle(timeout=60):
# 轮询目录,返回(耗时, 目录内容, 中途见过 .crdownload 吗)
t0 = time.time()
saw_partial = False
while time.time() - t0 < timeout:
names = os.listdir(tmp)
saw_partial |= any(n.endswith(".crdownload") for n in names)
if not any(n.endswith(".crdownload") for n in names) and names:
return time.time() - t0, names, saw_partial
time.sleep(0.1)
return time.time() - t0, os.listdir(tmp), saw_partial


driver = webdriver.Chrome(options=opts)
try:
driver.get(DL)

print("== 小文件:点带 download 属性的链接 ==")
driver.find_element(By.ID, "dl").click()
cost, _, saw_partial = settle()
print(" 完成耗时 %.2fs | 见过 .crdownload: %s | 大小 %d"
% (cost, saw_partial, os.path.getsize(os.path.join(tmp, "report.txt"))))

print("\n== 大文件:.crdownload 的时间线 ==")
t0 = time.time()
driver.find_element(By.ID, "dlbig").click()
last, next_print = None, 0.0
while time.time() - t0 < 60:
now = time.time() - t0
names = sorted((n, os.path.getsize(os.path.join(tmp, n)))
for n in os.listdir(tmp)
if n.startswith("big") or n.endswith(".crdownload"))
done = any(n == "big.bin" for n, _ in names)
if names and names != last and (now >= next_print or done):
print(" %5.2fs %s" % (now, names))
last = names
next_print = now + 2.0
if done:
break
time.sleep(0.1)
print(" 最终目录:", sorted((x, os.path.getsize(os.path.join(tmp, x)))
for x in os.listdir(tmp)))

print("\n== 同名文件再下一次 ==")
t0 = time.time()
driver.find_element(By.ID, "dlbig").click()
time.sleep(14)
print(" + %.1fs %s" % (time.time() - t0, sorted(os.listdir(tmp))))
finally:
driver.quit()
shutil.rmtree(tmp, ignore_errors=True)
1
2
3
4
5
6
7
8
9
10
11
12
13
14
== 小文件:点带 download 属性的链接 ==
完成耗时 0.10s | 见过 .crdownload: False | 大小 12400

== 大文件:.crdownload 的时间线 ==
0.13s [('未确认 319951.crdownload', 131072)]
2.14s [('未确认 319951.crdownload', 2752512)]
4.16s [('未确认 319951.crdownload', 5373952)]
6.18s [('未确认 319951.crdownload', 7995392)]
8.19s [('未确认 319951.crdownload', 8388608)]
13.52s [('big.bin', 8388608)]
最终目录: [('big.bin', 8388608), ('report.txt', 12400)]

== 同名文件再下一次 ==
+ 14.0s ['big (1).bin', 'big.bin', 'report.txt']
  • 8MB 的文件 0.13 秒就出现 .crdownload,8.19 秒字节数已经到位(8388608),但要到 13.52 秒才改名成 big.bin:中间那 5.3 秒不能按「字节数不再变」判完成,必须等扩展名落定。
  • 同名再下时 Chrome 自动加序号(实测第 14 秒出现 big (1).bin),要覆盖就得自己先清目录。
  • driver.get() 打开浏览器不渲染的类型也会触发下载(实测目录有变化),但拿不到进度;打开被防盗链拦住的图片时页面会变成靶站的拒绝页(实测 title: 404)。所以下载交给 requests,浏览器只负责需要点击才触发的下载。

文件上传

  • 上传不需要外部工具:input[type=file] 直接 send_keys 一个绝对路径;在线站不会随便给你一个能试的 file input,所以这节也用本机辅助页面。
upload.py
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
import os
from selenium import webdriver
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.common.by import By

path = os.path.abspath("data.csv")
opts = Options()
opts.add_argument("--headless=new")
driver = webdriver.Chrome(options=opts)
try:
driver.get("http://127.0.0.1:8013/upload")
box = driver.find_element(By.CSS_SELECTOR, "input[type=file]")
print("控件:", box.tag_name, "| 可见:", box.is_displayed())
box.send_keys(path)
print("回显:", driver.find_element(By.ID, "picked").text)
finally:
driver.quit()
1
2
3
4
5
6
7
== 上传:给 input[type=file] 直接 send_keys 绝对路径 ==
控件: tag=input type=file is_displayed=True
初始回显: '未选择文件'
-> 回显: 'books.csv|228字节'
换文件 -> 回显: 'report.txt|12400字节'
send_keys('abc') -> InvalidArgumentException | Message: invalid argument: File not found : abc
清空试试: clear 没报错 | value=''
  • 三个要点:路径必须存在且绝对,否则 InvalidArgumentException: File not found;很多站点把真正的 input 藏起来只留一个美化按钮,这时照样是对那个 input 送路径而不是点按钮;clear() 能清空且不报错。
  • 未实测:一次传多个文件、以及拖拽上传到 dropzone 的成功率。

代理与无头特征

无头的 UA 与指纹

  • 无头 Chrome 的 UA 里明晃晃写着 HeadlessChrome,服务端一眼能识别。
headless_fingerprint.py
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
import json

from selenium import webdriver
from selenium.webdriver.chrome.options import Options

WHOAMI = "http://127.0.0.1:8010/whoami"


def report(label, extra, full=True):
o = Options()
for a in extra:
o.add_argument(a)
driver = webdriver.Chrome(options=o)
try:
driver.get(WHOAMI)
outer = json.loads(driver.find_element("tag name", "pre").text)["headers"]
print(label)
print(" UA(外层) :", outer.get("User-Agent"))
if not full:
print(" navigator.webdriver:", driver.execute_script(
"return navigator.webdriver"))
print(" 屏幕分辨率 : %s" % driver.execute_script(
"return screen.width + 'x' + screen.height"))
print(" outer/inner : %s / %s" % (
driver.execute_script(
"return window.outerWidth + 'x' + window.outerHeight"),
driver.execute_script(
"return window.innerWidth + 'x' + window.innerHeight")))
return
inner = driver.execute_async_script(
"var cb = arguments[arguments.length - 1];"
"fetch('/whoami').then(function (r) { return r.json(); })"
".then(function (d) { cb(d.headers['User-Agent']); });")
print(" UA(页面内 fetch) :", inner)
print(" Accept-Language :", outer.get("Accept-Language"))
wd, has_chrome, plugins, screen_, outer_wh, inner_wh = driver.execute_script(
"return [navigator.webdriver, !!window.chrome,"
"navigator.plugins.length,"
"screen.width + 'x' + screen.height,"
"window.outerWidth + 'x' + window.outerHeight,"
"window.innerWidth + 'x' + window.innerHeight];")
print(" navigator.webdriver:", wd)
print(" window.chrome 有吗 :", has_chrome)
print(" plugins 数量 :", plugins)
print(" 屏幕分辨率 :", screen_)
print(" outer/inner : %s / %s" % (outer_wh, inner_wh))
finally:
driver.quit()


report("[headless=new]", ["--headless=new"])
report("[headless=none]", [], full=False)
1
2
3
4
5
6
7
8
9
10
11
12
13
14
[headless=new]
UA(外层) : Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) HeadlessChrome/156.0.0.0 Safari/537.36
UA(页面内 fetch) : Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) HeadlessChrome/156.0.0.0 Safari/537.36
Accept-Language : zh-CN,zh;q=0.9
navigator.webdriver: True
window.chrome 有吗 : True
plugins 数量 : 5
屏幕分辨率 : 800x600
outer/inner : 780x580 / 762x428
[headless=none]
UA(外层) : Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/156.0.0.0 Safari/537.36
navigator.webdriver: True
屏幕分辨率 : 2048x1152
outer/inner : 1010x1086 / 996x935
  • headless=new 和 headless=old 拿到的 UA 完全一样(都是 HeadlessChrome/156.0.0.0),差别只在功能完整度;navigator.webdriver 在三种模式下全部为真,这是比 UA 更硬的特征。
  • 无头固定 800x600 屏幕、780x580/762x428 窗口视口,也是「有头不会有的组合」。靶站的判定:requests 直接打 /ua 被拒,换个像样的 UA 才放行。
ua_check.py
1
2
3
4
5
6
7
8
9
10
import requests

UA = "http://127.0.0.1:8010/ua"
r = requests.get(UA)
print(" 响应:", r.text.replace("\n", " ").strip())
good = ("Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 "
"(KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36")
r2 = requests.get(UA, headers={"User-Agent": good})
print(" 换成正常 UA: %s %s"
% (r2.status_code, r2.text.replace("\n", " ").replace(" ", " ").strip()))
1
2
响应: {   "ok": false,   "ua": "python-requests/2.34.2",   "判定": "非浏览器 UA,拒绝" }
换成正常 UA: 200 { "ok": true, "ua": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36", "判定": "放行" }

抹掉 webdriver 标记

hide_webdriver.py
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
from selenium import webdriver
from selenium.webdriver.chrome.options import Options

WHOAMI = "http://127.0.0.1:8010/whoami"

for label, extra in (
("[--disable-blink-features=AutomationControlled]",
["--disable-blink-features=AutomationControlled"]),
("[excludeSwitches=enable-automation]", []),
):
o = Options()
o.add_argument("--headless=new")
for a in extra:
o.add_argument(a)
if label.startswith("[excludeSwitches"):
o.add_experimental_option("excludeSwitches", ["enable-automation"])
driver = webdriver.Chrome(options=o)
try:
driver.get(WHOAMI)
print(label)
if label.startswith("[--disable-blink"):
print(" UA :", driver.execute_script("return navigator.userAgent"))
print(" webdriver: %s | 'webdriver' in navigator: %s" % (
driver.execute_script("return navigator.webdriver"),
driver.execute_script("return 'webdriver' in navigator")))
if label.startswith("[--disable-blink"):
keys = driver.execute_script("return Object.getOwnPropertyNames(window);")
print(" chromedriver 注入的全局变量: %s"
% [k for k in keys if "cdc_" in k or "webdriver" in k][:6])
finally:
driver.quit()
1
2
3
4
5
6
[--disable-blink-features=AutomationControlled]
UA : Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) HeadlessChrome/156.0.0.0 Safari/537.36
webdriver: False | 'webdriver' in navigator: True
chromedriver 注入的全局变量: ['cdc_adoQpoasnfa76pfcZLmcfl_Array', 'cdc_adoQpoasnfa76pfcZLmcfl_Object', 'cdc_adoQpoasnfa76pfcZLmcfl_Promise', 'cdc_adoQpoasnfa76pfcZLmcfl_Proxy', 'cdc_adoQpoasnfa76pfcZLmcfl_Symbol', 'cdc_adoQpoasnfa76pfcZLmcfl_JSON']
[excludeSwitches=enable-automation]
webdriver: True | 'webdriver' in navigator: True
  • 真正管用的是 --disable-blink-features=AutomationControlled:navigator.webdriver 从 True 变 False,但 'webdriver' in navigator 仍是 True(属性还在,只是值为假),UA 里 HeadlessChrome 一个字符都没变——只关这一个开关等于没隐身。
  • 更要紧的是 chromedriver 注入的 cdc_adoQpoasnfa76pfcZLmcfl_* 那串全局变量还在(上面实测列出六个);要连这些一起处理得在页面加载前用 CDP 注入脚本,本篇未实测。excludeSwitches 单独用则完全没效果。

HTTP 代理

  • Chrome 的代理参数很好写,但对 127.0.0.1 这类回环地址默认不走代理。
proxy_loop.py
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
import json

from selenium import webdriver
from selenium.webdriver.chrome.options import Options

WHOAMI = "http://127.0.0.1:8010/whoami"


def start(extra):
o = Options()
o.add_argument("--headless=new")
for a in extra:
o.add_argument(a)
return webdriver.Chrome(options=o)


for label, extra in (("[不加代理]", []),
("[--proxy-server=127.0.0.1:8011]",
["--proxy-server=http://127.0.0.1:8011"]),
("[再加 --proxy-bypass-list=<-loopback>]",
["--proxy-server=http://127.0.0.1:8011",
"--proxy-bypass-list=<-loopback>"])):
driver = start(extra)
try:
driver.get(WHOAMI)
data = json.loads(driver.find_element("tag name", "pre").text)
head = data["headers"]
print(label)
print(" Via : %s" % head.get("Via", "(无)"))
print(" XFF : %s" % head.get("X-Forwarded-For", "(无)"))
print(" Host : %s | 请求路径: %s" % (head.get("Host"), data["path"]))
finally:
driver.quit()

print("[代理端口写错]", end=" ")
driver = start(["--proxy-server=http://127.0.0.1:1", "--proxy-bypass-list=<-loopback>"])
try:
driver.get(WHOAMI)
print("-> 页面照样打开了")
except Exception as exc:
print("-> %s | %s" % (type(exc).__name__, str(exc).splitlines()[0]))
finally:
driver.quit()
1
2
3
4
5
6
7
8
9
10
11
12
13
[不加代理]
Via : (无)
XFF : (无)
Host : 127.0.0.1:8010 | 请求路径: /whoami
[--proxy-server=127.0.0.1:8011]
Via : (无)
XFF : (无)
Host : 127.0.0.1:8010 | 请求路径: /whoami
[再加 --proxy-bypass-list=<-loopback>]
Via : 1.1 spider-proxy
XFF : 127.0.0.1
Host : 127.0.0.1:8010 | 请求路径: /whoami
[代理端口写错] -> WebDriverException | Message: unknown error: net::ERR_PROXY_CONNECTION_FAILED
proxy.py
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
import json

from selenium import webdriver
from selenium.webdriver.chrome.options import Options

opts = Options()
opts.add_argument("--headless=new")
opts.add_argument("--proxy-server=http://127.0.0.1:8011")
opts.add_argument("--proxy-bypass-list=<-loopback>") # 不加这行就还是直连,原因见下文
driver = webdriver.Chrome(options=opts)
try:
driver.get("http://127.0.0.1:8010/whoami")
data = json.loads(driver.find_element("tag name", "pre").text)
print("client:", data["client"])
for key in ("Via", "X-Forwarded-For"):
print("%s: %s" % (key, data["headers"].get(key, "-")))
print("收到的头总数:", len(data["headers"]))
finally:
driver.quit()
1
2
3
4
client: 127.0.0.1
Via: 1.1 spider-proxy
X-Forwarded-For: 127.0.0.1
收到的头总数: 15
  • 加上 <-loopback> 之后代理才真的生效:响应头出现 Via: 1.1 spider-proxy、X-Forwarded-For: 127.0.0.1;requests 走同一代理本来就能看到 Via,所以这是 Chrome 侧的策略不是代理的问题。
  • 代理端口写错时,请求回环地址默认绕过代理、页面照样打开;只有配合 <-loopback> 强制让回环也走代理,才抛 net::ERR_PROXY_CONNECTION_FAILED,报错很好认。
  • SOCKS5 写法 --proxy-server=socks5://127.0.0.1:8012 实测能启动、页面能开,但配合 <-loopback> 也没让回环请求走代理(Via 仍是「无」)。未实测公网代理下的真实出口 IP 和带认证的代理。

实战取数

主线:渲染页取数并落地

spa1_scrape.py
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
import csv
import os
import time

from selenium import webdriver
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.support.ui import WebDriverWait

CARD = (By.CSS_SELECTOR, "div.el-card.item")
opts = Options()
opts.add_argument("--headless=new")
driver = webdriver.Chrome(options=opts)
records, links = [], []
try:
t0 = time.time()
driver.get("https://spa1.scrape.center/")
print("get() 返回: %.2fs | 此刻 page_source %d 字节 | 卡片 %d 张" % (
time.time() - t0, len(driver.page_source), len(driver.find_elements(*CARD))))
cards = WebDriverWait(driver, 10).until(EC.presence_of_all_elements_located(CARD))
print("显式等待 %d 张卡片: %.2fs | 此刻 page_source %d 字节" % (
len(cards), time.time() - t0, len(driver.page_source)))
for i, card in enumerate(cards[:3], 1):
name = card.find_element(By.TAG_NAME, "h2").text
score = card.find_element(By.CSS_SELECTOR, "p.score").text
href = card.find_element(By.CSS_SELECTOR, "a").get_attribute("href")
print("%d | %s | %s | %s" % (i, name, score, href))
links.append(href)
print("片名锚点的 class: %r" % cards[0].find_element(
By.CSS_SELECTOR, "h2").find_element(By.XPATH, "./..").get_attribute("class"))
for url in links:
driver.get(url)
WebDriverWait(driver, 15).until(
EC.presence_of_element_located((By.CSS_SELECTOR, "div.info")))
infos = [x.text for x in driver.find_elements(By.CSS_SELECTOR, "div.info")]
records.append({
"片名": driver.find_element(By.TAG_NAME, "h2").text,
"评分": driver.find_element(By.CSS_SELECTOR, "p.score").text,
"地区时长": infos[0], "上映": infos[1],
"类型": "|".join(x.text for x in driver.find_elements(By.CSS_SELECTOR, "button.category"))})
print(records[-1])
finally:
driver.quit()

with open("movies.csv", "w", newline="", encoding="utf-8-sig") as f:
w = csv.DictWriter(f, fieldnames=["片名", "评分", "地区时长", "上映", "类型"])
w.writeheader()
w.writerows(records)
print("\nCSV %d 字节 | 文件头 12 字节: %r" % (
os.path.getsize("movies.csv"), open("movies.csv", "rb").read(12)))
1
2
3
4
5
6
7
8
9
10
11
get() 返回: 0.49s | 此刻 page_source 2544 字节 | 卡片 0 张
显式等待 10 张卡片: 2.55s | 此刻 page_source 27979 字节
1 | 霸王别姬 - Farewell My Concubine | 9.5 | https://spa1.scrape.center/detail/1
2 | 这个杀手不太冷 - Léon | 9.5 | https://spa1.scrape.center/detail/2
3 | 肖申克的救赎 - The Shawshank Redemption | 9.5 | https://spa1.scrape.center/detail/3
片名锚点的 class: ''
{'片名': '霸王别姬 - Farewell My Concubine', '评分': '9.5', '地区时长': '中国内地、中国香港 / 171 分钟', '上映': '1993-07-26 上映', '类型': '剧情|爱情'}
{'片名': '这个杀手不太冷 - Léon', '评分': '9.5', '地区时长': '法国 / 110 分钟', '上映': '1994-09-14 上映', '类型': '剧情|动作|犯罪'}
{'片名': '肖申克的救赎 - The Shawshank Redemption', '评分': '9.5', '地区时长': '美国 / 142 分钟', '上映': '1994-09-10 上映', '类型': '剧情|犯罪'}

CSV 357 字节 | 文件头 12 字节: b'\xef\xbb\xbf\xe7\x89\x87\xe5\x90\x8d,\xe8\xaf
  • get() 返回时那份 2544 字节的 DOM 里一张卡片都没有,presence_of_all_elements_located 等到 10 张齐了才往下走,page_source 也在这一步涨到 27979 字节。
  • 选择器只能照这套页面真实渲染出来的结构写:SPA 版片名锚点的 class 是空串,服务端渲染的 ssr1 同一段却是 class="name",照抄另一套直接 NoSuchElementException。
  • 先把三张卡片的链接取完再逐个 get():导航一次旧元素引用全部作废,边遍历卡片边跳转会直接撞上 stale(见常见报错那节)。详情页两个 div.info 分别是「地区 / 时长」和上映日期,类型是多个按钮拼 |。
  • 中文 CSV 用 utf-8-sig(带 BOM)Excel 才不乱码:文件头前三字节就是 b'\xef\xbb\xbf',整个文件 357 字节。

翻页两种路线

spa_paging.py
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
import json
import time
import urllib.request

from selenium import webdriver
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.support.ui import WebDriverWait

ROOT = "https://spa1.scrape.center/"
CARD = (By.CSS_SELECTOR, "div.el-card.item")
opts = Options()
opts.add_argument("--headless=new")


def wait_cards(driver, timeout=8.0):
# 轮询到 10 张卡片或超时,返回(等了多久, 卡片数)
t0 = time.time()
while time.time() - t0 < timeout:
n = len(driver.find_elements(*CARD))
if n >= 10:
return time.time() - t0, n
time.sleep(0.2)
return time.time() - t0, len(driver.find_elements(*CARD))


print("== 路线一:点「下一页」 ==")
driver = webdriver.Chrome(options=opts)
try:
driver.get(ROOT)
WebDriverWait(driver, 15).until(EC.presence_of_all_elements_located(CARD))
print("第 1 页 url: %s | 首条: %s"
% (driver.current_url, driver.find_element(By.CSS_SELECTOR, "h2").text))
btn = driver.find_element("css selector", "button.btn-next")
print("下一页按钮: tag=%s class=%r text=%r"
% (btn.tag_name, btn.get_attribute("class"), btn.text))
first = driver.find_element(By.CSS_SELECTOR, "h2").text
t0 = time.time()
btn.click()
WebDriverWait(driver, 15).until(
lambda d: d.find_element(By.CSS_SELECTOR, "h2").text != first)
print("翻页后 url: %s | 首条: %s | 用时 %.2fs | 卡片 %d 张" % (
driver.current_url, driver.find_element(By.CSS_SELECTOR, "h2").text,
time.time() - t0, len(driver.find_elements(*CARD))))
pager = [x.text for x in driver.find_elements(
"css selector", ".el-pager li")][:8]
print("页码条: %s | 当前页: %s" % (
pager, driver.execute_script(
"var e=document.querySelector('.el-pager li.active');"
"return e ? e.textContent : '没有 active'")))
finally:
driver.quit()

print("\n== 路线二:直接拼 URL ==")
for n in (2, 10, 11):
driver = webdriver.Chrome(options=opts)
try:
driver.get("%spage/%d" % (ROOT, n))
cost, cnt = wait_cards(driver)
if cnt:
print("/page/%-3d -> %d 张 | 首条 %s(等 %.2fs)"
% (n, cnt, driver.find_element(By.CSS_SELECTOR, "h2").text, cost))
else:
print("/page/%-3d -> 8 秒内 0 张卡片(等 %.2fs)" % (n, cost))
finally:
driver.quit()

r = urllib.request.urlopen(ROOT + "api/movie?limit=10&offset=100")
print("接口 offset=100 -> %s | 条数 %d"
% (r.getcode(), len(json.loads(r.read().decode("utf-8"))["results"])))
1
2
3
4
5
6
7
8
9
10
11
== 路线一:点「下一页」 ==
第 1 页 url: https://spa1.scrape.center/ | 首条: 霸王别姬 - Farewell My Concubine
下一页按钮: tag=button class='btn-next' text=''
翻页后 url: https://spa1.scrape.center/page/2 | 首条: V字仇杀队 - V for Vendetta | 用时 1.10s | 卡片 10 张
页码条: ['1', '2', '3', '4', '5', '6', '', '11'] | 当前页: 2

== 路线二:直接拼 URL ==
/page/2 -> 10 张 | 首条 V字仇杀队 - V for Vendetta(等 0.88s)
/page/10 -> 10 张 | 首条 哈利·波特与魔法石 - Harry Potter and the Sorcerer's Stone(等 1.04s)
/page/11 -> 8 秒内 0 张卡片(等 8.18s)
接口 offset=100 -> 200 | 条数 4
  • 路线一跟着浏览器走,停止条件是「下一页按钮不存在」;点完必须先确认列表第一个元素变了再取数,否则读到的是上一页残留。
  • 路线二快得多,但要自己判越界:/page/11 的接口是 200、还有 4 条数据,页面却一张卡片都不渲染,所以判据是「等不到卡片」而不是 HTTP 状态码。页码条里那个空字符串是省略号占位,照着它数总页数会错。

本地最小对照:ajax 页

  • 在线站的渲染流程走完了,留一个本机最小对照:数据由点击触发,loaded 这个隐藏域才是「取完了」的信号,完整链路是「点按钮 → 等 loaded=yes → 再取列表」。
ajax_loaded.py
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
from selenium import webdriver
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.support.ui import WebDriverWait

print("== 本地最小对照:ajax 页 loaded 信号 ==")
opts = Options()
opts.add_argument("--headless=new")
driver = webdriver.Chrome(options=opts)
try:
driver.get("http://127.0.0.1:8010/ajax")

def rows(d):
return d.find_elements(By.CSS_SELECTOR, "ul#rows li")

print("点之前 li 数量: %d | loaded: %s"
% (len(rows(driver)),
driver.find_element(By.ID, "loaded").get_attribute("value")))
driver.find_element(By.ID, "btn").click()
WebDriverWait(driver, 10).until(
EC.text_to_be_present_in_element_value((By.ID, "loaded"), "yes"))
print("点之后 li 数量: %d | loaded: %s"
% (len(rows(driver)),
driver.find_element(By.ID, "loaded").get_attribute("value")))
print("逐条 text:", [x.text for x in rows(driver)])
n = driver.execute_async_script(
"var cb = arguments[arguments.length - 1];"
"fetch('/api/books').then(function (r) { return r.json(); })"
".then(function (d) { cb(d.data.books.length); });")
print("用 JS 直接取接口数据:", n)
finally:
driver.quit()
1
2
3
4
5
== 本地最小对照:ajax 页 loaded 信号 ==
点之前 li 数量: 0 | loaded: no
点之后 li 数量: 4 | loaded: yes
逐条 text: ['三国演义/罗贯中', '水浒传/施耐庵', '西游记/吴承恩', '红楼梦/曹雪芹']
用 JS 直接取接口数据: 4
  • 「用 JS 直接取接口数据」是从页面已缓存的响应里数出 4 条;接口本身就是 JSON 时,看一眼 Network 面板,可能一段 requests 就替代掉整个浏览器。

常见报错

报错速查

场景 异常 报错里的关键字
元素不存在 NoSuchElementException Unable to locate element
XPath 或 CSS 写错 InvalidSelectorException invalid selector
JS 语法错 JavascriptException javascript error: Unexpected token
URL 少协议头 InvalidArgumentException invalid argument
给隐藏元素打字 ElementNotInteractableException element not interactable
被别的元素盖住 ElementClickInterceptedException element click intercepted
拖到视口外 MoveTargetOutOfBoundsException move target out of bounds
页面跳转后用旧引用 StaleElementReferenceException stale element reference
显式等待超时 TimeoutException 空 message + Stacktrace
加载超时 TimeoutException Timed out receiving message from renderer
异步脚本没回调 TimeoutException script timeout
弹窗开着就操作 UnexpectedAlertPresentException Alert Text:
切错窗口或框架 NoSuchWindowException / NoSuchFrameException no such window
驱动路径错 NoSuchDriverException Unable to obtain driver for chrome
代理连不上 WebDriverException net::ERR_PROXY_CONNECTION_FAILED

最脏的那个:stale

  • 页面重画(前端框架换组件、列表刷新)之后,之前拿到的元素引用全部作废,读写都炸。
stale_redraw.py
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
from selenium import webdriver
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.common.by import By

print("-- 整块重画导致的 stale --")
opts = Options()
opts.add_argument("--headless=new")
driver = webdriver.Chrome(options=opts)
try:
driver.get("http://127.0.0.1:8013/stale")
btn = driver.find_element(By.ID, "go")
print(" 点之前 text:", btn.text)
btn.click()
for label, fn in (("重画后用旧引用", lambda: btn.text),
("重画后 click", lambda: btn.click())):
try:
fn()
out = "没报错"
except Exception as exc:
out = "%s\n %s" % (type(exc).__name__, str(exc).splitlines()[0])
print("%-34s %s" % (label, out))
print(" 重新找一次:", driver.find_element(By.ID, "go").text)
finally:
driver.quit()
1
2
3
4
5
6
7
-- 整块重画导致的 stale --
点之前 text: 我是会被替换掉的按钮
重画后用旧引用 StaleElementReferenceException
Message: stale element reference: stale element not found in the current frame
重画后 click StaleElementReferenceException
Message: stale element reference: stale element not found in the current frame
重新找一次: 替换后的新按钮
  • 修法是重新定位一次,别在旧引用上重试;批量取数时把「定位 → 立刻读字段」贴紧,别把元素对象存进列表很久再用。跨页面用旧元素、在旧元素上找子元素都是同一个异常。

点不动与不可交互

covered_and_inert.py
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
from selenium import webdriver
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.common.by import By

opts = Options()
opts.add_argument("--headless=new")
driver = webdriver.Chrome(options=opts)
try:
print("-- 被别的元素盖住 --")
driver.get("http://127.0.0.1:8013/covered")
btn = driver.find_element(By.ID, "hidden-btn")
try:
btn.click()
print("点被遮挡的按钮 居然点动了")
except Exception as exc:
print("点被遮挡的按钮 %s" % type(exc).__name__)
print(" %s" % str(exc).splitlines()[0][:220])
driver.execute_script("arguments[0].click()", btn)
print(" 用 JS 点:", repr(driver.find_element(By.ID, "log").text))

print("\n-- 不可交互 --")
driver.get("http://127.0.0.1:8010/form")
driver.execute_script(
"var hid = document.createElement('input');"
"hid.id = 'hidden-css'; hid.style.display = 'none';"
"var ro = document.createElement('input');"
"ro.id = 'readonly-box'; ro.readOnly = true;"
"document.body.append(hid, ro);")
cases = [
("给 hidden input 打字", "form#post input[type=hidden]"),
("给 display:none 打字", "#hidden-css"),
("给 readonly 打字", "#readonly-box"),
]
for label, sel in cases:
el = driver.find_element(By.CSS_SELECTOR, sel)
try:
el.send_keys("x")
out = "没报错"
except Exception as exc:
out = "%s\n %s" % (type(exc).__name__, str(exc).splitlines()[0])
print("%-34s %s" % (label, out))
finally:
driver.quit()
1
2
3
4
5
6
7
8
9
10
11
-- 被别的元素盖住 --
点被遮挡的按钮 ElementClickInterceptedException
Message: element click intercepted: Element <button id="hidden-btn">...</button> is not clickable at point (55, 102). Other element would receive the click: <div id="mask" style="position:absolute;left:0;top:0;width:200p
用 JS 点: '按钮被点了'

-- 不可交互 --
给 hidden input 打字 ElementNotInteractableException
Message: element not interactable
给 display:none 打字 ElementNotInteractableException
Message: element not interactable
给 readonly 打字 没报错
  • 被遮住的报错直接告诉你谁挡住了;应急可以用 JS 点(execute_script("arguments[0].click()", el),实测能点动),但它绕过页面校验,提交类操作慎用,正规做法是先让遮罩消失或把元素滚进视口。
  • readonly 输入框 send_keys 不报错,别指望靠异常发现它,要读属性判断。

参考