293 lines
6.8 KiB
Markdown
293 lines
6.8 KiB
Markdown
# cloakfetch-gateway
|
||
|
||
CloakFetch Gateway 是局域网内使用的浏览器网络执行服务。其他局域网机器可以通过 HTTP API 调用真实 CloakBrowser Chromium 发起请求，并通过 Web Console 观察代理、Cookie、缓存、重定向、页面加载和 Chromium 内部网络事件。
|
||
|
||
它不是 HTTP/SOCKS 代理，也不是通用爬虫框架。当前只保留 GET document navigation，不提供 raw HTTP 引擎，不支持 image/fetch 专用模式。
|
||
|
||
## 运行
|
||
|
||
```powershell
|
||
npm install
|
||
npx cloakbrowser install
|
||
$env:CLOAKFETCH_HOST = "0.0.0.0"
|
||
$env:CLOAKFETCH_HEADLESS = "false"
|
||
$env:CLOAKFETCH_CONCURRENCY = "4"
|
||
$env:CLOAKFETCH_PROXY = "http://127.0.0.1:7890"
|
||
npm run dev
|
||
```
|
||
|
||
生产构建和运行：
|
||
|
||
```powershell
|
||
npm run build
|
||
npm run start
|
||
```
|
||
|
||
局域网访问控制台：
|
||
|
||
```text
|
||
http://<服务器局域网IP>:9230/console
|
||
```
|
||
|
||
服务不会自动获取或猜测服务器局域网 IP。
|
||
|
||
## 配置
|
||
|
||
```text
|
||
CLOAKFETCH_HOST=0.0.0.0
|
||
CLOAKFETCH_PORT=9230
|
||
CLOAKFETCH_PROFILE=profile
|
||
CLOAKFETCH_PROXY=http://127.0.0.1:7890
|
||
CLOAKFETCH_HEADLESS=false
|
||
CLOAKFETCH_CONCURRENCY=4
|
||
CLOAKFETCH_TIMEOUT_MS=30000
|
||
CLOAKFETCH_MAX_BODY_MB=64
|
||
CLOAKFETCH_HISTORY_LIMIT=1000
|
||
CLOAKFETCH_WS_HEARTBEAT_MS=15000
|
||
CLOAKFETCH_NETWORK_HISTORY_LIMIT=10000
|
||
CLOAKFETCH_NETWORK_BODY_CAPTURE_BYTES=1048576
|
||
```
|
||
|
||
`CLOAKFETCH_HOST` 默认是 `0.0.0.0`，也可以设置为 `127.0.0.1` 或具体 IPv4/IPv6 地址。
|
||
|
||
默认 source profile 路径是：
|
||
|
||
```text
|
||
profile/sources/<source_id>
|
||
```
|
||
|
||
所有项目内部路径都相对于项目根目录，不依赖启动时的当前工作目录。`profile/` 是运行数据，不应打包、上传或提交。
|
||
|
||
当前 CloakBrowser 版本在部分平台会把带认证代理转成 Chromium 命令行参数，因此 `CLOAKFETCH_PROXY` 中包含 username/password 时会在启动阶段被拒绝，错误码为 `proxy_auth_not_supported`。建议把 Mihomo 或其他上游代理暴露成本地无认证端口，例如 `http://127.0.0.1:7890`。
|
||
|
||
## 基础 API
|
||
|
||
`GET /health`
|
||
|
||
```json
|
||
{ "status": "ok" }
|
||
```
|
||
|
||
`POST /v1/fetch`
|
||
|
||
```json
|
||
{
|
||
"source_id": "default",
|
||
"url": "https://example.com/file.png",
|
||
"headers": {
|
||
"Referer": "https://example.com/"
|
||
},
|
||
"timeout_ms": 30000,
|
||
"cache_mode": "default"
|
||
}
|
||
```
|
||
|
||
未传 `source_id` 时使用 `default` SourceSession。响应状态码来自目标站，响应体是目标二进制内容。额外响应头：
|
||
|
||
```text
|
||
x-cloakfetch-final-url
|
||
x-cloakfetch-fetch-mode
|
||
```
|
||
|
||
`cache_mode`：
|
||
|
||
- `default`：允许 Chromium 使用正常缓存语义。
|
||
- `reload`：禁用当前页面缓存。
|
||
- `/v1/diagnostics/fetch` 会强制使用 `reload`，避免缓存影响代理和目标站诊断。
|
||
|
||
调用方传入的 headers 只应用到第一次主文档导航请求。页面后续 script、style、image、iframe、xhr、fetch、websocket 等请求完全由 Chromium 自己生成。
|
||
|
||
响应大小会先检查 `Content-Length`；没有 `Content-Length` 的响应只能在读取完成后检查实际大小，当前版本不支持真正流式限速。
|
||
|
||
`POST /v1/diagnostics/fetch`
|
||
|
||
返回 JSON 诊断信息，不返回二进制 body：
|
||
|
||
```json
|
||
{
|
||
"url": "https://example.com/file.jpg",
|
||
"final_url": "https://example.com/file.jpg",
|
||
"status": 200,
|
||
"ok": true,
|
||
"content_type": "image/jpeg",
|
||
"body_size": 44891,
|
||
"elapsed_ms": 421,
|
||
"fetch_mode": "document",
|
||
"error": null
|
||
}
|
||
```
|
||
|
||
`GET /v1/stats`
|
||
|
||
```json
|
||
{
|
||
"lifetime_total": 12,
|
||
"lifetime_failed": 1,
|
||
"active": 0,
|
||
"queued": 0
|
||
}
|
||
```
|
||
|
||
## SourceSession API
|
||
|
||
`default` SourceSession 会自动存在。每个 SourceSession 拥有独立 BrowserContext、独立 profile、独立 Cookie、缓存、LocalStorage、IndexedDB 和 Service Worker。
|
||
|
||
`POST /v1/sources`
|
||
|
||
```json
|
||
{
|
||
"id": "arcgis",
|
||
"warmup_url": "https://www.arcgis.com/",
|
||
"locale": "zh-CN",
|
||
"viewport": {
|
||
"width": 1920,
|
||
"height": 1080
|
||
}
|
||
}
|
||
```
|
||
|
||
`POST /v1/sources/:id/warmup`
|
||
|
||
```json
|
||
{
|
||
"url": "https://www.arcgis.com/",
|
||
"wait_until": "domcontentloaded",
|
||
"timeout_ms": 30000
|
||
}
|
||
```
|
||
|
||
其他 source API：
|
||
|
||
```text
|
||
GET /v1/sources
|
||
GET /v1/sources/:id
|
||
DELETE /v1/sources/:id
|
||
GET /v1/sources/:id/cookies
|
||
GET /v1/sources/:id/pages
|
||
```
|
||
|
||
`source_id` 只用于路径名称映射，只允许字母、数字、点、下划线和短横线。
|
||
|
||
## Network Observer
|
||
|
||
Chromium 内部网络事件使用独立 ring buffer，和外层 `/v1/fetch` 请求历史分开。默认保留最近 10000 条，可通过 `CLOAKFETCH_NETWORK_HISTORY_LIMIT` 调整。
|
||
|
||
记录字段包括：
|
||
|
||
```text
|
||
id, source_id, page_id, request_id, parent_request_id,
|
||
started_at, finished_at, duration_ms,
|
||
url, method, resource_type, navigation, frame_url, initiator,
|
||
request_headers, request_body_size, request_post_data,
|
||
status, status_text, protocol, response_headers, response_body_size,
|
||
mime_type, remote_address, from_cache, service_worker,
|
||
failure_text, redirect_from, redirect_to
|
||
```
|
||
|
||
请求头、响应头和 post data 默认完整记录。`CLOAKFETCH_NETWORK_BODY_CAPTURE_BYTES` 只限制观测存储大小，不限制真实请求。
|
||
|
||
```text
|
||
0 = 不抓取 body 内容，只记录大小
|
||
-1 = 不限制抓取大小
|
||
```
|
||
|
||
Network API：
|
||
|
||
```text
|
||
GET /v1/network/recent
|
||
GET /v1/network/:id
|
||
GET /v1/network/:id/body
|
||
GET /v1/network/summary
|
||
DELETE /v1/network/history
|
||
```
|
||
|
||
`GET /v1/network/recent` 支持：
|
||
|
||
```text
|
||
limit, source_id, page_id, method, resource_type, status, host, failed, from_cache, text
|
||
```
|
||
|
||
`GET /v1/network/summary` 返回 retained、active、finished、failed、method/resource/status/host/protocol 分布、cache hit、字节数和耗时统计。
|
||
|
||
`DELETE /v1/network/history` 只清空观测历史，不关闭 BrowserContext。
|
||
|
||
## WebSocket
|
||
|
||
`GET /v1/ws`
|
||
|
||
```js
|
||
const ws = new WebSocket("ws://127.0.0.1:9230/v1/ws");
|
||
ws.onmessage = event => console.log(JSON.parse(event.data));
|
||
```
|
||
|
||
消息格式：
|
||
|
||
```json
|
||
{
|
||
"version": 1,
|
||
"sequence": 123,
|
||
"type": "network.finished",
|
||
"time": "2026-07-24T12:00:00.000Z",
|
||
"data": {}
|
||
}
|
||
```
|
||
|
||
事件类型：
|
||
|
||
```text
|
||
hello
|
||
heartbeat
|
||
request
|
||
source.created
|
||
source.starting
|
||
source.ready
|
||
source.failed
|
||
source.removed
|
||
page.created
|
||
page.closed
|
||
network.request
|
||
network.response
|
||
network.finished
|
||
network.failed
|
||
browser.console
|
||
browser.pageerror
|
||
browser.crash
|
||
```
|
||
|
||
不提供 SSE，不提供 Socket.IO。
|
||
|
||
## Web Console
|
||
|
||
构建后由 Fastify 托管：
|
||
|
||
```powershell
|
||
npm run build
|
||
npm run start
|
||
```
|
||
|
||
访问：
|
||
|
||
```text
|
||
http://127.0.0.1:9230/console
|
||
http://<服务器局域网IP>:9230/console
|
||
```
|
||
|
||
控制台包含：
|
||
|
||
- Runtime 状态和代理状态
|
||
- Source Sessions 状态、Warmup、Remove、Open pages、View cookies
|
||
- 外层 Request Log
|
||
- Network Inspector
|
||
- Network 过滤、详情、Pause live update、Clear local view、清空服务端 network history
|
||
|
||
Network WebSocket 事件在前端按 150ms 批量合并，前端最多保留 5000 条 network record。
|
||
|
||
开发模式：
|
||
|
||
```powershell
|
||
npm run dev
|
||
npm run web:dev
|
||
```
|
||
|
||
如果 `web/dist` 不存在，`/console` 返回 `console_not_built` 的 404 JSON，不影响 `/health` 和 `/v1/*` API。
|