Pārlūkot izejas kodu

fix(deploy): 镜像源可配置 + 修复容器运行期路径与 externals 类型

- Dockerfile: 新增 USE_CN_MIRROR 构建参数(默认 1=国内镜像),统一门控
  4 个取源点(corepack/npm registry、apt+pypi、pnpm registry、Chrome 下载源);
  海外 release/CI 传 USE_CN_MIRROR=0 走官方源。lockfile 锁版本/integrity,registry 仅决定下载来源
- Dockerfile: 预置 Chrome 缓存软链到 /app/apps/web/node_modules/.remotion,
  修复服务运行 cwd=/app/apps/web 时 ensureBrowser() 探测路径不符、仍触发下载
- next.config.mjs: @pipeline/* 的 webpack external 由 commonjs 改为 module 类型,
  避免 await import() 被改写为 require() 触发 ERR_PACKAGE_PATH_NOT_EXPORTED
- run-render.ts: projectRoot 改用 findMonorepoRoot()(向上找 pnpm-workspace.yaml,cwd 无关),
  修复容器内 /app/apps/web 下 templatesEntry/assetsRoot 解析为 <root>/apps 的错误
- 文档同步:CLAUDE.md(module external 约束)、docs/DEPLOYMENT.md(USE_CN_MIRROR 用法)

Co-Authored-By: Claude <noreply@anthropic.com>
lkatzey 1 mēnesi atpakaļ
vecāks
revīzija
ee3582fbd3
5 mainītis faili ar 89 papildinājumiem un 30 dzēšanām
  1. 1 1
      CLAUDE.md
  2. 64 24
      Dockerfile
  3. 10 1
      apps/web/next.config.mjs
  4. 6 2
      apps/web/src/lib/run-render.ts
  5. 8 2
      docs/DEPLOYMENT.md

+ 1 - 1
CLAUDE.md

@@ -118,7 +118,7 @@ VideoSegment { id, kind("cover"|"content"|"outro"), title, desc, caption_origin,
 **核心逻辑在 Next.js 服务进程内运行,不再 spawn CLI**:
 
 - `apps/web/src/lib/run-render.ts` 的 `startRenderJob` 直接 `await runDocument(...)`(进程内),任务进 jobs 列表、走 OSS/飞书。`POST /api/render` 与调度器都走它。
-- **Next.js webpack 无法打包 `@remotion/bundler → esbuild`**,所以 `apps/web/next.config.mjs` 把所有服务端 workspace 包(`@pipeline/{core,text,audio,renderer,tts,collect}`)在 server 端强制 external(`webpack` 配置里往 `config.externals` 数组**追加**一个判别函数;`serverExternalPackages` 单独不够——Next 会顺着 workspace 软链深入 `@pipeline/renderer` 的真实路径去 bundle)。`@pipeline/shared` 仍 transpile(客户端页面要用 `TEMPLATE_TYPES`)。
+- **Next.js webpack 无法打包 `@remotion/bundler → esbuild`**,所以 `apps/web/next.config.mjs` 把所有服务端 workspace 包(`@pipeline/{core,text,audio,renderer,tts,collect}`)在 server 端强制 external(`webpack` 配置里往 `config.externals` 数组**追加**一个判别函数;`serverExternalPackages` 单独不够——Next 会顺着 workspace 软链深入 `@pipeline/renderer` 的真实路径去 bundle)。判别函数返回的 external 类型**必须是 `module`**(让 webpack 对 `await import()` emit 原生 `import()`),**绝不能用 `commonjs`**——这些包是 ESM-only(`"type":"module"`、`exports` 只有 `import` 条件),`commonjs` 会在运行期把 `import()` 改写成 `require()`,CJS 解析器找不到匹配条件 → `ERR_PACKAGE_PATH_NOT_EXPORTED`("No exports main defined")。`@pipeline/shared` 仍 transpile(客户端页面要用 `TEMPLATE_TYPES`)。
   - **重依赖必须懒加载**:web 服务端代码里 `@pipeline/core`、`@pipeline/collect` 用**动态 `await import()`**(在 handler 内部,`apps/web/src/lib/run-render.ts`、`app/api/collect/route.ts`、`app/api/collect/sources/route.ts`),**绝不在模块顶层静态 import**。否则 Next 构建期"收集页面数据"会尝试求值这些 externalized 的 ESM 包(CJS require ESM-only 的 exports 会失败)。类型用 `import type`(编译期擦除,不产生运行时 import)。
   - **turbo 配置**:`turbo.json` 的 `build.outputs` 必须含 `.next/**`(排除 `.next/cache/**`),否则 turbo 缓存命中时会跳过 `next build`、不产出 `.next`。`.dockerignore` 排除 `.turbo`,避免本地 turbo 缓存泄入镜像构建。
 - 调度器(`apps/web/src/lib/scheduler.ts` + `instrumentation.ts`)沿用 node-cron,到点调 `startRenderJob`。

+ 64 - 24
Dockerfile

@@ -1,17 +1,30 @@
 # Stage 1: Install dependencies (isolated layout, same as local dev)
 FROM node:22-bookworm AS deps
 
-# 使用阿里云镜像源
-RUN sed -i 's|deb.debian.org|mirrors.aliyun.com|g' /etc/apt/sources.list.d/debian.sources
+# Mirror switch: default ON (国内镜像源) for fast local/China builds.
+# CI/CD on the release branch builds with --build-arg USE_CN_MIRROR=0 to use
+# official sources (deb.debian.org / registry.npmjs.org / pypi.org /
+# storage.googleapis.com). Resolved per-stage below.
+ARG USE_CN_MIRROR=1
 
-ENV COREPACK_NPM_REGISTRY=https://registry.npmmirror.com
-RUN corepack enable && corepack prepare pnpm@10.24.0 --activate
+RUN set -eux; \
+    if [ "$USE_CN_MIRROR" = "1" ]; then REG=https://registry.npmmirror.com; \
+    else REG=https://registry.npmjs.org; fi; \
+    corepack enable && \
+    COREPACK_NPM_REGISTRY=$REG corepack prepare pnpm@10.24.0 --activate
 
-RUN apt-get update && \
+# 系统包 + Python 依赖:国内源时改 apt 源并走阿里云 pypi;官方源时全用默认
+RUN set -eux; \
+    if [ "$USE_CN_MIRROR" = "1" ]; then \
+      sed -i 's|deb.debian.org|mirrors.aliyun.com|g' /etc/apt/sources.list.d/debian.sources; \
+      PIP_INDEX=https://mirrors.aliyun.com/pypi/simple/; \
+    else \
+      PIP_INDEX=https://pypi.org/simple; \
+    fi; \
+    apt-get update && \
     apt-get install -y --no-install-recommends ffmpeg python3 python3-pip && \
-    rm -rf /var/lib/apt/lists/*
-
-RUN pip3 install --break-system-packages -i https://mirrors.aliyun.com/pypi/simple/ faster-whisper
+    rm -rf /var/lib/apt/lists/* && \
+    pip3 install --break-system-packages -i "$PIP_INDEX" faster-whisper
 
 WORKDIR /app
 
@@ -27,7 +40,10 @@ COPY packages/text/package.json packages/text/
 COPY packages/audio/package.json packages/audio/
 COPY packages/renderer/package.json packages/renderer/
 
-RUN pnpm config set registry https://registry.npmmirror.com && \
+RUN set -eux; \
+    if [ "$USE_CN_MIRROR" = "1" ]; then REG=https://registry.npmmirror.com; \
+    else REG=https://registry.npmjs.org; fi; \
+    pnpm config set registry "$REG" && \
     pnpm install --frozen-lockfile
 
 # Stage 2: Build
@@ -43,20 +59,32 @@ RUN pnpm build && rm -rf apps/web/.next/cache
 # Stage 3: Runtime — fresh prod install with hoisted layout
 FROM node:22-bookworm-slim AS runtime
 
-RUN sed -i 's|deb.debian.org|mirrors.aliyun.com|g' /etc/apt/sources.list.d/debian.sources
+# Mirror switch (see deps stage): default ON for local/China; CI passes
+# --build-arg USE_CN_MIRROR=0 for official sources.
+ARG USE_CN_MIRROR=1
 
-ENV COREPACK_NPM_REGISTRY=https://registry.npmmirror.com
-RUN corepack enable && corepack prepare pnpm@10.24.0 --activate
+RUN set -eux; \
+    if [ "$USE_CN_MIRROR" = "1" ]; then REG=https://registry.npmmirror.com; \
+    else REG=https://registry.npmjs.org; fi; \
+    corepack enable && \
+    COREPACK_NPM_REGISTRY=$REG corepack prepare pnpm@10.24.0 --activate
 
-RUN apt-get update && \
+# 系统包(ffmpeg + Chrome 运行库 + 字体)+ Python 依赖:国内源时改 apt 源并走阿里云 pypi;官方源时全用默认
+RUN set -eux; \
+    if [ "$USE_CN_MIRROR" = "1" ]; then \
+      sed -i 's|deb.debian.org|mirrors.aliyun.com|g' /etc/apt/sources.list.d/debian.sources; \
+      PIP_INDEX=https://mirrors.aliyun.com/pypi/simple/; \
+    else \
+      PIP_INDEX=https://pypi.org/simple; \
+    fi; \
+    apt-get update && \
     apt-get install -y --no-install-recommends \
       ffmpeg python3 python3-pip fontconfig \
       libnss3 libnspr4 libatk1.0-0 libatk-bridge2.0-0 libcups2 libdrm2 \
       libxkbcommon0 libxcomposite1 libxdamage1 libxfixes3 libxrandr2 \
       libgbm1 libpango-1.0-0 libcairo2 libasound2 && \
-    rm -rf /var/lib/apt/lists/*
-
-RUN pip3 install --break-system-packages -i https://mirrors.aliyun.com/pypi/simple/ faster-whisper
+    rm -rf /var/lib/apt/lists/* && \
+    pip3 install --break-system-packages -i "$PIP_INDEX" faster-whisper
 
 WORKDIR /app
 
@@ -73,12 +101,14 @@ COPY --from=build /app/packages/text/package.json packages/text/
 COPY --from=build /app/packages/audio/package.json packages/audio/
 COPY --from=build /app/packages/renderer/package.json packages/renderer/
 
-# Install production deps with hoisted layout (everything flat at root node_modules)
-RUN pnpm config set registry https://registry.npmmirror.com && \
-    pnpm install --frozen-lockfile --prod --config.node-linker=hoisted
-
-# Re-create workspace symlinks at root so any package can resolve @pipeline/* by walking up
-RUN mkdir -p node_modules/@pipeline && \
+# Install production deps with hoisted layout (everything flat at root node_modules),
+# then re-create the workspace symlinks at root so any package can resolve @pipeline/* by walking up.
+RUN set -eux; \
+    if [ "$USE_CN_MIRROR" = "1" ]; then REG=https://registry.npmmirror.com; \
+    else REG=https://registry.npmjs.org; fi; \
+    pnpm config set registry "$REG" && \
+    pnpm install --frozen-lockfile --prod --config.node-linker=hoisted && \
+    mkdir -p node_modules/@pipeline && \
     ln -s ../../packages/shared node_modules/@pipeline/shared && \
     ln -s ../../packages/core node_modules/@pipeline/core && \
     ln -s ../../packages/tts node_modules/@pipeline/tts && \
@@ -116,17 +146,21 @@ RUN mkdir -p /usr/share/fonts/truetype/noto && \
 # (storage.googleapis.com / remotion.media) on first render, but that is
 # ~5KB/s from CN networks and stalls indefinitely. Fetch the SAME binary
 # Remotion wants from the Aliyun-backed npmmirror binary mirror at build time
+# (host flips to storage.googleapis.com/chrome-for-testing-public when
+# USE_CN_MIRROR=0, for the overseas CI/release build)
 # and lay it out exactly as Remotion's BrowserFetcher expects, so
 # ensureBrowser() finds it present (revision.local && existsSync(executablePath))
 # and skips the download. Version is read from the installed @remotion/renderer
 # so this auto-tracks Remotion upgrades instead of hardcoding a number.
 RUN set -eux; \
+    if [ "$USE_CN_MIRROR" = "1" ]; then CHROME_BASE=https://registry.npmmirror.com/-/binary/chrome-for-testing; \
+    else CHROME_BASE=https://storage.googleapis.com/chrome-for-testing-public; fi; \
     SRC=/app/node_modules/@remotion/renderer/dist/browser/get-chrome-download-url.js; \
     VERSION=$(grep -oE "TESTED_VERSION = '[0-9.]+'" "$SRC" | grep -oE "[0-9]+\.[0-9]+\.[0-9]+\.[0-9]+" | head -1); \
     test -n "$VERSION"; \
     CACHE=/app/node_modules/.remotion/chrome-headless-shell; \
     mkdir -p "$CACHE/linux64"; \
-    URL="https://registry.npmmirror.com/-/binary/chrome-for-testing/${VERSION}/linux64/chrome-headless-shell-linux64.zip"; \
+    URL="$CHROME_BASE/${VERSION}/linux64/chrome-headless-shell-linux64.zip"; \
     node -e "const fs=require('fs');fetch(process.argv[1]).then(r=>r.arrayBuffer()).then(b=>fs.writeFileSync(process.argv[2],Buffer.from(b)))" "$URL" /tmp/chs.zip; \
     python3 -m zipfile -e /tmp/chs.zip "$CACHE/linux64/"; \
     rm -f /tmp/chs.zip; \
@@ -145,13 +179,19 @@ RUN set -eux; \
 #     umask 022 plus the explicit `chmod +x` on the binary), so the node user
 #     can read and execute it. We only chmod the .remotion *directory* 777 so a
 #     stray write (none expected once chrome is pre-placed) can still succeed.
+# Remotion resolves its chrome cache to <nearest-package.json>/node_modules/.remotion
+# from the SERVER's runtime cwd (/app/apps/web), not the image build dir (/app).
+# The binary lives at /app/node_modules/.remotion (pre-placed above); expose it at the
+# path the server actually probes so ensureBrowser() finds it and skips the download.
 ENV HOME=/home/node
 RUN mkdir -p /app/output /home/node \
       /app/node_modules/.remotion \
       /app/packages/templates/node_modules/.cache && \
     chown -R node:node /app/output /home/node \
       /app/packages/templates/node_modules/.cache && \
-    chmod 777 /app/node_modules/.remotion
+    chmod 777 /app/node_modules/.remotion && \
+    mkdir -p /app/apps/web/node_modules && \
+    ln -sfnT /app/node_modules/.remotion /app/apps/web/node_modules/.remotion
 USER node
 
 EXPOSE 3000

+ 10 - 1
apps/web/next.config.mjs

@@ -41,13 +41,22 @@ const nextConfig = {
   ],
   webpack: (config, { isServer }) => {
     if (isServer) {
+      // `@pipeline/*` are ESM-only workspace packages (type:module, exports
+      // with only `import`/`types` conditions). Next's webpack must NOT bundle
+      // them (they reach @remotion/bundler → esbuild). We force them external,
+      // but as **module**-type externals so webpack emits a NATIVE `import()`
+      // for the lazy `await import("@pipeline/...")` calls. A `commonjs`
+      // external would rewrite those to `require()`, which Node's CJS resolver
+      // rejects on an ESM-only package ("ERR_PACKAGE_PATH_NOT_EXPORTED: No
+      // 'exports' main defined"). `import()` is valid inside the CJS server
+      // bundle and resolves the `import` export condition at runtime.
       const prev = config.externals;
       const prevArr = Array.isArray(prev) ? prev : prev ? [prev] : [];
       config.externals = [
         ...prevArr,
         ({ request }, callback) => {
           if (EXTERNALS.some((p) => request === p || request.startsWith(p + "/"))) {
-            return callback(null, `commonjs ${request}`);
+            return callback(null, request, "module");
           }
           callback();
         },

+ 6 - 2
apps/web/src/lib/run-render.ts

@@ -1,6 +1,6 @@
 import { createJob, updateJob } from "@/lib/job-store";
 import { loadConfig } from "@/lib/config";
-import { resolveOutputDir } from "@pipeline/shared/node";
+import { resolveOutputDir, findMonorepoRoot } from "@pipeline/shared/node";
 import type { DocumentRunConfig } from "@pipeline/core";
 import { resolve } from "node:path";
 
@@ -47,7 +47,11 @@ export function startRenderJob(params: RenderJobParams): {
 
   const config = loadConfig();
   const outputDir = resolveOutputDir(config?.output?.dir);
-  const projectRoot = resolve(process.cwd(), "..");
+  // Monorepo root is cwd-independent (walks up to pnpm-workspace.yaml): works
+  // whether the server runs from <repo>/apps/web (local dev) or /app/apps/web
+  // (Docker). DO NOT use `resolve(cwd, "..")` — the server is two levels deep
+  // under the root, so that yields <root>/apps and breaks templatesEntry/assetsRoot.
+  const projectRoot = findMonorepoRoot();
   const templatesEntry = resolve(projectRoot, "packages/templates/src/entry.ts");
   const assetsRoot = resolve(projectRoot, "assets");
   const inputDir = resolve(process.cwd());

+ 8 - 2
docs/DEPLOYMENT.md

@@ -319,8 +319,13 @@ TAG=$(git rev-parse --short HEAD)   # 也可用日期或 latest
 # 2. 登录 Harbor
 docker login $HARBOR -u <用户名>     # 回车后输入密码 / Harbor CLI secret
 
-# 3. 用 Harbor 地址构建并打 tag(Dockerfile 在仓库根目录,多阶段构建)
-docker build -t $IMAGE:$TAG -t $IMAGE:latest .
+# 3. 构建并打 tag(Dockerfile 在仓库根目录,多阶段构建)
+#    Dockerfile 默认走国内镜像源(apt / pnpm / pip / chrome 二进制),本地与国内环境零配置即快。
+#    release / CI(海外环境)加 --build-arg USE_CN_MIRROR=0 切官方源
+#    (deb.debian.org / registry.npmjs.org / pypi.org / storage.googleapis.com)。
+#    两种模式装出的包完全一致——lockfile 锁了版本与 integrity,registry 只决定下载来源。
+docker build --build-arg USE_CN_MIRROR=0 -t $IMAGE:$TAG -t $IMAGE:latest .   # release / CI:官方源
+# docker build                       -t $IMAGE:$TAG -t $IMAGE:latest .        # 本地 / 国内:默认国内源
 
 # 4. 推送
 docker push $IMAGE:$TAG
@@ -330,6 +335,7 @@ docker push $IMAGE:latest
 ```
 
 要点:
+- **构建镜像源**:Dockerfile 默认国内镜像源(本地/国内构建快);release/CI 传 `--build-arg USE_CN_MIRROR=0` 走官方源(海外 CI 可达)。见上方第 3 步。
 - **Harbor 走 HTTP(非 HTTPS)** 时,需在 docker daemon 的 `/etc/docker/daemon.json` 加入 `"insecure-registries": ["harbor.example.com"]` 后重启 dockerd;或在 k8s 各节点同理配置 containerd。
 - **私有项目**:k8s 拉镜像需要 imagePullSecret,创建方法见 `deploy/k8s/README.md`。
 - 镜像约 1.5–2GB(含 ffmpeg、Python/faster-whisper、Chrome 依赖、字体)。首次推送较慢。