feat: add product refs and video candidate slots

This commit is contained in:
2026-05-17 16:15:48 +08:00
parent 9400db6952
commit c690979b58
2 changed files with 139 additions and 31 deletions

View File

@@ -589,7 +589,7 @@
<tr><td><code>web/next.config.mjs</code></td><td>Next.js 构建配置:静态导出、图片不走优化、禁用开发环境左下角 Next Dev Indicator并移除 Next 16 已不支持的 <code>eslint</code> 顶层配置,避免本地 dev 出现配置 Issue 提示。</td></tr>
<tr><td><code>web/app/globals.css</code></td><td>全局主题变量、登录页视觉样式、ReactFlow 样式引用,以及本地开发态 <code>nextjs-portal</code> 遮挡隐藏规则。</td></tr>
<tr><td><code>web/app/page.tsx</code></td><td>产品工作台主状态jobs、activeJobId、生成任务状态主渲染为全屏素材输入列 + 信息流广告复刻工作表;“开始”编排状态只负责在下载完成后自动触发 <code>triggerTranscribe</code>不再默认触发抽帧、Vision 扫描或分镜初稿保存;底部吸附音频条不再从主界面渲染。</td></tr>
<tr><td><code>web/components/ad-recreation-board.tsx</code></td><td>信息流广告复刻工作表:左侧素材输入;右侧展示视频下载状态、默认折叠的音频文案依据,以及统一的音频解析结果面板;面板顶部是一行讲话人/节奏/背景音摘要,下方左侧为原视频播放器、右侧为逐句时间轴,底部横向音频波形用参考图式的连续灰色包络显示响度、停顿和密集爆点。视频播放时通过 <code>requestAnimationFrame</code> 平滑驱动波形播放线,同时同步高亮并滚动当前句;点击音频波形或字幕行会跳转原视频时间。音频结果下方是信息流复刻分镜工作台:每条音频分镜纵向排列,行内从左到右串起原内容、新口播文案、画面规划/产品融入、参考帧/关键元素和生成视频;单条生成会先把该行规划保存为对应关键帧分镜,再复用现有生视频接口提交 Seedance 候选。旧分镜卡、抽帧控制和视频生成组件仍保留在文件里,但当前主路径不渲染。</td></tr>
<tr><td><code>web/components/ad-recreation-board.tsx</code></td><td>信息流广告复刻工作表:左侧素材输入;右侧展示视频下载状态、默认折叠的音频文案依据,以及统一的音频解析结果面板;面板顶部是一行讲话人/节奏/背景音摘要,下方左侧为原视频播放器、右侧为逐句时间轴,底部横向音频波形用参考图式的连续灰色包络显示响度、停顿和密集爆点。视频播放时通过 <code>requestAnimationFrame</code> 平滑驱动波形播放线,同时同步高亮并滚动当前句;点击音频波形或字幕行会跳转原视频时间。音频结果下方是信息流复刻分镜工作台:顶部可上传产品白底图,建议 5 张、最多 6 张,用来锁定正面、左右 45 度、厚度、内侧触点/佩戴比例,避免非对称产品被生成成左右镜像;每条音频分镜纵向排列,行内从左到右串起原内容、新口播文案、画面规划/产品融入、参考帧/关键元素和 6 个候选视频槽。单条生成会先把该行规划和已上传产品图保存为对应关键帧分镜,再复用现有生视频接口提交 Seedance 候选。旧分镜卡、抽帧控制和视频生成组件仍保留在文件里,但当前主路径不渲染。</td></tr>
<tr><td><code>web/app/login/page.tsx</code></td><td>生产登录页:访问账号/访问密钥表单、保持登录、错误/成功状态;当前只在原版 Digital Oasis 动态背景上叠加一个组合登录框,桌面端左侧是动态角色,右侧是图标化登录表单;面板左上角展示官网 SKG 字标和中文“营销内容工作台”系统标识。</td></tr>
<tr><td><code>web/app/login/layout.tsx</code></td><td>登录路由专属 layout覆盖全站默认网页标题和描述为空避免 <code>/login</code> 继承工作台 metadata 后在页面源码里继续出现登录界面文字以外的文案。</td></tr>
<tr><td><code>web/components/login/oasis-canvas.tsx</code></td><td>登录页全屏动态视觉层:用 iframe 直接承载下载包 <code>web/public/oasis-source/index.html</code> 的原 WebGPU / Three.js 草场源码;父级登录页只覆盖自己的文案和表单,并在捕获阶段把全局鼠标坐标同时用原生事件和 <code>postMessage</code> 转发给 iframe避免登录面板或输入框遮挡时草地失去鼠标响应。</td></tr>
@@ -627,7 +627,7 @@ web/app/page.tsx
-> 信息流广告复刻工作表web/components/ad-recreation-board.tsx
-> 开始:创建/激活 job → 下载完成后自动触发音频处理
-> 左侧素材输入列 + 右侧默认折叠的音频文案依据 + 统一音频解析结果面板(声音摘要在上,原视频与逐句时间轴并排,底部连续响度波形联动)
-> 信息流复刻分镜工作台:逐句时间轴 → 原内容 / 新口播文案 / 画面规划与产品融入 / 参考帧与关键元素 / 对应候选视频
-> 信息流复刻分镜工作台:产品白底图上传(建议 5、最多 6逐句时间轴 → 原内容 / 新口播文案 / 画面规划与产品融入 / 参考帧与关键元素 / 6 个候选视频
-> 底部音频条:不再渲染,音频结果集中到右侧工作表
-> 旧节点/深度素材面板web/components/nodes/index.tsx、web/components/lightbox.tsx、web/components/storyboard-workbench.tsx底层保留当前不作为主入口
-> API 契约web/lib/api.ts
@@ -654,8 +654,8 @@ api/main.py
</div>
<div class="flow-row">
<div><strong>你看到的区域</strong><span>信息流复刻分镜工作台</span></div>
<div><strong>主要源码</strong><span><code>AudioStoryboardPlanPanel</code><code>buildAudioStoryboardRows</code><code>buildStoryboardSceneFromAudioRow</code> in <code>web/components/ad-recreation-board.tsx</code>;逐行定向抽帧复用 <code>onAddManualFrameForJob</code>,单条生成复用 <code>onGenerateVideo</code><code>PUT /frames/{idx}/storyboard</code></span></div>
<div><strong>适合怎么描述</strong><span>“按音频逐句生成产品分镜、每行怎样改写口播、如何抽参考帧、生成的视频应该回显到哪一行”。</span></div>
<div><strong>主要源码</strong><span><code>AudioStoryboardPlanPanel</code><code>buildAudioStoryboardRows</code><code>buildStoryboardSceneFromAudioRow</code><code>StoryboardVideoSlots</code> in <code>web/components/ad-recreation-board.tsx</code>;逐行定向抽帧复用 <code>onAddManualFrameForJob</code>产品白底图上传复用 <code>uploadStoryboardAsset</code>单条生成复用 <code>onGenerateVideo</code><code>PUT /frames/{idx}/storyboard</code></span></div>
<div><strong>适合怎么描述</strong><span>“按音频逐句生成产品分镜、每行怎样改写口播、上传几张产品白底图、如何抽参考帧、生成的视频应该回显到哪一行”。</span></div>
</div>
<div class="flow-row">
<div><strong>你看到的区域</strong><span>旧深度素材面板(当前不作为主路径)</span></div>
@@ -948,6 +948,18 @@ SubjectAsset {
<h2>变更记录</h2>
<p>这个记录不是 git log 的替代品。它记录“产品理解发生了什么变化、影响了哪些源码、你以后描述需求时该怎么说”。后续每次改功能都要补一条。</p>
<div class="changelog">
<article class="change">
<header>
<h3>2026-05-17 · 压缩分镜行并加入产品白底图与多候选视频槽</h3>
<span class="tag rose">UI</span>
<span class="tag cyan">Workflow</span>
</header>
<div class="body">
<p><strong>问题:</strong>分镜行前半段占用横向空间偏多,右侧视频候选位太少;同时肩颈产品不是完全对称结构,只靠默认产品图容易生成左右一致、比例不准或佩戴尺寸跑偏。</p>
<p><strong>改动:</strong><code>AudioStoryboardPlanPanel</code> 顶部新增产品白底图上传条,建议上传 5 张、最多 6 张:正面、左 45、右 45、侧面厚度、内侧触点/佩戴比例,非对称明显时补背面或底部。分镜行前几列压缩宽度、字号和缩略图高度;右侧视频区改为 6 个候选槽,便于多次生成和筛选。生成本条时会把上传的产品图写入 <code>StoryboardScene.product_images</code>,并把“保留左右非对称、肩颈真实比例”加入产品融合提示。</p>
<p><strong>影响:</strong><code>web/components/ad-recreation-board.tsx</code><code>docs/source-analysis.html</code>。产品白底图当前作为当前页面会话内的复刻参考,单条生成时持久化进对应关键帧分镜;后续如果要跨刷新保留全局产品图组,应新增 job 级产品参考字段。</p>
</div>
</article>
<article class="change">
<header>
<h3>2026-05-17 · 新增信息流复刻分镜工作台</h3>

View File

@@ -11,6 +11,7 @@ import {
type FrameExtractTarget,
type FrameObject,
type GeneratedVideo,
type ImageRef,
type Job,
type KeyElement,
type KeyFrame,
@@ -24,8 +25,10 @@ import {
generatedImageUrl,
hasCutout,
representativeCutoutUrl,
resolveImageRefUrl,
sourceAudioUrl,
updateStoryboard,
uploadStoryboardAsset,
videoUrl,
} from "@/lib/api"
import { type NodeData } from "@/components/nodes"
@@ -287,14 +290,19 @@ function buildAudioStoryboardRows(job: Job | null): AudioStoryboardRow[] {
})
}
function buildStoryboardSceneFromAudioRow(row: AudioStoryboardRow, frame: KeyFrame, nextFrame?: KeyFrame | null): StoryboardScene {
function buildStoryboardSceneFromAudioRow(row: AudioStoryboardRow, frame: KeyFrame, nextFrame?: KeyFrame | null, productRefs: ImageRef[] = []): StoryboardScene {
const productGuidance = productRefs.length
? "产品白底图已上传:生成时必须同时参考正面、左侧、右侧、厚度和内侧触点/佩戴比例,保留左右非对称细节,不要把两边做成镜像对称;肩颈产品大小必须贴近真实佩戴比例,不能缩成耳机,也不能放大成护颈枕。"
: "未上传产品白底图时使用默认 SKG 产品图;生成前建议补 5 张白底图锁定左右差异、厚度和佩戴比例。"
return {
duration: Number(Math.max(3.2, Math.min(6.5, row.end - row.start || 4.5)).toFixed(1)),
first_image: { kind: "keyframe", frame_idx: frame.index, label: `分镜 ${row.index + 1} 参考帧` },
last_image: nextFrame ? { kind: "keyframe", frame_idx: nextFrame.index, label: `分镜 ${row.index + 1} 尾帧` } : null,
product_images: productRefs,
product_image: productRefs[0] ?? null,
subject: row.keyElements,
scene: `${row.visualPlan}\n原音频依据${row.source}`,
product: row.productIntegration,
product: `${row.productIntegration}\n${productGuidance}`,
action: row.skgCopy,
reference_ids: [],
}
@@ -911,9 +919,16 @@ function AudioStoryboardPlanPanel({
}) {
const [busyRow, setBusyRow] = useState<number | null>(null)
const [videoBusyRow, setVideoBusyRow] = useState<number | null>(null)
const [productRefs, setProductRefs] = useState<ImageRef[]>([])
const [productUploading, setProductUploading] = useState(false)
const productFileRef = useRef<HTMLInputElement | null>(null)
const rows = useMemo(() => buildAudioStoryboardRows(job), [job])
const orderedFrames = useMemo(() => job ? [...job.frames].sort((a, b) => a.timestamp - b.timestamp) : [], [job])
useEffect(() => {
setProductRefs([])
}, [job?.id])
const framesForRow = (row: AudioStoryboardRow) =>
orderedFrames.filter((frame) => frame.timestamp >= row.start - 0.2 && frame.timestamp <= row.end + 0.2).slice(0, 3)
@@ -933,11 +948,31 @@ function AudioStoryboardPlanPanel({
}
}
const uploadProductImages = async (files: FileList | null) => {
if (!job || !files?.length) return
const remaining = Math.max(0, 6 - productRefs.length)
if (remaining === 0) {
toast.info("产品白底图最多保留 6 张")
return
}
const selected = Array.from(files).slice(0, remaining)
setProductUploading(true)
try {
const refs = await Promise.all(selected.map((file) => uploadStoryboardAsset(job.id, file)))
setProductRefs((prev) => [...prev, ...refs].slice(0, 6))
toast.success(`已上传 ${refs.length} 张产品白底图`)
} catch (e) {
toast.error("产品白底图上传失败:" + (e instanceof Error ? e.message : String(e)))
} finally {
setProductUploading(false)
}
}
const generateRowVideo = async (row: AudioStoryboardRow, refs: KeyFrame[]) => {
if (!job || !refs.length || !onGenerateVideo) return
const frame = refs[0]
const nextFrame = orderedFrames.find((item) => item.timestamp > frame.timestamp) ?? null
const scene = buildStoryboardSceneFromAudioRow(row, frame, nextFrame)
const scene = buildStoryboardSceneFromAudioRow(row, frame, nextFrame, productRefs)
setVideoBusyRow(row.index)
try {
const updated = await updateStoryboard(job.id, frame.index, scene)
@@ -966,6 +1001,55 @@ function AudioStoryboardPlanPanel({
</div>
</div>
<div className="mb-2 grid gap-2 rounded-md border border-white/10 bg-black/24 p-2 xl:grid-cols-[minmax(0,1fr)_auto]">
<div className="min-w-0">
<div className="flex items-center gap-2">
<SectionTitle icon={<Package className="h-4 w-4" />} title="产品白底图" />
<span className="rounded-md border border-white/10 bg-white/[0.04] px-2 py-0.5 text-[10px] text-white/42"> 5 6 </span>
</div>
<p className="mt-1 line-clamp-2 text-[11px] leading-snug text-white/42">
45 45/
</p>
</div>
<div className="flex min-w-0 items-center justify-end gap-2">
<div className="flex max-w-[300px] gap-1 overflow-x-auto pb-0.5">
{productRefs.map((ref, index) => (
<img
key={`${ref.kind}:${ref.frame_idx}:${ref.element_id ?? ""}:${ref.cutout_id ?? ""}:${index}`}
src={resolveImageRefUrl(job.id, ref)}
alt={`产品白底图 ${index + 1}`}
className="h-12 w-12 shrink-0 rounded-md border border-white/10 bg-white object-contain"
/>
))}
{Array.from({ length: Math.max(0, Math.min(6, 5 - productRefs.length)) }).map((_, index) => (
<div key={`empty-product-${index}`} className="flex h-12 w-12 shrink-0 items-center justify-center rounded-md border border-dashed border-white/12 bg-black/25 text-[10px] text-white/25">
{productRefs.length + index + 1}
</div>
))}
</div>
<button
type="button"
onClick={() => productFileRef.current?.click()}
disabled={!job || productUploading || productRefs.length >= 6}
className="inline-flex h-9 shrink-0 items-center justify-center gap-1 rounded-md border border-white/10 bg-white/[0.055] px-2.5 text-[11px] font-semibold text-white/72 transition hover:border-white/25 hover:bg-white/[0.1] disabled:cursor-not-allowed disabled:opacity-40"
>
{productUploading ? <Loader2 className="h-3.5 w-3.5 animate-spin" /> : <Upload className="h-3.5 w-3.5" />}
</button>
<input
ref={productFileRef}
type="file"
accept="image/*"
multiple
className="hidden"
onChange={(event) => {
void uploadProductImages(event.currentTarget.files)
event.currentTarget.value = ""
}}
/>
</div>
</div>
{rows.length ? (
<div className="max-h-[560px] space-y-2 overflow-y-auto pr-1">
{rows.map((row) => {
@@ -976,7 +1060,7 @@ function AudioStoryboardPlanPanel({
return (
<article
key={row.index}
className="grid overflow-hidden rounded-md border border-white/10 bg-black/24 text-[11.5px] leading-snug text-white/64 xl:grid-cols-[82px_minmax(130px,0.8fr)_minmax(150px,1fr)_minmax(180px,1.1fr)_minmax(150px,0.9fr)_146px]"
className="grid overflow-hidden rounded-md border border-white/10 bg-black/24 text-[11px] leading-snug text-white/64 xl:grid-cols-[70px_minmax(112px,0.68fr)_minmax(128px,0.78fr)_minmax(154px,0.92fr)_minmax(126px,0.74fr)_230px]"
>
<StoryboardPlanCell label="分镜">
<div className="font-mono text-[11px] text-white/40">{row.start.toFixed(1)}-{row.end.toFixed(1)}s</div>
@@ -986,16 +1070,16 @@ function AudioStoryboardPlanPanel({
</StoryboardPlanCell>
<StoryboardPlanCell label="原内容">
<p className="line-clamp-5" title={row.source}>{row.source}</p>
<p className="line-clamp-4" title={row.source}>{row.source}</p>
</StoryboardPlanCell>
<StoryboardPlanCell label="新口播文案">
<p className="line-clamp-5 text-white/82" title={row.skgCopy}>{row.skgCopy}</p>
<p className="line-clamp-4 text-white/82" title={row.skgCopy}>{row.skgCopy}</p>
</StoryboardPlanCell>
<StoryboardPlanCell label="画面规划 / 产品融入">
<p className="line-clamp-3" title={row.visualPlan}>{row.visualPlan}</p>
<p className="mt-1.5 line-clamp-3 text-white/45" title={row.productIntegration}>
<p className="line-clamp-2" title={row.visualPlan}>{row.visualPlan}</p>
<p className="mt-1 line-clamp-3 text-white/45" title={row.productIntegration}>
<Package className="mr-1 inline h-3 w-3 text-rose-200/75" />
{row.productIntegration}
</p>
@@ -1003,13 +1087,13 @@ function AudioStoryboardPlanPanel({
<StoryboardPlanCell label="参考帧 / 关键元素">
{refs.length ? (
<div className="mb-2 flex gap-1.5 overflow-x-auto pb-1">
<div className="mb-1.5 flex gap-1.5 overflow-x-auto pb-1">
{refs.map((frame) => (
<button
key={frame.index}
type="button"
onClick={() => onOpenFrame?.(frame.index)}
className="h-16 w-10 shrink-0 overflow-hidden rounded border border-white/10 bg-black/45 transition hover:border-cyan-300/40"
className="h-14 w-9 shrink-0 overflow-hidden rounded border border-white/10 bg-black/45 transition hover:border-cyan-300/40"
title={`参考帧 ${frame.timestamp.toFixed(1)}s`}
>
<img src={effectiveFrameUrl(job.id, frame)} alt="" className="h-full w-full object-cover" />
@@ -1019,7 +1103,7 @@ function AudioStoryboardPlanPanel({
) : (
<p className="mb-2 line-clamp-2 text-white/34" title={row.referencePlan}>{row.referencePlan}</p>
)}
<div className="line-clamp-2 text-[10.5px] text-white/38" title={row.keyElements}>
<div className="line-clamp-2 text-[10px] text-white/38" title={row.keyElements}>
<ImageIcon className="mr-1 inline h-3 w-3" />
{row.keyElements}
</div>
@@ -1027,7 +1111,7 @@ function AudioStoryboardPlanPanel({
type="button"
onClick={() => addReferenceFrame(row)}
disabled={!onAddFrame || busy}
className="mt-2 inline-flex h-8 w-full items-center justify-center gap-1 rounded-md border border-white/10 bg-white/[0.055] px-2 text-[11px] text-white/70 transition hover:border-white/25 hover:bg-white/[0.1] disabled:cursor-not-allowed disabled:opacity-45"
className="mt-1.5 inline-flex h-7 w-full items-center justify-center gap-1 rounded-md border border-white/10 bg-white/[0.055] px-2 text-[11px] text-white/70 transition hover:border-white/25 hover:bg-white/[0.1] disabled:cursor-not-allowed disabled:opacity-45"
>
{busy ? <Loader2 className="h-3.5 w-3.5 animate-spin" /> : <Scissors className="h-3.5 w-3.5" />}
{refs.length ? "补抽参考帧" : "抽参考帧"}
@@ -1035,22 +1119,12 @@ function AudioStoryboardPlanPanel({
</StoryboardPlanCell>
<StoryboardPlanCell label="生成视频" className="xl:border-r-0">
{rowVideos.length > 0 ? (
<div className="mb-2 flex gap-1.5 overflow-x-auto pb-1">
{rowVideos.map((video) => (
<StoryboardVideoPreview key={video.id} job={job} video={video} />
))}
</div>
) : (
<div className="mb-2 flex h-20 items-center justify-center rounded-md border border-dashed border-white/12 bg-black/25 text-[11px] text-white/30">
{refs.length ? "等待生成" : "先抽参考帧"}
</div>
)}
<StoryboardVideoSlots job={job} videos={rowVideos} enabled={refs.length > 0} />
<button
type="button"
onClick={() => generateRowVideo(row, refs)}
disabled={!refs.length || !onGenerateVideo || generating}
className="inline-flex h-8 w-full items-center justify-center gap-1 rounded-md bg-white px-2 text-[11px] font-semibold text-black transition hover:bg-white/90 disabled:cursor-not-allowed disabled:opacity-40"
className="mt-1.5 inline-flex h-8 w-full items-center justify-center gap-1 rounded-md bg-white px-2 text-[11px] font-semibold text-black transition hover:bg-white/90 disabled:cursor-not-allowed disabled:opacity-40"
>
{generating ? <Loader2 className="h-3.5 w-3.5 animate-spin" /> : <Play className="h-3.5 w-3.5" />}
@@ -1069,14 +1143,36 @@ function AudioStoryboardPlanPanel({
function StoryboardPlanCell({ label, children, className = "" }: { label: string; children: ReactNode; className?: string }) {
return (
<div className={`min-w-0 border-b border-white/8 p-2.5 xl:border-b-0 xl:border-r ${className}`}>
<div className={`min-w-0 border-b border-white/8 p-2 xl:border-b-0 xl:border-r ${className}`}>
<div className="mb-1.5 text-[10px] font-semibold uppercase tracking-[0.12em] text-white/32">{label}</div>
{children}
</div>
)
}
function StoryboardVideoPreview({ job, video }: { job: Job; video: GeneratedVideo }) {
function StoryboardVideoSlots({ job, videos, enabled }: { job: Job; videos: GeneratedVideo[]; enabled: boolean }) {
const visible = videos.slice(0, 6)
const emptyCount = Math.max(0, 6 - visible.length)
return (
<div>
<div className="grid grid-cols-3 gap-1.5">
{visible.map((video) => (
<StoryboardVideoPreview key={video.id} job={job} video={video} className="h-[74px] w-full" />
))}
{Array.from({ length: emptyCount }).map((_, index) => (
<div key={`empty-video-${index}`} className="flex h-[74px] min-w-0 items-center justify-center rounded border border-dashed border-white/12 bg-black/25 px-1 text-center text-[10px] leading-tight text-white/26">
{enabled ? `候选 ${visible.length + index + 1}` : "先抽参考帧"}
</div>
))}
</div>
{videos.length > 6 && (
<div className="mt-1 text-[10px] text-white/34"> {videos.length - 6} </div>
)}
</div>
)
}
function StoryboardVideoPreview({ job, video, className = "h-20 w-12" }: { job: Job; video: GeneratedVideo; className?: string }) {
const src = videoSrc(video)
const poster = videoPoster(job, video)
const running = video.status === "queued" || video.status === "in_progress"
@@ -1085,7 +1181,7 @@ function StoryboardVideoPreview({ job, video }: { job: Job; video: GeneratedVide
href={src || undefined}
target={src ? "_blank" : undefined}
rel={src ? "noreferrer" : undefined}
className="group relative h-20 w-12 shrink-0 overflow-hidden rounded border border-white/10 bg-black/45"
className={`group relative shrink-0 overflow-hidden rounded border border-white/10 bg-black/45 ${className}`}
title={`${video.model} · ${video.status}`}
>
{src && video.status === "completed" ? (