fix: show real audio pitch waveform

2026-05-17 15:05:10 +08:00
parent 120dacf8b6
commit 365053a26f
2 changed files with 118 additions and 49 deletions
--- a/docs/source-analysis.html
+++ b/docs/source-analysis.html
@@ -588,7 +588,7 @@
                <tr><td><code>web/next.config.mjs</code></td><td>Next.js 构建配置：静态导出、图片不走优化、禁用开发环境左下角 Next Dev Indicator，并移除 Next 16 已不支持的 <code>eslint</code> 顶层配置，避免本地 dev 出现配置 Issue 提示。</td></tr>
                <tr><td><code>web/app/globals.css</code></td><td>全局主题变量、登录页视觉样式、ReactFlow 样式引用，以及本地开发态 <code>nextjs-portal</code> 遮挡隐藏规则。</td></tr>
                <tr><td><code>web/app/page.tsx</code></td><td>产品工作台主状态：jobs、activeJobId、生成任务状态；主渲染为全屏素材输入列 + 音频解析工作表；“开始”编排状态只负责在下载完成后自动触发 <code>triggerTranscribe</code>，不再默认触发抽帧、Vision 扫描或分镜初稿保存；底部吸附音频条不再从主界面渲染。</td></tr>
-                <tr><td><code>web/components/ad-recreation-board.tsx</code></td><td>信息流广告音频解析工作表：左侧素材输入；右侧展示视频下载状态、默认折叠的音频文案依据，以及统一的音频解析结果面板；面板顶部是一行讲话人/节奏/背景音摘要，下方左侧为原视频播放器、右侧为逐句时间轴，底部横向音频波形用于观察切割点和爆点节奏。视频播放会同步高亮并滚动当前句，点击波形或字幕行会跳转原视频时间。旧分镜卡、抽帧控制和视频生成组件仍保留在文件里，但当前主路径不渲染。</td></tr>
+                <tr><td><code>web/components/ad-recreation-board.tsx</code></td><td>信息流广告音频解析工作表：左侧素材输入；右侧展示视频下载状态、默认折叠的音频文案依据，以及统一的音频解析结果面板；面板顶部是一行讲话人/节奏/背景音摘要，下方左侧为原视频播放器、右侧为逐句时间轴，底部横向音频分析条用中线对称波形显示响度、用音高曲线显示高低音走势。视频播放会同步高亮并滚动当前句，点击音频分析条或字幕行会跳转原视频时间。旧分镜卡、抽帧控制和视频生成组件仍保留在文件里，但当前主路径不渲染。</td></tr>
                <tr><td><code>web/app/login/page.tsx</code></td><td>生产登录页：访问账号/访问密钥表单、保持登录、错误/成功状态；当前只在原版 Digital Oasis 动态背景上叠加一个组合登录框，桌面端左侧是动态角色，右侧是图标化登录表单；面板左上角展示官网 SKG 字标和中文“营销内容工作台”系统标识。</td></tr>
                <tr><td><code>web/app/login/layout.tsx</code></td><td>登录路由专属 layout：覆盖全站默认网页标题和描述为空，避免 <code>/login</code> 继承工作台 metadata 后在页面源码里继续出现登录界面文字以外的文案。</td></tr>
                <tr><td><code>web/components/login/oasis-canvas.tsx</code></td><td>登录页全屏动态视觉层：用 iframe 直接承载下载包 <code>web/public/oasis-source/index.html</code> 的原 WebGPU / Three.js 草场源码；父级登录页只覆盖自己的文案和表单，并在捕获阶段把全局鼠标坐标同时用原生事件和 <code>postMessage</code> 转发给 iframe，避免登录面板或输入框遮挡时草地失去鼠标响应。</td></tr>
@@ -625,7 +625,7 @@
 web/app/page.tsx
  -> 音频解析工作表：web/components/ad-recreation-board.tsx
  -> 开始：创建/激活 job → 下载完成后自动触发音频处理
-  -> 左侧素材输入列 + 右侧默认折叠的音频文案依据 + 统一音频解析结果面板（声音摘要在上，原视频与逐句时间轴并排，底部波形联动）
+  -> 左侧素材输入列 + 右侧默认折叠的音频文案依据 + 统一音频解析结果面板（声音摘要在上，原视频与逐句时间轴并排，底部响度波形和音高曲线联动）
  -> 底部音频条：不再渲染，音频结果集中到右侧工作表
  -> 旧节点/深度素材面板：web/components/nodes/index.tsx、web/components/lightbox.tsx、web/components/storyboard-workbench.tsx（底层保留，当前不作为主入口）
  -> API 契约：web/lib/api.ts
@@ -648,7 +648,7 @@ api/main.py
          <div class="flow-row">
            <div><strong>你看到的区域</strong><span>音频解析结果表</span></div>
            <div><strong>主要源码</strong><span><code>AudioIntakePanel</code> / <code>AudioIntakeStatus</code> in <code>web/components/ad-recreation-board.tsx</code>；复用 <code>triggerTranscribe</code> 和 <code>AudioScript</code>。</span></div>
-            <div><strong>适合怎么描述</strong><span>“原视频播放、音频波形、逐句时间轴滚动、高亮和跳转联动还需要怎么调整”。</span></div>
+            <div><strong>适合怎么描述</strong><span>“原视频播放、响度波形、音高曲线、逐句时间轴滚动、高亮和跳转联动还需要怎么调整”。</span></div>
          </div>
          <div class="flow-row">
            <div><strong>你看到的区域</strong><span>旧深度素材面板（当前不作为主路径）</span></div>
@@ -819,7 +819,7 @@ SubjectAsset {
            <tr><td>删除输入视频</td><td><code>DELETE /jobs/{id}</code></td><td><code>deleteJob</code></td><td>从任务队列、URL 和磁盘 <code>jobs/&lt;id&gt;</code> 目录移除整个 job，包括源视频、关键帧、元素提取图和生成视频。</td></tr>
            <tr><td>解析视频</td><td><code>POST /jobs/{id}/analyze?frames=&amp;target=&amp;mode=&amp;quality=</code></td><td><code>analyzeJob</code></td><td>后续阶段保留的抽帧能力。默认 <code>frames=12</code>；<code>target</code> 支持透明骨架人、综合、清晰主体、转场变化、表情瞬间、动作峰值。当前第一步主流程不自动调用该接口。</td></tr>
            <tr><td>音频文案轨</td><td><code>POST /jobs/{id}/transcribe</code></td><td><code>triggerTranscribe</code></td><td>若尚未拆轨，先从 <code>source.mp4</code> 提取 <code>audio.wav</code> 并回填 <code>source_audio_url</code>；随后用 ASR 提取原始文案，翻译成中文，写入 <code>audio_script.source_text</code>、<code>source_zh</code> 和逐句 <code>transcript</code>。远端 <code>ASR_MODEL</code> 失败后先走本机 <code>LOCAL_ASR_BIN</code>/<code>LOCAL_ASR_MODEL</code>（默认 <code>mlx_whisper</code>），再尝试 <code>ASR_FALLBACK_MODEL</code>。后端会拒绝重复文本、逐秒假字幕或覆盖率过低的结果，不再把不可听的多模态输出写进时间轴。再用 <code>ASR_FALLBACK_MODEL</code> 多模态音频分析讲话人、语速节奏、停顿、背景音乐/环境声/音效，写入 <code>speaker_profile</code>、<code>rhythm_profile</code>、<code>background_audio_profile</code>。当前第一步不默认生成 SKG 新口播和 MiniMax 配音。</td></tr>
-            <tr><td>原始音频文件</td><td><code>GET /jobs/{id}/audio.wav</code></td><td><code>sourceAudioUrl</code></td><td>返回拆轨得到的 wav；当前主界面不再渲染底部吸附音频条，右侧音频解析工作表会读取该文件生成横向波形，并和原视频、逐句时间轴联动。</td></tr>
+            <tr><td>原始音频文件</td><td><code>GET /jobs/{id}/audio.wav</code></td><td><code>sourceAudioUrl</code></td><td>返回拆轨得到的 wav；当前主界面不再渲染底部吸附音频条，右侧音频解析工作表会读取该文件生成横向响度波形和音高走势曲线，并和原视频、逐句时间轴联动。</td></tr>
            <tr><td>改写配音文件</td><td><code>GET /jobs/{id}/audio-script.mp3</code></td><td><code>apiAssetUrl(job.audio_script.voice_url)</code></td><td>后续新配音阶段保留的 MiniMax T2A 产物。当前第一步不默认生成该文件。</td></tr>
            <tr><td>手动加帧</td><td><code>POST /jobs/{id}/frames?t=</code></td><td><code>addManualFrame</code></td><td>按视频时间戳抽一帧，index 递增但 frames 按 timestamp 排序。</td></tr>
            <tr><td>Vision 识别</td><td><code>POST /frames/{idx}/describe</code></td><td><code>describeFrame</code></td><td>写入 frame.description，后续可从 objects 加候选元素。</td></tr>
@@ -943,14 +943,14 @@ SubjectAsset {
        <div class="changelog">
          <article class="change">
            <header>
-              <h3>2026-05-17 · 增加原视频与波形联动审片</h3>
+              <h3>2026-05-17 · 增加原视频与音频走势联动审片</h3>
              <span class="tag rose">UI</span>
              <span class="tag cyan">Workflow</span>
            </header>
            <div class="body">
-              <p><strong>问题：</strong>只有逐句时间轴难以判断原视频节奏、停顿和爆点位置；用户需要边看原版视频边看字幕，并通过音频波形快速识别结构切割点。</p>
-              <p><strong>改动：</strong><code>web/components/ad-recreation-board.tsx</code> 在音频解析结果面板中加入原视频播放器和横向音频波形，布局为左侧原视频、右侧逐句时间轴、底部波形。视频播放时根据 <code>currentTime</code> 高亮并自动滚动当前字幕行；点击波形或字幕行会跳转原视频时间。波形从 <code>audio.wav</code> 解码生成，失败时用本地 fallback peaks 保持布局可用。</p>
-              <p><strong>影响：</strong><code>web/components/ad-recreation-board.tsx</code>、<code>docs/source-analysis.html</code>。后续如果要做爆点标记，应基于当前波形和字幕时间轴继续加 marker，而不是恢复底部音频条。</p>
+              <p><strong>问题：</strong>只有逐句时间轴难以判断原视频节奏、停顿和爆点位置；普通柱状条也看不出高低音变化，无法辅助判断口播抬升、下沉和爆点。</p>
+              <p><strong>改动：</strong><code>web/components/ad-recreation-board.tsx</code> 在音频解析结果面板中加入原视频播放器和横向音频分析条，布局为左侧原视频、右侧逐句时间轴、底部音频走势。视频播放时根据 <code>currentTime</code> 高亮并自动滚动当前字幕行；点击音频分析条或字幕行会跳转原视频时间。音频分析条只从真实 <code>audio.wav</code> 解码生成：青色中线对称波形表示响度，琥珀曲线表示粗音高走势；解码前显示加载状态，失败时明确提示，不再用假波形兜底。</p>
+              <p><strong>影响：</strong><code>web/components/ad-recreation-board.tsx</code>、<code>docs/source-analysis.html</code>。后续如果要做爆点标记，应基于当前响度/音高走势和字幕时间轴继续加 marker，而不是恢复底部音频条。</p>
            </div>
          </article>
          <article class="change">
--- a/web/components/ad-recreation-board.tsx
+++ b/web/components/ad-recreation-board.tsx
@@ -60,6 +60,13 @@ type DraftSegment = {
  scene: StoryboardScene
 }

+type AudioFeature = {
+  loudness: number
+  pitch: number | null
+}
+
+type AudioFeatureStatus = "idle" | "loading" | "ready" | "failed"
+
 const controlClass =
  "h-10 rounded-md border border-white/10 bg-black/55 px-3 text-[12px] text-white outline-none transition focus:border-cyan-300/60 disabled:cursor-not-allowed disabled:opacity-40"

@@ -97,17 +104,7 @@ function clampNumber(value: number, min: number, max: number) {
  return Math.min(max, Math.max(min, value))
 }

-function fallbackPeaks(count: number, seedText: string) {
-  let seed = 0
-  for (let i = 0; i < seedText.length; i++) seed = (seed * 31 + seedText.charCodeAt(i)) % 9973
-  return Array.from({ length: count }, (_, i) => {
-    const wave = Math.sin((i + seed) * 0.39) * 0.35 + Math.sin((i + seed) * 0.12) * 0.24
-    const pulse = ((i + seed) % 11) / 24
-    return clampNumber(0.18 + Math.abs(wave) + pulse, 0.12, 1)
-  })
-}
-
-async function decodeWaveform(url: string, targetPeaks = 180) {
+async function decodeAudioFeatures(url: string, targetFrames = 180): Promise<AudioFeature[]> {
  const res = await fetch(url)
  if (!res.ok) throw new Error(`audio ${res.status}`)
  const arrayBuffer = await res.arrayBuffer()
@@ -117,18 +114,40 @@ async function decodeWaveform(url: string, targetPeaks = 180) {
  try {
    const buffer = await ctx.decodeAudioData(arrayBuffer.slice(0))
    const data = buffer.getChannelData(0)
-    const bucket = Math.max(1, Math.floor(data.length / targetPeaks))
-    let maxPeak = 0.01
-    const raw: number[] = []
-    for (let i = 0; i < targetPeaks; i++) {
+    const bucket = Math.max(1, Math.floor(data.length / targetFrames))
+    let maxLoudness = 0.01
+    const raw: Array<{ loudness: number; pitchHz: number | null }> = []
+    for (let i = 0; i < targetFrames; i++) {
      const start = i * bucket
      const end = Math.min(data.length, start + bucket)
      let peak = 0
-      for (let j = start; j < end; j++) peak = Math.max(peak, Math.abs(data[j] || 0))
-      raw.push(peak)
-      maxPeak = Math.max(maxPeak, peak)
+      let sumSq = 0
+      let crossings = 0
+      let prev = data[start] || 0
+      for (let j = start; j < end; j++) {
+        const sample = data[j] || 0
+        const abs = Math.abs(sample)
+        peak = Math.max(peak, abs)
+        sumSq += sample * sample
+        if (abs > 0.012 && Math.abs(prev) > 0.012 && Math.sign(sample) !== Math.sign(prev)) crossings += 1
+        prev = sample
+      }
+      const size = Math.max(end - start, 1)
+      const rms = Math.sqrt(sumSq / size)
+      const loudness = Math.max(rms * 1.8, peak * 0.62)
+      const seconds = size / buffer.sampleRate
+      const pitchHz = rms > 0.006 && seconds > 0 ? crossings / (2 * seconds) : null
+      raw.push({ loudness, pitchHz })
+      maxLoudness = Math.max(maxLoudness, loudness)
    }
-    return raw.map((p) => clampNumber(p / maxPeak, 0.08, 1))
+    const minHz = Math.log2(80)
+    const maxHz = Math.log2(520)
+    return raw.map((item) => ({
+      loudness: clampNumber(item.loudness / maxLoudness, 0.06, 1),
+      pitch: item.pitchHz && item.pitchHz >= 60 && item.pitchHz <= 720
+        ? clampNumber((Math.log2(item.pitchHz) - minHz) / (maxHz - minHz), 0.02, 0.98)
+        : null,
+    }))
  } finally {
    void ctx.close().catch(() => {})
  }
@@ -580,7 +599,8 @@ function AudioIntakeStatus({ job, audioReady }: { job: Job | null; audioReady: b
 function AudioIntakePanel({ job }: { job: Job | null }) {
  const [currentTime, setCurrentTime] = useState(0)
  const [mediaDuration, setMediaDuration] = useState(0)
-  const [peaks, setPeaks] = useState<number[]>(() => fallbackPeaks(180, "initial-waveform"))
+  const [audioFeatures, setAudioFeatures] = useState<AudioFeature[]>([])
+  const [audioFeatureStatus, setAudioFeatureStatus] = useState<AudioFeatureStatus>("idle")
  const videoRef = useRef<HTMLVideoElement | null>(null)
  const rowRefs = useRef<Record<number, HTMLDivElement | null>>({})
  const script = job?.audio_script
@@ -604,17 +624,28 @@ function AudioIntakePanel({ job }: { job: Job | null }) {
  const activeSegment = job?.transcript.find((segment) => currentTime >= segment.start && currentTime <= Math.max(segment.end, segment.start + 0.2))

  useEffect(() => {
-    if (!job?.id || !audioSrcUrl) return
+    if (!job?.id || !audioSrcUrl) {
+      setAudioFeatures([])
+      setAudioFeatureStatus("idle")
+      return
+    }
    setCurrentTime(0)
    setMediaDuration(0)
-    setPeaks(fallbackPeaks(180, `${job.id}-loading`))
+    setAudioFeatures([])
+    setAudioFeatureStatus("loading")
    let cancelled = false
-    decodeWaveform(audioSrcUrl)
+    decodeAudioFeatures(audioSrcUrl)
      .then((next) => {
-        if (!cancelled) setPeaks(next)
+        if (!cancelled) {
+          setAudioFeatures(next)
+          setAudioFeatureStatus("ready")
+        }
      })
      .catch(() => {
-        if (!cancelled) setPeaks(fallbackPeaks(180, job.id))
+        if (!cancelled) {
+          setAudioFeatures([])
+          setAudioFeatureStatus("failed")
+        }
      })
    return () => { cancelled = true }
  }, [audioSrcUrl, job?.id])
@@ -654,11 +685,12 @@ function AudioIntakePanel({ job }: { job: Job | null }) {
      <div className="grid gap-2 border-t border-white/8 pt-2">
        <div className="rounded-md border border-white/10 bg-black/32 p-2">
          <div className="mb-1 flex items-center justify-between text-[10px] text-white/40">
-            <span>音频波形 / 切点参考</span>
+            <span>音频波形 / 音高走势</span>
            <span className="font-mono">{currentTime.toFixed(1)}s</span>
          </div>
          <AudioWaveform
-            peaks={peaks}
+            features={audioFeatures}
+            status={audioFeatureStatus}
            currentTime={currentTime}
            duration={timelineDuration}
            segments={job.transcript}
@@ -738,38 +770,75 @@ function AudioIntakePanel({ job }: { job: Job | null }) {
 }

 function AudioWaveform({
-  peaks,
+  features,
+  status,
  currentTime,
  duration,
  segments,
  onSeek,
 }: {
-  peaks: number[]
+  features: AudioFeature[]
+  status: AudioFeatureStatus
  currentTime: number
  duration: number
  segments: Array<{ start: number; end: number }>
  onSeek: (time: number) => void
 }) {
  const pointerPct = clampNumber((currentTime / Math.max(duration, 1)) * 100, 0, 100)
+  const hasFeatures = features.length > 0
+  const pitchPoints = features.map((feature, index) => {
+    const x = features.length <= 1 ? 0 : (index / (features.length - 1)) * 100
+    const pitch = feature.pitch ?? 0.5
+    return `${x.toFixed(2)},${(100 - pitch * 100).toFixed(2)}`
+  }).join(" ")
  return (
    <div
-      className="relative h-16 cursor-pointer overflow-hidden rounded-md border border-white/10 bg-black/35 px-2"
+      className="relative h-24 cursor-pointer overflow-hidden rounded-md border border-white/10 bg-black/35 px-2"
+      aria-label="音频响度波形和音高走势"
      onClick={(event) => {
        const rect = event.currentTarget.getBoundingClientRect()
        onSeek(((event.clientX - rect.left) / Math.max(rect.width, 1)) * duration)
      }}
    >
-      <div className="absolute inset-y-1 left-2 right-2 flex items-center gap-[2px]">
-        {peaks.map((peak, index) => (
-          <div
-            key={index}
-            className="flex-1 rounded-full bg-cyan-200/55"
-            style={{
-              height: `${Math.round(6 + peak * 42)}px`,
-              opacity: 0.32 + peak * 0.48,
-            }}
-          />
-        ))}
+      <div className="pointer-events-none absolute left-2 top-1 text-[9px] text-amber-100/55">{hasFeatures ? "高" : ""}</div>
+      <div className="pointer-events-none absolute bottom-1 left-2 text-[9px] text-amber-100/38">{hasFeatures ? "低" : ""}</div>
+      <div className="absolute inset-y-2 left-7 right-2">
+        {!hasFeatures && (
+          <div className="absolute inset-0 flex items-center justify-center text-[11px] text-white/35">
+            {status === "failed" ? "audio.wav 解码失败" : status === "loading" ? "正在解码 audio.wav" : "等待音频文件"}
+          </div>
+        )}
+        <div className="absolute inset-x-0 top-1/2 h-px bg-white/14" />
+        <div className="absolute inset-x-0 top-1/4 h-px bg-white/6" />
+        <div className="absolute inset-x-0 top-3/4 h-px bg-white/6" />
+        {hasFeatures && (
+          <>
+            <div className="absolute inset-0 flex items-center gap-[2px]">
+              {features.map((feature, index) => (
+                <div key={index} className="relative h-full flex-1">
+                  <div
+                    className="absolute left-1/2 top-1/2 w-full -translate-x-1/2 -translate-y-1/2 rounded-full bg-cyan-200/60"
+                    style={{
+                      height: `${Math.round(8 + feature.loudness * 70)}%`,
+                      opacity: 0.26 + feature.loudness * 0.5,
+                    }}
+                  />
+                </div>
+              ))}
+            </div>
+            <svg className="pointer-events-none absolute inset-0 h-full w-full overflow-visible" viewBox="0 0 100 100" preserveAspectRatio="none">
+              <polyline
+                points={pitchPoints}
+                fill="none"
+                stroke="rgba(251,191,36,0.92)"
+                strokeWidth="1.8"
+                strokeLinecap="round"
+                strokeLinejoin="round"
+                vectorEffect="non-scaling-stroke"
+              />
+            </svg>
+          </>
+        )}
      </div>
      {segments.map((segment, index) => (
        <div