Sync openvpi upstream main + PR #326 (mixln_shuffle_speakers) - #4
Open
KakaruHayate wants to merge 24 commits into
Open
KakaruHayate wants to merge 24 commits into
KakaruHayate wants to merge 24 commits into
Conversation
…pi#313) * fix: pad mel with spec_min instead of 0.0 in acoustic collater Raw log-mel 0.0 is not a neutral padding value: norm_spec maps it to +1.0 — the very top of the normalized range, i.e. maximum loudness. Every batch therefore filled the padded tail of shorter samples with full-loudness garbage. Two consequences: 1. The diffusion backbones (WaveNet / LYNXNet / LYNXNet2) receive no padding mask, so their receptive field (~181 frames for LYNXNet2 with kernel_size=31 x 6 layers) leaks the fake signal into the trailing valid frames. The loss mask (mel2ph > 0) hides this from the loss on padding frames, but the contaminated valid frames near the boundary are fully counted — a systematic bias on utterance tails, exactly where breathy endings and vibrato decay live. 2. The aux decoder loss is not masked at all: with zero-padding the aux decoder was actively trained to predict maximum loudness on padding frames from near-zero condition. Padding with spec_min (-12 by default) maps to -1.0 (silence) and sits next to the mel extractor's true silence floor log(1e-5) = -11.51, so padded regions now look like ordinary trailing silence — consistent with what the model sees at inference time. Note: this changes the training data distribution slightly; models trained before/after this fix are checkpoint-compatible but their padded-region behavior differs. * Update mel padding strategy to use spec_min Replace zero-padding with spec_min for mel padding to avoid full-loudness garbage in shorter samples. * Change mel padding to use log scale
* some minor fixes * some minor fixes * fix
* some minor fixes * some minor fixes * fix * correct the length calculation * fix preder_ds data loading * fix the off-by-one issue * fix naming error
* some minor fixes * some minor fixes * fix * correct the length calculation * fix preder_ds data loading * fix the off-by-one issue * fix naming error * fill zero-frame phones with correct pitch values * fill zero-frame words with correct pitch values
* some minor fixes * some minor fixes * fix * correct the length calculation * fix preder_ds data loading * fix the off-by-one issue * fix naming error * fill zero-frame phones with correct pitch values * fill zero-frame words with correct pitch values * union-find over phonemes * remove dead code from prefix matching
* some minor fixes * some minor fixes * fix * correct the length calculation * fix preder_ds data loading * fix the off-by-one issue * fix naming error * fill zero-frame phones with correct pitch values * fill zero-frame words with correct pitch values * union-find over phonemes * remove dead code from prefix matching * fix the behavior of lang_seq under prefer_ds * update BestPractices.md * update GettingStarted.md * remove unused shuffle operations from preprocessing * update the template vocoder_ckpt to match the parent * update ConfigurationSchemas.md * update README.md
* support dual-timestep reflow * fix: preserve timestep rank in sinusoidal embedding (openvpi#316) --------- Co-authored-by: Kakaru <97896816+KakaruHayate@users.noreply.github.com>
* fix energy calculation * arrange configuration items alphabetically * add missed configuration items
* fix energy calculation * arrange configuration items alphabetically * add missed configuration items * Share manager across multiprocessing queues
# Conflicts: # deployment/exporters/variance_exporter.py # docs/GettingStarted.md # training/acoustic_task.py
…ning Mixed_LayerNorm mixes affine (beta/gamma) params across shuffled speakers in the batch at training time. Add a mixln_shuffle_speakers flag (default true = original behaviour) that keeps the conditional-affine network structure but skips the speaker shuffle when set to false, so each speaker's params stay consistent within a step (less inter-speaker leakage). The flag is threaded constructor-to-constructor (no new hparams imports in layer modules, per repo convention) and synced into both acoustic configs.
- variance_encoder: MelodyEncoder honors melody_encoder_args.rope_theta override (fallback chain: enc_hparams -> hparams -> 10000) - kernels/integration: patch_diffusion_module degrades to eager with a warning when Triton is unavailable instead of raising RuntimeError, so use_fused_kernels=true never crashes task construction - acoustic_binarizer: cap k_mutate at len(aug_list) before building aug_types/aug_items so zip() no longer silently drops type-2 entries - acoustic_task: read resolved self.model.backbone_args instead of hparams['backbone_args'] (KeyError on legacy-only configs; may also disagree with the backbone actually constructed)
Audit of configs/templates/*.yaml against docs/ConfigurationSchemas.md found 18 undocumented keys, now added in alphabetical order with the standard entry format (visibility / scope / customizability / type / default / constraints): - mixln_shuffle_speakers (synced from openvpi PR openvpi#326): controls the cross-batch speaker shuffle of Mixed_LayerNorm affine params; default false = each speaker's params stay its own, true = original Mix-LN Beta-mix behavior - voicing_domain / voicing_mu (fork mulaw voicing): 'db' | 'amplitude' | 'mulaw' plus the mu-law parameter; non-default domains are exported into dsconfig for inference frontends - use_mouth_opening_embed / mouth_opening_estimator_ckpt / mouth_opening_smooth_width (fork SHMC mouth-opening conditioning) - use_shift_mouth_opening_embed + shift_mouth_opening_args.* (6 keys, fork SHMC shift self-distillation: teacher ckpt, alpha sigma, replacement prob, opec bounds, teacher AMP) - predict_mouth_opening / mouth_opening_min / mouth_opening_max (variance-side mouth opening with normalization range) - use_acoustic_retake (fork note-level condition-level inpainting) Template coverage is now complete: 0 undocumented keys in all three templates (was 17), documented keys 184 -> 202.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
CodeRabbit review（在 KakaruHayate/DiffSinger 内进行，未在本仓库触发）
CodeRabbit 首轮 review 提出 4 条 Minor 意见，均已验证属实并在本 PR 中修复（提交
89fc557）：modules/fastspeech/variance_encoder.py—MelodyEncoder现在优先读取melody_encoder_args.rope_theta（回退链：enc_hparams → hparams → 10000）。modules/kernels/integration.py—patch_diffusion_module在 Triton 不可用时降级为 eager 并告警（返回 0），不再抛RuntimeError；use_fused_kernels=true在无 Triton 平台不再导致 task 构造失败。preprocessing/acoustic_binarizer.py—k_mutate先按len(aug_list)截断再构建aug_types/aug_items，避免zip静默丢弃多出的 type-2 增广任务。training/acoustic_task.py— fused kernels 的glu_type改读模型已解析的self.model.backbone_args，不再直接索引hparams['backbone_args']（兼容仅声明旧式 backbone 键的配置）。复审已通过（无新增意见）。
文档补充（提交
815a98c）对
configs/templates/*.yaml与docs/ConfigurationSchemas.md做了全覆盖审计，补齐了 18 个此前未文档化的配置项（文档键数 184 → 202，三个模板未覆盖键 17 → 0），按既有格式（visibility / scope / customizability / type / default / constraints）按字母序插入：mixln_shuffle_speakers（本次同步的 PR feat(mixln): add config switch to disable speaker-shuffle during training openvpi/DiffSinger#326）：控制Mixed_LayerNorm训练时跨 batch 的 speaker shuffle，默认false（各 speaker 仿射参数保持独立），true为原始 Mix-LN Beta 混合行为；voicing_domain/voicing_mu（mulaw voicing）：'db'（默认）/'amplitude'/'mulaw'及 mu 参数，非默认域会写入导出dsconfig供推理前端自动启用域转换；use_mouth_opening_embed、mouth_opening_estimator_ckpt、mouth_opening_smooth_width（SHMC 口型条件）；use_shift_mouth_opening_embed及shift_mouth_opening_args.*6 个子键（teacher_ckpt_path / alpha_sigma / opec_min / opec_max / replacement_prob / teacher_use_amp，SHMC 偏移自蒸馏）；predict_mouth_opening、mouth_opening_min、mouth_opening_max（variance 侧口型预测与归一化范围）；use_acoustic_retake（note 级条件级软 inpainting）。注意事项
binarize → train(少量步) → infer冒烟。configs/acoustic.yaml中use_dual_timestep: true（来自上游 Support dual-timestep reflow openvpi/DiffSinger#323 的默认开启行为）会改变 reflow 训练语义，如有既有实验需注意。mixln_shuffle_speakers默认false与上游 feat(mixln): add config switch to disable speaker-shuffle during training openvpi/DiffSinger#326 之前“训练时始终 shuffle”的行为不同，如需复现旧行为请显式置true。