feat: add portable LTX-2.5 audio-video inference - #1602
Open
NancyFyong wants to merge 1 commit into
Open
Conversation
Collaborator
|
Thank you very much for your contribution to this open-source project. We truly appreciate your effort and interest. However, we are unable to merge this pull request at this time for the following reasons:
Due to limited maintainer bandwidth, we anticipate a delay in integrating support for LTX-2.5. If you are willing, we would be happy for you to update this PR to address the above points. Alternatively, please feel free to wait, and we will incorporate the necessary changes ourselves when resources allow. Thank you again for your understanding and continued support! |
Contributor
Author
Thank you for the detailed feedback. I'll address the issues you mentioned. This PR is still under active development, and I'll resolve the merge conflicts and refactor the model code in line with the recent LTX-2.3 changes as soon as possible. |
Contributor
Author
|
Thank you for the review. I have updated this draft:
The revised PR description includes the precise scope and validation results. Thank you again for considering the updated implementation. |
NancyFyong
force-pushed
the
ltx2_5
branch
3 times, most recently
from
August 17, 2026 07:44
2ed0f8f to
e587d1e
Compare
This was referenced Aug 17, 2026
NancyFyong
force-pushed
the
ltx2_5
branch
4 times, most recently
from
August 18, 2026 12:35
fb3bc6a to
4bd0b04
Compare
NancyFyong
marked this pull request as ready for review
August 21, 2026 10:43
Merged
Artiprocher
pushed a commit
that referenced
this pull request
Sep 11, 2026
* feat: add LTX-2.5 inference pipeline * Support LTX-2.5: unified pipeline, flat DiffVAE, INT8 variants and inference examples - Merge LTX-2.5 into LTX2AudioVideoPipeline (drop separate ltx25_audio_video.py) - Register ltx25 DiT / Gemma4 text encoder / feature extractor + connectors / video VAE (DiffVAE + ConvVAE) / audio VAE / vocoder / duration head, BF16 and INT8 ConvRot variants - Flatten DiffVAE into diffsynth/models/ltx25_diffusion_video_vae.py with a single decode interface and auto tiling - Add keyframes abs pos embedding and tokenwise AV cross-attention scale/shift to ltx2_dit - Add stage1 distilled ancestral schedule unit, audio-only (T2A) and A2V freeze support, auto_duration in one pipe call - VRAM management maps for new modules; low-VRAM and standard inference examples for T2AV/I2AV/A2V/T2A/Retake/IC-LoRA/INT8 * Fix LTX-2.5 VRAM management, add training scripts and docs - Resolve aliased/preprocessor module references through the owner in ltx2_dit so VRAM-wrapped modules are used under CPU offload - Move STFT/mel buffers and DiffVAE raw parameters (scale_shift_table, fused QKV) to the input device/dtype at the use site - Read functional projection weights through the VRAM wrapper computation path; reduce the DiffVAE module map to leaf-level entries since the decoder calls block methods directly - Treat missing generate_video input param as True so training caches video positions (T2A keeps generate_video=False) - Add LTX-2.5 T2AV split training scripts (LoRA/full/debug), validate scripts and a series-local zero3 accelerate config - Rewrite LTX-2.5 docs for the unified pipeline, training and unsupported features; add README news and model table rows - Drop the PR leftover LTX-2.5-Keyframe-Interpolation low-VRAM script (keyframes are covered by I2AV) * Use ZeRO-2 with CPU offload for LTX-2.5 full training ZeRO-3 is incompatible with the Gemma4 text encoder: DiffSynth constructs models inside deepspeed.zero.Init, where transformers' _init_weights indexes weight[padding_idx] on an empty sharded embedding and raises IndexError. Replace the series-local zero3 config with the ZeRO-2 + CPU optimizer/param offload config used by the LTX-2.3 full scripts. * Remove decorative separator comment in ltx2_audio_vae * Fix LTX-2.5 A2V fidelity and two-stage refinement - Return the original input audio (resampled and trimmed) when the audio modality is fully frozen, matching the upstream A2V pipeline which skips the VAE/vocoder round trip to preserve fidelity (mel corr vs input 0.928 -> 0.998) - Run the second stage without classifier-free guidance, as upstream uses a simple denoiser there; the stage-1 cfg scale previously leaked into stage 2 - Align the stage-2 distilled LoRA strength default with upstream (1.0 instead of 0.8), which removes residual dithering in dev two-stage outputs * Remove dead code from the LTX-2.5 DiffVAE module Drop unreferenced helpers, tiling constructors and the NATTEN attention fallback that the portable eager path never selects. Verified bit-identical outputs (max abs pixel diff 0) for the distilled T2AV and I2AV keyframe examples before/after. * Point LTX-2.3 examples at the current spatial upscaler checkpoint The Lightricks/LTX-2.3 repo no longer ships ltx-2.3-spatial-upscaler-x2-1.0.safetensors; the registered ltx2_latent_upsampler hash matches the x2-1.1 file, so update the example scripts, docs and the registration example comment accordingly. * Skip video decoder validation during training caching stages The decoder selector validated the loaded decoder component at unit execution time, which breaks split training for LTX-2/2.3 repackaged checkpoints whose stage 1 loads only the VAE encoder. Caching stages never decode, so skip the validation when the scheduler is in training mode; inference behavior is unchanged. * Strip docstrings from the LTX-2.5 DiffVAE module Remove 192 docstring blocks (~1000 lines) from the portable DiffVAE implementation; the module keeps its inline WHY comments. Distilled T2AV output remains bit-identical (max abs pixel diff 0) to the pre-change baseline. File size 6391 -> 5036 lines. * Refine the LTX-2.5 example matrix and default negative prompt The dev weights are the general-purpose checkpoint, so they now cover the full feature set: OneStage/TwoStage T2AV and I2AV examples are added in both the standard and the low-VRAM directories. The distilled and INT8 checkpoints keep a single T2AV example each, so the distilled I2AV scripts are removed; IC-LoRA stays a distilled exception because upstream runs both of its stages on the distilled weights. Docs and README list the resulting ten-row matrix. LTX-2.5 also ships a longer default negative prompt than LTX-2.3: upstream prefixes the shared tag list with has_subtitles, has_blurbox, transition from black, transition to black and speech_ending_short. Add an "LTX-2.5" entry to pipe.default_negative_prompt and switch every 2.5 example, low-VRAM example and validation script to it (the T2A scripts previously used the placeholder "noise"). * Reformat the LTX-2.5 Gemma config literal LTX25_GEMMA_CONFIG was a raw pprint dump with single quotes and hanging indentation. Restyle it to match the other text encoder configs (double quotes, four-space indent, one key per line, inline leaf dicts) and collapse the 48-entry layer_types list into its repeating five-sliding-plus-one-full pattern. Formatting only: the resolved Gemma4UnifiedConfig.to_dict() is byte-identical to the previous one. * Restore upstream preprocessor init timing and stage-2 LoRA default Moving _init_preprocessors into __init__ made the argument preprocessors capture the plain modules before VRAM management replaces them with wrappers, so under CPU offload the preprocessors called stale modules whose weights never onload. PR #1602 built the preprocessors at the start of every forward, after wrapping; restore that timing and drop the OwnerModuleProxy workaround it had made unnecessary. Also restore the upstream stage2_lora_strength default of 0.8, which the LTX-2/2.3 two-stage examples rely on, and pass 1.0 explicitly in the LTX-2.5 two-stage examples whose distilled stage-2 LoRA needs full strength. * verified dit clean * Restore upstream VRAM maps and VAE code, merge the 2.5 tokenizer into the TE file The VRAM module maps and the audio/video VAE modules carried changes that only existed to support our own earlier fine-grained map experiment: PerChannelStatistics, Snake and sibling vocoder entries, the LTX2AudioEncoder entry, the LTXModel block-level entry and the use-site STFT/mel casts. With the maps back to the upstream layout the vocoder and audio encoder fall back to the whole-model wrap exactly as LTX-2.3 runs today, so those code changes are reverted too, together with the string-to-Enum coercions in the video VAE whose only consumers were string extra_kwargs we added for the 2.5 entries. The only remaining map delta is the LTX-2.5 DiffusionVideoDecoder entry, whose flat class path and leaf-level wrapping the 2.5 decoder requires. LTX25GemmaTokenizer moves verbatim into ltx25_text_encoder.py, mirroring how the LTX-2.3 tokenizer lives inside its TE file, and the 2.5 registry entries are regrouped by model hash with concrete dev-file example comments. * Unify LTX-2.5 model loading with the shared registry and a Repackage checkpoint Register the LTX-2.5 components under the same model names as LTX-2/2.3 so from_pretrained fetches every version through one shared path; only the tokenizer construction stays version-specific. The text encoder post modules are the single component whose weights are scattered across two upstream files (feature extractor in the TE checkpoint, connectors in the transformer checkpoint), so they are packed into DiffSynth-Studio/LTX-2.5-Repackage/text_encoder_post_modules.safetensors in target key layout, needing no state dict converter, and the packed Gemma4 tokenizer assets are unpacked into an HF-style tokenizer directory so tokenizer_config works like LTX-2.3. Pipeline cleanups along the way: the video decoder is selected by availability (conv decoder first, diffusion decoder as fallback), gemma_path and load_duration_head are gone, and the upstream section comments, import layout and upsampler fetch position are restored. * Drop the unused Gemma3 processor and unify tokenizer construction The Gemma3Processor was assigned but never read anywhere in the repository, so remove it together with its attribute and imports. Tokenizer construction collapses to picking the version-specific class and instantiating it with tokenizer_path, which is now the parameter name for LTX25GemmaTokenizer as well. * Simplify video decoding and auto duration handling The decoder selector unit is gone: the pipeline keeps a single video_vae_decoder attribute, filled with the ConvVAE decoder when loaded and the DiffVAE decoder otherwise, and __call__ passes the tiling and seed arguments straight to decode(). Both decoders accept **kwargs so the uniform argument set needs no branching; the DiffVAE decoder takes seed/rand_device and builds its sampling generator internally, offset by 42 from the denoise seed, and tiles automatically when tiled is set, matching the official pipeline default. Auto duration validation moves into the AutoDuration unit, which now declares onload_model_names and clamps the requested bounds into (0, 20] instead of raising. * Keep NoiseInitializer in upstream shape and rename generate_video to audio_only NoiseInitializer goes back to the upstream process_stage/process structure; the only additions are the generate-video guard and a build_video_keyframes_mask helper that returns the LTX-2.5 first-frame marker (None for other versions), so the mask is rebuilt at every stage resolution automatically. The token-layout noise, unpatchify bridges and ancestral transforms are gone: noise is drawn in the 5D latent layout like LTX-2.3. The generate_video flag is renamed to audio_only with inverted polarity, which removes the None-normalization trick: the unit runner passes None for missing keys, and a falsy value now correctly means "generate video" during training caching stages. * Give DiffVAE decode explicit tile sizes with a min-tile fallback The decode path accepts tile_size_in_pixels / tile_size_in_frames (overlaps stay halo-derived, since DiffVAE ramps need complementary masks) and validates them; any invalid configuration falls back to the automatic tiling instead of raising. The automatic path now returns the minimum legal tile (spatial floor 512 px, temporal 80 frames), which keeps the decode peak around 7 GB at 1024x1536x121 instead of the recommender's spare-VRAM-hungry choice (~68 GB measured). Inference examples pass tile_size_in_frames=80 and carry a comment pointing at the conv vae decoder as the low-VRAM alternative. * Add AncestralFlowMatchScheduler and revert non-essential pipeline deviations - Move the rectified-flow ancestral Euler step into AncestralFlowMatchScheduler (flow_match.py base classes stay byte-identical to upstream); installed in __call__ for LTX-2.5 distilled runs, stage 2 restores the base scheduler. - Revert base_pipeline.py to upstream: drop the dead generate_noise generator parameter and handle disabled-modality predictions with a 0 placeholder at the model_fn_ltx2 return site instead of a CFG None guard. - Remove LTX-2.5 special-casing from denoise_stage (timestep dtype/sigmas switch, cfg_scale read, timestep_scale), the PipelineChecker raises, and cosmetic reformats; denoise_stage and PipelineChecker now match upstream exactly. - Point LTX25TextEncoder.forward at the inner hidden-states pass so both text encoder generations share one call site in the prompt embedder unit. - Run LTX-2.5 one-stage examples at half resolution, matching LTX-2/2.3. - Ignore *.mp4 and *.wav test outputs. * Inherit LTX25TextEncoder from Gemma4UnifiedForConditionalGeneration - Drop the wrapper layout: the class now inherits the transformers model, the config literal lives in __init__ (no deepcopy, no module-level constant), and the forward override is gone since the inherited forward returns hidden states. - Delete reset_non_persistent_buffers: rope values match transformers' own init bit-for-bit and the embed_scale difference cancels in RMSNorm, verified by end-to-end bit-identical encodings with and without it. - Update the state dict converter key prefixes for the inherited layout (model.language_model.*, model.embed_vision.*, model.embed_audio.*, lm_head). * Mark appended reference tokens as non-keyframes in the keyframes mask Upstream extends the keyframes mask with zeros for given-content conditioning (keyframe_cond.py:86, reference_video_cond.py:102-104: "Reference tokens are never keyframes"); we concatenated ones, which added the learned keyframe embedding to tokens that must not receive it. Verified against the target library with per-channel probes (appended tokens stay unmarked, first latent frame stays marked unconditionally). * Decode frozen retake audio through the audio VAE like upstream The audio_fully_frozen shortcut returned the resampled input waveform and skipped the audio decoder when the retake mask was all-zero; upstream always decodes the (frozen) audio latent (retake.py:326). Restore the unconditional decode path and drop the now-unused resample_waveform import. * Revert .gitignore to the upstream version The *.mp4 / *.wav ignore rules were local-only convenience for test outputs and must not ship in the integration branch. * Merge LTX-2.5 docs into the LTX-2 page and keep position precision per generation - Treat LTX-2.5 as a version update: fold its content into docs/{zh,en}/Model_Details/LTX-2.md, delete the standalone LTX-2.5 pages, point README news/links at the merged page and drop the duplicated doc link entry. - Keep video positions in fp32 for LTX-2.5 and cast them to the model dtype only for LTX-2/2.3; bf16 time coordinates made the 2.5 outputs flicker while fp32 breaks the older generations. - Drop the dead ff_bias plumbing from LTX2TextEncoder and the redundant prose comments in the LTX-2.5 example scripts. * revert gitignore * refactor ltx vram * final check * Drop redundant cfg_scale from LTX-2.5 examples to use the pipeline default The LTX2AudioVideoPipeline defaults cfg_scale to 3.0. Remove the explicit cfg_scale=3.0 from the LTX-2.5 inference and validate scripts (no behavior change) and remove cfg_scale=4.0 from the T2AV validate scripts so they also fall back to 3.0. Keep cfg_scale=1.0 in the distilled / INT8-ConvRot / IC-LoRA-Pixel-Spatial-Upscaler scripts, where CFG is intentionally disabled. * Point LTX-2.5 T2AV training at its own dataset instead of LTX-2.3's The LTX-2.5 T2AV split-training scripts downloaded and read from ltx2/LTX-2.3-T2AV-splited. Give LTX-2.5 a dedicated example dataset (ltx2/LTX-2.5-T2AV-splited), matching the I2AV scripts, so each model version references its own dataset. --------- Co-authored-by: DiffSynth-Studio Bot <noreply@github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Scope
This PR adds LTX-2.5 inference only. It does not add training support or model weights.
Implemented inference
ltx_core, NATTEN, Triton, or Blackwell-DSL dependencyLTX25_VRAM_LIMIT_GBLTX25AudioVideoPipelineand a low-VRAM distilled T2AV examplenum_framesis omittedinput_imagesandinput_images_indexesValidation
python -m py_compilefor added and changed Python sourcesmax_diff=0.0max_diff=0.0max_diff=0.02.98e-08LTX25_VRAM_LIMIT_GB=16: 31.8 GiB peak allocated and 44.4 GiB peak reservedInference samples
All samples below are real full-resolution inference outputs:
960×576, 121 frames at 24 FPS, 5.041667 seconds, H.264 video, and 48 kHz stereo AAC audio. The GIF previews link to their original MP4s. SHA-256 andffprobemetadata are stored in the fork sample PR #1.Only full-resolution results are shown. In-context IC-LoRA is not shown because the separately gated official adapter is unavailable.
Not included
The following official LTX workflows are not implemented by this PR and are not claimed as supported:
The repository contains no checkpoint files, generated media, isolated environments, or authenticated inspection logs. Sample media is published separately in the fork sample PR linked above.