Skip to content

fix: Ensure that representative but not deployable snapshots use the Prod physical models consistently - #6065

Draft
davem-bis wants to merge 4 commits into
SQLMesh:mainfrom
davem-bis:feature/DRM/indirect-non-breaking-always-uses-prod-ledger
Draft

davem-bis wants to merge 4 commits into
SQLMesh:mainfrom
davem-bis:feature/DRM/indirect-non-breaking-always-uses-prod-ledger

Conversation

@davem-bis

@davem-bis davem-bis commented Sep 14, 2026

Copy link
Copy Markdown
Contributor

Description

Fixes #5793.

Ensures that all snapshots that are representative but not deployable (they contain the same data as prod, but can never become prod) are just pass through views of the production physical table. A by-product of this is that they can never have any missing intervals.

Test Plan

  • Expansion of existing unit tests for changed logic.
  • Run through of INDIRECT_BREAKING_CHANGES snapshots to prove idempotence.
    • Run plan:
      1. sqlmesh init

        Option 1 throughout - basic DuckDB implementation with the demo models.

      2. sqlmesh plan

      3. Add 'ABC' AS extra_column to incremental_model.sql

      4. sqlmesh plan indirect_non_breaking

      5. sqlmesh run

      6. sqlmesh run indirect_non_breaking

      7. sqlmesh fetchdf "UPDATE sqlmesh._intervals SET end_ts = end_ts - 86400000, start_ts = start_ts - 86400000, last_altered_ts = last_altered_ts - 86400000"

        86400000 is the number of milliseconds in a day (246060*1000)

      8. sqlmesh run indirect_non_breaking

      9. sqlmesh run indirect_non_breaking

        Previously this would result in the intervals being re-run, it now correctly does nothing.

      10. Plan a new model named extra_model as a SELECT * FROM sqlmesh_example.full_model and confirm that it is selecting from the full_model physical model without the __dev extension.

    • Comments:
      • Model Name Change Type Deployability Representativeness Version Used In indirect_non_breaking
        seed_model None True True Same as prod
        incremental_model NonBreaking True True Separate physical model
        full_model IndirectNonBreaking False True Prod physical model
      • full_model uses the non-__dev physical model, and has no missing intervals.
  • Run through of the above plan, but with vde_mode: dev_only.

Checklist

  • I have run make style and fixed any issues
  • I have added tests for my changes (if applicable)
  • All existing tests pass (make fast-test)
  • My commits are signed off (git commit -s) per the DCO

…any intervals and are a pure pass-through of prod.

Signed-off-by: davem-bis <68955845+davem-bis@users.noreply.github.com>
if not s.is_model or s.is_symbolic:
return False

# Do not create snapshots that contain production data but can never be deployed.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This needs an appropriate unit test.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Bug: INDIRECT_NON_BREAKING snapshots in dev environments write to a table nobody reads, causing repeated re-backfills

1 participant