Plain merge tree partition exports by arthurpassos · Pull Request #2290 · Altinity/ClickHouse

arthurpassos · 2026-08-28T17:30:25Z

List of changes:

Implements partition export for plain merge tree using Json task descriptors stored on disk, the structure is simplified because it is much simpler than the replicated case
Introduces the system.partition_exports table that keeps track of both plain and replicated cases. The old system.replicated_partition_exports is aliased to this new one for backwards compatibility.
Escapes each portion of the partition export key - this is not backwards compatible.
Stop shipping empty parquet files on plain object storage for exports with zero rows
Throws on a few errors zk that were left as todo
Refactor the test suite so that it is parametrized for both engines instead of duplicating the tests

Changelog category (leave one):

New Feature

Changelog entry (a user-readable short description of the changes that goes to CHANGELOG.md):

Export partition on plain merge tree. Should close #1908

User facing & functional changes:

Implements partition export for plain merge tree
Stop shipping empty parquet files on zero rows exports
New system.partition_exports table that covers both the previous replicated_partition_exports and the new one for plain merge tree

Documentation entry for user-facing changes

...

CI/CD Options

Exclude tests:

Regression jobs to run:

github-actions · 2026-08-28T17:31:49Z

Workflow [PR], commit [2662683]

arthurpassos · 2026-08-29T17:14:51Z

@codex review

chatgpt-codex-connector · 2026-08-29T17:15:05Z

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review	Status	Commit	Review trigger
📝 Code Review	✅ Completed 2026-09-04T12:10:15.243704Z	`c8d038d`	Manual request

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

arthurpassos · 2026-08-29T17:20:28Z

@codex review

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 240da8173c

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

chatgpt-codex-connector · 2026-08-29T17:37:02Z

+        catch (...)
+        {
+            replicated_error = std::current_exception();
+            tryLogCurrentException(getLogger("InterpreterKillQueryQuery"),
+                "KILL EXPORT PARTITION: could not read system.replicated_partition_exports (the WHERE may "
+                "reference columns that only exist for plain MergeTree); ignoring ReplicatedMergeTree tables");


Propagate system-table read failures during KILL

If either system-table query fails for an operational reason such as a memory limit, timeout, or internal exception, this broad catch treats it like an engine-specific unknown column and continues with the other table. The command can therefore report successful results while silently leaving matching exports from the failed table running. Only suppress the specific predicate-resolution error that proves the filter belongs exclusively to the other table; propagate all other failures.

AGENTS.md reference: AGENTS.md:L153-L153

Useful? React with 👍 / 👎.

…ya-26.6/export-partition-plain-merge-tree

arthurpassos · 2026-09-04T11:53:25Z

@codex review

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: c8d038d350

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

…n-plain-merge-tree

…n-plain-merge-tree Keep the skip-policy commit-file short-circuit from antalya-26.6 together with lazy first-sink creation, and port the new skip-policy tests into the unified object-storage suite. Co-authored-by: Cursor <cursoragent@cursor.com>

…n-plain-merge-tree

arthurpassos · 2026-09-11T11:50:18Z

+
+### Pending mutations {#plain-merge-tree-pending-mutations}
+
+The pending-mutation gate is more conservative than on a `Replicated*MergeTree`. A plain `MergeTree` does not scope its mutation snapshot by partition, so a mutation restricted with `IN PARTITION` still marks the parts of every other partition as having pending mutations, and exporting an unaffected partition is refused with `PENDING_MUTATIONS_NOT_ALLOWED`. The gate fails closed - it never exports data that a pending mutation would have changed - so the effect is that you may have to wait for an unrelated mutation to finish, or set `export_merge_tree_part_throw_on_pending_mutations` to `false`.


I should check ways around this later

arthurpassos · 2026-09-11T11:52:25Z

 ### Commit info columns

-These columns surface paths produced by the destination storage during commit, so it is possible to inspect what was written without consulting the destination directly:
+These columns surface paths produced by the destination storage during commit, so it is possible to inspect what was written without consulting the destination directly. They are populated for `Replicated*MergeTree` sources only; a plain `MergeTree` does not persist the commit paths, so they stay empty there even after a successful commit.


This shall be implemented for plain merge tree as well

Is this going to be covered in another PR?

I plan to cover it in this pr unless it is too complex (I doubt it is).

For now I am reviewing my own code (AI code :D)

DimensionWieldr · 2026-09-11T17:11:57Z

Iceberg export partition regression suite is fully passing on no/ice/glue catalogs with plain merge tree tables.

I will wait for dev review and the CI run to do a final CI triage to confirm no PR caused failures.

arthurpassos added 5 commits August 28, 2026 14:02

1st vibe coded version

69f071a

some vibe coded fixes

3190603

persist-then-apply and uuidv4 instead of snowflakeid

d94ea53

idk

b2af06c

possibly fix build

f2ee03f

arthurpassos mentioned this pull request Aug 28, 2026

Plain merge tree partition export #2032

Closed

28 tasks

svb-alt added antalya antalya-26.6 labels Aug 28, 2026

chmod

240da81

arthurpassos marked this pull request as ready for review August 29, 2026 17:20

chatgpt-codex-connector Bot reviewed Aug 29, 2026

View reviewed changes

arthurpassos added 9 commits August 31, 2026 14:44

fix kill operation

9ff4a4a

fix backoff on plain

6c86d2f

make code a bit more readable, ai

a622f44

release parts

23d90fa

fix endless commit due to part ref lost

fe7e61a

well, I don't like it, but ok

7c959fb

fix dispatch err handling

f06a7dd

fix

7482d6b

Merge remote-tracking branch 'origin/antalya-26.6' into feature/antal…

c8d038d

…ya-26.6/export-partition-plain-merge-tree

chatgpt-codex-connector Bot reviewed Sep 4, 2026

View reviewed changes

arthurpassos added 2 commits September 4, 2026 09:15

throw exceptions to the user

875cc47

throw on unknown values

9496d85

arthurpassos changed the title ~~Feature/antalya 26.6/export partition plain merge tree~~ Plain merge tree partition exports Sep 4, 2026

simpĺify docs

3af519a

arthurpassos added 6 commits September 4, 2026 10:19

throw on exporting unreadable entries

31b04b1

Merge branch 'antalya-26.6' into feature/antalya-26.6/export-partitio…

01b5d38

…n-plain-merge-tree

unify tables

a0badb4

fix old bug

2eab452

add no-cas

cd502a6

rmv unnecessary tests

3fc5e5f

arthurpassos added the port-antalya PRs to be ported to all new Antalya releases label Sep 8, 2026

arthurpassos and others added 7 commits September 8, 2026 13:51

fix long standing issue with key

cae460a

unify replicated and plain export tests

d4a3859

Merge branch 'antalya-26.6' into feature/antalya-26.6/export-partitio…

b50cee3

…n-plain-merge-tree

fix few bugs

b89bfa1

improve testing suite

00b1540

Merge branch 'antalya-26.6' into feature/antalya-26.6/export-partitio…

487b674

…n-plain-merge-tree

arthurpassos commented Sep 11, 2026

View reviewed changes

arthurpassos added 4 commits September 11, 2026 08:57

some documentation fixes

0433384

fix some more docs

b9f4406

improve naming

ff54cf5

more comments docs

7a8b440

arthurpassos added 4 commits September 11, 2026 14:34

rename method

968163b

documentation

8ef5a76

ai fixes

c70028c

persist commit info plain mt

2662683


		### Pending mutations {#plain-merge-tree-pending-mutations}

		The pending-mutation gate is more conservative than on a `Replicated*MergeTree`. A plain `MergeTree` does not scope its mutation snapshot by partition, so a mutation restricted with `IN PARTITION` still marks the parts of every other partition as having pending mutations, and exporting an unaffected partition is refused with `PENDING_MUTATIONS_NOT_ALLOWED`. The gate fails closed - it never exports data that a pending mutation would have changed - so the effect is that you may have to wait for an unrelated mutation to finish, or set `export_merge_tree_part_throw_on_pending_mutations` to `false`.

Uh oh!

Conversation

arthurpassos commented Aug 28, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Changelog category (leave one):

Changelog entry (a user-readable short description of the changes that goes to CHANGELOG.md):

Documentation entry for user-facing changes

CI/CD Options

Exclude tests:

Regression jobs to run:

Uh oh!

github-actions Bot commented Aug 28, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

arthurpassos commented Aug 29, 2026

Uh oh!

chatgpt-codex-connector Bot commented Aug 29, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codex Review Summary

Uh oh!

arthurpassos commented Aug 29, 2026

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

Uh oh!

Uh oh!

Uh oh!

chatgpt-codex-connector Bot Aug 29, 2026

Choose a reason for hiding this comment

Uh oh!

arthurpassos commented Sep 4, 2026

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

arthurpassos Sep 11, 2026

Choose a reason for hiding this comment

Uh oh!

arthurpassos Sep 11, 2026

Choose a reason for hiding this comment

Uh oh!

DimensionWieldr Sep 11, 2026

Choose a reason for hiding this comment

Uh oh!

arthurpassos Sep 11, 2026

Choose a reason for hiding this comment

Uh oh!

DimensionWieldr commented Sep 11, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

arthurpassos commented Aug 28, 2026 •

edited

Loading

github-actions Bot commented Aug 28, 2026 •

edited

Loading

chatgpt-codex-connector Bot commented Aug 29, 2026 •

edited

Loading