Repository navigation
[iceberg] Add opt-out of REST catalog drop+recreate recovery - #9349
Draft
vbabenkoru wants to merge 6 commits into
Draft
vbabenkoru wants to merge 6 commits into
vbabenkoru wants to merge 6 commits into
Conversation
vbabenkoru
marked this pull request as draft
August 21, 2026 20:00
vbabenkoru
force-pushed
the
iceberg-rest-reconcile
branch
from
August 24, 2026 15:09
cb42c7e to
58a1bff
Compare
vbabenkoru
force-pushed
the
iceberg-rest-reconcile
branch
from
August 31, 2026 18:07
58a1bff to
6c14d2b
Compare
When Iceberg metadata is created from scratch (compatibility enabled on a table that already has snapshots, or the previous metadata is unusable), only the latest Paimon snapshot was exposed to Iceberg, losing time travel and tags (apache#6107). With the new opt-in option the whole retained Paimon history is replayed instead: metadata is created afresh for the earliest retained snapshot, then every following snapshot is applied on top of its predecessor, so schemas, tags and the v3 row-id space accumulate exactly like live commits. Each step persists its metadata file, making an interrupted rebuild resumable; a resume base is first validated to cover the retained history prefix, so single-snapshot metadata from a plain rebuild never truncates the replayed history. Intermediate steps skip the version hint, the external catalog commit and cleanup, which only the final step performs. The rollback and self-heal floors (inherited table uuid, last-column-id and next-row-id) are threaded through every replay step so a rebuild never reuses ids handed out by abandoned metadata. Replayed snapshots stay subject to the snapshot retention policy exactly like live commits.
…scratch Creating metadata from scratch dropped every DataSplit that is not rawConvertible. For primary key tables this discards whole buckets whenever they contain level-0 files or overlapping key ranges, silently losing rows that the incremental commit path would have published (files above level 0 with their deletion vectors, or max-level files), and a full-history replay bakes the loss into every replayed snapshot. Collect files per file instead of per split: raw-convertible splits keep their exact export, and non-raw-convertible splits now contribute every file that shouldAddFileToIceberg accepts, together with its deletion vector. Files that genuinely cannot be read without merging (unmerged level-0 data) are counted and reported in a warning with exact file and row counts, pointing at full compaction as the immediate remedy.
A from-scratch full-history rebuild deleted the old build's manifest lists and manifests up front, so an external catalog still pointing at the old metadata could resolve files that no longer existed while the replay was running, or forever if it failed midway. The rebuild also could not recover from a corrupted metadata file: the resume scan tolerated it, but the cleanup walk parsed the same file unguarded and failed the commit. The up-front deletion is replaced by a tolerant collection of the old build's file names; each replay step removes only its own target file just before writing the replacement (a leftover from a regenerated build can match the step's commit identity while carrying other content), and the collected files are deleted only after the final step has published, skipping anything the replayed chain references. The expiration walk and the with-base read of a base metadata file now tolerate unreadable files as well, so a corrupted base self-heals like a structurally invalid one instead of failing every commit. Reported by JingsongLi in review.
A from-scratch replay wrote each step to the published metadata path, deleting the previous file just before: the file referenced by Hive's metadata location or the version hint could briefly disappear, and a replay failing midway left the pointer at a missing file. Replay steps now write to uniquely named staging paths; the published chain, hint and catalog stay untouched until the final staged metadata is durable and validated, and only then is the chain promoted, position by position. A failed rebuild leaves the published metadata fully readable and only staged garbage behind, which the next rebuild removes. The resume-candidate scan now applies the same usability checks as the replay itself (live-timeline identity and schema definitions, on top of format and lineage): an abandoned or re-evolved base was previously selected, rejected by the first replay step, and re-entered the rebuild without terminating. An unusable base hit inside a replay now fails the commit instead of writing reduced metadata, which silently truncated the replayed history; a truncation guard also validates the staged head against the retained prefix before promotion. An unreadable base metadata file now fails the commit on format version 3, since the file may have issued row ids no other file records and rebuilding without its high-water mark would reuse them; format version 2 recreates as before, keeping the identity of the newest readable metadata so external catalogs keep tracking the same table. Reported by JingsongLi in review.
…ommits Promotion deleted each published metadata file before renaming its staged replacement into place. A crash in between, or a rename that returned false (which FileIO permits), left the version hint and an external catalog pointing at a missing head. Published positions are now replaced with FileIO#overwriteFileUtf8, the atomic replacement Paimon already relies on for hint files, so no position is ever missing. The position the pointer names when promotion starts is replaced only after the pointer has moved to the new head; until then it keeps the old build, whose manifests are removed only after a successful promotion. The latest-snapshot check of a replay ran before promotion and before the pointer switch, so a newer commit could publish in between and the slower replay then moved the pointer back. Every publication, live and replayed, now runs the check and the switch under the table lock that Paimon snapshot commits take, so no newer snapshot can commit and publish in between. Without a configured lock, a pointer that was moved back is handed forward to the newest complete head right after the switch. Staging names now carry the rebuild's head snapshot. A rebuild removes only staging of rebuilds it supersedes, and a superseded rebuild whose staging was removed stops quietly instead of failing its commit. Reported by JingsongLi in review.
…8875) The REST metadata committer treated any catalog state that was not exactly one snapshot behind as an invalid base and recovered by dropping and recreating the catalog table. On AWS Glue this requires glue:DeleteTable, replaces the table's identity for downstream consumers (Glue, Snowflake), and turns a transient publication failure into a destructive operation: the local metadata file is written before the catalog commit, so an ambiguous failure (CommitStateUnknown) leaves the catalog behind while retries skipped publication entirely because the file already existed. With metadata.iceberg.rest-auto-recreate=false (default true keeps the legacy behavior) the committer now reconciles the catalog in place: * A commit whose metadata file already exists re-drives the REST publication; the committer no-ops when the catalog already holds the intended snapshot (in every mode - previously this recreated the table just to reproduce identical content). * CommitStateUnknownException is resolved by reloading the catalog: if the commit landed it is treated as success, otherwise it is retried once against the unchanged base before surfacing. * A catalog that is behind is brought up to date by replaying, in order, every retained snapshot it is missing - each Iceberg snapshot is self-contained through its manifest list, so this reproduces exactly what consecutive successful commits would have written. Snapshots the catalog already holds are re-activated by a ref move instead of an add (Iceberg rejects adding an existing snapshot id). * Regenerated local history (same ids, different manifest lists, e.g. after a full-history rebuild) is detected by comparing manifest lists, not just snapshot ids; stale catalog entries are removed, and entries whose sequence numbers were already consumed are skipped with a warning since Iceberg sequence numbers are single-use. * States that cannot be reconciled non-destructively - catalog ahead, or a different snapshot under the id being published - fail with IcebergRestCatalogOutOfSyncException naming both states and the manual remediation. The catalog table is never dropped. Initial table creation, empty-table recovery (apache#6906/apache#8335), field-id mapping, schema deduplication and next-row-id handling are unchanged.
vbabenkoru
force-pushed
the
iceberg-rest-reconcile
branch
from
November 1, 2026 07:20
6c14d2b to
c47b367
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Purpose
This PR is stacked on #9348 (the first two commits). Review only the top commit.
The REST metadata committer expects the catalog to be exactly one snapshot behind. It treats any other catalog state as an invalid base. To recover, it drops and recreates the catalog table. Since #9245, it may drop and then re-register the table instead. On AWS Glue, this requires
glue:DeleteTable. It also gives the catalog table a new identity for downstream users such as Glue and Snowflake. This turns a temporary publishing failure into a destructive operation. The local metadata file is written before the catalog commit. An unclear failure (CommitStateUnknownException) can therefore leave the catalog behind. Today, recovery requires dropping the table (#8875).This PR adds
metadata.iceberg.rest-auto-recreate. Its default value istrue, which keeps the current behavior. When this option isfalse, the committer fixes the catalog in place and never drops the catalog table:CommitStateUnknownException), the committer reloads the catalog. If the commit succeeded, it reports success. Otherwise, it retries the same updates once using the unchanged base. If the result is still unclear, it reports the ambiguity.IcebergRestCatalogOutOfSyncException. These states include a catalog that is ahead, a different snapshot stored under the id being published, or a v3 row-id space that the catalog cannot accept. The error names both states and explains the manual fix.Initial table creation, empty-table recovery, the v3 nonzero-watermark registration path from #9245, field-id mapping, schema deduplication, and next-row-id handling are unchanged. The v3 nonzero-watermark registration path remains available only in the default mode.
Tests
Five new tests were added to
IcebergRestMetadataCommitterTest: a behind catalog catches up by replaying multiple snapshots; an already-published snapshot is a no-op in both modes; an ahead catalog fails with the out-of-sync error without dropping the table; regenerated history is found by comparing manifest lists and stale entries are removed; and a retry publishes existing metadata again. The fullpaimon-icebergsuite passes with both configurations: JDK 11 / Iceberg 1.8.1 and JDK 17 /-Piceberg-ga(Iceberg 1.11).API and Format
New optional table option:
metadata.iceberg.rest-auto-recreate(defaulttrue). The default behavior is unchanged, except that the table is no longer recreated when the catalog already contains the commit's snapshot.Documentation
The option is documented in its description, which is used to generate the configuration docs.
AI notice: The code is generated using Fable 5 (with reviews from Codex) but has been verified to run on a real cluster with Flink, Paimon, Iceberg, StarRocks and Snowflake.