Repository navigation
[iceberg] Add metadata.iceberg.sync-full-history to rebuild full history - #9348
vbabenkoru wants to merge 5 commits into
Conversation
|
JingsongLi
left a comment
JingsongLi
left a comment
There was a problem hiding this comment.
Inline review comments on full-history replay publication and recovery safety.
| Snapshot snapshot = snapshotManager.snapshot(snapshotId); | ||
| if (freshRebuild && snapshotId != currentSnapshotId) { | ||
| // see above; the final step replaces its twin through the regular write path | ||
| table.fileIO().deleteQuietly(pathFactory.toMetadataPath(snapshotId)); |
There was a problem hiding this comment.
[P1] Keep the published metadata file intact until replay commits
At this point v${snapshotId}.metadata.json can still be the file referenced by Hive's metadata_location or Hadoop's version hint, especially when snapshotId == currentSnapshotId - 1. Deleting it before createMetadataWithBase finishes creates a reader outage, and any later manifest/DV/metadata write failure leaves the external pointer aimed at a missing file until another callback succeeds. intermediate suppresses hint/catalog updates, but it does not make mutations to the already-published path invisible. Please build the replay under unique staging paths and switch the published head only after the final metadata is durable.
| if (isSameFormatVersion(metadata.formatVersion()) | ||
| && (formatVersion < IcebergMetadata.FORMAT_VERSION_V3 | ||
| || metadata.nextRowId() != null) | ||
| && coversRetainedPrefix(metadata, id, startId)) { |
There was a problem hiding this comment.
[P1] Reject unusable resume bases before selecting them
This candidate is not checked against the live snapshot identity or schema definitions. If an abandoned-timeline/schema-incompatible base triggered the rebuild, this loop selects the same base again; the final createMetadataWithBase rejects it and recursively enters rebuildFullHistory through recreateFromUnusableBase. I reproduced this with rollback, an Iceberg-disabled commit that reused snapshot 2, and a full-history commit for snapshot 3: it ends in StackOverflowError. Please apply all extension-usability checks here, or explicitly exclude the rejected base and restart a non-recursive fresh replay.
| "Unreadable base Iceberg metadata {}, recreating metadata.", | ||
| baseMetadataPath, | ||
| e); | ||
| recreateFromUnusableBase( |
There was a problem hiding this comment.
[P1] Do not reset v3 row lineage after an unreadable base
For a normal next commit, the caller floors are commonly zero. If the published v3 base is corrupt, this catch therefore rebuilds with a null UUID and nextRowIdFloor == 0, even though the unreadable metadata may already have issued higher row IDs. A regression test with v2.next-row-id == 3 produces snapshot 3 with first-row-id == 0, reusing IDs 0-2; before this change the commit failed instead of publishing reset lineage. For v3, recover the UUID/high-water mark from trustworthy older or catalog metadata, and fail safely when it cannot be recovered.
| boolean intermediate) | ||
| throws IOException { | ||
| if (intermediate) { | ||
| createMetadataWithoutBase( |
There was a problem hiding this comment.
[P1] Do not truncate history after an intermediate base-read failure
When intermediate is true, this fallback writes metadata containing only snapshotId. The replay loop then extends that truncated base and can publish the final file without revalidating coversRetainedPrefix, so one transient I/O error, corruption, or concurrent replacement silently drops all earlier retained snapshots and tags. Please propagate/retry the failure or restart from startId, and validate the retained prefix again before final publication.
| Path staged = stagedMetadataPath(rebuildUuid, id); | ||
| Path published = pathFactory.toMetadataPath(id); | ||
| if (table.fileIO().exists(published)) { | ||
| table.fileIO().deleteQuietly(published); |
There was a problem hiding this comment.
[P1] Keep the published head crash-safe during promotion. This still deletes an existing vN.metadata.json before the staged rename. If the process dies in that window, or both rename attempts return false/throw (which FileIO explicitly permits), Hive/REST and the version hint can remain pointed at a missing head indefinitely—the same availability failure staging was meant to eliminate. Please use an atomic overwrite/replace primitive for published positions, or publish a generation-specific immutable head and switch only the external pointer; add a failure-injection test at the head promotion.
There was a problem hiding this comment.
Fixed in the latest commit. Promotion no longer deletes or renames published files. Each one is replaced with FileIO#overwriteFileUtf8, the existing atomic replace (overwrite rename on HDFS, single PUT on object stores), so a position is never missing. The file the hint points to is replaced last, after the hint moves to the new head, and the old manifests are deleted only after promotion succeeds.
testHeadPromotionFailureKeepsPublishedHead makes the head overwrite fail and checks that the hint and the old head are unchanged and still readable.
| "Staged replay head for snapshot %s does not cover the retained history", | ||
| currentSnapshotId); | ||
|
|
||
| Long latest = table.snapshotManager().latestSnapshotId(); |
There was a problem hiding this comment.
[P1] Prevent an older replay from publishing after a newer commit. This latest-snapshot check is separated from both the per-file promotion and the hint/catalog update. Snapshot N+1 can commit after this check, extend/publish metadata, and then this slow N replay overwrites the hint and external catalog back to N; depending on timing, N+1 can also extend the old partial vN chain and permanently lose the rebuilt history. Recheck is still TOCTOU unless publication is serialized/claimed conditionally. Please make replay publication mutually exclusive with later callbacks (or use a CAS against the exact catalog/base head), and test an N+1 callback interleaved after this check.
There was a problem hiding this comment.
Fixed in the same commit. Every publication, live or replayed, now runs the latest-snapshot check, the hint switch and the catalog commit under the table lock from CatalogEnvironment#snapshotCommit, the same lock snapshot commits take. Without a configured lock, the pointer moves forward after each switch to the newest snapshot with complete metadata.
testNewerCommitDuringPromotionIsNotRolledBack commits snapshot 4 during the snapshot-3 switch and checks that the hint ends at 4. Concurrent rebuilds also no longer delete each other's staging files.
When Iceberg metadata is created from scratch (compatibility enabled on a table that already has snapshots, or the previous metadata is unusable), only the latest Paimon snapshot was exposed to Iceberg, losing time travel and tags (apache#6107). With the new opt-in option the whole retained Paimon history is replayed instead: metadata is created afresh for the earliest retained snapshot, then every following snapshot is applied on top of its predecessor, so schemas, tags and the v3 row-id space accumulate exactly like live commits. Each step persists its metadata file, making an interrupted rebuild resumable; a resume base is first validated to cover the retained history prefix, so single-snapshot metadata from a plain rebuild never truncates the replayed history. Intermediate steps skip the version hint, the external catalog commit and cleanup, which only the final step performs. The rollback and self-heal floors (inherited table uuid, last-column-id and next-row-id) are threaded through every replay step so a rebuild never reuses ids handed out by abandoned metadata. Replayed snapshots stay subject to the snapshot retention policy exactly like live commits.
…scratch Creating metadata from scratch dropped every DataSplit that is not rawConvertible. For primary key tables this discards whole buckets whenever they contain level-0 files or overlapping key ranges, silently losing rows that the incremental commit path would have published (files above level 0 with their deletion vectors, or max-level files), and a full-history replay bakes the loss into every replayed snapshot. Collect files per file instead of per split: raw-convertible splits keep their exact export, and non-raw-convertible splits now contribute every file that shouldAddFileToIceberg accepts, together with its deletion vector. Files that genuinely cannot be read without merging (unmerged level-0 data) are counted and reported in a warning with exact file and row counts, pointing at full compaction as the immediate remedy.
A from-scratch full-history rebuild deleted the old build's manifest lists and manifests up front, so an external catalog still pointing at the old metadata could resolve files that no longer existed while the replay was running, or forever if it failed midway. The rebuild also could not recover from a corrupted metadata file: the resume scan tolerated it, but the cleanup walk parsed the same file unguarded and failed the commit. The up-front deletion is replaced by a tolerant collection of the old build's file names; each replay step removes only its own target file just before writing the replacement (a leftover from a regenerated build can match the step's commit identity while carrying other content), and the collected files are deleted only after the final step has published, skipping anything the replayed chain references. The expiration walk and the with-base read of a base metadata file now tolerate unreadable files as well, so a corrupted base self-heals like a structurally invalid one instead of failing every commit. Reported by JingsongLi in review.
A from-scratch replay wrote each step to the published metadata path, deleting the previous file just before: the file referenced by Hive's metadata location or the version hint could briefly disappear, and a replay failing midway left the pointer at a missing file. Replay steps now write to uniquely named staging paths; the published chain, hint and catalog stay untouched until the final staged metadata is durable and validated, and only then is the chain promoted, position by position. A failed rebuild leaves the published metadata fully readable and only staged garbage behind, which the next rebuild removes. The resume-candidate scan now applies the same usability checks as the replay itself (live-timeline identity and schema definitions, on top of format and lineage): an abandoned or re-evolved base was previously selected, rejected by the first replay step, and re-entered the rebuild without terminating. An unusable base hit inside a replay now fails the commit instead of writing reduced metadata, which silently truncated the replayed history; a truncation guard also validates the staged head against the retained prefix before promotion. An unreadable base metadata file now fails the commit on format version 3, since the file may have issued row ids no other file records and rebuilding without its high-water mark would reuse them; format version 2 recreates as before, keeping the identity of the newest readable metadata so external catalogs keep tracking the same table. Reported by JingsongLi in review.
…ommits Promotion deleted each published metadata file before renaming its staged replacement into place. A crash in between, or a rename that returned false (which FileIO permits), left the version hint and an external catalog pointing at a missing head. Published positions are now replaced with FileIO#overwriteFileUtf8, the atomic replacement Paimon already relies on for hint files, so no position is ever missing. The position the pointer names when promotion starts is replaced only after the pointer has moved to the new head; until then it keeps the old build, whose manifests are removed only after a successful promotion. The latest-snapshot check of a replay ran before promotion and before the pointer switch, so a newer commit could publish in between and the slower replay then moved the pointer back. Every publication, live and replayed, now runs the check and the switch under the table lock that Paimon snapshot commits take, so no newer snapshot can commit and publish in between. Without a configured lock, a pointer that was moved back is handed forward to the newest complete head right after the switch. Staging names now carry the rebuild's head snapshot. A rebuild removes only staging of rebuilds it supersedes, and a superseded rebuild whose staging was removed stops quietly instead of failing its commit. Reported by JingsongLi in review.
37b0660 to
dadf0b3
Compare
Purpose
When Iceberg metadata is created from scratch, only the latest Paimon snapshot was available in Iceberg. This happens when Iceberg compatibility is enabled on a table that already has snapshots, or when the old metadata can no longer be used. Time travel and tags were lost (#6107).
This PR has two commits.
1.
metadata.iceberg.sync-full-history(opt-in, defaultfalse): This option replays the full retained Paimon history. It creates new metadata for the earliest retained snapshot. It then applies each later snapshot on top of the previous one. This makes schemas, tags, and the row-id space for format version 3 build up in the same way as live commits.Design points:
inheritUuid,lastColumnIdFloor,nextRowIdFloor) are passed through every replay step. This prevents a rebuild from reusing row ids or column ids that abandoned metadata already assigned.snapshot.num-retained.*,snapshot.time-retained) works on replayed history in the same way as on live commits.2. Keep live-parity files when creating metadata from scratch. Creating metadata from scratch used to drop every
DataSplitthat is not raw-convertible. For primary key tables, this removed entire buckets when they contained level-0 files or overlapping key ranges. This silently lost rows that the incremental commit path would have published. A full-history replay then included that loss in every replayed snapshot. Files are now collected one file at a time instead of one split at a time. Raw-convertible splits keep their exact export. Non-raw-convertible splits add every file that the incremental path would accept, along with its deletion vector. Only unmerged level-0 data is excluded. These files are counted, and a warning recommends full compaction as the fix.Tests
IcebergSyncFullHistoryTest(paimon-core): checks that the default still exposes only the latest snapshot; checks a full replay of retained snapshots with schema changes and a tag in the middle of the history that an Iceberg client can read; checks that an interrupted replay resumes from the newest metadata; checks that a resume candidate without the retained history prefix is rejected; checks that changing the format version rebuilds history while keeping v3 row lineage correct.IcebergBootstrapNonRawSplitsTest(paimon-core): checks that creation from scratch exports compacted files from non-raw-convertible splits; checks full-history replay with non-raw splits.IcebergFullHistoryCompatibilityTest(paimon-iceberg): checks that enabling the option on an existing v3 DV table rebuilds history correctly with the Iceberg 1.8/1.11 readers; checks that an uncompacted DV bucket exports its compacted files.paimon-icebergtest suite passes on both configurations: JDK 11 / Iceberg 1.8.1 and JDK 17 /-Piceberg-ga(Iceberg 1.11).API and Format
Adds the optional table option
metadata.iceberg.sync-full-historywith a default offalse. The default behavior stays the same, except for the live-parity fix in commit 2. That fix only adds files that the incremental commit path would already publish.Documentation
The option is documented through its description in the generated configuration docs.
AI notice: The code is generated using Fable 5 (with reviews from Codex) but has been verified to run on a real cluster with Flink, Paimon, Iceberg, StarRocks and Snowflake.