Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Reenable proper trimming based on completed checkpoints #1812

Open
Tracked by #1807
tillrohrmann opened this issue Aug 8, 2024 · 2 comments · May be fixed by #2390
Open
Tracked by #1807

Reenable proper trimming based on completed checkpoints #1812

tillrohrmann opened this issue Aug 8, 2024 · 2 comments · May be fixed by #2390
Assignees

Comments

@tillrohrmann
Copy link
Contributor

tillrohrmann commented Aug 8, 2024

Remove temporary band-aid introduced via #1781. Once we upload state checkpoints to the object store, they are accessible to all partition processor replicas. Hence, it is theoretically possible to trim the log once the checkpoint was stored. However, if some replicas are lagging behind, this can lead to them encountering trim gaps. So either, we wait until all replicas have reached the lsn of the checkpoint before trimming or we add support to handle trim gaps by downloading the latest checkpoint.

@tillrohrmann tillrohrmann changed the title Reenable proper trimming based on completed snapshots Reenable proper trimming based on completed checkpoints Nov 1, 2024
@tillrohrmann
Copy link
Contributor Author

What the partition processor needs to report to the CC is the last snapshotted lsn. Based on this information, the CC can make a decision when it is safe to trim the log.

@pcholakov pcholakov self-assigned this Nov 22, 2024
@pcholakov
Copy link
Contributor

While we're at it, we should also clean up the restatectl display; we can eliminate PERSISTED-LSN and hide ARCHIVED-LSN behind a --extra flag.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants