[Bifrost] LocalLoglet push-down filtering and new record format #1826
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This was referenced Aug 12, 2024
Test Results 5 files - 97 5 suites - 97 7m 12s ⏱️ - 16m 20s Results for commit 37d20d4. ± Comparison against base commit 5db1cba. This pull request removes 84 and adds 41 tests. Note that renamed tests count towards both.♻️ This comment has been updated with latest results. |
tillrohrmann
approved these changes
Aug 13, 2024
- Design for lazy serialization and deserialization - In-memory loglet now skips serialization entirely - LogEntry api enables future key-filtering (upcoming) - Fixed benchmarks - Decouple local-loglet's storage layout from Bifrost - Pave for in-memory record cache - Simplified loglet's API, everything now goes through the append_batch API - Support for passing payloads through an Arc since our payloads are pretty large in inefficient to pass through mpsc channels by value. Using Arc internally in bifrost allows near-zero cost clones on retries, caching of records, and zero-cost delivery to readers with no memory copies or deserialization. - Removes requirement for metadata store's value to be `Clone` since StorageEncode is now object-safe As a result of all recent changes, benchmarks show nearly _almost_ double the append throughput, although impact on production workloads will likely be small at this stage
- Upgrade derive_more - Use derive_more as much as possible (it has a comprehensive list of powerful derive macros) - Some `strum::Display` macros still exist whenever we need parity to serde's kebab-case style. - Replaces `strum::EnumIs` with `derive_more::IsVariant` - Replaces `strum::FromRepr` with `derive_more::TryFrom` - Clean ups: strum_macros -> strum (using `derive` feature on `strum` instead of a separate crate. Easier to grep, and one version to maintain) - Remove unused deps in bifrost to speed up builds - Removed a couple of Debug fields in bifrost where we were printing a value under mutex
dashmap v6.0.1 brings nice performance improvements anywhere in 5-40% range
This introduces push-down key-based filtering support in local-loglet along with a new on-disk record layout that supports key checking without copying the payload over. It's extensible through an unused flags field and is currently used to store the key length and the key itself. The new record layout is not enabled by default. This allows the next release to be the first safe version to rollback into once the new format is enabled. Keys are now stored in the legacy payload holder as well, in a backward compatible fashion. Code is not my prettiest, but the logic is solid ;)
|
Thanks @tillrohrmann for the review. I'll merge the stack once GHA is green. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
[Bifrost] LocalLoglet push-down filtering and new record format
This introduces push-down key-based filtering support in local-loglet along with a new on-disk record layout that supports key checking without copying the payload over. It's extensible through an unused flags field and is currently used to store the key length and the key itself.
The new record layout is not enabled by default. This allows the next release to be the first safe version to rollback into once the new format is enabled.
Keys are now stored in the legacy payload holder as well, in a backward compatible fashion.
Code is not my prettiest, but the logic is solid ;)
Stack created with Sapling. Best reviewed with ReviewStack.