Documenting memory usage/management by BatchLogProcessor #2469
Conversation
Codecov ReportAll modified and coverable lines are covered by tests ✅
Additional details and impacted files@@ Coverage Diff @@
## main #2469 +/- ##
=====================================
Coverage 77.1% 77.1%
=====================================
Files 124 124
Lines 23021 23021
=====================================
Hits 17771 17771
Misses 5250 5250 ☔ View full report in Codecov by Sentry. |
| @@ -167,6 +167,72 @@ enum BatchMessage { | |||
|
|
|||
| /// A [`LogProcessor`] that buffers log records and reports | |||
| /// them at a pre-configured interval from a dedicated background thread. | |||
| // **Memory Management in BatchLogProcessor** | |||
There was a problem hiding this comment.
a lot of these can be moved off to docs.rs itself, as they are good for users to be aware of.
| // 1. **Record Ingestion**: | ||
| // - Each `LogRecord` is **cloned** upon entering the processor. | ||
| // - `LogRecordAttributes` utilize a hybrid memory model: | ||
| // - Attributes up to `PREALLOCATED_ATTRIBUTE_CAPACITY` are **stack-allocated**. |
There was a problem hiding this comment.
Since PREALLOCATED_ATTRIBUTE_CAPACITY is not a public const, lets write the exact number of attributes here. I would suggest something like this:
- The first 5 attributes are allocated on the stack.
- Any additional attributes are allocated on the heap in a dynamically growing vector.
| // 4. **Export Process**: | ||
| // - A temporary `Vec` is created during the export process to hold ownership of the boxed records: | ||
| // - The contents of the worker thread's buffer (`Vec<Box<(LogRecord, InstrumentationScope)>>`) | ||
| // are moved into this temporary vector using `logs.split_off(0)`. |
There was a problem hiding this comment.
There was a problem hiding this comment.
@utpilla, yes that's a good point - the original buffer is not getting used/updated during export process, so it's good idea to reuse it for export, and then clear it up.
| // | ||
| /// 5. **Memory Limits**: | ||
| /// - **Worst-Case Memory Usage**: | ||
| /// - **Queue Memory** = `max_queue_size * size of boxed (LogRecord + InstrumentationScope)`. |
There was a problem hiding this comment.
It'd be nice to mention the actual size of LogRecord and InstrumentationScope in bytes to get a better idea of the memory consumption.
| // are moved into this temporary vector using `logs.split_off(0)`. | ||
| // - Ownership of the boxed records is transferred to the new vector, ensuring efficient | ||
| // memory usage without additional cloning. | ||
| // - The temporary vector is then used to construct references passed to the exporter via `LogBatch`. |
There was a problem hiding this comment.
It'd be good to mention that there two temporary vectors created during each export:
- One vector is created by
split_off(0)(we could avoid this if we pursue Avoid vec allocation during each export for BatchLogProcessor #2483) - The second vector is created by collecting references from the first vector mentioned above.
I think we should be able to avoid the second vector creation as well by updating LogBatch constructors.
Changes
Felt it would be good to document it for better understanding of the memory management by BatchLogProcessor. This is not the doc-comment so won't go the docs.rs. If there is better place to add this, please suggest.
Merge requirement checklist
CHANGELOG.mdfiles updated for non-trivial, user-facing changes