[testbed] Add batcher perf tests for heavy processing

[swiatekm]

Description

Add batching performance test simulating heavy processing. The primary intent of these is to help verify that we're not introducing performance regressions with open-telemetry/opentelemetry-collector#8122.

I've added two additional benchmarks:

• The first one uses a single transform processor with 50 statements.
• The second one uses 20 transform processors with 2 statements each.

The idea here is to capture both the overhead of doing a lot of work in a single processor, and the function call overhead of many processors.

I've also refactored the tests to get rid of duplication when generating test cases.

Current results

This benchmark was run with the exporter.UsePullingBasedExporterQueueBatcher feature gate enabled.

Test	Result	Duration	CPU Avg%	CPU Max%	RAM Avg MiB	RAM Max MiB	Sent Items	Received Items
Log10kDPSWithHeavyProcessing/No_batching,_no_queue	PASS	15s	56.3	57.3	66	93	150100	150100
Log10kDPSWithHeavyProcessing/No_batching,_queue	PASS	15s	56.5	57.0	66	92	150100	150100
Log10kDPSWithHeavyProcessing/Batch_size_1000_with_batch_processor,_no_queue	PASS	15s	51.7	53.0	69	98	150100	150100
Log10kDPSWithHeavyProcessing/Batch_size_1000_with_batch_processor,_queue	PASS	15s	51.4	52.3	69	100	150100	150100
Log10kDPSWithHeavyProcessing/Batch_size_1000_with_exporter_batcher,_no_queue	PASS	15s	52.9	54.0	68	96	150100	150100
Log10kDPSWithHeavyProcessing/Batch_size_1000_with_exporter_batcher,_queue	PASS	15s	52.9	53.3	69	98	150100	150100
Log10kDPSWith20Processors/No_batching,_no_queue	PASS	15s	42.1	43.3	68	96	150100	150100
Log10kDPSWith20Processors/No_batching,_queue	PASS	15s	41.4	42.0	68	96	150100	150100
Log10kDPSWith20Processors/Batch_size_1000_with_batch_processor,_no_queue	PASS	15s	30.0	30.3	68	98	150100	150100
Log10kDPSWith20Processors/Batch_size_1000_with_batch_processor,_queue	PASS	15s	29.7	29.7	67	97	150100	150100
Log10kDPSWith20Processors/Batch_size_1000_with_exporter_batcher,_no_queue	PASS	15s	36.4	38.3	68	98	150100	150100
Log10kDPSWith20Processors/Batch_size_1000_with_exporter_batcher,_queue	PASS	15s	35.7	36.7	68	97	150100	150100

Relative to the batch processor, the new exporter batcher loses when there are many processors in the pipeline. I think this is just due to nested function call overhead, but haven't investigated very deeply. Worth noting that if I bump the initial batch size to 100, the differences basically go away.

Link to tracking issue

Fixes open-telemetry/opentelemetry-collector#10836

[swiatekm]

@jmacd @sfc-gh-sili could you have a look? Is this benchmark representative enough to actually help?