Continuous performance unit testing

[gunnarmorling]

No description provided.

[hpgrahsl]

really enjoyed this early stage read! my comments are mostly referring to trivial things like typos etc. plus a few questions. nothing serious I would say.

[hpgrahsl]

note: I stopped processing here ;-)

[github-actions]

♻️ PR Preview ddf9d24 has been successfully destroyed since this PR has been closed.

_{🤖 By surge-preview}

[gunnarmorling]

Hey @hpgrahsl, thanks a lot for that first round of review! I've addressed most of your remarks, and I've also added the missing parts to the post. I.e. it's ready now for a review of that remainder, should you have the time and interest. There's also a rendered preview available now. I'm planning to do some more polishing and also updates to the images, but overall, it's 95% of what I had in mind. Thank you so much!

[Sanne]

While I understand what you mean, I'd say that these specific examples are a bit off since they are very much metrics which I would NOT test using your new tool?

There's a need for testing at both levels:

• very micro, such as "This sort operation here can be performed in less than Z memory allocated even for an N sized array"
• system wide impact, such as system design and integration is such that overall throughput (on certain machine) is within X,Y with some margins.

I'd use a tool like JfrUnit for the first cathefory only, it seems a slippery slope to try abusing it for beyond this and that's probably a claim I'd be umcofortable with :)

[gunnarmorling]

There's a need for testing at both levels

Agreed. I think JfrUnit's testing approach can play a role for both, though. For the second, it wouldn't help you answering that question "overall throughput is within X,Y with some margins" directly, but it would help you to identify potential regressions, going against that goal. It should be more clear in the discussion towards the end, perhaps I need to reword here a bit, too.

[Sanne]

But best to split what you're going to measure here in cathegories. Since JfrUnit is about "indirect" metrics, you can't really verify that not having a certain Sleep(1000) or not having a certain GC event will necessarily give you the throughput you're after.

[Sanne]

well N+1 is meh.. it's certainly good to test against it (and maybe we should see to make this easier with some helper and a blog) but that could be done in much simpler / traditional ways.

I feel it's a bit distracting from the real potential of the Jfr metrics.

[gunnarmorling]

Happy to discuss another example around Hibernate/SQL, if you have one? N+1 was the first thing that came to my mind. It's going to be discussed in a follow-up post (in January), so you got some time to think about it ;)

[Sanne]

hum I don't remember the limit defaults, but I think it's a bit misleading to let people think it will just keep growing, as the whole point of our most important optimisations is to not exceed the limit.

[gunnarmorling]

I'm not quite following on that one; what is the "it" in "it will just keep growing"? The TLAB doesn't grow, it will be used up by allocations, and when the next allocation doesn't fit into it, the thread will get a new TLAB for the next set of allocations. You won't avoid that, unless you do zero allocations or allocate in your own self-managed byte[].

[Sanne]

Suggested change

* *Pro-active idenfication of performance issues:* Asserting a metric like memory allocation can help to identify future performance problems before they actual materialize; while the additional allocation rate may make no difference with the system's load as of today, it may negatively impact latency and throughput as the system reaches its limits with increased load; being able to identify the increased allocation rate early on allows for a more efficient handling of the situation while working on the code, compared to when finding out about such regression only later on

* *Pro-active identification of performance issues:* Asserting a metric like memory allocation can help to identify future performance problems before they actual materialize; while the additional allocation rate may make no difference with the system's load as of today, it may negatively impact latency and throughput as the system reaches its limits with increased load; being able to identify the increased allocation rate early on allows for a more efficient handling of the situation while working on the code, compared to when finding out about such regression only later on

[Sanne]

Also might want to consider: in a complete production system there might be more components being integrated then the ones we have in our testbeds.

When measuring allocation rate of components individually, they might not seem problematic as it's easier to keep the allocation budget within the cheaper TLAB range, so while their individual allocation rate might not seem to have a problematic impact on throughput measurements performed on simple benchmarks, an excessive use of allocations could still translate in problematic bottlenecks on a more complex system as the same TLAB region is shared with other libraries.

[gunnarmorling]

That's an interesting one; any idea how you'd identify such bottleneck? I don't think it needs discussion in this post (it's super long already), but would like to better understand and see what perhaps even can be done in JfrUnit itself towards that end.

[gunnarmorling]

Ok, so I think I've addressed the critical review remarks by all of you. Thank you so much. Latest update just pushed, going to look into zoomable images next.

[gunnarmorling]

Some more word-smithing and fixing. Pushing now. Thank you all, I'm deeply grateful for all the feedback you provided on such short notice 🙏 !