Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Reconfigured run_benchmarks CI so that it can be conducted all moveit_benchmarks reliably #21

Merged

Conversation

CihatAltiparmak
Copy link
Member

@CihatAltiparmak CihatAltiparmak commented Jul 28, 2024

With this PR, it's aimed to conduct benchmark in github action and then to visualize all the benchmark results as well.

  1. Using this repo's own docker image, we just run run_benchmarks.sh for each middleware. But all benchmarks run inside one job because it's not guarantied that github uses same machine in every job. This blocks the reliability of benchmarks which runs on github action's runners. For example:

Screenshot from 2024-07-28 03-15-04

In above picture, you can see the plots related to the results of scenario basic service client. Until 133944c commit, you can see the unreliable results. After 133944c commit, graph starts to show the real plots. we still cannot guarantee comparison between commits, but seems we can at least guarantee the comparison in same commit between middlewares.

Copy link
Member

@henningkayser henningkayser left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This looks good to me, overall. I would only suggest combining the runs into a matrix by using value substitution for cyclonedds and fastrtps, similar how the MoveIt job runs the same thing for different distros https://github.com/moveit/moveit2/blob/main/.github/workflows/ci.yaml#L18

@CihatAltiparmak CihatAltiparmak merged commit a6e71c4 into main Aug 12, 2024
5 of 11 checks passed
@CihatAltiparmak CihatAltiparmak deleted the fix/reconfigure_ci_jobs_for_conducting_benchmarks branch October 16, 2024 13:55
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants