ITKPerformanceBenchmarking

Real-world tests to benchmark ITK performance.

New classes increase operating system process priority to minimize the impact of other processes running on the system.

These classes are used by a suite of example ITK benchmarks to quantify toolkit performance.

For more information, see the Insight Journal article:

McCormick M., Kang H.J., Barre S.
Performance Benchmarking the Insight Toolkit
The Insight Journal. January-December. 2016.
https://hdl.handle.net/10380/3557
https://insight-journal.org/browse/publication/972

Since ITK 4.11.0, this module is available in the ITK source tree as a Remote module. To enable it, set:

Module_PerformanceBenchmarking:BOOL=ON

in ITK's CMake build configuration.

Requirements

• CMake
• Ninja
• Python

Notes for running the benchmarks

An example to call the benchmarking script is:

$ python ./evaluate-itk-performance.py run -g {ITK-version} {ITK-source} {ITK-build} {ITKPerformanceBenchmarking-build}

where {ITK-version} is the ITK version that the user wishes to evaluate.

Also, note that the ITK source folder (ITK-source}, where the specific version is fetched, needs to exist.

The generated/result JSON files are placed in:

./{ITKPerformanceBenchmarking-build}/BenchmarkResults/{machine-name}

Notes for benchmarking in Windows

CMake and Ninja need to be in the PATH. Also, the C++ compiler has to be in the PATH so that Ninja can find it.

For the Microsoft Visual Studio compiler, the C++ compiler is a file that is usually under:

C:/Program Files/Microsoft Visual Studio {version}/VC/bin/cl.exe

The user will need to change the path to the vcvars*.bat command file in the RunWithVisualStudio.cmd command line script to the specific location of their vcvars file, e.g.:

C:/Program Files/Microsoft Visual Studio {version}/VC/vcvarsall.bat

Finally, the user will need to start the Git bash by double-clicking on the RunWithVisualStudio.cmd script.

Note that the module is built with static libraries to allow for ITK benchmarking in Windows: the BUILD_SHARED_LIBS flag in the evaluate-itk-performance.py script is set to OFF.

License

This software is distributed under the Apache 2.0 license. Please see the LICENSE file for details.