[Perf] Re-rendering multiple dicom images at once

[shunia]

High-level description

I'm building a filmer app that the customer would be able to interact with multiple dicom images at once for adjusting say ww/wc, pan, scale etc, and I was aiming for 60fps for smooth real-time interactions.

I'm using vtkOpenGLRenderWindow under the hood and creating multiple renderWindows and each window holds a dicom image (pixel data) for interactions, which is quite similar to the MultipleRenderWindow example, but with some modifications:

• use custom managed input event handlers instead of vtk interactors
• re-render by picking nessesary child renderWindows and call the .traverseAllPasses() api instead of rootWindow.render()
• split render calls into batches with a size about 40 between frames (with requestAnimationFrame), and ensures each renderWindow will only be re-rendered once within the re-render calls

I'm stuck into a sutuation where in my developing machine I can only achive 60fps when re-rendering mostly 40 images at once with strategies implemented as above, when adjusting ww/wl with:

// the code is definitly much complex than this but the calling sequence is much like this
actor.getProperty().setColorWindow(ww);
actor.getProperty().setColorLevel(wl);
renderWindow.traverseAllPasses();

the hardware may be better in production machine but still the number is far less than what I would want to achive, which I would like to keep 60fps when re-rendering about 100 images at once, because the whole app would load over 2000 images at once (the loading performance and the first rendering performance is not a problem here)

With above senerios, what could I do to improve the performance? Or is there anything I do wrong? Is there ways to batch the re-rendering under the hood in vtkjs to squeeze the performance?

Steps to reproduce

It's somehow a little bit complicate to extract the logic out of my app so for now there's none re-produciable steps.

Detailed Current Behavior

Environment

• vtk.js version: "@kitware/vtk.js": "^30.9.3",
• Browsers: Chrome Version 129.0.6668.100 (Official Build) (64-bit)
• OS: Window 10 Enterprise 10.0.19045 Build 19045

[finetjul]

What exactly does the screenshot represent ?

It's odd that you have to create a new texture each time you re-render. Do you confirm that updatelabelOutlineThicknessTexture() creates vtkOpenGLTexture at each render ?
(fyi @sedghi)

Did you check that MultipleRenderer is any faster ?
If your render windows do not need to share GPU resources, did you try to not make them shared ?

[sedghi]

when re-rendering mostly 40 images at once

You mean you have 40 textures and 40 render window? I'm not surprised if that does not achieved 60 fps

[shunia]

Hi @finetjul, @sedghi,

Thank you for your quick responses. I’d like to provide some additional context regarding our current rendering challenges.

Summary of Current Situation:
Our application renders between 200 to 2000 renderWindows simultaneously, each displaying a DICOM image with adjustable parameters like window width/level, scale, pan, etc. Users can select multiple renderWindows to modify images collectively. However, achieving 60 fps with 2000 renderWindows is impractical. To address this, I am currently batching the render calls, processing 40 renderWindows per batch using the traverseAllPasses function for each renderWindow.

Questions and Clarifications:

1. Regarding updatelabelOutlineThicknessTexture: I have not called this API. What steps should I take to confirm this?
2. MultipleRenderer Approach: Implementing a MultipleRenderer is not feasible at this stage as our app is based on a MultipleRenderWindow architecture. Reworking the entire implementation is not an option right now.
3. Render Windows: While I mentioned having 40 renderWindows in the batch process, I need to clarify that I'm managing 200 to 2000 renderWindows simultaneously, rendering only 40 at a time to optimize performance.

I would greatly appreciate your feedback and any suggestions on alternative approaches we might explore. If MultipleRender is truly the only viable solution, I am open to considering it, despite the potential need for extensive codebase modifications. However, I hope we can identify other strategies to test before pursuing that route.

[finetjul]

If you have that many views in your single page application, it seems that you will get the best performances with having a unique render window with a unique renderer. You would then have many ImageResliceMappers... (or potentially a unique one with a tiled image you created on the cpu)

[shunia]

...with having a unique render window with a unique renderer...

Could you please elaborate more on this?

...You would then have many ImageResliceMappers...

I did not find any examples on how to use vtkImageResliceMapper, could you please direct me to resources/docs on how to use it?

[finetjul]

Could you please elaborate more on this?

const fullScreenRenderer = vtkFullScreenRenderWindow.newInstance();
const renderer = fullScreenRenderer.getRenderer();
const renderWindow = fullScreenRenderer.getRenderWindow();

for (let i = 0; i < 1000; ++i) {
  const mapper = vtkImageResliceMapper.newInstance();
  mapper.setSlicePlane(vtkPlane.newInstance({normal: ..., origin: ...));

  const actor = vtkImageSlice.newInstance();
  actor.setMapper(mapper);
  renderer.addActor(actor);
}

I did not find any examples on how to use vtkImageResliceMapper, could you please direct me to resources/docs on how to use it?

https://kitware.github.io/vtk-js/examples/ImageResliceMapper.html

[shunia]

@finetjul Sorry for the late response, I've been away from this project and just getting back.

For the one renderer solution I believe the point is to reduce the number of canvas to only one, and use multiple mapper to render the images into the unique canvas to gain performance, right? This is clear and definitly would work somehow, the downside would be we need to re-implement a lot of basic dom stuff like layout/event handling/intersections, etc, we need to discuss the possibiltis internally of course.

For now, the rendering performance is acceptable after batching + intersection observers, which means the maximum windows rendered at one screen are less than 500, which is managable.

But at the same time, we hit another wall which is the rendering is blocking the main thread a little bit, especially when initializing all these images into the canvases. I know that it is possible to use OffscreenCanvas or even better Web Worker to transfer the canvas manuplations to the another thread. Is it possible for us to do so? Does the library provide any APIs for this kind of process, or how can we 'hack' into the process to do it?

[finetjul]

The ManyRenderWindows shows offscreen rendering: https://kitware.github.io/vtk-js/examples/ManyRenderWindows.html

[shunia]

I did not find any OffscreenCanvas in the said example.

I did find out that cornerstone3D is using OffscreenCanvas to render multiple viewports at one time with a custom render window implementation though: vtkOffscreenMultiRenderWindow

[finetjul]

The offscreen rendering is done under the hood when you have a parent render window (mainRenderWindow.addRenderWindow(renderWindow);)

[shunia]

Sorry but I can not find any evidence in the source codes for the usage of OffscreenCanvas.
I've searched direct ref to OffscreenCanvas, but got only result is in ORMTexture.worker.js which is not directly used in rendering.
I've searched the API transferControlToOffscreen which is another way to create a OffscreenCanvas, there is no result at all.

If I'm mistaken here please bare with me.

[finetjul]

Indeed, it does not rely on OffscreenCanvas. That could be a nice addition though...