Improving Performance - Maven Artifact Caching

[goneall]

Question

I'm running some very large Trivy scans with several hundred Maven POM files being analyzed.

It is taking over 11 hours to complete.

From analysis, almost all of the time is waiting for Maven artifact downloads.

I was wondering if I could configure Trivy to use a local artifact repository which would essentially be a very large cache or mirror of Maven to speed things up. Something like Sonatype Nexus.

Is it possible to configure Trivy to utilize such a repository?

I did find an old article on implementing a caching forward proxy to solve a slightly different problem. This is something I could try if you think it would help.

Target

Filesystem

Scanner

License

Output Format

SPDX

Mode

Standalone

Operating System

Ubuntu

Version

No response

[knqyf263]

@DmitriyLewen Can you please take a look? You already have many tasks, so anytime you have a minute is fine.

[DmitriyLewen]

I will try to check that on this weak.

[DmitriyLewen]

Hello @goneall
I looked at your case.

I did find an old article on implementing a caching forward proxy to solve a slightly different problem

That instruction was for jar files.
No, it doesn't work now because we use trivy-java-db.

About pom.xml files:
Trivy doesn't have an env (like MAVEN_CENTRAL_URL) to set repositories to look for pom files.
Trivy only checks repositories from pom.xml files (so you can temporarily add your local repository to the scanned pom file (but I'm not sure if that's ok for you)).
I think the best solution is to use the local repository from the settings file. We already have #7807 with a request to add this functionality.

[goneall]

Thanks @DmitriyLewen - I'll look into your suggestion.

In looking at the Trivy and trivy-java-db code, it looks like Trivy is already doing quite a bit of caching and network access optimization.

I'll do some further analysis to see if I can come up with some additional optimizations.

[DmitriyLewen]

Trivy is already doing quite a bit of caching

There is problem with cache for pom.xml files:
Trivy doesn't use cache for fs mode.

---

Actually i was thinking about this case again last night.
Maybe you are not the only one who faces such a case.
Maybe we should make an exception and add env (e.g. TRIVY_MAVEN_MIRROR) with repository to add it first in the list of repositories (because way with settings.xml is also not perfect, since for scanning users will have to change this file (and they can forget to change it back, etc.))
@knqyf263 wdyt?

[knqyf263]

@goneall How do you configure mirror repositories with Maven? If it's already configured for Maven, it sounds straightforward that Trivy respects that.

[goneall]

We're not running the scans on the same environment as the builds. This may be a somewhat unique situation - the project is scanning all of the Linux Foundation key projects, so we have to deal with a wide variety of languages and configurations.

From looking at the code - it seems like adding a configuration to the Maven settings.xml should direct the requests to a local repo. Over the next week or two, I'll setup a couple experiments to see if I can speed it up. I'll let you know how it goes.