Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Optimize provenance generation speed #122

Merged
merged 1 commit into from
Aug 29, 2024

Conversation

joinemm
Copy link
Contributor

@joinemm joinemm commented Aug 29, 2024

This PR gives the provenance generator ~60x speed increase when generating recursive provenance for a big system image, from 5 minutes to 5 seconds, without removing any data. In fact, the detail of the provenance is even increased now with the addition of --include-outputs flag, adding previously missing dependencies to the array.

The nix commands themselves are fast but the subprocess.run() calls for every dependency in the loop was causing significant overhead. Upon reading the nix docs I found out that you can pass as many derivations as you want in one nix-store query, and you will get the result for all of them, so we can compute all hashes before the loop in single command.

@joinemm joinemm requested a review from a team August 29, 2024 06:58
@joinemm joinemm merged commit 444ba45 into tiiuae:main Aug 29, 2024
3 checks passed
@joinemm joinemm deleted the pr-provenance-faster branch August 29, 2024 11:06
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants