Introduce /run/modules/uname -r path to be able to preserve modules from initramfs in the main system

[vittyvk]

I've opened kmod-project/kmod#202 already but just want to raise the visibility.

The idea is to make UKIs self sufficient in terms of kernel modules. Currently, modules from initramfs are not available in the main system and users are forced to keep a second copy (for the matching uname -r) on the rootfs. To make a real world example of the usage, Fedora/CentOS/RHEL distros now ship 'kernel-uki-virt' package with a UKI which is self sufficient in many cases. The package, however, has to have a dependency on 'kernel-modules-core' so basically the same set of modules is available post boot. This is needed when e.g. a filesystem module which was not used for root filesystem need to be loaded to mount some other volume (and ESP is the most common example). The desire is to be able to break this dependency for 'minimal' cloud/VM images but still support the case when people want to install ''kernel-modules-core'/'kernel-modules'/'kernel-modules-extra' post boot and enjoy additional modules.

[cgwalters]

I think it'd be useful beyond just this UKI case; we have cases where people are getting additional kmods fetched dynamically from elsewhere, and having a place to drop them in /run by default would make sense to me.

That said I share the thought that it's a bit weird to have modules in the initramfs that aren't builtin anyways but maybe it does save some potential bug blast radius?

[vittyvk]

While UKI is a good example, I think the idea can also be employed with the traditional kernel/initramfs for specialized immutable setups where all potentially required modules can be shoved into initramfs.
Building modules in certainly has some security implications. E.g. when there's a CVE in a module, we can say that only the deployments which have the corresponding module loaded are affected. It is also possible to denylist a module and prevent it from loading (for non-critical ones, obviously). For built-in modules, you have the code in memory and proving that it cannot be executed is extremely hard. This is significant if you want to have a kernel/initramfs (or a UKI) which are capable of booting at least on all hyperscalers and not in just one specific environment.

[poettering]

That said I share the thought that it's a bit weird to have modules in the initramfs that aren't builtin anyways but maybe it does save some potential bug blast radius?

initrd contents can be flushed from memory. kernel code cannot. hence, if you care about a set of different devices it makes sense to place some kmods in the initrd of the UKI, so that they can be loaded where necessary, but not where they aren't, and can then be flushed out subsequently.

[poettering]

I think generally it makes a ton of sense to teach the various low-level tools on this planet to read from /run/ in addition to /usr/.

The issue I personally dislike about these specific approaches is that we have to copy all the files from initrd to /run for this, which is not precisely free (not because copying memory was slow, but because we do it for a lot of inodes). I wished there was a better way. for example, if we could bind mount the module tree over instead of copying, that would be a lot nicer. But that conflicts with the messy way how initrd→host transition works (which involves basically an rm -rf of the initrd).

hence, if you want to do this, and you have a lot of modules, maybe that should be addressed somehow. For example, what we might be able to do, is to maybe set an xattr on some inode. When "rm -rf" we'd skip inodes marked like that (and their parents) or so. (but not sure whether initrd cpios support xattrs, i think tehy don't).

[vittyvk]

I agree, I wish there was a cleaner solution which doesn't involve copying at all but I was thinking that we can tolerate copying (or moving, actually, as we can do it the last thing initramfs does before the root switch and thus all required modules are already loaded) in the interim. Like RHEL UKI now has 113 kernel modules included. Also, we can only do the dance if there's no /lib/modules/uname -r on the rootfs -- which is already mounted pre-switch so it's cheap to check.

[evelikov]

Paraphrasing few questions from the linked kmod issue:

• IIRC there was some discussion about unmounting the initrd from the host. How would that interact with the proposal, or is that orthogonal?
• Currently kmod allows the user to override the modules directory. Would it make sense to have this opt-in aka kmod_new("/run/modules/...", NULL) as opposed to hard-coding it?
• Usually files in /run take higher precedence than ones in /usr, although the proposal here does the opposite. Is that something we should be concerned about?

[evelikov]

One more that I forgot:

• If we have the extra /run handling in kmod, should we allow to "mix-and-match" and overrides in the usual way - any symlinks to /dev/null overrides the equivalently named file in /usr, files missing from /run are picked from /usr

[vittyvk]

I was hoping someone else will chime in :-)

My take on these great questions:

• Initramfs is already nuked before the root switch today but we have /run which persists. So an easy solution would be to just move modules there (it's tmpfs so it's pretty fast). @poettering has already commented on this point above. A more advanced mechanism preserving some parts (or all) of the initramfs when needed can probably be developed, especially if someone find a use-case for not just modules (we have duplication in binaries and libs too, don't we?). Alternatively, we can try to find a way to mount initramfs from the main system post-boot (like a fuse mount).
• We can certainly have module location as a configuration option. Then, we can set the default to /run/modules but then when kernel-modules-core or something similar is installed we drop a snippet to /lib/depmod.d/ which changes the path to /lib/modules.
• There are multiple reasons why /run should take precedence here. First, if we don't mix then the number of modules in initramfs is supposed to me much lower than what you may need in the main system. You may, for example, want to install some third party modules (Nvidia, virtualbox, ...) for your kernel and these go to /lib. Second, 'developer' use-case. If you are rebuilding and reinstalling your kernel and modules locally, you kid of expect that if you say 'rmmod foo && modprobe foo' that the new version of the module will get loaded.

[esposem]

Hi all, I tried to implement the suggested changes in this PR: kmod-project/kmod#261
Please take a look if you have a minute, thanks!