Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

BPF Recorder: Exclude Container Initialization from Recorded Profile #2623

Draft
wants to merge 3 commits into
base: main
Choose a base branch
from
Draft

BPF Recorder: Exclude Container Initialization from Recorded Profile #2623

wants to merge 3 commits into from
+150 −38

Conversation

mhils
Copy link
Contributor

@mhils mhils commented Dec 13, 2024 •
edited
Loading

What type of PR is this?

/kind feature

What this PR does / why we need it:

This PR makes us 1) detect when containers are (about to be) initialized, and 2) makes sure the permissions exercised before this point are not included in the recorded profile.

Which issue(s) this PR fixes:

Fixes #2576

Does this PR have test?

Covered by existing E2E tests.

Special notes for your reviewer:

99% sure that some kernels won't support sched_prepare_exec, but let's see if it works in the first place before we revise.

Switched from sched_prepare_exec to an execve tracepoint.

Switched from execve to setsid to make sure we capture the last part of runc startup for seccomp.

Switched from execve to getppid in an attempt to make things more reliable.

Switched to an approach where we detect the initial unshare and then start suppressing events until runc reaches getppid. This avoids data races on the Go side.

Does this PR introduce a user-facing change?

The BPF profile recorder now excludes unnecessary permissions exercised during container init.

@k8s-ci-robot k8s-ci-robot added release-note Denotes a PR that will be considered when it comes time to generate release notes. kind/feature Categorizes issue or PR as related to a new feature. cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. labels Dec 13, 2024
@k8s-ci-robot k8s-ci-robot added the needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. label Dec 13, 2024
@k8s-ci-robot
Copy link
Contributor

Hi @mhils. Thanks for your PR.

I'm waiting for a kubernetes-sigs member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

@k8s-ci-robot k8s-ci-robot added the size/M Denotes a PR that changes 30-99 lines, ignoring generated files. label Dec 13, 2024
@codecov-commenter
Copy link

codecov-commenter commented Dec 13, 2024 •
edited
Loading

Codecov Report

Attention: Patch coverage is 0% with 3 lines in your changes missing coverage. Please review.

Project coverage is 41.42%. Comparing base (11d77f4) to head (cc422d7).
Report is 588 commits behind head on main.

Additional details and impacted files 
@@            Coverage Diff             @@
##             main    #2623      +/-   ##
==========================================
- Coverage   45.50%   41.42%   -4.09%     
==========================================
  Files          79      109      +30     
  Lines        7782    18227   +10445     
==========================================
+ Hits         3541     7550    +4009     
- Misses       4099    10177    +6078     
- Partials      142      500     +358

@k8s-ci-robot k8s-ci-robot added size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. and removed size/M Denotes a PR that changes 30-99 lines, ignoring generated files. labels Dec 13, 2024
ccojocar
ccojocar approved these changes Dec 16, 2024
View reviewed changes
internal/pkg/daemon/bpfrecorder/bpf/recorder.bpf.c Outdated Show resolved Hide resolved
internal/pkg/daemon/bpfrecorder/bpfrecorder_apparmor.go Outdated Show resolved Hide resolved
internal/pkg/daemon/bpfrecorder/bpfrecorder_apparmor.go Outdated Show resolved Hide resolved
internal/pkg/daemon/bpfrecorder/bpf/recorder.bpf.c Outdated Show resolved Hide resolved
@k8s-ci-robot k8s-ci-robot added lgtm "Looks good to me", indicates that a PR is ready to be merged. approved Indicates a PR has been approved by an approver from all required OWNERS files. labels Dec 16, 2024
@k8s-ci-robot
Copy link
Contributor

@mhils: The following test failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name Commit Details Required Rerun command
pull-security-profiles-operator-verify ab5a2d3 link true /test pull-security-profiles-operator-verify

Full PR test history. Your PR dashboard. Please help us cut down on flakes by linking to an open issue when you hit one in your PR.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

@k8s-ci-robot k8s-ci-robot added needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. and removed lgtm "Looks good to me", indicates that a PR is ready to be merged. labels Dec 16, 2024
@k8s-ci-robot k8s-ci-robot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Dec 16, 2024
@mhils mhils marked this pull request as draft December 16, 2024 15:00
@k8s-ci-robot k8s-ci-robot added the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Dec 16, 2024
@mhils mhils force-pushed the container-init branch 3 times, most recently from d710ac4 to ed678c9 Compare December 18, 2024 09:07
@k8s-ci-robot k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Dec 18, 2024
@ccojocar
Copy link
Contributor

It is an impressive reduction in permissions in both apapmror and secocmp. 🤞 that this works!

@k8s-ci-robot k8s-ci-robot removed the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Dec 18, 2024
@mhils mhils mentioned this pull request Dec 18, 2024
Merged
@mhils mhils force-pushed the container-init branch 4 times, most recently from fd8559b to ddee8ef Compare December 20, 2024 15:00
@k8s-ci-robot k8s-ci-robot added size/L Denotes a PR that changes 100-499 lines, ignoring generated files. and removed size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. labels Dec 20, 2024
This was referenced Dec 20, 2024
Merged
Merged
@mhils mhils force-pushed the container-init branch 3 times, most recently from 4c10716 to d9e0956 Compare December 20, 2024 17:06
ccojocar
ccojocar approved these changes Dec 23, 2024
View reviewed changes
internal/pkg/daemon/bpfrecorder/bpf/recorder.bpf.c
@@ -51,6 +53,9 @@ char LICENSE[] SEC("license") = "Dual BSD/GPL";
#define unlikely(x) __builtin_expect((x), 0)
#endif

#define trace_hook(...)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: this look like an empty macro, I would remove it if it is not used.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The intent here is that one can quickly toggle between no-op and bpf_printk, which makes debugging BPF much nicer. Does that make sense, any better suggestions?

@k8s-ci-robot k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Dec 23, 2024
@k8s-ci-robot
Copy link
Contributor

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: ccojocar, mhils

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@k8s-ci-robot
Copy link
Contributor

New changes are detected. LGTM label has been removed.

@k8s-ci-robot k8s-ci-robot removed the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Dec 23, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@saschagrunert saschagrunert saschagrunert left review comments

@ccojocar ccojocar ccojocar approved these changes

@Vincent056 Vincent056 Awaiting requested review from Vincent056

Assignees

@ccojocar ccojocar

Labels
approved Indicates a PR has been approved by an approver from all required OWNERS files. cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. kind/feature Categorizes issue or PR as related to a new feature. needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. release-note Denotes a PR that will be considered when it comes time to generate release notes. size/L Denotes a PR that changes 100-499 lines, ignoring generated files.
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Apparmor recorder allows extra capabilities in the profile which doesn't seem to be required
5 participants
@mhils @k8s-ci-robot @codecov-commenter @ccojocar @saschagrunert