devices: drop cilium/ebpf{,link} deps - #64
Conversation
d5f40e9 to
733c596
Compare
…ranch Temporarily point the opencontainers/cgroups dependency at the drop-cilium-ebpf branch (opencontainers/cgroups#64) via a go.mod replace, and re-vendor, so CI can exercise the cilium/ebpf main+link package removal end-to-end in runc. This must not be merged: the replace directive points at a personal fork branch. Once opencontainers/cgroups#64 lands and is tagged, this should be replaced by a normal dependency bump. Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
733c596 to
cf4738d
Compare
Let's drop the second biggest (by size) runc dependency and reduce the binary size by another ~1MB. Draft/DNM until opencontainers/cgroups#64 is merged/released. Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
There was a problem hiding this comment.
Pull request overview
This PR refactors the devices eBPF cgroup-device filter implementation to stop importing github.com/cilium/ebpf’s main and link packages, and instead perform the required operations via direct bpf(2) syscalls while continuing to use github.com/cilium/ebpf/asm for instruction assembly. This aligns with the stated goal of reducing consumer binary size by avoiding heavy transitive dependencies.
Changes:
- Replaced
cilium/ebpfprogram/link usage with thin wrappers aroundBPF_PROG_*commands and raw program fds. - Updated cgroup device filter attach/query logic to operate on fds (with explicit closes) instead of
*ebpf.Program. - Added
nativeEndianselection via build-tagged endian-specific files to satisfyasm.Instructions.Marshalrequirements.
Reviewed changes
Copilot reviewed 3 out of 4 changed files in this pull request and generated 2 comments.
| File | Description |
|---|---|
| go.sum | Removes now-unused transitive dependencies after dropping cilium/ebpf main/link usage. |
| devices/endian_le.go | Provides nativeEndian = binary.LittleEndian under little-endian arch build tags. |
| devices/endian_be.go | Provides nativeEndian = binary.BigEndian under big-endian arch build tags. |
| devices/ebpf_linux.go | Replaces ebpf/link usage with direct bpf(2) syscall wrappers and fd-based program management. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| err = bpfProgAttach(dirFd, progFd, attachFlags, replaceFd) | ||
| if err != nil { | ||
| return fmt.Errorf("failed to call BPF_PROG_ATTACH (BPF_CGROUP_DEVICE, BPF_F_ALLOW_MULTI): %w", err) | ||
| } |
There was a problem hiding this comment.
ell, since this is the only non-test^ call to bpfProgAttach we can just omit the flags (and BPF_CGROUP_DEVICE, too -- it is kind of expected here).
^ by the test call I mean the one in haveBpfProgReplace
cf4738d to
0fd3600
Compare
Let's drop the second biggest (by size) runc dependency and reduce the binary size by another ~1MB. Draft/DNM until opencontainers/cgroups#64 is merged/released. Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
Let's drop the second biggest (by size) runc dependency and reduce the binary size by another ~1MB. Draft/DNM until opencontainers/cgroups#64 is merged/released. Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
0fd3600 to
8decd05
Compare
There was a problem hiding this comment.
@kolyshkin nice, thanks! Did you c&p the definitions? In that case, can you link them so it's simpler to review?
Also, don't we have overlapping functionality in the cgroups package?
Also, I guess no go.mod changes because it is used for the asm part? Nice that the size is decreased anyways :)
Again I'm afraid I fail to understand what you mean here @rata, can you please elaborate or point to whatever you have in mind? |
8decd05 to
ae8789d
Compare
I did not, I just recreated a bare minimum (poor boy) version of functionality that we used from cilium/ebpf and cilium/ebpf/link, basically wiring I have added a separate second commit, linking to the original cilium/ebpf functions. Let me know if you want it or not so I will squash or remove it. |
|
@AkihiroSuda @thaJeztah PTAL (I think I've addressed all of your comments). The second commit is optional and can either be squashed or removed. |
kolyshkin
left a comment
There was a problem hiding this comment.
Looking into kernel's tools/lib/bpf/bpf.c, I see that BPF_PROG_LOAD is retried up to 5 times when EAGAIN is received. Also, cilium/ebpf does that (indefinitely).
Implemented the same.
1bb7901 to
b364e3b
Compare
Let's drop the second biggest (by size) runc dependency and reduce the binary size by another ~1MB. Draft/DNM until opencontainers/cgroups#64 is merged/released. NB: to update this: go mod edit -replace github.com/opencontainers/cgroups=github.com/kolyshkin/oc-cgroups@drop-cilium-ebpf make vendor git add vendor git commit --amend -a Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
b364e3b to
94c4808
Compare
Let's drop the second biggest (by size) runc dependency and reduce the binary size by another ~1MB. Draft/DNM until opencontainers/cgroups#64 is merged/released. NB: to update this: go mod edit -replace github.com/opencontainers/cgroups=github.com/kolyshkin/oc-cgroups@drop-cilium-ebpf make vendor git add vendor git commit --amend -a Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
|
Rebased on top of merged #69; updated opencontainers/runc#5340. |
thaJeztah
left a comment
There was a problem hiding this comment.
Mostly nits and suggestions; ebpf itself is not my forte, but overall looks good.
| maskProfilerSignal() | ||
| defer unmaskProfilerSignal() |
There was a problem hiding this comment.
Perhaps consider making maskProfilerSignal() return the unmask func. This forces anyone using it to consider how to handle it, and makes it more organic to call the unmask after wards (similar to context.WithCancel and similar);
unmask := maskProfilerSignal()
defer unmask()There was a problem hiding this comment.
This is taken from cilium, there are benefits on keeping it as it is there so we can sync. The func is not exporter either.
But no strong opinion here,
There was a problem hiding this comment.
Ahm right; yeah, that makes sense. It could be something we could try if it's accepted in upstream.
I generally don't like functions that come with a "don't forget to close the door using <some other function> when you leave"; it's easy to not read the docs, and especially in cases where the consequences may not always be immediately apparent.
There was a problem hiding this comment.
fwiw I agree with @thaJeztah but am similarly not so strict about it
There was a problem hiding this comment.
Yeah, we can still do these kind of changes in a follow-up; it's all non-exported, so no breaking changes involved.
| // very strict SELinux policies). | ||
| if errors.Is(err, os.ErrPermission) { | ||
| logrus.Debugf("ignoring existing CGROUP_DEVICE program (prog_id=%v) which cannot be accessed by runc -- likely due to LSM policy: %v", progId, err) | ||
| logrus.Debugf("ignoring existing CGROUP_DEVICE program (prog_id=%v) which cannot be accessed by runc -- likely due to LSM policy: %v", progID, err) |
There was a problem hiding this comment.
Unrelated; we could consider using more structured logs for these (probably also remove "by runc");
| logrus.Debugf("ignoring existing CGROUP_DEVICE program (prog_id=%v) which cannot be accessed by runc -- likely due to LSM policy: %v", progID, err) | |
| logrus.WithFields(logrus.Fields{ | |
| "error": err, | |
| "progID": progID, | |
| }).Debug("ignoring existing CGROUP_DEVICE program which cannot be accessed -- likely due to LSM policy") |
| }, "MIT") | ||
| if err != nil { | ||
| logrus.Warnf("checking for BPF_F_REPLACE support: ebpf.NewProgram failed: %v", err) | ||
| logrus.Warnf("checking for BPF_F_REPLACE support: bpfProgLoad failed: %v", err) |
There was a problem hiding this comment.
Same here;
| logrus.Warnf("checking for BPF_F_REPLACE support: bpfProgLoad failed: %v", err) | |
| logrus.WithError(err).Warn("checking for BPF_F_REPLACE support: bpfProgLoad failed") |
There was a problem hiding this comment.
I'd rather address this separately as this is already a big change.
| progFd, err := bpfProgLoad(asm.Instructions{ | ||
| asm.Mov.Imm(asm.R0, 0), | ||
| asm.Return(), | ||
| }, "MIT") |
There was a problem hiding this comment.
Could make license the first argument to make formatting slightly neater, or remove it altogether as argument (if it's only internal and always using MIT);
| progFd, err := bpfProgLoad(asm.Instructions{ | |
| asm.Mov.Imm(asm.R0, 0), | |
| asm.Return(), | |
| }, "MIT") | |
| progFd, err := bpfProgLoad("MIT", asm.Instructions{ | |
| asm.Mov.Imm(asm.R0, 0), | |
| asm.Return(), | |
| }) |
There was a problem hiding this comment.
I'd rather keep this as is because the signature matches whatever the deviceFilter returns. If we are to drop this from deviceFilter, the change is too much (and this PR is already non-trivial).
Here's the draft diff BTW:
diff --git a/devices/devicefilter.go b/devices/devicefilter.go
index aafa0d0..7b505fe 100644
--- a/devices/devicefilter.go
+++ b/devices/devicefilter.go
@@ -17,13 +17,8 @@ import (
"golang.org/x/sys/unix"
)
-const (
- // license string format is same as kernel MODULE_LICENSE macro
- license = "Apache"
-)
-
-// deviceFilter returns eBPF device filter program and its license string.
-func deviceFilter(rules []*devices.Rule) (asm.Instructions, string, error) {
+// deviceFilter returns eBPF device filter program.
+func deviceFilter(rules []*devices.Rule) (asm.Instructions, error) {
// Generate the minimum ruleset for the device rules we are given. While we
// don't care about minimum transitions in cgroupv2, using the emulator
// gives us a guarantee that the behaviour of devices filtering is the same
@@ -32,12 +27,12 @@ func deviceFilter(rules []*devices.Rule) (asm.Instructions, string, error) {
emu := new(emulator)
for _, rule := range rules {
if err := emu.Apply(*rule); err != nil {
- return nil, "", err
+ return nil, err
}
}
cleanRules, err := emu.Rules()
if err != nil {
- return nil, "", err
+ return nil, err
}
p := &program{
@@ -51,20 +46,20 @@ func deviceFilter(rules []*devices.Rule) (asm.Instructions, string, error) {
// only be one (at most) at the very start to instruct cgroupv1 to
// go into allow-list mode. However we do double-check this here.
if idx != 0 || rule.Allow != emu.IsBlacklist() {
- return nil, "", fmt.Errorf("[internal error] emulated cgroupv2 devices ruleset had bad wildcard at idx %v (%s)", idx, rule.CgroupString())
+ return nil, fmt.Errorf("[internal error] emulated cgroupv2 devices ruleset had bad wildcard at idx %v (%s)", idx, rule.CgroupString())
}
continue
}
if rule.Allow == p.defaultAllow {
// There should be no rules which have an action equal to the
// default action, the emulator removes those.
- return nil, "", fmt.Errorf("[internal error] emulated cgroupv2 devices ruleset had no-op rule at idx %v (%s)", idx, rule.CgroupString())
+ return nil, fmt.Errorf("[internal error] emulated cgroupv2 devices ruleset had no-op rule at idx %v (%s)", idx, rule.CgroupString())
}
if err := p.appendRule(rule); err != nil {
- return nil, "", err
+ return nil, err
}
}
- return p.finalize(), license, nil
+ return p.finalize(), nil
}
type program struct {
diff --git a/devices/devicefilter_test.go b/devices/devicefilter_test.go
index 6df6af2..6a5f93d 100644
--- a/devices/devicefilter_test.go
+++ b/devices/devicefilter_test.go
@@ -20,7 +20,7 @@ func hash(s, comm string) string {
}
func testDeviceFilter(t testing.TB, devices []*devices.Rule, expectedStr string) {
- insts, _, err := deviceFilter(devices)
+ insts, err := deviceFilter(devices)
if err != nil {
t.Fatalf("%s: %v (devices: %+v)", t.Name(), err, devices)
}
diff --git a/devices/ebpf_linux.go b/devices/ebpf_linux.go
index 96e1379..3e312d7 100644
--- a/devices/ebpf_linux.go
+++ b/devices/ebpf_linux.go
@@ -65,17 +65,18 @@ func bpfFD(cmd uintptr, attr unsafe.Pointer, size uintptr) (int, error) {
//
// It is roughly equivalent to [github.com/cilium/ebpf/internal/sys.ProgLoad],
// and the "retry with verifier log" is taken from [github.com/cilium/ebpf.NewProgram].
-func bpfProgLoad(insns asm.Instructions, license string) (int, error) {
+func bpfProgLoad(insns asm.Instructions) (int, error) {
buf := bytes.NewBuffer(make([]byte, 0, insns.Size()))
if err := insns.Marshal(buf, nativeEndian); err != nil {
return -1, err
}
insnsBytes := buf.Bytes()
- licensePtr, err := unix.BytePtrFromString(license)
- if err != nil {
- return -1, err
- }
+ // License string, in the same format as the kernel MODULE_LICENSE macro.
+ const license = "MIT\000"
// Subset of struct bpf_attr for BPF_PROG_LOAD. Fields past the ones we set
// are left zero; the kernel zero-fills any part of bpf_attr beyond the size
@@ -92,14 +93,14 @@ func bpfProgLoad(insns asm.Instructions, license string) (int, error) {
progType: unix.BPF_PROG_TYPE_CGROUP_DEVICE,
insnCnt: uint32(len(insnsBytes) / asm.InstructionSize),
insns: uint64(uintptr(unsafe.Pointer(&insnsBytes[0]))),
- license: uint64(uintptr(unsafe.Pointer(licensePtr))),
+ license: uint64(uintptr(unsafe.Pointer(unsafe.StringData(license)))),
}
fd, err := bpfFD(unix.BPF_PROG_LOAD, unsafe.Pointer(&attr), unsafe.Sizeof(attr))
// attr holds the pointers as integers, so the GC can't see them; keep the
- // referenced objects alive until the syscall returns.
+ // referenced objects alive until the syscall returns. The license string is
+ // a constant, so it needs no such treatment.
runtime.KeepAlive(insnsBytes)
- runtime.KeepAlive(licensePtr)
if err == nil {
return fd, nil
}
@@ -120,7 +121,6 @@ func bpfProgLoad(insns asm.Instructions, license string) (int, error) {
fd, err = bpfFD(unix.BPF_PROG_LOAD, unsafe.Pointer(&attr), unsafe.Sizeof(attr))
runtime.KeepAlive(insnsBytes)
- runtime.KeepAlive(licensePtr)
runtime.KeepAlive(log)
if err == nil { // Totally unexpected.
logrus.Warnf("BPF_PROG_LOAD retry unexpectedly succeeded after failing with %v earlier", origErr)
@@ -290,7 +290,7 @@ func haveBpfProgReplace() bool {
progFd, err := bpfProgLoad(asm.Instructions{
asm.Mov.Imm(asm.R0, 0),
asm.Return(),
- }, "MIT")
+ })
if err != nil {
logrus.Warnf("checking for BPF_F_REPLACE support: bpfProgLoad failed: %v", err)
return
@@ -334,7 +334,7 @@ func haveBpfProgReplace() bool {
// Requires the system to be running in cgroup2 unified-mode with kernel >= 4.15 .
//
// https://github.com/torvalds/linux/commit/ebc614f687369f9df99828572b1d85a7c2de3d92
-func loadAttachCgroupDeviceFilter(insts asm.Instructions, license string, dirFd int) error {
+func loadAttachCgroupDeviceFilter(insts asm.Instructions, dirFd int) error {
// Increase `ulimit -l` limit to avoid BPF_PROG_LOAD error (#2167).
// This limit is not inherited into the container.
memlockLimit := &unix.Rlimit{
@@ -357,7 +357,7 @@ func loadAttachCgroupDeviceFilter(insts asm.Instructions, license string, dirFd
useReplaceProg := haveBpfProgReplace() && len(oldFds) == 1
// Generate new program.
- progFd, err := bpfProgLoad(insts, license)
+ progFd, err := bpfProgLoad(insts)
if err != nil {
return fmt.Errorf("failed to call BPF_PROG_LOAD: %w", err)
}
diff --git a/devices/v2.go b/devices/v2.go
index 508f3dd..6da257a 100644
--- a/devices/v2.go
+++ b/devices/v2.go
@@ -55,7 +55,7 @@ func setV2(dirPath string, r *cgroups.Resources) error {
if r.SkipDevices {
return nil
}
- insts, license, err := deviceFilter(r.Devices)
+ insts, err := deviceFilter(r.Devices)
if err != nil {
return err
}
@@ -64,7 +64,7 @@ func setV2(dirPath string, r *cgroups.Resources) error {
return fmt.Errorf("cannot get dir FD for %s", dirPath)
}
defer unix.Close(dirFD)
- if err := loadAttachCgroupDeviceFilter(insts, license, dirFD); err != nil {
+ if err := loadAttachCgroupDeviceFilter(insts, dirFD); err != nil {
if !canSkipEBPFError(r) {
return err
}| err = bpfProgDetach(dirFd, oldFd) | ||
| if err != nil { |
There was a problem hiding this comment.
Maybe this was done to please the "shadow" linter, but it's good to clearly scope the var (to avoid ambiguity)
| err = bpfProgDetach(dirFd, oldFd) | |
| if err != nil { | |
| if err := bpfProgDetach(dirFd, oldFd); err != nil { |
Also wondering if for these we should always fail early, or use a multi error;
var errs []error
...
errs = append(errs, err)
if err := errors.Join(errs...); err != nil {
return fmt.Errorf("failed to call BPF_PROG_DETACH (BPF_CGROUP_DEVICE) on old filter program: %w", err)
}There was a problem hiding this comment.
Failing early is better I guess -- the current state is broken in any way, and trying to remove other old progs makes no sense.
rata
left a comment
There was a problem hiding this comment.
I'm no maintainer here, but LGTM! :)
The comments @thaJeztah seem nice, but this LGTM already and with those changes it still LGTM :)
|
@opencontainers/cgroups-maintainers @cyphar PTAL |
94c4808 to
501a5cb
Compare
|
Would really love to have this released in time for runc v1.6.0-rc.1 (ETA 31 August); PTAL @opencontainers/cgroups-maintainers |
|
FYI packages that depend on this one (e.g. https://github.com/apptainer/apptainer) today got a low severity dependabot alert asking to upgrade to github.com/cilium/ebpf version 0.22.0, but that's failing because type |
Replace the use of the cilium/ebpf and cilium/ebpf/link with direct bpf(2) syscalls. Keep cilium/ebpf/asm for instruction assembly. Notes: - the eBPF device-filter programs are now tracked by raw file descriptors instead of *ebpf.Program handles; - asm.Instructions.Marshal requires a concrete binary.LittleEndian or binary.BigEndian, so endian.go detects the native byte order at runtime as a workaround. This could be done during compile time but requires maintaining a list of all GOARCHes; - the "removing old filter %d from cgroup" log messages are removed: this always happens when using systemd and the messages are not useful, plus obtaining the details would add more code; This reduces the runc binary size by about ~1M. Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
501a5cb to
713912d
Compare
|
Ran claude's ultrareview on this, found one nit:
I had this deliberately written this way (pinned object is small, retries are rare, and there should not be more than one, so one defer for simplicity), but decided to implement review's comment. Rebased, pushed, PTAL @opencontaienrs/runc-maintainers (note we have one LGTM from @rata (who is currently not a maintainer). |
|
@opencontainers/cgroups-maintainers as a side effect, this PR also solves a compatibility issue with cilium/ebpf@v0.22 (by removing its use). |
thaJeztah
left a comment
There was a problem hiding this comment.
LGTM! Sorry thought I already did 😂
Can we fix that? I think the intent of this repository was mostly "mechanical" (extract it from runc), and still maintained by the runc maintainers; Idealy maintainers here would be "runc maintainers" + any "cgroup-only" maintainers. So I'm tempted to make @opencontainers/cgroups-maintainers "extend" @opencontainers/runc-maintainers I think that can be done by making |
|
3 reviews; one "non-binding", but works for me; let's bring this one in. @kolyshkin did you want to tag a new version with this already, or do some of the follow-ups first? (either way works for me; incremental releases are fine) |
This currently contains #69; will rebase once that one is merged.Replace the use of the cilium/ebpf and cilium/ebpf/link with direct
bpf(2) syscalls. Keep cilium/ebpf/asm for instruction assembly.
Notes:
descriptors instead of *ebpf.Program handles;
binary.BigEndian, thus endian_{le,be}.go are introduced as a
workaround.
This reduces the runc binary size by about ~1M.
NOTE that this is probably limited to runc, because for other users (k8s, cri-o) this was already solved by opencontainers/runc#4248.
Being tested in opencontainers/runc#5340.
For initial discussion about this, see opencontainers/runc#5218.