Hi all,
I believe 5.10.168 and 5.15.93 introduced a regression that impacts the Cilium project. Some information on the nature of the regression is available at https://github.com/cilium/cilium/issues/25500. The primary symptom seems to be the error `BPF program is too large.`
My colleague has found that reverting the following two commits:
8de8c4a "bpf: Support <8-byte scalar spill and refill" 9ff2beb "bpf: Fix incorrect state pruning for <8B spill/fill"
resolves the regression.
If we revert these in the stable tree, there may be a few changes that depend on those that also need to be reverted, but I'm not sure yet.
Would it make sense to revert these changes (and any dependent ones) in the 5.10 and 5.15 trees? If anyone has other ideas, I can help test possible solutions.
Thanks, -Robert
On Wed, Jun 14, 2023 at 11:23:52AM -0700, Robert Kolchmeyer wrote:
Hi all,
I believe 5.10.168 and 5.15.93 introduced a regression that impacts the Cilium project. Some information on the nature of the regression is available at https://github.com/cilium/cilium/issues/25500. The primary symptom seems to be the error `BPF program is too large.`
My colleague has found that reverting the following two commits:
8de8c4a "bpf: Support <8-byte scalar spill and refill" 9ff2beb "bpf: Fix incorrect state pruning for <8B spill/fill"
resolves the regression.
If we revert these in the stable tree, there may be a few changes that depend on those that also need to be reverted, but I'm not sure yet.
Would it make sense to revert these changes (and any dependent ones) in the 5.10 and 5.15 trees? If anyone has other ideas, I can help test possible solutions.
Can you actually test if those reverts work properly for you and if there are other dependencies involved?
And is this issue also in 6.1.y and Linus's tree? If not, why not, are we just missing a commit? We can't revert something from a stable release if you are going to hit the same issue when moving to a new release, right?
thanks,
greg k-h
On Wed, Jun 14, 2023 at 1:57 PM Greg KH gregkh@linuxfoundation.org wrote:
On Wed, Jun 14, 2023 at 11:23:52AM -0700, Robert Kolchmeyer wrote:
Hi all,
I believe 5.10.168 and 5.15.93 introduced a regression that impacts the Cilium project. Some information on the nature of the regression is available at https://github.com/cilium/cilium/issues/25500. The primary symptom seems to be the error `BPF program is too large.`
My colleague has found that reverting the following two commits:
8de8c4a "bpf: Support <8-byte scalar spill and refill" 9ff2beb "bpf: Fix incorrect state pruning for <8B spill/fill"
resolves the regression.
If we revert these in the stable tree, there may be a few changes that depend on those that also need to be reverted, but I'm not sure yet.
Would it make sense to revert these changes (and any dependent ones) in the 5.10 and 5.15 trees? If anyone has other ideas, I can help test possible solutions.
Can you actually test if those reverts work properly for you and if there are other dependencies involved?
And is this issue also in 6.1.y and Linus's tree? If not, why not, are we just missing a commit? We can't revert something from a stable release if you are going to hit the same issue when moving to a new release, right?
thanks,
greg k-h
Before jumping to reverts.. how is it fixed in 6.0+ kernels?
"BPF program is too large" can probably be worked around on the cilium side. The kernel cannot guarantee that a particular program will always be verifiable. We find safety bugs in the verifier and often enough the fixes to such issues make the verifier work harder to prove the safety of the program. This is one of such cases. These two commits are necessary. Reverting them will prevent loading of valid programs. So reverts is a dangerous path. The best is to identify the other patches from 6.0+ and backport them. The second best path is to bump 1M limit to something higher to mitigate "more work by the verifier".
On Wed, Jun 14, 2023 at 2:48 PM Alexei Starovoitov alexei.starovoitov@gmail.com wrote:
On Wed, Jun 14, 2023 at 1:57 PM Greg KH gregkh@linuxfoundation.org wrote:
On Wed, Jun 14, 2023 at 11:23:52AM -0700, Robert Kolchmeyer wrote:
Hi all,
I believe 5.10.168 and 5.15.93 introduced a regression that impacts the Cilium project. Some information on the nature of the regression is available at https://github.com/cilium/cilium/issues/25500. The primary symptom seems to be the error `BPF program is too large.`
My colleague has found that reverting the following two commits:
8de8c4a "bpf: Support <8-byte scalar spill and refill" 9ff2beb "bpf: Fix incorrect state pruning for <8B spill/fill"
resolves the regression.
If we revert these in the stable tree, there may be a few changes that depend on those that also need to be reverted, but I'm not sure yet.
Would it make sense to revert these changes (and any dependent ones) in the 5.10 and 5.15 trees? If anyone has other ideas, I can help test possible solutions.
Can you actually test if those reverts work properly for you and if there are other dependencies involved?
And is this issue also in 6.1.y and Linus's tree? If not, why not, are we just missing a commit? We can't revert something from a stable release if you are going to hit the same issue when moving to a new release, right?
thanks,
greg k-h
Before jumping to reverts.. how is it fixed in 6.0+ kernels?
"BPF program is too large" can probably be worked around on the cilium side. The kernel cannot guarantee that a particular program will always be verifiable. We find safety bugs in the verifier and often enough the fixes to such issues make the verifier work harder to prove the safety of the program. This is one of such cases. These two commits are necessary. Reverting them will prevent loading of valid programs. So reverts is a dangerous path. The best is to identify the other patches from 6.0+ and backport them. The second best path is to bump 1M limit to something higher to mitigate "more work by the verifier".
Thanks all for the pointers, this is super helpful.
I'm working on getting more details, and I'll post here when I have them.
-Robert
linux-stable-mirror@lists.linaro.org
-
Alexei Starovoitov -
Greg KH -
Robert Kolchmeyer