Hi,
(prefix, I was not aware of the regression reporting process and incorrectly reported this informally with the developers mentioned in the change)
I upgraded from 6.1.38 to 6.1.55 this morning and it broke my traffic shaping script, leaving me with a non-functional uplink on a remote router.
The script errors out like this:
Oct 06 05:49:22 wendy00 isp-setup-shaping-start[2053]: + ext=ispA Oct 06 05:49:22 wendy00 isp-setup-shaping-start[2053]: + ext_ingress=ifb0 Oct 06 05:49:22 wendy00 isp-setup-shaping-start[2053]: + modprobe ifb Oct 06 05:49:22 wendy00 isp-setup-shaping-start[2053]: + modprobe act_mirred Oct 06 05:49:22 wendy00 isp-setup-shaping-start[2053]: + tc qdisc del dev ispA root Oct 06 05:49:22 wendy00 isp-setup-shaping-start[2061]: Error: Cannot delete qdisc with handle of zero. Oct 06 05:49:22 wendy00 isp-setup-shaping-start[2053]: + true Oct 06 05:49:22 wendy00 isp-setup-shaping-start[2053]: + tc qdisc del dev ispA ingress Oct 06 05:49:22 wendy00 isp-setup-shaping-start[2064]: Error: Cannot find specified qdisc on specified device. Oct 06 05:49:22 wendy00 isp-setup-shaping-start[2053]: + true Oct 06 05:49:22 wendy00 isp-setup-shaping-start[2053]: + tc qdisc del dev ifb0 root Oct 06 05:49:22 wendy00 isp-setup-shaping-start[2066]: Error: Cannot delete qdisc with handle of zero. Oct 06 05:49:22 wendy00 isp-setup-shaping-start[2053]: + true Oct 06 05:49:22 wendy00 isp-setup-shaping-start[2053]: + tc qdisc del dev ifb0 ingress Oct 06 05:49:22 wendy00 isp-setup-shaping-start[2067]: Error: Cannot find specified qdisc on specified device. Oct 06 05:49:22 wendy00 isp-setup-shaping-start[2053]: + true Oct 06 05:49:22 wendy00 isp-setup-shaping-start[2053]: + tc qdisc add dev ispA handle ffff: ingress Oct 06 05:49:22 wendy00 isp-setup-shaping-start[2053]: + ifconfig ifb0 up Oct 06 05:49:22 wendy00 isp-setup-shaping-start[2053]: + tc filter add dev ispA parent ffff: protocol all u32 match u32 0 0 action mirred egress redirect dev ifb0 Oct 06 05:49:22 wendy00 isp-setup-shaping-start[2053]: + tc qdisc add dev ifb0 root handle 1: hfsc default 1 Oct 06 05:49:22 wendy00 isp-setup-shaping-start[2053]: + tc class add dev ifb0 parent 1: classid 1:999 hfsc rt m2 2.5gbit Oct 06 05:49:22 wendy00 isp-setup-shaping-start[2053]: + tc class add dev ifb0 parent 1:999 classid 1:1 hfsc sc rate 50mbit Oct 06 05:49:22 wendy00 isp-setup-shaping-start[2077]: Error: Invalid parent - parent class must have FSC.
The error message is also a bit weird (but that’s likely due to iproute2 being weird) as the CLI interface for `tc` and the error message do not map well. (I think I would have to choose `hfsc sc` on the parent to enable the FSC option which isn’t mentioned anywhere in the hfsc manpage).
The breaking change was introduced in 6.1.53[1] and a multitude of other currently supported kernels:
---- commit a1e820fc7808e42b990d224f40e9b4895503ac40 Author: Budimir Markovic markovicbudimir@gmail.com Date: Thu Aug 24 01:49:05 2023 -0700
net/sched: sch_hfsc: Ensure inner classes have fsc curve
[ Upstream commit b3d26c5702c7d6c45456326e56d2ccf3f103e60f ]
HFSC assumes that inner classes have an fsc curve, but it is currently possible for classes without an fsc curve to become parents. This leads to bugs including a use-after-free.
Don't allow non-root classes without HFSC_FSC to become parents.
Fixes: 1da177e4c3f4 ("Linux-2.6.12-rc2") Reported-by: Budimir Markovic markovicbudimir@gmail.com Signed-off-by: Budimir Markovic markovicbudimir@gmail.com Acked-by: Jamal Hadi Salim jhs@mojatatu.com Link: https://lore.kernel.org/r/20230824084905.422-1-markovicbudimir@gmail.com Signed-off-by: Jakub Kicinski kuba@kernel.org Signed-off-by: Sasha Levin sashal@kernel.org ----
Regards, Christian
[1] https://cdn.kernel.org/pub/linux/kernel/v6.x/ChangeLog-6.1.53
#regzbot introduced: a1e820fc7808e42b990d224f40e9b4895503ac40
On 06.10.23 10:37, Christian Theune wrote:
(prefix, I was not aware of the regression reporting process and incorrectly reported this informally with the developers mentioned in the change)
Don't worry too much about that, but thx for taking care of all the details. FWIW, there is one more thing that would be good to know:
Does the problem happen with mainline (e.g. 6.6-rc4) as well? That's relevant, as different people might care[1].
Ciao, Thorsten
[1] this among others is explained here: https://linux-regtracking.leemhuis.info/post/frequent-reasons-why-linux-kern...
I upgraded from 6.1.38 to 6.1.55 this morning and it broke my traffic shaping script, leaving me with a non-functional uplink on a remote router.
The script errors out like this:
Oct 06 05:49:22 wendy00 isp-setup-shaping-start[2053]: + ext=ispA Oct 06 05:49:22 wendy00 isp-setup-shaping-start[2053]: + ext_ingress=ifb0 Oct 06 05:49:22 wendy00 isp-setup-shaping-start[2053]: + modprobe ifb Oct 06 05:49:22 wendy00 isp-setup-shaping-start[2053]: + modprobe act_mirred Oct 06 05:49:22 wendy00 isp-setup-shaping-start[2053]: + tc qdisc del dev ispA root Oct 06 05:49:22 wendy00 isp-setup-shaping-start[2061]: Error: Cannot delete qdisc with handle of zero. Oct 06 05:49:22 wendy00 isp-setup-shaping-start[2053]: + true Oct 06 05:49:22 wendy00 isp-setup-shaping-start[2053]: + tc qdisc del dev ispA ingress Oct 06 05:49:22 wendy00 isp-setup-shaping-start[2064]: Error: Cannot find specified qdisc on specified device. Oct 06 05:49:22 wendy00 isp-setup-shaping-start[2053]: + true Oct 06 05:49:22 wendy00 isp-setup-shaping-start[2053]: + tc qdisc del dev ifb0 root Oct 06 05:49:22 wendy00 isp-setup-shaping-start[2066]: Error: Cannot delete qdisc with handle of zero. Oct 06 05:49:22 wendy00 isp-setup-shaping-start[2053]: + true Oct 06 05:49:22 wendy00 isp-setup-shaping-start[2053]: + tc qdisc del dev ifb0 ingress Oct 06 05:49:22 wendy00 isp-setup-shaping-start[2067]: Error: Cannot find specified qdisc on specified device. Oct 06 05:49:22 wendy00 isp-setup-shaping-start[2053]: + true Oct 06 05:49:22 wendy00 isp-setup-shaping-start[2053]: + tc qdisc add dev ispA handle ffff: ingress Oct 06 05:49:22 wendy00 isp-setup-shaping-start[2053]: + ifconfig ifb0 up Oct 06 05:49:22 wendy00 isp-setup-shaping-start[2053]: + tc filter add dev ispA parent ffff: protocol all u32 match u32 0 0 action mirred egress redirect dev ifb0 Oct 06 05:49:22 wendy00 isp-setup-shaping-start[2053]: + tc qdisc add dev ifb0 root handle 1: hfsc default 1 Oct 06 05:49:22 wendy00 isp-setup-shaping-start[2053]: + tc class add dev ifb0 parent 1: classid 1:999 hfsc rt m2 2.5gbit Oct 06 05:49:22 wendy00 isp-setup-shaping-start[2053]: + tc class add dev ifb0 parent 1:999 classid 1:1 hfsc sc rate 50mbit Oct 06 05:49:22 wendy00 isp-setup-shaping-start[2077]: Error: Invalid parent - parent class must have FSC.
The error message is also a bit weird (but that’s likely due to iproute2 being weird) as the CLI interface for `tc` and the error message do not map well. (I think I would have to choose `hfsc sc` on the parent to enable the FSC option which isn’t mentioned anywhere in the hfsc manpage).
The breaking change was introduced in 6.1.53[1] and a multitude of other currently supported kernels:
commit a1e820fc7808e42b990d224f40e9b4895503ac40 Author: Budimir Markovic markovicbudimir@gmail.com Date: Thu Aug 24 01:49:05 2023 -0700
net/sched: sch_hfsc: Ensure inner classes have fsc curve
[ Upstream commit b3d26c5702c7d6c45456326e56d2ccf3f103e60f ]
HFSC assumes that inner classes have an fsc curve, but it is currently possible for classes without an fsc curve to become parents. This leads to bugs including a use-after-free.
Don't allow non-root classes without HFSC_FSC to become parents.
Fixes: 1da177e4c3f4 ("Linux-2.6.12-rc2") Reported-by: Budimir Markovic markovicbudimir@gmail.com Signed-off-by: Budimir Markovic markovicbudimir@gmail.com Acked-by: Jamal Hadi Salim jhs@mojatatu.com Link: https://lore.kernel.org/r/20230824084905.422-1-markovicbudimir@gmail.com Signed-off-by: Jakub Kicinski kuba@kernel.org Signed-off-by: Sasha Levin sashal@kernel.org
Regards, Christian
[1] https://cdn.kernel.org/pub/linux/kernel/v6.x/ChangeLog-6.1.53
#regzbot introduced: a1e820fc7808e42b990d224f40e9b4895503ac40
Hi,
it seems that 6.6rc4 is affected as well:
---- commit b3d26c5702c7d6c45456326e56d2ccf3f103e60f Author: Budimir Markovic markovicbudimir@gmail.com Date: Thu Aug 24 01:49:05 2023 -0700
net/sched: sch_hfsc: Ensure inner classes have fsc curve ——
I have not found newer commits that would suggest they change any behaviour around this in any way, but I might be wrong.
Christian
On 6. Oct 2023, at 11:01, Linux regression tracking (Thorsten Leemhuis) regressions@leemhuis.info wrote:
On 06.10.23 10:37, Christian Theune wrote:
(prefix, I was not aware of the regression reporting process and incorrectly reported this informally with the developers mentioned in the change)
Don't worry too much about that, but thx for taking care of all the details. FWIW, there is one more thing that would be good to know:
Does the problem happen with mainline (e.g. 6.6-rc4) as well? That's relevant, as different people might care[1].
Ciao, Thorsten
[1] this among others is explained here: https://linux-regtracking.leemhuis.info/post/frequent-reasons-why-linux-kern...
I upgraded from 6.1.38 to 6.1.55 this morning and it broke my traffic shaping script, leaving me with a non-functional uplink on a remote router.
The script errors out like this:
Oct 06 05:49:22 wendy00 isp-setup-shaping-start[2053]: + ext=ispA Oct 06 05:49:22 wendy00 isp-setup-shaping-start[2053]: + ext_ingress=ifb0 Oct 06 05:49:22 wendy00 isp-setup-shaping-start[2053]: + modprobe ifb Oct 06 05:49:22 wendy00 isp-setup-shaping-start[2053]: + modprobe act_mirred Oct 06 05:49:22 wendy00 isp-setup-shaping-start[2053]: + tc qdisc del dev ispA root Oct 06 05:49:22 wendy00 isp-setup-shaping-start[2061]: Error: Cannot delete qdisc with handle of zero. Oct 06 05:49:22 wendy00 isp-setup-shaping-start[2053]: + true Oct 06 05:49:22 wendy00 isp-setup-shaping-start[2053]: + tc qdisc del dev ispA ingress Oct 06 05:49:22 wendy00 isp-setup-shaping-start[2064]: Error: Cannot find specified qdisc on specified device. Oct 06 05:49:22 wendy00 isp-setup-shaping-start[2053]: + true Oct 06 05:49:22 wendy00 isp-setup-shaping-start[2053]: + tc qdisc del dev ifb0 root Oct 06 05:49:22 wendy00 isp-setup-shaping-start[2066]: Error: Cannot delete qdisc with handle of zero. Oct 06 05:49:22 wendy00 isp-setup-shaping-start[2053]: + true Oct 06 05:49:22 wendy00 isp-setup-shaping-start[2053]: + tc qdisc del dev ifb0 ingress Oct 06 05:49:22 wendy00 isp-setup-shaping-start[2067]: Error: Cannot find specified qdisc on specified device. Oct 06 05:49:22 wendy00 isp-setup-shaping-start[2053]: + true Oct 06 05:49:22 wendy00 isp-setup-shaping-start[2053]: + tc qdisc add dev ispA handle ffff: ingress Oct 06 05:49:22 wendy00 isp-setup-shaping-start[2053]: + ifconfig ifb0 up Oct 06 05:49:22 wendy00 isp-setup-shaping-start[2053]: + tc filter add dev ispA parent ffff: protocol all u32 match u32 0 0 action mirred egress redirect dev ifb0 Oct 06 05:49:22 wendy00 isp-setup-shaping-start[2053]: + tc qdisc add dev ifb0 root handle 1: hfsc default 1 Oct 06 05:49:22 wendy00 isp-setup-shaping-start[2053]: + tc class add dev ifb0 parent 1: classid 1:999 hfsc rt m2 2.5gbit Oct 06 05:49:22 wendy00 isp-setup-shaping-start[2053]: + tc class add dev ifb0 parent 1:999 classid 1:1 hfsc sc rate 50mbit Oct 06 05:49:22 wendy00 isp-setup-shaping-start[2077]: Error: Invalid parent - parent class must have FSC.
The error message is also a bit weird (but that’s likely due to iproute2 being weird) as the CLI interface for `tc` and the error message do not map well. (I think I would have to choose `hfsc sc` on the parent to enable the FSC option which isn’t mentioned anywhere in the hfsc manpage).
The breaking change was introduced in 6.1.53[1] and a multitude of other currently supported kernels:
commit a1e820fc7808e42b990d224f40e9b4895503ac40 Author: Budimir Markovic markovicbudimir@gmail.com Date: Thu Aug 24 01:49:05 2023 -0700
net/sched: sch_hfsc: Ensure inner classes have fsc curve
[ Upstream commit b3d26c5702c7d6c45456326e56d2ccf3f103e60f ]
HFSC assumes that inner classes have an fsc curve, but it is currently possible for classes without an fsc curve to become parents. This leads to bugs including a use-after-free.
Don't allow non-root classes without HFSC_FSC to become parents.
Fixes: 1da177e4c3f4 ("Linux-2.6.12-rc2") Reported-by: Budimir Markovic markovicbudimir@gmail.com Signed-off-by: Budimir Markovic markovicbudimir@gmail.com Acked-by: Jamal Hadi Salim jhs@mojatatu.com Link: https://lore.kernel.org/r/20230824084905.422-1-markovicbudimir@gmail.com Signed-off-by: Jakub Kicinski kuba@kernel.org Signed-off-by: Sasha Levin sashal@kernel.org
Regards, Christian
[1] https://cdn.kernel.org/pub/linux/kernel/v6.x/ChangeLog-6.1.53
#regzbot introduced: a1e820fc7808e42b990d224f40e9b4895503ac40
Liebe Grüße, Christian Theune
On 06.10.23 11:07, Christian Theune wrote:
it seems that 6.6rc4 is affected as well:
commit b3d26c5702c7d6c45456326e56d2ccf3f103e60f Author: Budimir Markovic markovicbudimir@gmail.com Date: Thu Aug 24 01:49:05 2023 -0700
net/sched: sch_hfsc: Ensure inner classes have fsc curve——
Did you actually try if the problem occurs with 6.6-rc4? That the commit is in there is expected (everything that lands in stable trees has to go to mainline first).
Ciao, Thorsten
I have not found newer commits that would suggest they change any behaviour around this in any way, but I might be wrong.
Christian
On 6. Oct 2023, at 11:01, Linux regression tracking (Thorsten Leemhuis) regressions@leemhuis.info wrote:
On 06.10.23 10:37, Christian Theune wrote:
(prefix, I was not aware of the regression reporting process and incorrectly reported this informally with the developers mentioned in the change)
Don't worry too much about that, but thx for taking care of all the details. FWIW, there is one more thing that would be good to know:
Does the problem happen with mainline (e.g. 6.6-rc4) as well? That's relevant, as different people might care[1].
Ciao, Thorsten
[1] this among others is explained here: https://linux-regtracking.leemhuis.info/post/frequent-reasons-why-linux-kern...
I upgraded from 6.1.38 to 6.1.55 this morning and it broke my traffic shaping script, leaving me with a non-functional uplink on a remote router.
The script errors out like this:
Oct 06 05:49:22 wendy00 isp-setup-shaping-start[2053]: + ext=ispA Oct 06 05:49:22 wendy00 isp-setup-shaping-start[2053]: + ext_ingress=ifb0 Oct 06 05:49:22 wendy00 isp-setup-shaping-start[2053]: + modprobe ifb Oct 06 05:49:22 wendy00 isp-setup-shaping-start[2053]: + modprobe act_mirred Oct 06 05:49:22 wendy00 isp-setup-shaping-start[2053]: + tc qdisc del dev ispA root Oct 06 05:49:22 wendy00 isp-setup-shaping-start[2061]: Error: Cannot delete qdisc with handle of zero. Oct 06 05:49:22 wendy00 isp-setup-shaping-start[2053]: + true Oct 06 05:49:22 wendy00 isp-setup-shaping-start[2053]: + tc qdisc del dev ispA ingress Oct 06 05:49:22 wendy00 isp-setup-shaping-start[2064]: Error: Cannot find specified qdisc on specified device. Oct 06 05:49:22 wendy00 isp-setup-shaping-start[2053]: + true Oct 06 05:49:22 wendy00 isp-setup-shaping-start[2053]: + tc qdisc del dev ifb0 root Oct 06 05:49:22 wendy00 isp-setup-shaping-start[2066]: Error: Cannot delete qdisc with handle of zero. Oct 06 05:49:22 wendy00 isp-setup-shaping-start[2053]: + true Oct 06 05:49:22 wendy00 isp-setup-shaping-start[2053]: + tc qdisc del dev ifb0 ingress Oct 06 05:49:22 wendy00 isp-setup-shaping-start[2067]: Error: Cannot find specified qdisc on specified device. Oct 06 05:49:22 wendy00 isp-setup-shaping-start[2053]: + true Oct 06 05:49:22 wendy00 isp-setup-shaping-start[2053]: + tc qdisc add dev ispA handle ffff: ingress Oct 06 05:49:22 wendy00 isp-setup-shaping-start[2053]: + ifconfig ifb0 up Oct 06 05:49:22 wendy00 isp-setup-shaping-start[2053]: + tc filter add dev ispA parent ffff: protocol all u32 match u32 0 0 action mirred egress redirect dev ifb0 Oct 06 05:49:22 wendy00 isp-setup-shaping-start[2053]: + tc qdisc add dev ifb0 root handle 1: hfsc default 1 Oct 06 05:49:22 wendy00 isp-setup-shaping-start[2053]: + tc class add dev ifb0 parent 1: classid 1:999 hfsc rt m2 2.5gbit Oct 06 05:49:22 wendy00 isp-setup-shaping-start[2053]: + tc class add dev ifb0 parent 1:999 classid 1:1 hfsc sc rate 50mbit Oct 06 05:49:22 wendy00 isp-setup-shaping-start[2077]: Error: Invalid parent - parent class must have FSC.
The error message is also a bit weird (but that’s likely due to iproute2 being weird) as the CLI interface for `tc` and the error message do not map well. (I think I would have to choose `hfsc sc` on the parent to enable the FSC option which isn’t mentioned anywhere in the hfsc manpage).
The breaking change was introduced in 6.1.53[1] and a multitude of other currently supported kernels:
commit a1e820fc7808e42b990d224f40e9b4895503ac40 Author: Budimir Markovic markovicbudimir@gmail.com Date: Thu Aug 24 01:49:05 2023 -0700
net/sched: sch_hfsc: Ensure inner classes have fsc curve
[ Upstream commit b3d26c5702c7d6c45456326e56d2ccf3f103e60f ]
HFSC assumes that inner classes have an fsc curve, but it is currently possible for classes without an fsc curve to become parents. This leads to bugs including a use-after-free.
Don't allow non-root classes without HFSC_FSC to become parents.
Fixes: 1da177e4c3f4 ("Linux-2.6.12-rc2") Reported-by: Budimir Markovic markovicbudimir@gmail.com Signed-off-by: Budimir Markovic markovicbudimir@gmail.com Acked-by: Jamal Hadi Salim jhs@mojatatu.com Link: https://lore.kernel.org/r/20230824084905.422-1-markovicbudimir@gmail.com Signed-off-by: Jakub Kicinski kuba@kernel.org Signed-off-by: Sasha Levin sashal@kernel.org
Regards, Christian
[1] https://cdn.kernel.org/pub/linux/kernel/v6.x/ChangeLog-6.1.53
#regzbot introduced: a1e820fc7808e42b990d224f40e9b4895503ac40
Liebe Grüße, Christian Theune
Hi,
sorry, no I didn’t. I don’t have a testbed available right now to try this out quickly.
Christian
On 6. Oct 2023, at 11:16, Linux regression tracking (Thorsten Leemhuis) regressions@leemhuis.info wrote:
On 06.10.23 11:07, Christian Theune wrote:
it seems that 6.6rc4 is affected as well:
commit b3d26c5702c7d6c45456326e56d2ccf3f103e60f Author: Budimir Markovic markovicbudimir@gmail.com Date: Thu Aug 24 01:49:05 2023 -0700
net/sched: sch_hfsc: Ensure inner classes have fsc curve ——
Did you actually try if the problem occurs with 6.6-rc4? That the commit is in there is expected (everything that lands in stable trees has to go to mainline first).
Ciao, Thorsten
I have not found newer commits that would suggest they change any behaviour around this in any way, but I might be wrong.
Christian
On 6. Oct 2023, at 11:01, Linux regression tracking (Thorsten Leemhuis) regressions@leemhuis.info wrote:
On 06.10.23 10:37, Christian Theune wrote:
(prefix, I was not aware of the regression reporting process and incorrectly reported this informally with the developers mentioned in the change)
Don't worry too much about that, but thx for taking care of all the details. FWIW, there is one more thing that would be good to know:
Does the problem happen with mainline (e.g. 6.6-rc4) as well? That's relevant, as different people might care[1].
Ciao, Thorsten
[1] this among others is explained here: https://linux-regtracking.leemhuis.info/post/frequent-reasons-why-linux-kern...
I upgraded from 6.1.38 to 6.1.55 this morning and it broke my traffic shaping script, leaving me with a non-functional uplink on a remote router.
The script errors out like this:
Oct 06 05:49:22 wendy00 isp-setup-shaping-start[2053]: + ext=ispA Oct 06 05:49:22 wendy00 isp-setup-shaping-start[2053]: + ext_ingress=ifb0 Oct 06 05:49:22 wendy00 isp-setup-shaping-start[2053]: + modprobe ifb Oct 06 05:49:22 wendy00 isp-setup-shaping-start[2053]: + modprobe act_mirred Oct 06 05:49:22 wendy00 isp-setup-shaping-start[2053]: + tc qdisc del dev ispA root Oct 06 05:49:22 wendy00 isp-setup-shaping-start[2061]: Error: Cannot delete qdisc with handle of zero. Oct 06 05:49:22 wendy00 isp-setup-shaping-start[2053]: + true Oct 06 05:49:22 wendy00 isp-setup-shaping-start[2053]: + tc qdisc del dev ispA ingress Oct 06 05:49:22 wendy00 isp-setup-shaping-start[2064]: Error: Cannot find specified qdisc on specified device. Oct 06 05:49:22 wendy00 isp-setup-shaping-start[2053]: + true Oct 06 05:49:22 wendy00 isp-setup-shaping-start[2053]: + tc qdisc del dev ifb0 root Oct 06 05:49:22 wendy00 isp-setup-shaping-start[2066]: Error: Cannot delete qdisc with handle of zero. Oct 06 05:49:22 wendy00 isp-setup-shaping-start[2053]: + true Oct 06 05:49:22 wendy00 isp-setup-shaping-start[2053]: + tc qdisc del dev ifb0 ingress Oct 06 05:49:22 wendy00 isp-setup-shaping-start[2067]: Error: Cannot find specified qdisc on specified device. Oct 06 05:49:22 wendy00 isp-setup-shaping-start[2053]: + true Oct 06 05:49:22 wendy00 isp-setup-shaping-start[2053]: + tc qdisc add dev ispA handle ffff: ingress Oct 06 05:49:22 wendy00 isp-setup-shaping-start[2053]: + ifconfig ifb0 up Oct 06 05:49:22 wendy00 isp-setup-shaping-start[2053]: + tc filter add dev ispA parent ffff: protocol all u32 match u32 0 0 action mirred egress redirect dev ifb0 Oct 06 05:49:22 wendy00 isp-setup-shaping-start[2053]: + tc qdisc add dev ifb0 root handle 1: hfsc default 1 Oct 06 05:49:22 wendy00 isp-setup-shaping-start[2053]: + tc class add dev ifb0 parent 1: classid 1:999 hfsc rt m2 2.5gbit Oct 06 05:49:22 wendy00 isp-setup-shaping-start[2053]: + tc class add dev ifb0 parent 1:999 classid 1:1 hfsc sc rate 50mbit Oct 06 05:49:22 wendy00 isp-setup-shaping-start[2077]: Error: Invalid parent - parent class must have FSC.
The error message is also a bit weird (but that’s likely due to iproute2 being weird) as the CLI interface for `tc` and the error message do not map well. (I think I would have to choose `hfsc sc` on the parent to enable the FSC option which isn’t mentioned anywhere in the hfsc manpage).
The breaking change was introduced in 6.1.53[1] and a multitude of other currently supported kernels:
commit a1e820fc7808e42b990d224f40e9b4895503ac40 Author: Budimir Markovic markovicbudimir@gmail.com Date: Thu Aug 24 01:49:05 2023 -0700
net/sched: sch_hfsc: Ensure inner classes have fsc curve
[ Upstream commit b3d26c5702c7d6c45456326e56d2ccf3f103e60f ]
HFSC assumes that inner classes have an fsc curve, but it is currently possible for classes without an fsc curve to become parents. This leads to bugs including a use-after-free.
Don't allow non-root classes without HFSC_FSC to become parents.
Fixes: 1da177e4c3f4 ("Linux-2.6.12-rc2") Reported-by: Budimir Markovic markovicbudimir@gmail.com Signed-off-by: Budimir Markovic markovicbudimir@gmail.com Acked-by: Jamal Hadi Salim jhs@mojatatu.com Link: https://lore.kernel.org/r/20230824084905.422-1-markovicbudimir@gmail.com Signed-off-by: Jakub Kicinski kuba@kernel.org Signed-off-by: Sasha Levin sashal@kernel.org
Regards, Christian
[1] https://cdn.kernel.org/pub/linux/kernel/v6.x/ChangeLog-6.1.53
#regzbot introduced: a1e820fc7808e42b990d224f40e9b4895503ac40
Liebe Grüße, Christian Theune
Liebe Grüße, Christian Theune
On 06/10/2023 17:51, Christian Theune wrote:
Hi,
sorry, no I didn’t. I don’t have a testbed available right now to try this out quickly.
Please don't top-post; reply inline with appropriate context instead.
You need to have testing system, unfortunately. It should mimic your production setup as much as possible. Your organization may have one already, but if not, you have to arrange for it.
Thanks.
Hi,
On 6. Oct 2023, at 14:07, Bagas Sanjaya bagasdotme@gmail.com wrote:
On 06/10/2023 17:51, Christian Theune wrote:
Hi,
sorry, no I didn’t. I don’t have a testbed available right now to try this out quickly.
Please don't top-post; reply inline with appropriate context instead.
Sorry, I will avoid that in the future - I was a bit in a hurry and thus negligent, but you’re right of course.
You need to have testing system, unfortunately. It should mimic your production setup as much as possible. Your organization may have one already, but if not, you have to arrange for it.
I’m able to do that, I just didn’t have one at hand at this very moment and no time to prepare one within a few minutes. I’ll try to reproduce with a 6.6rc in the next days.
Christian
On 06/10/2023 19:37, Christian Theune wrote:
Hi,
On 6. Oct 2023, at 14:07, Bagas Sanjaya bagasdotme@gmail.com wrote:
On 06/10/2023 17:51, Christian Theune wrote:
Hi,
sorry, no I didn’t. I don’t have a testbed available right now to try this out quickly.
Please don't top-post; reply inline with appropriate context instead.
Sorry, I will avoid that in the future - I was a bit in a hurry and thus negligent, but you’re right of course.
You need to have testing system, unfortunately. It should mimic your production setup as much as possible. Your organization may have one already, but if not, you have to arrange for it.
I’m able to do that, I just didn’t have one at hand at this very moment and no time to prepare one within a few minutes. I’ll try to reproduce with a 6.6rc in the next days.
OK, thanks!
Hi,
On 6. Oct 2023, at 16:13, Bagas Sanjaya bagasdotme@gmail.com wrote:
You need to have testing system, unfortunately. It should mimic your production setup as much as possible. Your organization may have one already, but if not, you have to arrange for it.
I’m able to do that, I just didn’t have one at hand at this very moment and no time to prepare one within a few minutes. I’ll try to reproduce with a 6.6rc in the next days.
OK, thanks!
So, I got around running this with 6.6-rc5 today and yes, it’s broken there as well:
Here’s the output from my testing environment:
#### snip subtest: try hfsc machine: must succeed: uname -a (finished: must succeed: uname -a, in 0.02 seconds) Linux machine 6.6.0-rc5 #1-NixOS SMP PREEMPT_DYNAMIC Tue Jan 1 00:00:00 UTC 1980 x86_64 GNU/Linux
machine: must succeed: modprobe ifb (finished: must succeed: modprobe ifb, in 0.02 seconds) machine: must succeed: modprobe act_mirred machine # [ 9.187251] (udev-worker)[851]: ifb0: Could not set Alias=, MACAddress=/MACAddressPolicy=, TransmitQueues=, ReceiveQueues=, TransmitQueueLength=, MTUBytes=, GenericSegmentOffloadMaxBytes= or GenericSegmentOffloadMaxSegments=, ignoring: Operation not supported machine # [ 9.190616] (udev-worker)[851]: Network interface NamePolicy= disabled on kernel command line. machine # [ 9.192605] (udev-worker)[928]: ifb1: Could not set Alias=, MACAddress=/MACAddressPolicy=, TransmitQueues=, ReceiveQueues=, TransmitQueueLength=, MTUBytes=, GenericSegmentOffloadMaxBytes= or GenericSegmentOffloadMaxSegments=, ignoring: Operation not supported machine # [ 9.197607] (udev-worker)[928]: Network interface NamePolicy= disabled on kernel command line. machine # [ 9.292073] Mirror/redirect action on (finished: must succeed: modprobe act_mirred, in 0.04 seconds) machine: must succeed: tc qdisc add dev eth0 handle ffff: ingress (finished: must succeed: tc qdisc add dev eth0 handle ffff: ingress, in 0.05 seconds) machine: must succeed: ifconfig ifb0 up machine # [ 9.276585] dhcpcd[672]: ifb0: waiting for carrier machine # [ 9.278532] dhcpcd[672]: ifb0: carrier acquired machine # [ 9.287692] dhcpcd[672]: ifb0: IAID df:2e:ad:2b machine # [ 9.288225] dhcpcd[672]: ifb0: adding address fe80::40d2:dfff:fe2e:ad2b (finished: must succeed: ifconfig ifb0 up, in 0.02 seconds) machine: must succeed: tc filter add dev eth0 parent ffff: protocol all u32 match u32 0 0 action mirred egress redirect dev ifb0 machine # [ 9.396408] u32 classifier machine # [ 9.396613] Performance counters on machine # [ 9.396895] input device check on machine # [ 9.397148] Actions configured (finished: must succeed: tc filter add dev eth0 parent ffff: protocol all u32 match u32 0 0 action mirred egress redirect dev ifb0, in 0.04 seconds) machine: must succeed: tc qdisc add dev ifb0 root handle 1: hfsc default 1 machine # [ 9.330784] dhcpcd[672]: ifb1: waiting for carrier machine # [ 9.332246] dhcpcd[672]: ifb1: carrier acquired machine # [ 9.343280] dhcpcd[672]: ifb1: IAID ab:f5:e8:5d machine # [ 9.343868] dhcpcd[672]: ifb1: adding address fe80::2c7f:abff:fef5:e85d (finished: must succeed: tc qdisc add dev ifb0 root handle 1: hfsc default 1, in 0.03 seconds) machine: must succeed: tc class add dev ifb0 parent 1: classid 1:999 hfsc rt m2 2.5gbit (finished: must succeed: tc class add dev ifb0 parent 1: classid 1:999 hfsc rt m2 2.5gbit, in 0.01 seconds) machine: must succeed: tc class add dev ifb0 parent 1:999 classid 1:1 hfsc sc rate 50mbit machine # Error: Invalid parent - parent class must have FSC. machine: output: Test "try hfsc" failed with error: "command `tc class add dev ifb0 parent 1:999 classid 1:1 hfsc sc rate 50mbit` failed (exit code 2)" cleanup kill machine (pid 6) #### snap
Liebe Grüße, Christian Theune
On 06/10/2023 05:37, Christian Theune wrote:
Hi,
(prefix, I was not aware of the regression reporting process and incorrectly reported this informally with the developers mentioned in the change)
I upgraded from 6.1.38 to 6.1.55 this morning and it broke my traffic shaping script, leaving me with a non-functional uplink on a remote router.
The script errors out like this:
Oct 06 05:49:22 wendy00 isp-setup-shaping-start[2053]: + ext=ispA Oct 06 05:49:22 wendy00 isp-setup-shaping-start[2053]: + ext_ingress=ifb0 Oct 06 05:49:22 wendy00 isp-setup-shaping-start[2053]: + modprobe ifb Oct 06 05:49:22 wendy00 isp-setup-shaping-start[2053]: + modprobe act_mirred Oct 06 05:49:22 wendy00 isp-setup-shaping-start[2053]: + tc qdisc del dev ispA root Oct 06 05:49:22 wendy00 isp-setup-shaping-start[2061]: Error: Cannot delete qdisc with handle of zero. Oct 06 05:49:22 wendy00 isp-setup-shaping-start[2053]: + true Oct 06 05:49:22 wendy00 isp-setup-shaping-start[2053]: + tc qdisc del dev ispA ingress Oct 06 05:49:22 wendy00 isp-setup-shaping-start[2064]: Error: Cannot find specified qdisc on specified device. Oct 06 05:49:22 wendy00 isp-setup-shaping-start[2053]: + true Oct 06 05:49:22 wendy00 isp-setup-shaping-start[2053]: + tc qdisc del dev ifb0 root Oct 06 05:49:22 wendy00 isp-setup-shaping-start[2066]: Error: Cannot delete qdisc with handle of zero. Oct 06 05:49:22 wendy00 isp-setup-shaping-start[2053]: + true Oct 06 05:49:22 wendy00 isp-setup-shaping-start[2053]: + tc qdisc del dev ifb0 ingress Oct 06 05:49:22 wendy00 isp-setup-shaping-start[2067]: Error: Cannot find specified qdisc on specified device. Oct 06 05:49:22 wendy00 isp-setup-shaping-start[2053]: + true Oct 06 05:49:22 wendy00 isp-setup-shaping-start[2053]: + tc qdisc add dev ispA handle ffff: ingress Oct 06 05:49:22 wendy00 isp-setup-shaping-start[2053]: + ifconfig ifb0 up Oct 06 05:49:22 wendy00 isp-setup-shaping-start[2053]: + tc filter add dev ispA parent ffff: protocol all u32 match u32 0 0 action mirred egress redirect dev ifb0 Oct 06 05:49:22 wendy00 isp-setup-shaping-start[2053]: + tc qdisc add dev ifb0 root handle 1: hfsc default 1 Oct 06 05:49:22 wendy00 isp-setup-shaping-start[2053]: + tc class add dev ifb0 parent 1: classid 1:999 hfsc rt m2 2.5gbit Oct 06 05:49:22 wendy00 isp-setup-shaping-start[2053]: + tc class add dev ifb0 parent 1:999 classid 1:1 hfsc sc rate 50mbit Oct 06 05:49:22 wendy00 isp-setup-shaping-start[2077]: Error: Invalid parent - parent class must have FSC.
The error message is also a bit weird (but that’s likely due to iproute2 being weird) as the CLI interface for `tc` and the error message do not map well. (I think I would have to choose `hfsc sc` on the parent to enable the FSC option which isn’t mentioned anywhere in the hfsc manpage).
The breaking change was introduced in 6.1.53[1] and a multitude of other currently supported kernels:
commit a1e820fc7808e42b990d224f40e9b4895503ac40 Author: Budimir Markovic markovicbudimir@gmail.com Date: Thu Aug 24 01:49:05 2023 -0700
net/sched: sch_hfsc: Ensure inner classes have fsc curve
[ Upstream commit b3d26c5702c7d6c45456326e56d2ccf3f103e60f ]
HFSC assumes that inner classes have an fsc curve, but it is currently possible for classes without an fsc curve to become parents. This leads to bugs including a use-after-free.
Don't allow non-root classes without HFSC_FSC to become parents.
Fixes: 1da177e4c3f4 ("Linux-2.6.12-rc2") Reported-by: Budimir Markovic markovicbudimir@gmail.com Signed-off-by: Budimir Markovic markovicbudimir@gmail.com Acked-by: Jamal Hadi Salim jhs@mojatatu.com Link: https://lore.kernel.org/r/20230824084905.422-1-markovicbudimir@gmail.com Signed-off-by: Jakub Kicinski kuba@kernel.org Signed-off-by: Sasha Levin sashal@kernel.org
Regards, Christian
[1] https://cdn.kernel.org/pub/linux/kernel/v6.x/ChangeLog-6.1.53
#regzbot introduced: a1e820fc7808e42b990d224f40e9b4895503ac40
I will take a look, Thanks!
On 06/10/2023 05:37, Christian Theune wrote:
Hi,
(prefix, I was not aware of the regression reporting process and incorrectly reported this informally with the developers mentioned in the change)
I upgraded from 6.1.38 to 6.1.55 this morning and it broke my traffic shaping script, leaving me with a non-functional uplink on a remote router.
The script errors out like this:
Oct 06 05:49:22 wendy00 isp-setup-shaping-start[2053]: + ext=ispA Oct 06 05:49:22 wendy00 isp-setup-shaping-start[2053]: + ext_ingress=ifb0 Oct 06 05:49:22 wendy00 isp-setup-shaping-start[2053]: + modprobe ifb Oct 06 05:49:22 wendy00 isp-setup-shaping-start[2053]: + modprobe act_mirred Oct 06 05:49:22 wendy00 isp-setup-shaping-start[2053]: + tc qdisc del dev ispA root Oct 06 05:49:22 wendy00 isp-setup-shaping-start[2061]: Error: Cannot delete qdisc with handle of zero. Oct 06 05:49:22 wendy00 isp-setup-shaping-start[2053]: + true Oct 06 05:49:22 wendy00 isp-setup-shaping-start[2053]: + tc qdisc del dev ispA ingress Oct 06 05:49:22 wendy00 isp-setup-shaping-start[2064]: Error: Cannot find specified qdisc on specified device. Oct 06 05:49:22 wendy00 isp-setup-shaping-start[2053]: + true Oct 06 05:49:22 wendy00 isp-setup-shaping-start[2053]: + tc qdisc del dev ifb0 root Oct 06 05:49:22 wendy00 isp-setup-shaping-start[2066]: Error: Cannot delete qdisc with handle of zero. Oct 06 05:49:22 wendy00 isp-setup-shaping-start[2053]: + true Oct 06 05:49:22 wendy00 isp-setup-shaping-start[2053]: + tc qdisc del dev ifb0 ingress Oct 06 05:49:22 wendy00 isp-setup-shaping-start[2067]: Error: Cannot find specified qdisc on specified device. Oct 06 05:49:22 wendy00 isp-setup-shaping-start[2053]: + true Oct 06 05:49:22 wendy00 isp-setup-shaping-start[2053]: + tc qdisc add dev ispA handle ffff: ingress Oct 06 05:49:22 wendy00 isp-setup-shaping-start[2053]: + ifconfig ifb0 up Oct 06 05:49:22 wendy00 isp-setup-shaping-start[2053]: + tc filter add dev ispA parent ffff: protocol all u32 match u32 0 0 action mirred egress redirect dev ifb0 Oct 06 05:49:22 wendy00 isp-setup-shaping-start[2053]: + tc qdisc add dev ifb0 root handle 1: hfsc default 1 Oct 06 05:49:22 wendy00 isp-setup-shaping-start[2053]: + tc class add dev ifb0 parent 1: classid 1:999 hfsc rt m2 2.5gbit Oct 06 05:49:22 wendy00 isp-setup-shaping-start[2053]: + tc class add dev ifb0 parent 1:999 classid 1:1 hfsc sc rate 50mbit Oct 06 05:49:22 wendy00 isp-setup-shaping-start[2077]: Error: Invalid parent - parent class must have FSC.
The error message is also a bit weird (but that’s likely due to iproute2 being weird) as the CLI interface for `tc` and the error message do not map well. (I think I would have to choose `hfsc sc` on the parent to enable the FSC option which isn’t mentioned anywhere in the hfsc manpage).
The breaking change was introduced in 6.1.53[1] and a multitude of other currently supported kernels:
Hi,
Your script is actually incorrect. `man 7 tc-hfsc` goes in depth into why, but I just wanna highlight this section: SEPARATE LS / RT SCs Another difference from the original HFSC paper is that RT and LS SCs can be specified separately. Moreover, leaf classes are allowed to have only either RT SC or LS SC. For interior classes, only LS SCs make sense: any RT SC will be ignored.
The last part ("For interior classes...") was what the referenced patch fixed. We were mistakenly allowing RTs into "interior classes" which the implementation never accounted for and this was a source of crashes. I'm surprised you were lucky enough to never crash the kernel ;) -= I believe the script could be updated to the following and still achieve the same results: tc class add dev ifb0 parent 1: classid 1:999 hfsc ls m2 2.5gbit tc class add dev ifb0 parent 1:999 classid 1:1 hfsc rt rate 50mbit
Hi,
On 7. Oct 2023, at 00:12, Pedro Tammela pctammela@mojatatu.com wrote:
On 06/10/2023 05:37, Christian Theune wrote:
Hi, (prefix, I was not aware of the regression reporting process and incorrectly reported this informally with the developers mentioned in the change) I upgraded from 6.1.38 to 6.1.55 this morning and it broke my traffic shaping script, leaving me with a non-functional uplink on a remote router. The script errors out like this: Oct 06 05:49:22 wendy00 isp-setup-shaping-start[2053]: + ext=ispA Oct 06 05:49:22 wendy00 isp-setup-shaping-start[2053]: + ext_ingress=ifb0 Oct 06 05:49:22 wendy00 isp-setup-shaping-start[2053]: + modprobe ifb Oct 06 05:49:22 wendy00 isp-setup-shaping-start[2053]: + modprobe act_mirred Oct 06 05:49:22 wendy00 isp-setup-shaping-start[2053]: + tc qdisc del dev ispA root Oct 06 05:49:22 wendy00 isp-setup-shaping-start[2061]: Error: Cannot delete qdisc with handle of zero. Oct 06 05:49:22 wendy00 isp-setup-shaping-start[2053]: + true Oct 06 05:49:22 wendy00 isp-setup-shaping-start[2053]: + tc qdisc del dev ispA ingress Oct 06 05:49:22 wendy00 isp-setup-shaping-start[2064]: Error: Cannot find specified qdisc on specified device. Oct 06 05:49:22 wendy00 isp-setup-shaping-start[2053]: + true Oct 06 05:49:22 wendy00 isp-setup-shaping-start[2053]: + tc qdisc del dev ifb0 root Oct 06 05:49:22 wendy00 isp-setup-shaping-start[2066]: Error: Cannot delete qdisc with handle of zero. Oct 06 05:49:22 wendy00 isp-setup-shaping-start[2053]: + true Oct 06 05:49:22 wendy00 isp-setup-shaping-start[2053]: + tc qdisc del dev ifb0 ingress Oct 06 05:49:22 wendy00 isp-setup-shaping-start[2067]: Error: Cannot find specified qdisc on specified device. Oct 06 05:49:22 wendy00 isp-setup-shaping-start[2053]: + true Oct 06 05:49:22 wendy00 isp-setup-shaping-start[2053]: + tc qdisc add dev ispA handle ffff: ingress Oct 06 05:49:22 wendy00 isp-setup-shaping-start[2053]: + ifconfig ifb0 up Oct 06 05:49:22 wendy00 isp-setup-shaping-start[2053]: + tc filter add dev ispA parent ffff: protocol all u32 match u32 0 0 action mirred egress redirect dev ifb0 Oct 06 05:49:22 wendy00 isp-setup-shaping-start[2053]: + tc qdisc add dev ifb0 root handle 1: hfsc default 1 Oct 06 05:49:22 wendy00 isp-setup-shaping-start[2053]: + tc class add dev ifb0 parent 1: classid 1:999 hfsc rt m2 2.5gbit Oct 06 05:49:22 wendy00 isp-setup-shaping-start[2053]: + tc class add dev ifb0 parent 1:999 classid 1:1 hfsc sc rate 50mbit Oct 06 05:49:22 wendy00 isp-setup-shaping-start[2077]: Error: Invalid parent - parent class must have FSC. The error message is also a bit weird (but that’s likely due to iproute2 being weird) as the CLI interface for `tc` and the error message do not map well. (I think I would have to choose `hfsc sc` on the parent to enable the FSC option which isn’t mentioned anywhere in the hfsc manpage). The breaking change was introduced in 6.1.53[1] and a multitude of other currently supported kernels:
Hi,
Your script is actually incorrect. `man 7 tc-hfsc` goes in depth into why, but I just wanna highlight this section: SEPARATE LS / RT SCs Another difference from the original HFSC paper is that RT and LS SCs can be specified separately. Moreover, leaf classes are allowed to have only either RT SC or LS SC. For interior classes, only LS SCs make sense: any RT SC will be ignored.
The last part ("For interior classes...") was what the referenced patch fixed. We were mistakenly allowing RTs into "interior classes" which the implementation never accounted for and this was a source of crashes. I'm surprised you were lucky enough to never crash the kernel ;)
I believe the script could be updated to the following and still achieve the same results: tc class add dev ifb0 parent 1: classid 1:999 hfsc ls m2 2.5gbit tc class add dev ifb0 parent 1:999 classid 1:1 hfsc rt rate 50mbit
I’m absolutely with you on this point and I’m very sure that I may have gotten the configuration wrong. I was already diving into fixing the script. Nevertheless, this end of the system is definitely not my strongest discipline and I tried get it right when setting this up initially scrambling through the man pages and the complexities of this topic. Unfortunately trying to learn this stuff is really hard and if the system finally accepts your config and does what you want and doesn’t complain in any way, then it’s hard to try and track down whether anything else could have been missed - somewhat a case of the halting problem. ;)
Point in case: I didn’t even notice tc-hfsc(7) was there and that I was only reading tc-hfsc(8).
Nevertheless, I’m curious and registered this as a regression, because it seems to fit the description and broke a running system. Looking at the change I’m wondering whether the kernel stability rule implies that the change has to be more forgiving towards user land, even if the user(land) might be wrong?
The idea of not bricking your system by upgrading the Linux kernel seems to apply here. IMHO the change could maybe done in a way that keeps the system running but informs the user that something isn’t working as intended?
Liebe Grüße, Christian Theune
On Sat, 7 Oct 2023 06:10:42 +0200 Christian Theune wrote:
The idea of not bricking your system by upgrading the Linux kernel seems to apply here. IMHO the change could maybe done in a way that keeps the system running but informs the user that something isn’t working as intended?
Herm, how did we get this far without CCing the author of the patch. Adding Budimir.
Pedro, Budimir, any idea what the original bug was? There isn't much info in the commit message.
On 09/10/2023 12:06, Jakub Kicinski wrote:
On Sat, 7 Oct 2023 06:10:42 +0200 Christian Theune wrote:
The idea of not bricking your system by upgrading the Linux kernel seems to apply here. IMHO the change could maybe done in a way that keeps the system running but informs the user that something isn’t working as intended?
Herm, how did we get this far without CCing the author of the patch. Adding Budimir.
Pedro, Budimir, any idea what the original bug was? There isn't much info in the commit message.
We had a UAF with a very straight forward way to trigger it. Setting 'rt' as a parent is incorrect and the man page is explicit about it as it doesn't make sense 'qdisc wise'. Being able to set it has always been wrong unfortunately...
Hi,
On 9. Oct 2023, at 17:31, Pedro Tammela pctammela@mojatatu.com wrote:
We had a UAF with a very straight forward way to trigger it. Setting 'rt' as a parent is incorrect and the man page is explicit about it as it doesn't make sense 'qdisc wise'. Being able to set it has always been wrong unfortunately...
Well - this is a complex thing and even though I took care when setting it up I did not find this in the manpages sufficiently highlighted and built such a system: this has been running for more than a year, it did what I wanted AFAICT and upgrading a micro version broke it catastrophically.
The argument that this never should have been built, feels a bit like “this has been written for 50 years in the basement of the office for planetary destruction on Alpha Centauri and earthlings had more than enough time to come and complain” … ;)
Hugs, Christian
On Mon, 9 Oct 2023 12:31:57 -0300 Pedro Tammela wrote:
Herm, how did we get this far without CCing the author of the patch. Adding Budimir.
Pedro, Budimir, any idea what the original bug was? There isn't much info in the commit message.
We had a UAF with a very straight forward way to trigger it.
Any details?
Setting 'rt' as a parent is incorrect and the man page is explicit about it as it doesn't make sense 'qdisc wise'. Being able to set it has always been wrong unfortunately...
Sure but unfortunately "we don't break backward compat" means we can't really argue. It will take us more time to debate this than to fix it (assuming we understand the initial problem).
Frankly one can even argue whether "exploitable by root / userns" is more important than single user's init scripts breaking. The "security" issues for root are dime a dozen.
On Mon, Oct 9, 2023 at 8:28 PM Jakub Kicinski kuba@kernel.org wrote:
On Mon, 9 Oct 2023 12:31:57 -0300 Pedro Tammela wrote:
Herm, how did we get this far without CCing the author of the patch. Adding Budimir.
Pedro, Budimir, any idea what the original bug was? There isn't much info in the commit message.
We had a UAF with a very straight forward way to trigger it.
Any details?
As in you want the sequence of commands that caused the fault posted? Budimir, lets wait for Jakub's response before you do that. I have those details as well of course.
Setting 'rt' as a parent is incorrect and the man page is explicit about it as it doesn't make sense 'qdisc wise'. Being able to set it has always been wrong unfortunately...
Sure but unfortunately "we don't break backward compat" means we can't really argue. It will take us more time to debate this than to fix it (assuming we understand the initial problem).
Frankly one can even argue whether "exploitable by root / userns" is more important than single user's init scripts breaking. The "security" issues for root are dime a dozen.
This is a tough one - as it stands right now we dont see a good way out. It's either "exploitable by root / userns" or break uapi. Christian - can you send your "working" scripts, simplified if possible, and we'll take a look.
cheers, jamal
Hi,
On 10. Oct 2023, at 17:02, Jamal Hadi Salim jhs@mojatatu.com wrote:
This is a tough one - as it stands right now we dont see a good way out. It's either "exploitable by root / userns" or break uapi. Christian - can you send your "working" scripts, simplified if possible, and we'll take a look.
Sure, what kind of simplification are we talking about? Something like this?
#### snip #!/bin/bash modprobe ifb modprobe act_mirred
uplink=eth0 uplink_ingress=ifb0
tc qdisc add dev $uplink handle ffff: ingress ifconfig $uplink up
tc filter add dev $uplink parent ffff: protocol all u32 match u32 0 0 action mirred egress redirect dev $uplink_ingress
tc qdisc add dev $uplink_ingress root handle 1: hfsc default 1 tc class add dev $uplink_ingress parent 1: classid 1:999 hfsc rt m2 2.5gbit tc class add dev $uplink_ingress parent 1:999 classid 1:1 hfsc sc rate 50mbit #### snap
This should provoke the error reliably. You might need to point it at whatever network interface is available but need to be prepared to loose connectivity.
Christian
Liebe Grüße, Christian Theune
On Tue, Oct 10, 2023 at 1:32 PM Christian Theune ct@flyingcircus.io wrote:
Hi,
On 10. Oct 2023, at 17:02, Jamal Hadi Salim jhs@mojatatu.com wrote:
This is a tough one - as it stands right now we dont see a good way out. It's either "exploitable by root / userns" or break uapi. Christian - can you send your "working" scripts, simplified if possible, and we'll take a look.
Sure, what kind of simplification are we talking about? Something like this?
#### snip #!/bin/bash modprobe ifb modprobe act_mirred
uplink=eth0 uplink_ingress=ifb0
tc qdisc add dev $uplink handle ffff: ingress ifconfig $uplink up
tc filter add dev $uplink parent ffff: protocol all u32 match u32 0 0 action mirred egress redirect dev $uplink_ingress
tc qdisc add dev $uplink_ingress root handle 1: hfsc default 1 tc class add dev $uplink_ingress parent 1: classid 1:999 hfsc rt m2 2.5gbit tc class add dev $uplink_ingress parent 1:999 classid 1:1 hfsc sc rate 50mbit #### snap
This should provoke the error reliably. You might need to point it at whatever network interface is available but need to be prepared to loose connectivity.
Ok - thanks, we'll look at this from the perspective of both ensuring UAF is gone and making your config happy. TBH, in my view UAF comes first but we can debate that later.
cheers, jamal
Christian
Liebe Grüße, Christian Theune
-- Christian Theune · ct@flyingcircus.io · +49 345 219401 0 Flying Circus Internet Operations GmbH · https://flyingcircus.io Leipziger Str. 70/71 · 06108 Halle (Saale) · Deutschland HR Stendal HRB 21169 · Geschäftsführer: Christian Theune, Christian Zagrodnick
On Tue, 10 Oct 2023 11:02:25 -0400 Jamal Hadi Salim wrote:
We had a UAF with a very straight forward way to trigger it.
Any details?
As in you want the sequence of commands that caused the fault posted? Budimir, lets wait for Jakub's response before you do that. I have those details as well of course.
More - the sequence of events which leads to the UAF, and on what object it occurs. If there's an embargo or some such we can wait a little longer before discussing?
I haven't looked at the code for more than a minute. If this is super trivial to spot let me know, I'll stare harder. Didn't seem like the qdisc as a whole is all that trivial.
On Tue, Oct 10, 2023 at 2:26 PM Jakub Kicinski kuba@kernel.org wrote:
On Tue, 10 Oct 2023 11:02:25 -0400 Jamal Hadi Salim wrote:
We had a UAF with a very straight forward way to trigger it.
Any details?
As in you want the sequence of commands that caused the fault posted? Budimir, lets wait for Jakub's response before you do that. I have those details as well of course.
More - the sequence of events which leads to the UAF, and on what object it occurs. If there's an embargo or some such we can wait a little longer before discussing?
I haven't looked at the code for more than a minute. If this is super trivial to spot let me know, I'll stare harder. Didn't seem like the qdisc as a whole is all that trivial.
The qdisc is non-trivial. The good news is we now know there's at least one user for this qdisc ;->
cheers, jamal
On 11. Oct 2023, at 19:27, Jamal Hadi Salim jhs@mojatatu.com wrote:
On Tue, Oct 10, 2023 at 2:26 PM Jakub Kicinski kuba@kernel.org wrote:
The qdisc is non-trivial. The good news is we now know there's at least one user for this qdisc ;->
Thanks everyone for taking the time to look at it. I hope and guess I’m not the only user … :)
Hugs, Christian
[TLDR: This mail in primarily relevant for Linux kernel regression tracking. See link in footer if these mails annoy you.]
On 06.10.23 10:37, Christian Theune wrote:
I upgraded from 6.1.38 to 6.1.55 this morning and it broke my traffic shaping script, leaving me with a non-functional uplink on a remote router. [...] #regzbot introduced: a1e820fc7808e42b990d224f40e9b4895503ac40
#regzbot fix: net/sched: sch_hfsc: upgrade rt to sc when it becomes a inner curve #regzbot ignore-activity
Ciao, Thorsten (wearing his 'the Linux kernel's regression tracker' hat) -- Everything you wanna know about Linux kernel regression tracking: https://linux-regtracking.leemhuis.info/about/#tldr That page also explains what to do if mails like this annoy you.
On 01.11.23 20:05, Linux regression tracking #update (Thorsten Leemhuis) wrote:
[TLDR: This mail in primarily relevant for Linux kernel regression tracking. See link in footer if these mails annoy you.]
On 06.10.23 10:37, Christian Theune wrote:
I upgraded from 6.1.38 to 6.1.55 this morning and it broke my traffic shaping script, leaving me with a non-functional uplink on a remote router. [...] #regzbot introduced: a1e820fc7808e42b990d224f40e9b4895503ac40
#regzbot fix: net/sched: sch_hfsc: upgrade rt to sc when it becomes a inner curve #regzbot ignore-activity
Sorry, that had a typo:
#regzbot fix: d80bc191420a2edecb555b #regzbot ignore-activity
#regzbot ignore-activity
Ciao, Thorsten (wearing his 'the Linux kernel's regression tracker' hat) -- Everything you wanna know about Linux kernel regression tracking: https://linux-regtracking.leemhuis.info/about/#tldr That page also explains what to do if mails like this annoy you.
linux-stable-mirror@lists.linaro.org
-
Bagas Sanjaya -
Christian Theune -
Jakub Kicinski -
Jamal Hadi Salim -
Linux regression tracking #update (Thorsten Leemhuis) -
Linux regression tracking (Thorsten Leemhuis)
-
Pedro Tammela