igc_configure_rx_ring() function will be called as part of XDP program setup. If Rx hardware timestamp is enabled prio to XDP program setup, this timestamp enablement will be overwritten when buffer size is written into SRRCTL register.
Thus, this commit read the register value before write to SRRCTL register. This commit is tested by using xdp_hw_metadata bpf selftest tool. The tool enables Rx hardware timestamp and then attach XDP program to igc driver. It will display hardware timestamp of UDP packet with port number 9092. Below are detail of test steps and results.
Command on DUT: sudo ./xdp_hw_metadata <interface name>
Command on Link Partner: echo -n skb | nc -u -q1 <destination IPv4 addr> 9092
Result before this patch: skb hwtstamp is not found!
Result after this patch: found skb hwtstamp = 1677800973.642836757
Optionally, read PHC to confirm the values obtained are almost the same: Command: sudo ./testptp -d /dev/ptp0 -g Result: clock time: 1677800973.913598978 or Fri Mar 3 07:49:33 2023
Fixes: fc9df2a0b520 ("igc: Enable RX via AF_XDP zero-copy") Cc: stable@vger.kernel.org # 5.14+ Signed-off-by: Song Yoong Siang yoong.siang.song@intel.com Reviewed-by: Jacob Keller jacob.e.keller@intel.com Reviewed-by: Jesper Dangaard Brouer brouer@redhat.com --- v2 -> v3: Refactor SRRCTL definitions to more human readable definitions v1 -> v2: Fix indention --- drivers/net/ethernet/intel/igc/igc_base.h | 11 ++++++++--- drivers/net/ethernet/intel/igc/igc_main.c | 7 +++++-- 2 files changed, 13 insertions(+), 5 deletions(-)
diff --git a/drivers/net/ethernet/intel/igc/igc_base.h b/drivers/net/ethernet/intel/igc/igc_base.h index 7a992befca24..9f3827eda157 100644 --- a/drivers/net/ethernet/intel/igc/igc_base.h +++ b/drivers/net/ethernet/intel/igc/igc_base.h @@ -87,8 +87,13 @@ union igc_adv_rx_desc { #define IGC_RXDCTL_SWFLUSH 0x04000000 /* Receive Software Flush */
/* SRRCTL bit definitions */ -#define IGC_SRRCTL_BSIZEPKT_SHIFT 10 /* Shift _right_ */ -#define IGC_SRRCTL_BSIZEHDRSIZE_SHIFT 2 /* Shift _left_ */ -#define IGC_SRRCTL_DESCTYPE_ADV_ONEBUF 0x02000000 +#define IGC_SRRCTL_BSIZEPKT_MASK GENMASK(6, 0) +#define IGC_SRRCTL_BSIZEPKT(x) FIELD_PREP(IGC_SRRCTL_BSIZEPKT_MASK, \ + (x) / 1024) /* in 1 KB resolution */ +#define IGC_SRRCTL_BSIZEHDR_MASK GENMASK(13, 8) +#define IGC_SRRCTL_BSIZEHDR(x) FIELD_PREP(IGC_SRRCTL_BSIZEHDR_MASK, \ + (x) / 64) /* in 64 bytes resolution */ +#define IGC_SRRCTL_DESCTYPE_MASK GENMASK(27, 25) +#define IGC_SRRCTL_DESCTYPE_ADV_ONEBUF FIELD_PREP(IGC_SRRCTL_DESCTYPE_MASK, 1)
#endif /* _IGC_BASE_H */ diff --git a/drivers/net/ethernet/intel/igc/igc_main.c b/drivers/net/ethernet/intel/igc/igc_main.c index 25fc6c65209b..a2d823e64609 100644 --- a/drivers/net/ethernet/intel/igc/igc_main.c +++ b/drivers/net/ethernet/intel/igc/igc_main.c @@ -641,8 +641,11 @@ static void igc_configure_rx_ring(struct igc_adapter *adapter, else buf_size = IGC_RXBUFFER_2048;
- srrctl = IGC_RX_HDR_LEN << IGC_SRRCTL_BSIZEHDRSIZE_SHIFT; - srrctl |= buf_size >> IGC_SRRCTL_BSIZEPKT_SHIFT; + srrctl = rd32(IGC_SRRCTL(reg_idx)); + srrctl &= ~(IGC_SRRCTL_BSIZEPKT_MASK | IGC_SRRCTL_BSIZEHDR_MASK | + IGC_SRRCTL_DESCTYPE_MASK); + srrctl |= IGC_SRRCTL_BSIZEHDR(IGC_RX_HDR_LEN); + srrctl |= IGC_SRRCTL_BSIZEPKT(buf_size); srrctl |= IGC_SRRCTL_DESCTYPE_ADV_ONEBUF;
wr32(IGC_SRRCTL(reg_idx), srrctl);
On 14/04/2023 17.49, Song Yoong Siang wrote:
igc_configure_rx_ring() function will be called as part of XDP program setup. If Rx hardware timestamp is enabled prio to XDP program setup, this timestamp enablement will be overwritten when buffer size is written into SRRCTL register.
Thus, this commit read the register value before write to SRRCTL register. This commit is tested by using xdp_hw_metadata bpf selftest tool. The tool enables Rx hardware timestamp and then attach XDP program to igc driver. It will display hardware timestamp of UDP packet with port number 9092. Below are detail of test steps and results.
Command on DUT: sudo ./xdp_hw_metadata <interface name>
Command on Link Partner: echo -n skb | nc -u -q1 <destination IPv4 addr> 9092
Result before this patch: skb hwtstamp is not found!
Result after this patch: found skb hwtstamp = 1677800973.642836757
Optionally, read PHC to confirm the values obtained are almost the same: Command: sudo ./testptp -d /dev/ptp0 -g Result: clock time: 1677800973.913598978 or Fri Mar 3 07:49:33 2023
Fixes: fc9df2a0b520 ("igc: Enable RX via AF_XDP zero-copy") Cc: stable@vger.kernel.org # 5.14+ Signed-off-by: Song Yoong Siang yoong.siang.song@intel.com Reviewed-by: Jacob Keller jacob.e.keller@intel.com Reviewed-by: Jesper Dangaard Brouer brouer@redhat.com
LGTM, thank for the adjustments :-)
Acked-by: Jesper Dangaard Brouer brouer@redhat.com
v2 -> v3: Refactor SRRCTL definitions to more human readable definitions v1 -> v2: Fix indention
drivers/net/ethernet/intel/igc/igc_base.h | 11 ++++++++--- drivers/net/ethernet/intel/igc/igc_main.c | 7 +++++-- 2 files changed, 13 insertions(+), 5 deletions(-)
diff --git a/drivers/net/ethernet/intel/igc/igc_base.h b/drivers/net/ethernet/intel/igc/igc_base.h index 7a992befca24..9f3827eda157 100644 --- a/drivers/net/ethernet/intel/igc/igc_base.h +++ b/drivers/net/ethernet/intel/igc/igc_base.h @@ -87,8 +87,13 @@ union igc_adv_rx_desc { #define IGC_RXDCTL_SWFLUSH 0x04000000 /* Receive Software Flush */ /* SRRCTL bit definitions */ -#define IGC_SRRCTL_BSIZEPKT_SHIFT 10 /* Shift _right_ */ -#define IGC_SRRCTL_BSIZEHDRSIZE_SHIFT 2 /* Shift _left_ */ -#define IGC_SRRCTL_DESCTYPE_ADV_ONEBUF 0x02000000 +#define IGC_SRRCTL_BSIZEPKT_MASK GENMASK(6, 0) +#define IGC_SRRCTL_BSIZEPKT(x) FIELD_PREP(IGC_SRRCTL_BSIZEPKT_MASK, \
(x) / 1024) /* in 1 KB resolution */+#define IGC_SRRCTL_BSIZEHDR_MASK GENMASK(13, 8) +#define IGC_SRRCTL_BSIZEHDR(x) FIELD_PREP(IGC_SRRCTL_BSIZEHDR_MASK, \
(x) / 64) /* in 64 bytes resolution */+#define IGC_SRRCTL_DESCTYPE_MASK GENMASK(27, 25) +#define IGC_SRRCTL_DESCTYPE_ADV_ONEBUF FIELD_PREP(IGC_SRRCTL_DESCTYPE_MASK, 1) #endif /* _IGC_BASE_H */ diff --git a/drivers/net/ethernet/intel/igc/igc_main.c b/drivers/net/ethernet/intel/igc/igc_main.c index 25fc6c65209b..a2d823e64609 100644 --- a/drivers/net/ethernet/intel/igc/igc_main.c +++ b/drivers/net/ethernet/intel/igc/igc_main.c @@ -641,8 +641,11 @@ static void igc_configure_rx_ring(struct igc_adapter *adapter, else buf_size = IGC_RXBUFFER_2048;
- srrctl = IGC_RX_HDR_LEN << IGC_SRRCTL_BSIZEHDRSIZE_SHIFT;
- srrctl |= buf_size >> IGC_SRRCTL_BSIZEPKT_SHIFT;
- srrctl = rd32(IGC_SRRCTL(reg_idx));
- srrctl &= ~(IGC_SRRCTL_BSIZEPKT_MASK | IGC_SRRCTL_BSIZEHDR_MASK |
IGC_SRRCTL_DESCTYPE_MASK);- srrctl |= IGC_SRRCTL_BSIZEHDR(IGC_RX_HDR_LEN);
- srrctl |= IGC_SRRCTL_BSIZEPKT(buf_size); srrctl |= IGC_SRRCTL_DESCTYPE_ADV_ONEBUF;
wr32(IGC_SRRCTL(reg_idx), srrctl);
On 14/04/2023 22.05, Jesper Dangaard Brouer wrote:
On 14/04/2023 17.49, Song Yoong Siang wrote:
igc_configure_rx_ring() function will be called as part of XDP program setup. If Rx hardware timestamp is enabled prio to XDP program setup, this timestamp enablement will be overwritten when buffer size is written into SRRCTL register.
Thus, this commit read the register value before write to SRRCTL register. This commit is tested by using xdp_hw_metadata bpf selftest tool. The tool enables Rx hardware timestamp and then attach XDP program to igc driver. It will display hardware timestamp of UDP packet with port number 9092. Below are detail of test steps and results.
[...]
Fixes: fc9df2a0b520 ("igc: Enable RX via AF_XDP zero-copy") Cc: stable@vger.kernel.org # 5.14+ Signed-off-by: Song Yoong Siang yoong.siang.song@intel.com Reviewed-by: Jacob Keller jacob.e.keller@intel.com Reviewed-by: Jesper Dangaard Brouer brouer@redhat.com
LGTM, thank for the adjustments :-)
Acked-by: Jesper Dangaard Brouer brouer@redhat.com
Tested-by: Jesper Dangaard Brouer brouer@redhat.com
I can confirm that this patch fix the issue I experienced with igc.
This patch clearly fixes a bug in igc when writing the SRRCTL register. (as bit 30 in register is "Timestamp Received Packet" which got cleared before).
Florian might have found another bug around RX timestamps, but this patch should be safe and sane to apply as is.
v2 -> v3: Refactor SRRCTL definitions to more human readable definitions v1 -> v2: Fix indention
drivers/net/ethernet/intel/igc/igc_base.h | 11 ++++++++--- drivers/net/ethernet/intel/igc/igc_main.c | 7 +++++-- 2 files changed, 13 insertions(+), 5 deletions(-)
On Mon, 2023-04-17 at 16:24 +0200, Jesper Dangaard Brouer wrote:
On 14/04/2023 22.05, Jesper Dangaard Brouer wrote:
On 14/04/2023 17.49, Song Yoong Siang wrote:
igc_configure_rx_ring() function will be called as part of XDP program setup. If Rx hardware timestamp is enabled prio to XDP program setup, this timestamp enablement will be overwritten when buffer size is written into SRRCTL register.
Thus, this commit read the register value before write to SRRCTL register. This commit is tested by using xdp_hw_metadata bpf selftest tool. The tool enables Rx hardware timestamp and then attach XDP program to igc driver. It will display hardware timestamp of UDP packet with port number 9092. Below are detail of test steps and results.
[...]
Fixes: fc9df2a0b520 ("igc: Enable RX via AF_XDP zero-copy") Cc: stable@vger.kernel.org # 5.14+ Signed-off-by: Song Yoong Siang yoong.siang.song@intel.com Reviewed-by: Jacob Keller jacob.e.keller@intel.com Reviewed-by: Jesper Dangaard Brouer brouer@redhat.com
LGTM, thank for the adjustments :-)
Acked-by: Jesper Dangaard Brouer brouer@redhat.com
Tested-by: Jesper Dangaard Brouer brouer@redhat.com
I can confirm that this patch fix the issue I experienced with igc.
This patch clearly fixes a bug in igc when writing the SRRCTL register. (as bit 30 in register is "Timestamp Received Packet" which got cleared before).
Florian might have found another bug around RX timestamps, but this patch should be safe and sane to apply as is.
After a closer look I'm quite sure now that this patch should fix my issue as well. The register will be overwritten when setting up a XSK_POOL as well:
igc_bpf igc_xdp_setup_pool igc_enable_rx_ring igc_configure_rx_ring wr32(IGC_SRRCTL)
I already removed the BPF loading (which is the use case that the patch description mentions) from my setup to limit the search scope. If you like you could extend the patch description, but I'm fine with it.
Thanks a lot for all the support / ideas! Highly appreciated!
Florian
v2 -> v3: Refactor SRRCTL definitions to more human readable definitions v1 -> v2: Fix indention
drivers/net/ethernet/intel/igc/igc_base.h | 11 ++++++++--- drivers/net/ethernet/intel/igc/igc_main.c | 7 +++++-- 2 files changed, 13 insertions(+), 5 deletions(-)
On 14.04.23 17:49, Song Yoong Siang wrote:
igc_configure_rx_ring() function will be called as part of XDP program setup. If Rx hardware timestamp is enabled prio to XDP program setup, this timestamp enablement will be overwritten when buffer size is written into SRRCTL register.
Hi all,
I'm actually searching for the root cause of a similar problem (RX timestamp lost) that I can reproduce here, but the setup is slightly different. The setup:
- igc driver - i225/226 network card - When starting to transmit frames using XDP with zero copy enabled another application (ptp4l) complains about missing RX timestamps - Loading the XDP program has already been removed (not needed for TX only) - ptp4l is using the traditional socket API - The RX timestamps seem to come back once we stop sending frames using XDP
The "zero copy support" enabled part is important. If ZC support is not requested everything works fine.
Any ideas?
Best regards, Florian
On Saturday, April 15, 2023 5:20 PM, Florian Bezdeka florian.bezdeka@siemens.com wrote:
On 14.04.23 17:49, Song Yoong Siang wrote:
igc_configure_rx_ring() function will be called as part of XDP program setup. If Rx hardware timestamp is enabled prio to XDP program setup, this timestamp enablement will be overwritten when buffer size is written into SRRCTL register.
Hi all,
I'm actually searching for the root cause of a similar problem (RX timestamp lost) that I can reproduce here, but the setup is slightly different. The setup:
- igc driver
- i225/226 network card
- When starting to transmit frames using XDP with zero copy enabled
another application (ptp4l) complains about missing RX timestamps
- Loading the XDP program has already been removed (not needed for
TX only)
- ptp4l is using the traditional socket API
- The RX timestamps seem to come back once we stop sending frames
using XDP
The "zero copy support" enabled part is important. If ZC support is not requested everything works fine.
Any ideas?
Best regards, Florian
Hi Florian,
You means this patch does not help on your issue? Need to understand more on the setup and behavior to tell. Are ptp4l and XDP ZC Tx apps running on same queue or separate queue? I suggest you to run " sudo hwstamp_ctl -i eth0 -r 1 " multiple times in the middle to check the behavior.
Thanks & Regards Siang
Hi Florian,
-----Original Message----- From: Intel-wired-lan intel-wired-lan-bounces@osuosl.org On Behalf Of Song, Yoong Siang Sent: Sunday, 16 April, 2023 10:19 AM To: Bezdeka, Florian florian.bezdeka@siemens.com; Brandeburg, Jesse jesse.brandeburg@intel.com; Nguyen, Anthony L anthony.l.nguyen@intel.com; David S . Miller davem@davemloft.net; Eric Dumazet edumazet@google.com; Jakub Kicinski kuba@kernel.org; Paolo Abeni pabeni@redhat.com; Alexei Starovoitov ast@kernel.org; Daniel Borkmann daniel@iogearbox.net; Jesper Dangaard Brouer hawk@kernel.org; John Fastabend john.fastabend@gmail.com; Fijalkowski, Maciej maciej.fijalkowski@intel.com; Vedang Patel vedang.patel@intel.com; Joseph, Jithu jithu.joseph@intel.com; Andre Guedes andre.guedes@intel.com; Brouer, Jesper brouer@redhat.com; Stanislav Fomichev sdf@google.com; Keller, Jacob E jacob.e.keller@intel.com; David Laight David.Laight@ACULAB.COM Cc: xdp-hints@xdp-project.net; netdev@vger.kernel.org; linux- kernel@vger.kernel.org; stable@vger.kernel.org; intel-wired- lan@lists.osuosl.org; bpf@vger.kernel.org Subject: Re: [Intel-wired-lan] [PATCH net v3 1/1] igc: read before write to SRRCTL register
On Saturday, April 15, 2023 5:20 PM, Florian Bezdeka florian.bezdeka@siemens.com wrote:
On 14.04.23 17:49, Song Yoong Siang wrote:
igc_configure_rx_ring() function will be called as part of XDP program setup. If Rx hardware timestamp is enabled prio to XDP program setup, this timestamp enablement will be overwritten when buffer size is written into SRRCTL register.
Hi all,
I'm actually searching for the root cause of a similar problem (RX timestamp lost) that I can reproduce here, but the setup is slightly
different. The setup:
- igc driver
- i225/226 network card
- When starting to transmit frames using XDP with zero copy enabled
another application (ptp4l) complains about missing RX timestamps
- Loading the XDP program has already been removed (not needed for
TX only)
- ptp4l is using the traditional socket API
- The RX timestamps seem to come back once we stop sending frames
using XDP
The "zero copy support" enabled part is important. If ZC support is not requested everything works fine.
Any ideas?
Are you observing similar issue like below? ptp4l: timed out while polling for tx timestamp ptp4l: increasing tx_timestamp_timeout may correct this issue
If yes, only TXSTAMPO register is used for both PTP and non-PTP packets in the current driver code. There is a possibility that the time stamp for a PTP packet will be lost when there is a lot of traffic when multiple applications request for hardware transmission timestamps. Few months back, I submitted a patch series to enable the DMA Timestamp for non-ptp packet which can resolve the above issue. https://lore.kernel.org/netdev/20221018010733.4765-1-muhammad.husaini.zulkif... Will continuing back the activity soon.
But you might want to try with series submitted by Vinicius as well. This patch series add support for four sets of timestamping register. https://patchwork.ozlabs.org/project/intel-wired-lan/patch/20230228054534.10... https://patchwork.ozlabs.org/project/intel-wired-lan/patch/20230228054534.10... https://patchwork.ozlabs.org/project/intel-wired-lan/patch/20230228054534.10...
Would it be possible for you to try any of the above patch series and see if that fixes your problem?
Thanks, Husaini
Best regards, Florian
Hi Florian,
You means this patch does not help on your issue? Need to understand more on the setup and behavior to tell. Are ptp4l and XDP ZC Tx apps running on same queue or separate queue? I suggest you to run " sudo hwstamp_ctl -i eth0 -r 1 " multiple times in the middle to check the behavior.
Thanks & Regards Siang _______________________________________________ Intel-wired-lan mailing list Intel-wired-lan@osuosl.org https://lists.osuosl.org/mailman/listinfo/intel-wired-lan
On Mon, 17 Apr 2023 02:53:13 +0000 Zulkifli, Muhammad Husaini wrote:
Are you observing similar issue like below? ptp4l: timed out while polling for tx timestamp ptp4l: increasing tx_timestamp_timeout may correct this issue
If yes, only TXSTAMPO register is used for both PTP and non-PTP packets in the current driver code. There is a possibility that the time stamp for a PTP packet will be lost when there is a lot of traffic when multiple applications request for hardware transmission timestamps. Few months back, I submitted a patch series to enable the DMA Timestamp for non-ptp packet which can resolve the above issue. https://lore.kernel.org/netdev/20221018010733.4765-1-muhammad.husaini.zulkif... Will continuing back the activity soon.
FWIW the work on selecting the source of the timestamp is progressing slowly:
https://lore.kernel.org/all/20230406173308.401924-1-kory.maincent@bootlin.co...
On 4/14/2023 18:49, Song Yoong Siang wrote:
igc_configure_rx_ring() function will be called as part of XDP program setup. If Rx hardware timestamp is enabled prio to XDP program setup, this timestamp enablement will be overwritten when buffer size is written into SRRCTL register.
Thus, this commit read the register value before write to SRRCTL register. This commit is tested by using xdp_hw_metadata bpf selftest tool. The tool enables Rx hardware timestamp and then attach XDP program to igc driver. It will display hardware timestamp of UDP packet with port number 9092. Below are detail of test steps and results.
Command on DUT: sudo ./xdp_hw_metadata <interface name>
Command on Link Partner: echo -n skb | nc -u -q1 <destination IPv4 addr> 9092
Result before this patch: skb hwtstamp is not found!
Result after this patch: found skb hwtstamp = 1677800973.642836757
Optionally, read PHC to confirm the values obtained are almost the same: Command: sudo ./testptp -d /dev/ptp0 -g Result: clock time: 1677800973.913598978 or Fri Mar 3 07:49:33 2023
Fixes: fc9df2a0b520 ("igc: Enable RX via AF_XDP zero-copy") Cc: stable@vger.kernel.org # 5.14+ Signed-off-by: Song Yoong Siang yoong.siang.song@intel.com Reviewed-by: Jacob Keller jacob.e.keller@intel.com Reviewed-by: Jesper Dangaard Brouer brouer@redhat.com
v2 -> v3: Refactor SRRCTL definitions to more human readable definitions v1 -> v2: Fix indention
drivers/net/ethernet/intel/igc/igc_base.h | 11 ++++++++--- drivers/net/ethernet/intel/igc/igc_main.c | 7 +++++-- 2 files changed, 13 insertions(+), 5 deletions(-)
Tested-by: Naama Meir naamax.meir@linux.intel.com
linux-stable-mirror@lists.linaro.org
-
Florian Bezdeka -
Jakub Kicinski -
Jesper Dangaard Brouer -
naamax.meir -
Song Yoong Siang -
Song, Yoong Siang
-
Zulkifli, Muhammad Husaini