[PATCH v6 0/6] Introduce sendpage_ok() to detect misused sendpage in network related drivers
This series was original by a bug fix in nvme-over-tcp driver which only checked whether a page was allocated from slab allcoator, but forgot to check its page_count: The page handled by sendpage should be neither a Slab page nor 0 page_count page.
As Sagi Grimberg suggested, the original fix is refind to a more common inline routine: static inline bool sendpage_ok(struct page *page) { return (!PageSlab(page) && page_count(page) >= 1); } If sendpage_ok() returns true, the checking page can be handled by the zero copy sendpage method in network layer.
The first patch in this series introduces sendpage_ok() in header file include/linux/net.h, the second patch fixes the page checking issue in nvme-over-tcp driver, the third patch adds page_count check by using sendpage_ok() in do_tcp_sendpages() as Eric Dumazet suggested, and all rested patches just replace existing open coded checks with the inline sendpage_ok() routine.
Coly Li
Cc: Chaitanya Kulkarni chaitanya.kulkarni@wdc.com Cc: Chris Leech cleech@redhat.com Cc: Christoph Hellwig hch@lst.de Cc: Cong Wang amwang@redhat.com Cc: David S. Miller davem@davemloft.net Cc: Hannes Reinecke hare@suse.de Cc: Ilya Dryomov idryomov@gmail.com Cc: Jan Kara jack@suse.com Cc: Jeff Layton jlayton@kernel.org Cc: Jens Axboe axboe@kernel.dk Cc: Lee Duncan lduncan@suse.com Cc: Mike Christie michaelc@cs.wisc.edu Cc: Mikhail Skorzhinskii mskorzhinskiy@solarflare.com Cc: Philipp Reisner philipp.reisner@linbit.com Cc: Sagi Grimberg sagi@grimberg.me Cc: stable@vger.kernel.org Cc: Vasily Averin vvs@virtuozzo.com Cc: Vlastimil Babka vbabka@suse.com --- Changelog: v6: fix page check in do_tcp_sendpages(), and replace other open coded checks with sendpage_ok() in libceph, iscsi drivers. v5, include linux/mm.h in include/linux/net.h v4, change sendpage_ok() as an inline helper, and post it as separate patch. v3, introduce a more common sendpage_ok() v2, fix typo in patch subject v1, the initial version.
Coly Li (6): net: introduce helper sendpage_ok() in include/linux/net.h nvme-tcp: check page by sendpage_ok() before calling kernel_sendpage() tcp: use sendpage_ok() to detect misused .sendpage drbd: code cleanup by using sendpage_ok() to check page for kernel_sendpage() scsi: libiscsi: use sendpage_ok() in iscsi_tcp_segment_map() libceph: use sendpage_ok() in ceph_tcp_sendpage()
drivers/block/drbd/drbd_main.c | 2 +- drivers/nvme/host/tcp.c | 7 +++---- drivers/scsi/libiscsi_tcp.c | 2 +- include/linux/net.h | 16 ++++++++++++++++ net/ceph/messenger.c | 2 +- net/ipv4/tcp.c | 3 ++- 6 files changed, 24 insertions(+), 8 deletions(-)
The original problem was from nvme-over-tcp code, who mistakenly uses kernel_sendpage() to send pages allocated by __get_free_pages() without __GFP_COMP flag. Such pages don't have refcount (page_count is 0) on tail pages, sending them by kernel_sendpage() may trigger a kernel panic from a corrupted kernel heap, because these pages are incorrectly freed in network stack as page_count 0 pages.
This patch introduces a helper sendpage_ok(), it returns true if the checking page, - is not slab page: PageSlab(page) is false. - has page refcount: page_count(page) is not zero
All drivers who want to send page to remote end by kernel_sendpage() may use this helper to check whether the page is OK. If the helper does not return true, the driver should try other non sendpage method (e.g. sock_no_sendpage()) to handle the page.
Signed-off-by: Coly Li colyli@suse.de Cc: Chaitanya Kulkarni chaitanya.kulkarni@wdc.com Cc: Christoph Hellwig hch@lst.de Cc: Hannes Reinecke hare@suse.de Cc: Jan Kara jack@suse.com Cc: Jens Axboe axboe@kernel.dk Cc: Mikhail Skorzhinskii mskorzhinskiy@solarflare.com Cc: Philipp Reisner philipp.reisner@linbit.com Cc: Sagi Grimberg sagi@grimberg.me Cc: Vlastimil Babka vbabka@suse.com Cc: stable@vger.kernel.org --- include/linux/net.h | 16 ++++++++++++++++ 1 file changed, 16 insertions(+)
diff --git a/include/linux/net.h b/include/linux/net.h index d48ff1180879..a807fad31958 100644 --- a/include/linux/net.h +++ b/include/linux/net.h @@ -21,6 +21,7 @@ #include <linux/rcupdate.h> #include <linux/once.h> #include <linux/fs.h> +#include <linux/mm.h> #include <linux/sockptr.h>
#include <uapi/linux/net.h> @@ -286,6 +287,21 @@ do { \ #define net_get_random_once_wait(buf, nbytes) \ get_random_once_wait((buf), (nbytes))
+/* + * E.g. XFS meta- & log-data is in slab pages, or bcache meta + * data pages, or other high order pages allocated by + * __get_free_pages() without __GFP_COMP, which have a page_count + * of 0 and/or have PageSlab() set. We cannot use send_page for + * those, as that does get_page(); put_page(); and would cause + * either a VM_BUG directly, or __page_cache_release a page that + * would actually still be referenced by someone, leading to some + * obscure delayed Oops somewhere else. + */ +static inline bool sendpage_ok(struct page *page) +{ + return (!PageSlab(page) && page_count(page) >= 1); +} + int kernel_sendmsg(struct socket *sock, struct msghdr *msg, struct kvec *vec, size_t num, size_t len); int kernel_sendmsg_locked(struct sock *sk, struct msghdr *msg,
Currently nvme_tcp_try_send_data() doesn't use kernel_sendpage() to send slab pages. But for pages allocated by __get_free_pages() without __GFP_COMP, which also have refcount as 0, they are still sent by kernel_sendpage() to remote end, this is problematic.
The new introduced helper sendpage_ok() checks both PageSlab tag and page_count counter, and returns true if the checking page is OK to be sent by kernel_sendpage().
This patch fixes the page checking issue of nvme_tcp_try_send_data() with sendpage_ok(). If sendpage_ok() returns true, send this page by kernel_sendpage(), otherwise use sock_no_sendpage to handle this page.
Signed-off-by: Coly Li colyli@suse.de Cc: Chaitanya Kulkarni chaitanya.kulkarni@wdc.com Cc: Christoph Hellwig hch@lst.de Cc: Hannes Reinecke hare@suse.de Cc: Jan Kara jack@suse.com Cc: Jens Axboe axboe@kernel.dk Cc: Mikhail Skorzhinskii mskorzhinskiy@solarflare.com Cc: Philipp Reisner philipp.reisner@linbit.com Cc: Sagi Grimberg sagi@grimberg.me Cc: Vlastimil Babka vbabka@suse.com Cc: stable@vger.kernel.org --- drivers/nvme/host/tcp.c | 7 +++---- 1 file changed, 3 insertions(+), 4 deletions(-)
diff --git a/drivers/nvme/host/tcp.c b/drivers/nvme/host/tcp.c index 62fbaecdc960..902fe742762b 100644 --- a/drivers/nvme/host/tcp.c +++ b/drivers/nvme/host/tcp.c @@ -912,12 +912,11 @@ static int nvme_tcp_try_send_data(struct nvme_tcp_request *req) else flags |= MSG_MORE | MSG_SENDPAGE_NOTLAST;
- /* can't zcopy slab pages */ - if (unlikely(PageSlab(page))) { - ret = sock_no_sendpage(queue->sock, page, offset, len, + if (sendpage_ok(page)) { + ret = kernel_sendpage(queue->sock, page, offset, len, flags); } else { - ret = kernel_sendpage(queue->sock, page, offset, len, + ret = sock_no_sendpage(queue->sock, page, offset, len, flags); } if (ret <= 0)
commit a10674bf2406 ("tcp: detecting the misuse of .sendpage for Slab objects") adds the checks for Slab pages, but the pages don't have page_count are still missing from the check.
Network layer's sendpage method is not designed to send page_count 0 pages neither, therefore both PageSlab() and page_count() should be both checked for the sending page. This is exactly what sendpage_ok() does.
This patch uses sendpage_ok() in do_tcp_sendpages() to detect misused .sendpage, to make the code more robust.
Fixes: a10674bf2406 ("tcp: detecting the misuse of .sendpage for Slab objects") Suggested-by: Eric Dumazet eric.dumazet@gmail.com Signed-off-by: Coly Li colyli@suse.de Cc: Vasily Averin vvs@virtuozzo.com Cc: David S. Miller davem@davemloft.net Cc: stable@vger.kernel.org --- net/ipv4/tcp.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-)
diff --git a/net/ipv4/tcp.c b/net/ipv4/tcp.c index 31f3b858db81..d96c7549895a 100644 --- a/net/ipv4/tcp.c +++ b/net/ipv4/tcp.c @@ -970,7 +970,8 @@ ssize_t do_tcp_sendpages(struct sock *sk, struct page *page, int offset, long timeo = sock_sndtimeo(sk, flags & MSG_DONTWAIT);
if (IS_ENABLED(CONFIG_DEBUG_VM) && - WARN_ONCE(PageSlab(page), "page must not be a Slab one")) + WARN_ONCE(!sendpage_ok(page)), + "page must not be a Slab one and have page_count > 0") return -EINVAL;
/* Wait for a connection to finish. One exception is TCP Fast Open
linux-stable-mirror@lists.linaro.org
-
Coly Li