Under heavy load if we don't have any pre-allocated rsps left, we dynamically allocate a rsp, but we are not actually allocating memory for nvme_completion (rsp->req.rsp). In such a case, accessing pointer fields (req->rsp->status) in nvmet_req_init() will result in crash.
To fix this, allocate the memory for nvme_completion by calling nvmet_rdma_alloc_rsp()
Fixes: 8407879c("nvmet-rdma:fix possible bogus dereference under heavy load")
Cc: stable@vger.kernel.org Reviewed-by: Max Gurtovoy maxg@mellanox.com Signed-off-by: Raju Rangoju rajur@chelsio.com
-- Changes from v2: - Dropped the unneeded initialization of ret variable
Changes from v1: - Moved integer to 'if' block - Used 'unlikely' in datapath flow condition --- drivers/nvme/target/rdma.c | 15 ++++++++++++++- 1 file changed, 14 insertions(+), 1 deletion(-)
diff --git a/drivers/nvme/target/rdma.c b/drivers/nvme/target/rdma.c index a8d23eb80192..a884e3a0e8af 100644 --- a/drivers/nvme/target/rdma.c +++ b/drivers/nvme/target/rdma.c @@ -139,6 +139,10 @@ static void nvmet_rdma_recv_done(struct ib_cq *cq, struct ib_wc *wc); static void nvmet_rdma_read_data_done(struct ib_cq *cq, struct ib_wc *wc); static void nvmet_rdma_qp_event(struct ib_event *event, void *priv); static void nvmet_rdma_queue_disconnect(struct nvmet_rdma_queue *queue); +static void nvmet_rdma_free_rsp(struct nvmet_rdma_device *ndev, + struct nvmet_rdma_rsp *r); +static int nvmet_rdma_alloc_rsp(struct nvmet_rdma_device *ndev, + struct nvmet_rdma_rsp *r);
static const struct nvmet_fabrics_ops nvmet_rdma_ops;
@@ -182,9 +186,17 @@ nvmet_rdma_get_rsp(struct nvmet_rdma_queue *queue) spin_unlock_irqrestore(&queue->rsps_lock, flags);
if (unlikely(!rsp)) { - rsp = kmalloc(sizeof(*rsp), GFP_KERNEL); + int ret; + + rsp = kzalloc(sizeof(*rsp), GFP_KERNEL); if (unlikely(!rsp)) return NULL; + ret = nvmet_rdma_alloc_rsp(queue->dev, rsp); + if (unlikely(ret)) { + kfree(rsp); + return NULL; + } + rsp->allocated = true; }
@@ -197,6 +209,7 @@ nvmet_rdma_put_rsp(struct nvmet_rdma_rsp *rsp) unsigned long flags;
if (unlikely(rsp->allocated)) { + nvmet_rdma_free_rsp(rsp->queue->dev, rsp); kfree(rsp); return; }
Wonder how the original fix got a tested-by...
Reviewed-by: Sagi Grimberg sagi@grimberg.me
On 1/3/2019 5:13 PM, Sagi Grimberg wrote:
Wonder how the original fix got a tested-by...
Reviewed-by: Sagi Grimberg sagi@grimberg.me
That was my bad. But the testers said they couldn't reproduce the issue, so I assumed that meant it was fixed. I guess they weren't stressing it enough.
Steve.
Hi Christoph,
In case you missed, could you please pull this patch ?
-Raju
-----Original Message----- From: Steve Wise swise@opengridcomputing.com Sent: 07 January 2019 21:29 To: Sagi Grimberg sagi@grimberg.me; Raju Rangoju rajur@chelsio.com; linux-nvme@lists.infradead.org Cc: maxg@mellanox.com; stable@vger.kernel.org Subject: Re: [PATCH v3] nvmet-rdma: fix null dereference under heavy load
On 1/3/2019 5:13 PM, Sagi Grimberg wrote:
Wonder how the original fix got a tested-by...
Reviewed-by: Sagi Grimberg sagi@grimberg.me
That was my bad. But the testers said they couldn't reproduce the issue, so I assumed that meant it was fixed. I guess they weren't stressing it enough.
Steve.
This looks ok to me:
Reviewed-by: Christoph Hellwig hch@lst.de
In the long run I wonder if we should get rid of the ѕtatic preallocation entirely and just use a free pool to avoid constant reallocations..
This looks ok to me:
Reviewed-by: Christoph Hellwig hch@lst.de
Applied to nvme-5.0
In the long run I wonder if we should get rid of the ѕtatic preallocation entirely and just use a free pool to avoid constant reallocations..
you mean a kmem_cache? very possible, we can also make the command struct to contain the rsp behind it to avoid the extra allocation...
On 1/21/2019 11:10 AM, Sagi Grimberg wrote:
This looks ok to me:
Reviewed-by: Christoph Hellwig hch@lst.de
Applied to nvme-5.0
In the long run I wonder if we should get rid of the ѕtatic preallocation entirely and just use a free pool to avoid constant reallocations..
you mean a kmem_cache? very possible, we can also make the command struct to contain the rsp behind it to avoid the extra allocation...
yes, the container idea sounds good..
On 1/21/2019 3:10 AM, Sagi Grimberg wrote:
This looks ok to me:
Reviewed-by: Christoph Hellwig hch@lst.de
Applied to nvme-5.0
In the long run I wonder if we should get rid of the ѕtatic preallocation entirely and just use a free pool to avoid constant reallocations..
you mean a kmem_cache? very possible, we can also make the command struct to contain the rsp behind it to avoid the extra allocation...
Hey guys,
If every command struct requires a rsp, then it seems simpler to contain it in the command struct.
My 2 cents...
Steve.
linux-stable-mirror@lists.linaro.org