From: Justin Sanders jsanders.devel@gmail.com
[ Upstream commit cffc873d68ab09a0432b8212008c5613f8a70a2c ]
When aoe's rexmit_timer() notices that an aoe target fails to respond to commands for more than aoe_deadsecs, it calls aoedev_downdev() which cleans the outstanding aoe and block queues. This can involve sleeping, such as in blk_mq_freeze_queue(), which should not occur in irq context.
This patch defers that aoedev_downdev() call to the aoe device's workqueue.
Link: https://bugzilla.kernel.org/show_bug.cgi?id=212665 Signed-off-by: Justin Sanders jsanders.devel@gmail.com Link: https://lore.kernel.org/r/20250610170600.869-2-jsanders.devel@gmail.com Tested-By: Valentin Kleibel valentin@vrvis.at Signed-off-by: Jens Axboe axboe@kernel.dk Signed-off-by: Sasha Levin sashal@kernel.org ---
Based on my analysis of the commit and the kernel repository context, here is my determination:
**YES**
This commit should be backported to stable kernel trees. Here's my extensive explanation:
## 1. **Critical Bug Fix for Kernel Crash/Hang**
The commit fixes a serious "scheduling while atomic" bug that can cause kernel crashes. The issue occurs when: - The `rexmit_timer()` function (timer callback, runs in softirq context) detects a device timeout - It directly calls `aoedev_downdev()` which contains `blk_mq_freeze_queue()` - a sleeping operation - Calling a sleeping function from atomic context violates kernel rules and causes crashes
## 2. **Clear Root Cause and Minimal Fix**
The fix is minimal and targeted: - Adds a new flag `DEVFL_DEAD` to mark devices that need to be downed - Instead of calling `aoedev_downdev()` directly from timer context, it sets the flag and queues work - The actual `aoedev_downdev()` call happens in workqueue context where sleeping is allowed
## 3. **Part of a Series Fixing Related Issues**
This is part of a series addressing bugzilla #212665, including: - This commit: Fixes the timer context sleeping issue - Commit 7f90d45e57cb: Fixes the hang in `blk_mq_freeze_queue()` by properly cleaning request lists
Both commits are needed to fully resolve the reported issue.
## 4. **Regression from blk-mq Conversion**
The issue was introduced by commit 3582dd291788 ("aoe: convert aoeblk to blk-mq"), making this a regression fix. The blk-mq conversion added sleeping operations that weren't safe to call from existing timer contexts.
## 5. **Similar to Previously Backported Fixes**
Looking at the similar commits: - Commit 430380b4637a (fixing atomic sleep in flush) was marked as suitable for backporting - It addressed the same class of bug in a different code path - This establishes precedent for backporting such fixes
## 6. **Minimal Risk of Regression**
The changes are: - Well-contained to the aoe driver - Follow established patterns (deferring to workqueue) - Don't change any user-visible behavior - Only change the execution context of an existing operation
## 7. **Meets Stable Criteria**
According to stable tree rules, this commit: - ✓ Fixes a real bug (kernel crash/scheduling while atomic) - ✓ Has been tested (per commit message) - ✓ Is minimal and targeted - ✓ Fixes a regression from previous changes - ✓ Has clear explanation and reproducer (bugzilla link)
The only consideration is that it should be backported together with commit 7f90d45e57cb to ensure the complete fix for bugzilla #212665 is applied.
drivers/block/aoe/aoe.h | 1 + drivers/block/aoe/aoecmd.c | 8 ++++++-- drivers/block/aoe/aoedev.c | 5 ++++- 3 files changed, 11 insertions(+), 3 deletions(-)
diff --git a/drivers/block/aoe/aoe.h b/drivers/block/aoe/aoe.h index 749ae1246f4cf..d35caa3c69e15 100644 --- a/drivers/block/aoe/aoe.h +++ b/drivers/block/aoe/aoe.h @@ -80,6 +80,7 @@ enum { DEVFL_NEWSIZE = (1<<6), /* need to update dev size in block layer */ DEVFL_FREEING = (1<<7), /* set when device is being cleaned up */ DEVFL_FREED = (1<<8), /* device has been cleaned up */ + DEVFL_DEAD = (1<<9), /* device has timed out of aoe_deadsecs */ };
enum { diff --git a/drivers/block/aoe/aoecmd.c b/drivers/block/aoe/aoecmd.c index d1f4ddc576451..c4c5cf1ec71ba 100644 --- a/drivers/block/aoe/aoecmd.c +++ b/drivers/block/aoe/aoecmd.c @@ -754,7 +754,7 @@ rexmit_timer(struct timer_list *timer)
utgts = count_targets(d, NULL);
- if (d->flags & DEVFL_TKILL) { + if (d->flags & (DEVFL_TKILL | DEVFL_DEAD)) { spin_unlock_irqrestore(&d->lock, flags); return; } @@ -786,7 +786,8 @@ rexmit_timer(struct timer_list *timer) * to clean up. */ list_splice(&flist, &d->factive[0]); - aoedev_downdev(d); + d->flags |= DEVFL_DEAD; + queue_work(aoe_wq, &d->work); goto out; }
@@ -898,6 +899,9 @@ aoecmd_sleepwork(struct work_struct *work) { struct aoedev *d = container_of(work, struct aoedev, work);
+ if (d->flags & DEVFL_DEAD) + aoedev_downdev(d); + if (d->flags & DEVFL_GDALLOC) aoeblk_gdalloc(d);
diff --git a/drivers/block/aoe/aoedev.c b/drivers/block/aoe/aoedev.c index 3523dd82d7a00..5f42c1a14b805 100644 --- a/drivers/block/aoe/aoedev.c +++ b/drivers/block/aoe/aoedev.c @@ -199,8 +199,11 @@ aoedev_downdev(struct aoedev *d) struct aoetgt *t, **tt, **te; struct list_head *head, *pos, *nx; int i; + unsigned long flags;
- d->flags &= ~DEVFL_UP; + spin_lock_irqsave(&d->lock, flags); + d->flags &= ~(DEVFL_UP | DEVFL_DEAD); + spin_unlock_irqrestore(&d->lock, flags);
/* clean out active and to-be-retransmitted buffers */ for (i = 0; i < NFACTIVE; i++) {