[ Upstream commit 99143f82a255e7f054bead8443462fae76dd829e ]
While reviewing another read_slowpath patch, both Will and I noticed another missing ACQUIRE, namely:
X = 0;
CPU0 CPU1
rwsem_down_read() for (;;) { set_current_state(TASK_UNINTERRUPTIBLE);
X = 1; rwsem_up_write(); rwsem_mark_wake() atomic_long_add(adjustment, &sem->count); smp_store_release(&waiter->task, NULL);
if (!waiter.task) break;
... }
r = X;
Allows 'r == 0'.
Reported-by: Peter Zijlstra (Intel) peterz@infradead.org Reported-by: Will Deacon will@kernel.org Signed-off-by: Peter Zijlstra (Intel) peterz@infradead.org Acked-by: Will Deacon will@kernel.org Cc: Linus Torvalds torvalds@linux-foundation.org Cc: Peter Zijlstra peterz@infradead.org Cc: Thomas Gleixner tglx@linutronix.de Fixes: 1da177e4c3f4 ("Linux-2.6.12-rc2") Signed-off-by: Ingo Molnar mingo@kernel.org Acked-by: Jan Stancek jstancek@redhat.com Signed-off-by: Sasha Levin sashal@kernel.org --- kernel/locking/rwsem-xadd.c | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-)
diff --git a/kernel/locking/rwsem-xadd.c b/kernel/locking/rwsem-xadd.c index 397dedc58432d..385ebcfc31a6d 100644 --- a/kernel/locking/rwsem-xadd.c +++ b/kernel/locking/rwsem-xadd.c @@ -485,8 +485,10 @@ __rwsem_down_read_failed_common(struct rw_semaphore *sem, int state) /* wait to be given the lock */ while (true) { set_current_state(state); - if (!waiter.task) + if (!smp_load_acquire(&waiter.task)) { + /* Orders against rwsem_mark_wake()'s smp_store_release() */ break; + } if (signal_pending_state(state, current)) { raw_spin_lock_irq(&sem->wait_lock); if (waiter.task)