Hi all,
Here are fixes that worth to have in the @stable, as they were hit by different people, including Arista on v4.9 stable.
And for linux-next - adding lockdep asserts for line discipline changing code, verifying that write ldisc sem will be held forthwith.
Mikulas, can you add your tested-by on this patches set again, please? I tried to reproduce reboot issue on qemu-hppa and even built cross-compiler for pa-risc.. but was unlucky in reproducing.
Thanks, Dima
Changes since v5: - Better commit tags - Hopefully fixed issue with reboot on pa-risc with Debian 5
Changes since v4: - back to lock ldisc with (5*HZ) timeout in tty_reopen() (LKP report link: lkml.kernel.org/r/1536940609.3185.29.camel@arista.com) - reordered 3/7 with 2/7 for LKP robot
Changes since v3: - Added tested-by Mark Rutland (thanks!) - Dropped patch with smp_wmb() - wrong idea - lockdep_assert_held() should be actually lockdep_assert_held_exclusive() - Described why tty_ldisc_open() can be called without ldisc_sem held for pty slave end (o_tty). - Added Peter's patch for dropping self-made lockdep annotations - Fix for a reader(s) of ldisc semaphore waiting for an active reader(s)
Changes since v2: - Added reviewed-by tags - Hopefully, fixed reported by 0-day issue. - Added optional fix for wait_readers decrement
Changes since v1: - Added tested-by/reported-by tags - Dropped 3/4 (locking tty pair for lockdep sake), Because of that - not adding lockdep_assert_held() in tty_ldisc_open() - Added 4/4 cleanup to inc tty->count only on success of tty_ldisc_reinit() - lock ldisc without (5*HZ) timeout in tty_reopen()
v1 link: lkml.kernel.org/r/20180829022353.23568-1-dima@arista.com v2 link: lkml.kernel.org/r/20180903165257.29227-1-dima@arista.com v3 link: lkml.kernel.org/r/20180911014821.26286-1-dima@arista.com v4 link: lkml.kernel.org/r/20180912001702.18522-1-dima@arista.com
Cc: Daniel Axtens dja@axtens.net Cc: Dmitry Vyukov dvyukov@google.com Cc: Mark Rutland mark.rutland@arm.com Cc: Michael Neuling mikey@neuling.org Cc: Mikulas Patocka mpatocka@redhat.com Cc: Nathan March nathan@gt.net Cc: Pasi Kärkkäinen pasik@iki.fi Cc: Peter Hurley peter@hurleysoftware.com Cc: Peter Zijlstra peterz@infradead.org Cc: "Rong, Chen" rong.a.chen@intel.com Cc: Sergey Senozhatsky sergey.senozhatsky.work@gmail.com Cc: Tan Xiaojun tanxiaojun@huawei.com Cc: Tetsuo Handa penguin-kernel@I-love.SAKURA.ne.jp (please, ignore if I Cc'ed you mistakenly)
Dmitry Safonov (6): tty/ldsem: Wake up readers after timed out down_write() tty: Hold tty_ldisc_lock() during tty_reopen() tty: Don't block on IO when ldisc change is pending tty: Simplify tty->count math in tty_reopen() tty/ldsem: Add lockdep asserts for ldisc_sem tty/ldsem: Decrement wait_readers on timeouted down_read()
Peter Zijlstra (1): tty/ldsem: Convert to regular lockdep annotations
drivers/tty/n_hdlc.c | 4 +-- drivers/tty/n_r3964.c | 2 +- drivers/tty/n_tty.c | 8 +++--- drivers/tty/tty_io.c | 14 ++++++---- drivers/tty/tty_ldisc.c | 16 +++++++++++ drivers/tty/tty_ldsem.c | 62 +++++++++++++++++------------------------ include/linux/tty.h | 7 +++++ 7 files changed, 63 insertions(+), 50 deletions(-)
ldsem_down_read() will sleep if there is pending writer in the queue. If the writer times out, readers in the queue should be woken up, otherwise they may miss a chance to acquire the semaphore until the last active reader will do ldsem_up_read().
There was a couple of reports where there was one active reader and other readers soft locked up: Showing all locks held in the system: 2 locks held by khungtaskd/17: #0: (rcu_read_lock){......}, at: watchdog+0x124/0x6d1 #1: (tasklist_lock){.+.+..}, at: debug_show_all_locks+0x72/0x2d3 2 locks held by askfirst/123: #0: (&tty->ldisc_sem){.+.+.+}, at: ldsem_down_read+0x46/0x58 #1: (&ldata->atomic_read_lock){+.+...}, at: n_tty_read+0x115/0xbe4
Prevent readers wait for active readers to release ldisc semaphore.
Link: lkml.kernel.org/r/20171121132855.ajdv4k6swzhvktl6@wfg-t540p.sh.intel.com Link: lkml.kernel.org/r/20180907045041.GF1110@shao2-debian Cc: Greg Kroah-Hartman gregkh@linuxfoundation.org Cc: Jiri Slaby jslaby@suse.com Cc: Peter Zijlstra peterz@infradead.org Cc: stable@vger.kernel.org Reported-by: kernel test robot rong.a.chen@intel.com Signed-off-by: Dmitry Safonov dima@arista.com --- drivers/tty/tty_ldsem.c | 10 ++++++++++ 1 file changed, 10 insertions(+)
diff --git a/drivers/tty/tty_ldsem.c b/drivers/tty/tty_ldsem.c index 0c98d88f795a..b989ca26fc78 100644 --- a/drivers/tty/tty_ldsem.c +++ b/drivers/tty/tty_ldsem.c @@ -293,6 +293,16 @@ down_write_failed(struct ld_semaphore *sem, long count, long timeout) if (!locked) atomic_long_add_return(-LDSEM_WAIT_BIAS, &sem->count); list_del(&waiter.list); + + /* + * In case of timeout, wake up every reader who gave the right of way + * to writer. Prevent separation readers into two groups: + * one that helds semaphore and another that sleeps. + * (in case of no contention with a writer) + */ + if (!locked && list_empty(&sem->write_wait)) + __ldsem_wake_readers(sem); + raw_spin_unlock_irq(&sem->wait_lock);
__set_current_state(TASK_RUNNING);
tty_ldisc_reinit() doesn't race with neither tty_ldisc_hangup() nor set_ldisc() nor tty_ldisc_release() as they use tty lock. But it races with anyone who expects line discipline to be the same after hoding read semaphore in tty_ldisc_ref().
We've seen the following crash on v4.9.108 stable:
BUG: unable to handle kernel paging request at 0000000000002260 IP: [..] n_tty_receive_buf_common+0x5f/0x86d Workqueue: events_unbound flush_to_ldisc Call Trace: [..] n_tty_receive_buf2 [..] tty_ldisc_receive_buf [..] flush_to_ldisc [..] process_one_work [..] worker_thread [..] kthread [..] ret_from_fork
tty_ldisc_reinit() should be called with ldisc_sem hold for writing, which will protect any reader against line discipline changes.
Cc: Greg Kroah-Hartman gregkh@linuxfoundation.org Cc: Jiri Slaby jslaby@suse.com Cc: stable@vger.kernel.org # b027e2298bd5 ("tty: fix data race between tty_init_dev and flush of buf") Reviewed-by: Jiri Slaby jslaby@suse.cz Reported-by: syzbot+3aa9784721dfb90e984d@syzkaller.appspotmail.com Tested-by: Mark Rutland mark.rutland@arm.com Tested-by: Tetsuo Handa penguin-kernel@i-love.sakura.ne.jp Signed-off-by: Dmitry Safonov dima@arista.com --- drivers/tty/tty_io.c | 9 +++++++-- 1 file changed, 7 insertions(+), 2 deletions(-)
diff --git a/drivers/tty/tty_io.c b/drivers/tty/tty_io.c index ee80dfbd5442..f73d8fa7f02b 100644 --- a/drivers/tty/tty_io.c +++ b/drivers/tty/tty_io.c @@ -1268,15 +1268,20 @@ static int tty_reopen(struct tty_struct *tty) if (test_bit(TTY_EXCLUSIVE, &tty->flags) && !capable(CAP_SYS_ADMIN)) return -EBUSY;
- tty->count++; + retval = tty_ldisc_lock(tty, 5 * HZ); + if (retval) + return retval;
+ tty->count++; if (tty->ldisc) - return 0; + goto out_unlock;
retval = tty_ldisc_reinit(tty, tty->termios.c_line); if (retval) tty->count--;
+out_unlock: + tty_ldisc_unlock(tty); return retval; }
Hi,
On Thu, Nov 01, 2018 at 12:24:47AM +0000, Dmitry Safonov wrote:
tty_ldisc_reinit() doesn't race with neither tty_ldisc_hangup() nor set_ldisc() nor tty_ldisc_release() as they use tty lock. But it races with anyone who expects line discipline to be the same after hoding read semaphore in tty_ldisc_ref().
We've seen the following crash on v4.9.108 stable:
BUG: unable to handle kernel paging request at 0000000000002260 IP: [..] n_tty_receive_buf_common+0x5f/0x86d Workqueue: events_unbound flush_to_ldisc Call Trace: [..] n_tty_receive_buf2 [..] tty_ldisc_receive_buf [..] flush_to_ldisc [..] process_one_work [..] worker_thread [..] kthread [..] ret_from_fork
tty_ldisc_reinit() should be called with ldisc_sem hold for writing, which will protect any reader against line discipline changes.
Cc: Greg Kroah-Hartman gregkh@linuxfoundation.org Cc: Jiri Slaby jslaby@suse.com Cc: stable@vger.kernel.org # b027e2298bd5 ("tty: fix data race between tty_init_dev and flush of buf") Reviewed-by: Jiri Slaby jslaby@suse.cz Reported-by: syzbot+3aa9784721dfb90e984d@syzkaller.appspotmail.com Tested-by: Mark Rutland mark.rutland@arm.com Tested-by: Tetsuo Handa penguin-kernel@i-love.sakura.ne.jp
Feel free to add
Tested-by: Tycho Andersen tycho@tycho.ws
to this as well. We've recently seen this bug (well, the one that syzbot reported), and this patch fixes it.
Tycho
Hello Mikulas,
Any chance you could re-test these v6 patches? It seems you're able to relatively easily trigger the bug on your (pa-risc) environment.. I'd be good to have this series finally acked/merged!
Thanks a lot,
-- Pasi
On Thu, Nov 01, 2018 at 12:24:45AM +0000, Dmitry Safonov wrote:
Hi all,
Here are fixes that worth to have in the @stable, as they were hit by different people, including Arista on v4.9 stable.
And for linux-next - adding lockdep asserts for line discipline changing code, verifying that write ldisc sem will be held forthwith.
Mikulas, can you add your tested-by on this patches set again, please? I tried to reproduce reboot issue on qemu-hppa and even built cross-compiler for pa-risc.. but was unlucky in reproducing.
Thanks, Dima
Changes since v5:
- Better commit tags
- Hopefully fixed issue with reboot on pa-risc with Debian 5
Changes since v4:
- back to lock ldisc with (5*HZ) timeout in tty_reopen() (LKP report link: lkml.kernel.org/r/1536940609.3185.29.camel@arista.com)
- reordered 3/7 with 2/7 for LKP robot
Changes since v3:
- Added tested-by Mark Rutland (thanks!)
- Dropped patch with smp_wmb() - wrong idea
- lockdep_assert_held() should be actually lockdep_assert_held_exclusive()
- Described why tty_ldisc_open() can be called without ldisc_sem held for pty slave end (o_tty).
- Added Peter's patch for dropping self-made lockdep annotations
- Fix for a reader(s) of ldisc semaphore waiting for an active reader(s)
Changes since v2:
- Added reviewed-by tags
- Hopefully, fixed reported by 0-day issue.
- Added optional fix for wait_readers decrement
Changes since v1:
- Added tested-by/reported-by tags
- Dropped 3/4 (locking tty pair for lockdep sake), Because of that - not adding lockdep_assert_held() in tty_ldisc_open()
- Added 4/4 cleanup to inc tty->count only on success of tty_ldisc_reinit()
- lock ldisc without (5*HZ) timeout in tty_reopen()
v1 link: lkml.kernel.org/r/20180829022353.23568-1-dima@arista.com v2 link: lkml.kernel.org/r/20180903165257.29227-1-dima@arista.com v3 link: lkml.kernel.org/r/20180911014821.26286-1-dima@arista.com v4 link: lkml.kernel.org/r/20180912001702.18522-1-dima@arista.com
Cc: Daniel Axtens dja@axtens.net Cc: Dmitry Vyukov dvyukov@google.com Cc: Mark Rutland mark.rutland@arm.com Cc: Michael Neuling mikey@neuling.org Cc: Mikulas Patocka mpatocka@redhat.com Cc: Nathan March nathan@gt.net Cc: Pasi Kärkkäinen pasik@iki.fi Cc: Peter Hurley peter@hurleysoftware.com Cc: Peter Zijlstra peterz@infradead.org Cc: "Rong, Chen" rong.a.chen@intel.com Cc: Sergey Senozhatsky sergey.senozhatsky.work@gmail.com Cc: Tan Xiaojun tanxiaojun@huawei.com Cc: Tetsuo Handa penguin-kernel@I-love.SAKURA.ne.jp (please, ignore if I Cc'ed you mistakenly)
Dmitry Safonov (6): tty/ldsem: Wake up readers after timed out down_write() tty: Hold tty_ldisc_lock() during tty_reopen() tty: Don't block on IO when ldisc change is pending tty: Simplify tty->count math in tty_reopen() tty/ldsem: Add lockdep asserts for ldisc_sem tty/ldsem: Decrement wait_readers on timeouted down_read()
Peter Zijlstra (1): tty/ldsem: Convert to regular lockdep annotations
drivers/tty/n_hdlc.c | 4 +-- drivers/tty/n_r3964.c | 2 +- drivers/tty/n_tty.c | 8 +++--- drivers/tty/tty_io.c | 14 ++++++---- drivers/tty/tty_ldisc.c | 16 +++++++++++ drivers/tty/tty_ldsem.c | 62 +++++++++++++++++------------------------ include/linux/tty.h | 7 +++++ 7 files changed, 63 insertions(+), 50 deletions(-)
-- 2.19.1
linux-stable-mirror@lists.linaro.org
-
Dmitry Safonov -
Pasi Kärkkäinen -
Tycho Andersen