The code cleaning transaction's lists of checkpoint buffers has a bug where it increases bh refcount only after releasing journal->j_list_lock. Thus the following race is possible:
CPU0 CPU1 jbd2_log_do_checkpoint() jbd2_journal_try_to_free_buffers() __journal_try_to_free_buffer(bh) ... while (transaction->t_checkpoint_io_list) ... if (buffer_locked(bh)) {
<-- IO completes now, buffer gets unlocked -->
spin_unlock(&journal->j_list_lock); spin_lock(&journal->j_list_lock); __jbd2_journal_remove_checkpoint(jh); spin_unlock(&journal->j_list_lock); try_to_free_buffers(page); get_bh(bh) <-- accesses freed bh
Fix the problem by grabbing bh reference before unlocking journal->j_list_lock.
Fixes: dc6e8d669cf5cb3ff84707c372c0a2a8a5e80845 Fixes: be1158cc615fd723552f0d9912087423c7cadda5 Reported-by: syzbot+7f4a27091759e2fe7453@syzkaller.appspotmail.com CC: stable@vger.kernel.org Signed-off-by: Jan Kara jack@suse.cz --- fs/jbd2/checkpoint.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)
diff --git a/fs/jbd2/checkpoint.c b/fs/jbd2/checkpoint.c index c125d662777c..26f8d7e46462 100644 --- a/fs/jbd2/checkpoint.c +++ b/fs/jbd2/checkpoint.c @@ -251,8 +251,8 @@ int jbd2_log_do_checkpoint(journal_t *journal) bh = jh2bh(jh);
if (buffer_locked(bh)) { - spin_unlock(&journal->j_list_lock); get_bh(bh); + spin_unlock(&journal->j_list_lock); wait_on_buffer(bh); /* the journal_head may have gone by now */ BUFFER_TRACE(bh, "brelse"); @@ -333,8 +333,8 @@ int jbd2_log_do_checkpoint(journal_t *journal) jh = transaction->t_checkpoint_io_list; bh = jh2bh(jh); if (buffer_locked(bh)) { - spin_unlock(&journal->j_list_lock); get_bh(bh); + spin_unlock(&journal->j_list_lock); wait_on_buffer(bh); /* the journal_head may have gone by now */ BUFFER_TRACE(bh, "brelse");
On Thu, Oct 04, 2018 at 12:46:40PM +0200, Jan Kara wrote:
The code cleaning transaction's lists of checkpoint buffers has a bug where it increases bh refcount only after releasing journal->j_list_lock. Thus the following race is possible:
CPU0 CPU1 jbd2_log_do_checkpoint() jbd2_journal_try_to_free_buffers() __journal_try_to_free_buffer(bh) ... while (transaction->t_checkpoint_io_list) ... if (buffer_locked(bh)) {
<-- IO completes now, buffer gets unlocked -->
spin_unlock(&journal->j_list_lock); spin_lock(&journal->j_list_lock); __jbd2_journal_remove_checkpoint(jh); spin_unlock(&journal->j_list_lock); try_to_free_buffers(page); get_bh(bh) <-- accesses freed bhFix the problem by grabbing bh reference before unlocking journal->j_list_lock.
Hi Jan,
nice catch. The patch looks good, you can add
Reviewed-by: Lukas Czerner lczerner@redhat.com
Btw, do you by any chance have a reproducer for this ?
-Lukas
Fixes: dc6e8d669cf5cb3ff84707c372c0a2a8a5e80845 Fixes: be1158cc615fd723552f0d9912087423c7cadda5 Reported-by: syzbot+7f4a27091759e2fe7453@syzkaller.appspotmail.com CC: stable@vger.kernel.org Signed-off-by: Jan Kara jack@suse.cz
fs/jbd2/checkpoint.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)
diff --git a/fs/jbd2/checkpoint.c b/fs/jbd2/checkpoint.c index c125d662777c..26f8d7e46462 100644 --- a/fs/jbd2/checkpoint.c +++ b/fs/jbd2/checkpoint.c @@ -251,8 +251,8 @@ int jbd2_log_do_checkpoint(journal_t *journal) bh = jh2bh(jh); if (buffer_locked(bh)) {
spin_unlock(&journal->j_list_lock); get_bh(bh);
spin_unlock(&journal->j_list_lock); wait_on_buffer(bh); /* the journal_head may have gone by now */ BUFFER_TRACE(bh, "brelse");@@ -333,8 +333,8 @@ int jbd2_log_do_checkpoint(journal_t *journal) jh = transaction->t_checkpoint_io_list; bh = jh2bh(jh); if (buffer_locked(bh)) {
spin_unlock(&journal->j_list_lock); get_bh(bh);
spin_unlock(&journal->j_list_lock); wait_on_buffer(bh); /* the journal_head may have gone by now */ BUFFER_TRACE(bh, "brelse");-- 2.16.4
On Thu 04-10-18 13:50:12, Lukas Czerner wrote:
On Thu, Oct 04, 2018 at 12:46:40PM +0200, Jan Kara wrote:
The code cleaning transaction's lists of checkpoint buffers has a bug where it increases bh refcount only after releasing journal->j_list_lock. Thus the following race is possible:
CPU0 CPU1 jbd2_log_do_checkpoint() jbd2_journal_try_to_free_buffers() __journal_try_to_free_buffer(bh) ... while (transaction->t_checkpoint_io_list) ... if (buffer_locked(bh)) {
<-- IO completes now, buffer gets unlocked -->
spin_unlock(&journal->j_list_lock); spin_lock(&journal->j_list_lock); __jbd2_journal_remove_checkpoint(jh); spin_unlock(&journal->j_list_lock); try_to_free_buffers(page); get_bh(bh) <-- accesses freed bhFix the problem by grabbing bh reference before unlocking journal->j_list_lock.
Hi Jan,
nice catch. The patch looks good, you can add
Reviewed-by: Lukas Czerner lczerner@redhat.com
Btw, do you by any chance have a reproducer for this ?
No, syzbot hit it but the race window is really small so I don't think you can create reasonably reliable reproducer...
Honza
On Thu, Oct 04, 2018 at 12:46:40PM +0200, Jan Kara wrote:
The code cleaning transaction's lists of checkpoint buffers has a bug where it increases bh refcount only after releasing journal->j_list_lock. Thus the following race is possible:
CPU0 CPU1 jbd2_log_do_checkpoint() jbd2_journal_try_to_free_buffers() __journal_try_to_free_buffer(bh) ... while (transaction->t_checkpoint_io_list) ... if (buffer_locked(bh)) {
<-- IO completes now, buffer gets unlocked -->
spin_unlock(&journal->j_list_lock); spin_lock(&journal->j_list_lock); __jbd2_journal_remove_checkpoint(jh); spin_unlock(&journal->j_list_lock); try_to_free_buffers(page); get_bh(bh) <-- accesses freed bhFix the problem by grabbing bh reference before unlocking journal->j_list_lock.
Fixes: dc6e8d669cf5cb3ff84707c372c0a2a8a5e80845 Fixes: be1158cc615fd723552f0d9912087423c7cadda5
Nit, this normally looks like:
Fixes: dc6e8d669cf5 ("jbd2: don't call get_bh() before calling __jbd2_journal_remove_checkpoint()")
And this is created by: git show -s --abbrev-commit --abbrev=12 --pretty=format:"%h ("%s")%n"
thanks
greg k-h
On Thu 04-10-18 08:44:02, Greg KH wrote:
On Thu, Oct 04, 2018 at 12:46:40PM +0200, Jan Kara wrote:
The code cleaning transaction's lists of checkpoint buffers has a bug where it increases bh refcount only after releasing journal->j_list_lock. Thus the following race is possible:
CPU0 CPU1 jbd2_log_do_checkpoint() jbd2_journal_try_to_free_buffers() __journal_try_to_free_buffer(bh) ... while (transaction->t_checkpoint_io_list) ... if (buffer_locked(bh)) {
<-- IO completes now, buffer gets unlocked -->
spin_unlock(&journal->j_list_lock); spin_lock(&journal->j_list_lock); __jbd2_journal_remove_checkpoint(jh); spin_unlock(&journal->j_list_lock); try_to_free_buffers(page); get_bh(bh) <-- accesses freed bhFix the problem by grabbing bh reference before unlocking journal->j_list_lock.
Fixes: dc6e8d669cf5cb3ff84707c372c0a2a8a5e80845 Fixes: be1158cc615fd723552f0d9912087423c7cadda5
Nit, this normally looks like:
Fixes: dc6e8d669cf5 ("jbd2: don't call get_bh() before calling __jbd2_journal_remove_checkpoint()")
And this is created by: git show -s --abbrev-commit --abbrev=12 --pretty=format:"%h ("%s")%n"
OK, ok, will do next time. The full git commit ID is faster to cut-n-paste which shows how lazy I'm ;)
Honza
On Thu, Oct 04, 2018 at 06:05:46PM +0200, Jan Kara wrote:
On Thu 04-10-18 08:44:02, Greg KH wrote:
On Thu, Oct 04, 2018 at 12:46:40PM +0200, Jan Kara wrote:
The code cleaning transaction's lists of checkpoint buffers has a bug where it increases bh refcount only after releasing journal->j_list_lock. Thus the following race is possible:
CPU0 CPU1 jbd2_log_do_checkpoint() jbd2_journal_try_to_free_buffers() __journal_try_to_free_buffer(bh) ... while (transaction->t_checkpoint_io_list) ... if (buffer_locked(bh)) {
<-- IO completes now, buffer gets unlocked -->
spin_unlock(&journal->j_list_lock); spin_lock(&journal->j_list_lock); __jbd2_journal_remove_checkpoint(jh); spin_unlock(&journal->j_list_lock); try_to_free_buffers(page); get_bh(bh) <-- accesses freed bhFix the problem by grabbing bh reference before unlocking journal->j_list_lock.
Fixes: dc6e8d669cf5cb3ff84707c372c0a2a8a5e80845 Fixes: be1158cc615fd723552f0d9912087423c7cadda5
Nit, this normally looks like:
Fixes: dc6e8d669cf5 ("jbd2: don't call get_bh() before calling __jbd2_journal_remove_checkpoint()")
And this is created by: git show -s --abbrev-commit --abbrev=12 --pretty=format:"%h ("%s")%n"
OK, ok, will do next time. The full git commit ID is faster to cut-n-paste which shows how lazy I'm ;)
I have this in the .gitconfig
[pretty] fixes = Fixes: %h ("%s")
which helps. I think I found it in the docs somewhere.
-Lukas
Honza-- Jan Kara jack@suse.com SUSE Labs, CR
On Thu 04-10-18 17:22:18, Theodore Y. Ts'o wrote:
On Thu, Oct 04, 2018 at 06:05:46PM +0200, Jan Kara wrote:
OK, ok, will do next time. The full git commit ID is faster to cut-n-paste which shows how lazy I'm ;)
No worires, I'll fix that up for you when I apply it.
Thanks!
Honza
On Thu, Oct 04, 2018 at 12:46:40PM +0200, Jan Kara wrote:
The code cleaning transaction's lists of checkpoint buffers has a bug where it increases bh refcount only after releasing journal->j_list_lock. Thus the following race is possible:
CPU0 CPU1 jbd2_log_do_checkpoint() jbd2_journal_try_to_free_buffers() __journal_try_to_free_buffer(bh) ... while (transaction->t_checkpoint_io_list) ... if (buffer_locked(bh)) {
<-- IO completes now, buffer gets unlocked -->
spin_unlock(&journal->j_list_lock); spin_lock(&journal->j_list_lock); __jbd2_journal_remove_checkpoint(jh); spin_unlock(&journal->j_list_lock); try_to_free_buffers(page); get_bh(bh) <-- accesses freed bhFix the problem by grabbing bh reference before unlocking journal->j_list_lock.
Fixes: dc6e8d669cf5cb3ff84707c372c0a2a8a5e80845 Fixes: be1158cc615fd723552f0d9912087423c7cadda5 Reported-by: syzbot+7f4a27091759e2fe7453@syzkaller.appspotmail.com CC: stable@vger.kernel.org Signed-off-by: Jan Kara jack@suse.cz
Applied, thanks (with the Fixes field adjusted).
- Ted
linux-stable-mirror@lists.linaro.org
-
Greg KH -
Jan Kara -
Lukas Czerner -
Theodore Y. Ts'o