November 2022 - Linux-stable-mirror

[PATCH v2 1/2] mm/migrate: Fix read-only page got writable when recover pte

by Peter Xu

Ives van Hoorne from codesandbox.io reported an issue regarding possible data loss of uffd-wp when applied to memfds on heavily loaded systems. The sympton is some read page got data mismatch from the snapshot child VMs. Here I can also reproduce with a Rust reproducer that was provided by Ives that keeps taking snapshot of a 256MB VM, on a 32G system when I initiate 80 instances I can trigger the issues in ten minutes. It turns out that we got some pages write-through even if uffd-wp is applied to the pte. The problem is, when removing migration entries, we didn't really worry about write bit as long as we know it's not a write migration entry. That may not be true, for some memory types (e.g. writable shmem) mk_pte can return a pte with write bit set, then to recover the migration entry to its original state we need to explicit wr-protect the pte or it'll has the write bit set if it's a read migration entry. For uffd it can cause write-through. I didn't verify, but I think it'll be the same for mprotect()ed pages and after migration we can miss the sigbus instead. The relevant code on uffd was introduced in the anon support, which is commit f45ec5ff16a7 ("userfaultfd: wp: support swap and page migration", 2020-04-07). However anon shouldn't suffer from this problem because anon should already have the write bit cleared always, so that may not be a proper Fixes target. To satisfy the need on the backport, I'm attaching the Fixes tag to the uffd-wp shmem support. Since no one had issue with mprotect, so I assume that's also the kernel version we should start to backport for stable, and we shouldn't need to worry before that. Cc: Andrea Arcangeli <aarcange(a)redhat.com> Cc: stable(a)vger.kernel.org Fixes: b1f9e876862d ("mm/uffd: enable write protection for shmem & hugetlbfs") Reported-by: Ives van Hoorne <ives(a)codesandbox.io> Signed-off-by: Peter Xu <peterx(a)redhat.com> --- mm/migrate.c | 8 +++++++- 1 file changed, 7 insertions(+), 1 deletion(-) diff --git a/mm/migrate.c b/mm/migrate.c index dff333593a8a..8b6351c08c78 100644 --- a/mm/migrate.c +++ b/mm/migrate.c @@ -213,8 +213,14 @@ static bool remove_migration_pte(struct folio *folio, pte = pte_mkdirty(pte); if (is_writable_migration_entry(entry)) pte = maybe_mkwrite(pte, vma); - else if (pte_swp_uffd_wp(*pvmw.pte)) + else + /* NOTE: mk_pte can have write bit set */ + pte = pte_wrprotect(pte); + + if (pte_swp_uffd_wp(*pvmw.pte)) { + WARN_ON_ONCE(pte_write(pte)); pte = pte_mkuffd_wp(pte); + } if (folio_test_anon(folio) && !is_readable_migration_entry(entry)) rmap_flags |= RMAP_EXCLUSIVE; -- 2.37.3

3 years, 1 month

4
15
0 0

[PATCH v2] spi: spi-imx: Fix spi_bus_clk if requested clock is higher than input clock

by Frieder Schrempf

From: Frieder Schrempf <frieder.schrempf(a)kontron.de> In case the requested bus clock is higher than the input clock, the correct dividers (pre = 0, post = 0) are returned from mx51_ecspi_clkdiv(), but *fres is left uninitialized and therefore contains an arbitrary value. This causes trouble for the recently introduced PIO polling feature as the value in spi_imx->spi_bus_clk is used there to calculate for which transfers to enable PIO polling. Fix this by setting *fres even if no clock dividers are in use. This issue was observed on Kontron BL i.MX8MM with an SPI peripheral clock set to 50 MHz by default and a requested SPI bus clock of 80 MHz for the SPI NOR flash. With the fix applied the debug message from mx51_ecspi_clkdiv() now prints the following: spi_imx 30820000.spi: mx51_ecspi_clkdiv: fin: 50000000, fspi: 50000000, post: 0, pre: 0 Fixes: 07e759387788 ("spi: spi-imx: add PIO polling support") Cc: Marc Kleine-Budde <mkl(a)pengutronix.de> Cc: David Jander <david(a)protonic.nl> Cc: Fabio Estevam <festevam(a)gmail.com> Cc: Mark Brown <broonie(a)kernel.org> Cc: Marek Vasut <marex(a)denx.de> Cc: stable(a)vger.kernel.org Signed-off-by: Frieder Schrempf <frieder.schrempf(a)kontron.de> --- Changes for v2: * Remove the reference and the Fixes tag for commit 6fd8b8503a0d as it is incorrect. --- drivers/spi/spi-imx.c | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/drivers/spi/spi-imx.c b/drivers/spi/spi-imx.c index 30d82cc7300b..468ce0a2b282 100644 --- a/drivers/spi/spi-imx.c +++ b/drivers/spi/spi-imx.c @@ -444,8 +444,7 @@ static unsigned int mx51_ecspi_clkdiv(struct spi_imx_data *spi_imx, unsigned int pre, post; unsigned int fin = spi_imx->spi_clk; - if (unlikely(fspi > fin)) - return 0; + fspi = min(fspi, fin); post = fls(fin) - fls(fspi); if (fin > fspi << post) -- 2.38.1

3 years, 1 month

4
4
0 0

[PATCH 5.15 1/3] bitops: always define asm-generic non-atomic bitops

by Mikulas Patocka

commit 21bb8af513d35c005c401706030f4eb469538d1d upstream. Move generic non-atomic bitops from the asm-generic header which gets included only when there are no architecture-specific alternatives, to a separate independent file to make them always available. Almost no actual code changes, only one comment added to generic_test_bit() saying that it's an atomic operation itself and thus `volatile` must always stay there with no cast-aways. Suggested-by: Andy Shevchenko <andriy.shevchenko(a)linux.intel.com> # comment Suggested-by: Marco Elver <elver(a)google.com> # reference to kernel-doc Signed-off-by: Alexander Lobakin <alexandr.lobakin(a)intel.com> Reviewed-by: Andy Shevchenko <andriy.shevchenko(a)linux.intel.com> Reviewed-by: Marco Elver <elver(a)google.com> Signed-off-by: Yury Norov <yury.norov(a)gmail.com> --- include/asm-generic/bitops/generic-non-atomic.h | 130 ++++++++++++++++++++++++ include/asm-generic/bitops/non-atomic.h | 110 +------------------- 2 files changed, 138 insertions(+), 102 deletions(-) Index: linux-stable/include/asm-generic/bitops/generic-non-atomic.h =================================================================== --- /dev/null 1970-01-01 00:00:00.000000000 +0000 +++ linux-stable/include/asm-generic/bitops/generic-non-atomic.h 2022-11-14 22:17:30.000000000 +0100 @@ -0,0 +1,130 @@ +/* SPDX-License-Identifier: GPL-2.0-only */ + +#ifndef __ASM_GENERIC_BITOPS_GENERIC_NON_ATOMIC_H +#define __ASM_GENERIC_BITOPS_GENERIC_NON_ATOMIC_H + +#include <linux/bits.h> + +#ifndef _LINUX_BITOPS_H +#error only <linux/bitops.h> can be included directly +#endif + +/* + * Generic definitions for bit operations, should not be used in regular code + * directly. + */ + +/** + * generic___set_bit - Set a bit in memory + * @nr: the bit to set + * @addr: the address to start counting from + * + * Unlike set_bit(), this function is non-atomic and may be reordered. + * If it's called on the same region of memory simultaneously, the effect + * may be that only one operation succeeds. + */ +static __always_inline void +generic___set_bit(unsigned int nr, volatile unsigned long *addr) +{ + unsigned long mask = BIT_MASK(nr); + unsigned long *p = ((unsigned long *)addr) + BIT_WORD(nr); + + *p |= mask; +} + +static __always_inline void +generic___clear_bit(unsigned int nr, volatile unsigned long *addr) +{ + unsigned long mask = BIT_MASK(nr); + unsigned long *p = ((unsigned long *)addr) + BIT_WORD(nr); + + *p &= ~mask; +} + +/** + * generic___change_bit - Toggle a bit in memory + * @nr: the bit to change + * @addr: the address to start counting from + * + * Unlike change_bit(), this function is non-atomic and may be reordered. + * If it's called on the same region of memory simultaneously, the effect + * may be that only one operation succeeds. + */ +static __always_inline +void generic___change_bit(unsigned int nr, volatile unsigned long *addr) +{ + unsigned long mask = BIT_MASK(nr); + unsigned long *p = ((unsigned long *)addr) + BIT_WORD(nr); + + *p ^= mask; +} + +/** + * generic___test_and_set_bit - Set a bit and return its old value + * @nr: Bit to set + * @addr: Address to count from + * + * This operation is non-atomic and can be reordered. + * If two examples of this operation race, one can appear to succeed + * but actually fail. You must protect multiple accesses with a lock. + */ +static __always_inline int +generic___test_and_set_bit(unsigned int nr, volatile unsigned long *addr) +{ + unsigned long mask = BIT_MASK(nr); + unsigned long *p = ((unsigned long *)addr) + BIT_WORD(nr); + unsigned long old = *p; + + *p = old | mask; + return (old & mask) != 0; +} + +/** + * generic___test_and_clear_bit - Clear a bit and return its old value + * @nr: Bit to clear + * @addr: Address to count from + * + * This operation is non-atomic and can be reordered. + * If two examples of this operation race, one can appear to succeed + * but actually fail. You must protect multiple accesses with a lock. + */ +static __always_inline int +generic___test_and_clear_bit(unsigned int nr, volatile unsigned long *addr) +{ + unsigned long mask = BIT_MASK(nr); + unsigned long *p = ((unsigned long *)addr) + BIT_WORD(nr); + unsigned long old = *p; + + *p = old & ~mask; + return (old & mask) != 0; +} + +/* WARNING: non atomic and it can be reordered! */ +static __always_inline int +generic___test_and_change_bit(unsigned int nr, volatile unsigned long *addr) +{ + unsigned long mask = BIT_MASK(nr); + unsigned long *p = ((unsigned long *)addr) + BIT_WORD(nr); + unsigned long old = *p; + + *p = old ^ mask; + return (old & mask) != 0; +} + +/** + * generic_test_bit - Determine whether a bit is set + * @nr: bit number to test + * @addr: Address to start counting from + */ +static __always_inline int +generic_test_bit(unsigned int nr, const volatile unsigned long *addr) +{ + /* + * Unlike the bitops with the '__' prefix above, this one *is* atomic, + * so `volatile` must always stay here with no cast-aways. See + * `Documentation/atomic_bitops.txt` for the details. + */ + return 1UL & (addr[BIT_WORD(nr)] >> (nr & (BITS_PER_LONG-1))); +} + +#endif /* __ASM_GENERIC_BITOPS_GENERIC_NON_ATOMIC_H */ Index: linux-stable/include/asm-generic/bitops/non-atomic.h =================================================================== --- linux-stable.orig/include/asm-generic/bitops/non-atomic.h 2022-11-14 22:17:30.000000000 +0100 +++ linux-stable/include/asm-generic/bitops/non-atomic.h 2022-11-14 22:17:30.000000000 +0100 @@ -2,121 +2,27 @@ #ifndef _ASM_GENERIC_BITOPS_NON_ATOMIC_H_ #define _ASM_GENERIC_BITOPS_NON_ATOMIC_H_ -#include <asm/types.h> +#include <asm-generic/bitops/generic-non-atomic.h> -/** - * arch___set_bit - Set a bit in memory - * @nr: the bit to set - * @addr: the address to start counting from - * - * Unlike set_bit(), this function is non-atomic and may be reordered. - * If it's called on the same region of memory simultaneously, the effect - * may be that only one operation succeeds. - */ -static __always_inline void -arch___set_bit(unsigned int nr, volatile unsigned long *addr) -{ - unsigned long mask = BIT_MASK(nr); - unsigned long *p = ((unsigned long *)addr) + BIT_WORD(nr); - - *p |= mask; -} +#define arch___set_bit generic___set_bit #define __set_bit arch___set_bit -static __always_inline void -arch___clear_bit(unsigned int nr, volatile unsigned long *addr) -{ - unsigned long mask = BIT_MASK(nr); - unsigned long *p = ((unsigned long *)addr) + BIT_WORD(nr); - - *p &= ~mask; -} +#define arch___clear_bit generic___clear_bit #define __clear_bit arch___clear_bit -/** - * arch___change_bit - Toggle a bit in memory - * @nr: the bit to change - * @addr: the address to start counting from - * - * Unlike change_bit(), this function is non-atomic and may be reordered. - * If it's called on the same region of memory simultaneously, the effect - * may be that only one operation succeeds. - */ -static __always_inline -void arch___change_bit(unsigned int nr, volatile unsigned long *addr) -{ - unsigned long mask = BIT_MASK(nr); - unsigned long *p = ((unsigned long *)addr) + BIT_WORD(nr); - - *p ^= mask; -} +#define arch___change_bit generic___change_bit #define __change_bit arch___change_bit -/** - * arch___test_and_set_bit - Set a bit and return its old value - * @nr: Bit to set - * @addr: Address to count from - * - * This operation is non-atomic and can be reordered. - * If two examples of this operation race, one can appear to succeed - * but actually fail. You must protect multiple accesses with a lock. - */ -static __always_inline int -arch___test_and_set_bit(unsigned int nr, volatile unsigned long *addr) -{ - unsigned long mask = BIT_MASK(nr); - unsigned long *p = ((unsigned long *)addr) + BIT_WORD(nr); - unsigned long old = *p; - - *p = old | mask; - return (old & mask) != 0; -} +#define arch___test_and_set_bit generic___test_and_set_bit #define __test_and_set_bit arch___test_and_set_bit -/** - * arch___test_and_clear_bit - Clear a bit and return its old value - * @nr: Bit to clear - * @addr: Address to count from - * - * This operation is non-atomic and can be reordered. - * If two examples of this operation race, one can appear to succeed - * but actually fail. You must protect multiple accesses with a lock. - */ -static __always_inline int -arch___test_and_clear_bit(unsigned int nr, volatile unsigned long *addr) -{ - unsigned long mask = BIT_MASK(nr); - unsigned long *p = ((unsigned long *)addr) + BIT_WORD(nr); - unsigned long old = *p; - - *p = old & ~mask; - return (old & mask) != 0; -} +#define arch___test_and_clear_bit generic___test_and_clear_bit #define __test_and_clear_bit arch___test_and_clear_bit -/* WARNING: non atomic and it can be reordered! */ -static __always_inline int -arch___test_and_change_bit(unsigned int nr, volatile unsigned long *addr) -{ - unsigned long mask = BIT_MASK(nr); - unsigned long *p = ((unsigned long *)addr) + BIT_WORD(nr); - unsigned long old = *p; - - *p = old ^ mask; - return (old & mask) != 0; -} +#define arch___test_and_change_bit generic___test_and_change_bit #define __test_and_change_bit arch___test_and_change_bit -/** - * arch_test_bit - Determine whether a bit is set - * @nr: bit number to test - * @addr: Address to start counting from - */ -static __always_inline int -arch_test_bit(unsigned int nr, const volatile unsigned long *addr) -{ - return 1UL & (addr[BIT_WORD(nr)] >> (nr & (BITS_PER_LONG-1))); -} +#define arch_test_bit generic_test_bit #define test_bit arch_test_bit #endif /* _ASM_GENERIC_BITOPS_NON_ATOMIC_H_ */

3 years, 1 month

2
1
0 0

[PATCH 4.9 3/3] provide arch_test_bit_acquire for architectures that define test_bit

by Mikulas Patocka

commit d6ffe6067a54972564552ea45d320fb98db1ac5e upstream. Some architectures define their own arch_test_bit and they also need arch_test_bit_acquire, otherwise they won't compile. We also clean up the code by using the generic test_bit if that is equivalent to the arch-specific version. Signed-off-by: Mikulas Patocka <mpatocka(a)redhat.com> Cc: stable(a)vger.kernel.org Fixes: 8238b4579866 ("wait_on_bit: add an acquire memory barrier") Signed-off-by: Linus Torvalds <torvalds(a)linux-foundation.org> --- arch/alpha/include/asm/bitops.h | 8 +++----- arch/arc/include/asm/bitops.h | 17 +++-------------- arch/frv/include/asm/bitops.h | 26 +++----------------------- arch/h8300/include/asm/bitops.h | 3 ++- arch/hexagon/include/asm/bitops.h | 15 +++++++++++++++ arch/ia64/include/asm/bitops.h | 8 +++----- arch/m68k/include/asm/bitops.h | 7 +++---- arch/mn10300/include/asm/bitops.h | 10 +++------- arch/s390/include/asm/bitops.h | 11 +++-------- arch/sh/include/asm/bitops-op32.h | 11 ++--------- 10 files changed, 40 insertions(+), 76 deletions(-) Index: linux-stable/arch/alpha/include/asm/bitops.h =================================================================== --- linux-stable.orig/arch/alpha/include/asm/bitops.h 2022-11-14 22:15:18.000000000 +0100 +++ linux-stable/arch/alpha/include/asm/bitops.h 2022-11-14 22:15:18.000000000 +0100 @@ -7,6 +7,7 @@ #include <asm/compiler.h> #include <asm/barrier.h> +#include <asm-generic/bitops/generic-non-atomic.h> /* * Copyright 1994, Linus Torvalds. @@ -282,11 +283,8 @@ __test_and_change_bit(unsigned long nr, return (old & mask) != 0; } -static inline int -test_bit(int nr, const volatile void * addr) -{ - return (1UL & (((const int *) addr)[nr >> 5] >> (nr & 31))) != 0UL; -} +#define test_bit generic_test_bit +#define test_bit_acquire generic_test_bit_acquire /* * ffz = Find First Zero in word. Undefined if no zero exists, Index: linux-stable/arch/hexagon/include/asm/bitops.h =================================================================== --- linux-stable.orig/arch/hexagon/include/asm/bitops.h 2022-11-14 22:15:18.000000000 +0100 +++ linux-stable/arch/hexagon/include/asm/bitops.h 2022-11-14 22:15:18.000000000 +0100 @@ -186,7 +186,22 @@ static inline int __test_bit(int nr, con return retval; } +static inline int __test_bit_acquire(int nr, const volatile unsigned long *addr) +{ + int retval; + + asm volatile( + "{P0 = tstbit(%1,%2); if (P0.new) %0 = #1; if (!P0.new) %0 = #0;}\n" + : "=&r" (retval) + : "r" (addr[BIT_WORD(nr)]), "r" (nr % BITS_PER_LONG) + : "p0", "memory" + ); + + return retval; +} + #define test_bit(nr, addr) __test_bit(nr, addr) +#define test_bit_acquire(nr, addr) __test_bit_acquire(nr, addr) /* * ffz - find first zero in word. Index: linux-stable/arch/ia64/include/asm/bitops.h =================================================================== --- linux-stable.orig/arch/ia64/include/asm/bitops.h 2022-11-14 22:15:18.000000000 +0100 +++ linux-stable/arch/ia64/include/asm/bitops.h 2022-11-14 22:15:18.000000000 +0100 @@ -17,6 +17,7 @@ #include <linux/types.h> #include <asm/intrinsics.h> #include <asm/barrier.h> +#include <asm-generic/bitops/generic-non-atomic.h> /** * set_bit - Atomically set a bit in memory @@ -330,11 +331,8 @@ __test_and_change_bit (int nr, void *add return (old & bit) != 0; } -static __inline__ int -test_bit (int nr, const volatile void *addr) -{ - return 1 & (((const volatile __u32 *) addr)[nr >> 5] >> (nr & 31)); -} +#define test_bit generic_test_bit +#define test_bit_acquire generic_test_bit_acquire /** * ffz - find the first zero bit in a long word Index: linux-stable/arch/m68k/include/asm/bitops.h =================================================================== --- linux-stable.orig/arch/m68k/include/asm/bitops.h 2022-11-14 22:15:18.000000000 +0100 +++ linux-stable/arch/m68k/include/asm/bitops.h 2022-11-14 22:15:18.000000000 +0100 @@ -14,6 +14,7 @@ #include <linux/compiler.h> #include <asm/barrier.h> +#include <asm-generic/bitops/generic-non-atomic.h> /* * Bit access functions vary across the ColdFire and 68k families. @@ -148,10 +149,8 @@ static inline void bfchg_mem_change_bit( #define __change_bit(nr, vaddr) change_bit(nr, vaddr) -static inline int test_bit(int nr, const unsigned long *vaddr) -{ - return (vaddr[nr >> 5] & (1UL << (nr & 31))) != 0; -} +#define test_bit generic_test_bit +#define test_bit_acquire generic_test_bit_acquire static inline int bset_reg_test_and_set_bit(int nr, Index: linux-stable/arch/s390/include/asm/bitops.h =================================================================== --- linux-stable.orig/arch/s390/include/asm/bitops.h 2022-11-14 22:15:18.000000000 +0100 +++ linux-stable/arch/s390/include/asm/bitops.h 2022-11-14 22:15:18.000000000 +0100 @@ -43,6 +43,7 @@ #include <linux/typecheck.h> #include <linux/compiler.h> #include <asm/barrier.h> +#include <asm-generic/bitops/generic-non-atomic.h> #define __BITOPS_NO_BARRIER "\n" @@ -261,14 +262,8 @@ __test_and_change_bit(unsigned long nr, return (ch >> (nr & 7)) & 1; } -static inline int test_bit(unsigned long nr, const volatile unsigned long *ptr) -{ - const volatile unsigned char *addr; - - addr = ((const volatile unsigned char *)ptr); - addr += (nr ^ (BITS_PER_LONG - 8)) >> 3; - return (*addr >> (nr & 7)) & 1; -} +#define test_bit generic_test_bit +#define test_bit_acquire generic_test_bit_acquire static inline int test_and_set_bit_lock(unsigned long nr, volatile unsigned long *ptr) Index: linux-stable/arch/sh/include/asm/bitops-op32.h =================================================================== --- linux-stable.orig/arch/sh/include/asm/bitops-op32.h 2022-11-14 22:15:18.000000000 +0100 +++ linux-stable/arch/sh/include/asm/bitops-op32.h 2022-11-14 22:15:18.000000000 +0100 @@ -129,14 +129,7 @@ static inline int __test_and_change_bit( return (old & mask) != 0; } -/** - * test_bit - Determine whether a bit is set - * @nr: bit number to test - * @addr: Address to start counting from - */ -static inline int test_bit(int nr, const volatile unsigned long *addr) -{ - return 1UL & (addr[BIT_WORD(nr)] >> (nr & (BITS_PER_LONG-1))); -} +#define test_bit generic_test_bit +#define test_bit_acquire generic_test_bit_acquire #endif /* __ASM_SH_BITOPS_OP32_H */ Index: linux-stable/arch/h8300/include/asm/bitops.h =================================================================== --- linux-stable.orig/arch/h8300/include/asm/bitops.h 2022-11-14 22:15:18.000000000 +0100 +++ linux-stable/arch/h8300/include/asm/bitops.h 2022-11-14 22:15:18.000000000 +0100 @@ -88,7 +88,8 @@ static inline int test_bit(int nr, const return ret; } -#define __test_bit(nr, addr) test_bit(nr, addr) +#define __test_bit(nr, addr) test_bit(nr, addr) +#define test_bit_acquire(nr, addr) test_bit(nr, addr) #define H8300_GEN_TEST_BITOP(FNNAME, OP) \ static inline int FNNAME(int nr, void *addr) \ Index: linux-stable/arch/arc/include/asm/bitops.h =================================================================== --- linux-stable.orig/arch/arc/include/asm/bitops.h 2022-11-14 22:15:18.000000000 +0100 +++ linux-stable/arch/arc/include/asm/bitops.h 2022-11-14 22:15:18.000000000 +0100 @@ -21,6 +21,7 @@ #ifndef CONFIG_ARC_HAS_LLSC #include <asm/smp.h> #endif +#include <asm-generic/bitops/generic-non-atomic.h> #ifdef CONFIG_ARC_HAS_LLSC @@ -239,20 +240,8 @@ BIT_OPS(clear, & ~, CTOP_INST_AAND_DI_R2 BIT_OPS(change, ^, CTOP_INST_AXOR_DI_R2_R2_R3) #endif -/* - * This routine doesn't need to be atomic. - */ -static inline int -test_bit(unsigned int nr, const volatile unsigned long *addr) -{ - unsigned long mask; - - addr += nr >> 5; - - mask = 1UL << (nr & 0x1f); - - return ((mask & *addr) != 0); -} +#define test_bit generic_test_bit +#define test_bit_acquire generic_test_bit_acquire #ifdef CONFIG_ISA_ARCOMPACT Index: linux-stable/arch/frv/include/asm/bitops.h =================================================================== --- linux-stable.orig/arch/frv/include/asm/bitops.h 2022-11-14 22:15:18.000000000 +0100 +++ linux-stable/arch/frv/include/asm/bitops.h 2022-11-14 22:15:18.000000000 +0100 @@ -23,6 +23,7 @@ #error only <linux/bitops.h> can be included directly #endif +#include <asm-generic/bitops/generic-non-atomic.h> #include <asm-generic/bitops/ffz.h> #include <asm/atomic.h> @@ -132,29 +133,8 @@ static inline int __test_and_change_bit( return retval; } -/* - * This routine doesn't need to be atomic. - */ -static inline int -__constant_test_bit(unsigned long nr, const volatile void *addr) -{ - return ((1UL << (nr & 31)) & (((const volatile unsigned int *) addr)[nr >> 5])) != 0; -} - -static inline int __test_bit(unsigned long nr, const volatile void *addr) -{ - int * a = (int *) addr; - int mask; - - a += nr >> 5; - mask = 1 << (nr & 0x1f); - return ((mask & *a) != 0); -} - -#define test_bit(nr,addr) \ -(__builtin_constant_p(nr) ? \ - __constant_test_bit((nr),(addr)) : \ - __test_bit((nr),(addr))) +#define test_bit generic_test_bit +#define test_bit_acquire generic_test_bit_acquire #include <asm-generic/bitops/find.h> Index: linux-stable/arch/mn10300/include/asm/bitops.h =================================================================== --- linux-stable.orig/arch/mn10300/include/asm/bitops.h 2022-11-14 22:15:18.000000000 +0100 +++ linux-stable/arch/mn10300/include/asm/bitops.h 2022-11-14 22:15:18.000000000 +0100 @@ -19,6 +19,7 @@ #include <asm/cpu-regs.h> #include <asm/barrier.h> +#include <asm-generic/bitops/generic-non-atomic.h> /* * set bit @@ -65,13 +66,8 @@ static inline void __clear_bit(unsigned *a &= ~mask; } -/* - * test bit - */ -static inline int test_bit(unsigned long nr, const volatile void *addr) -{ - return 1UL & (((const volatile unsigned int *) addr)[nr >> 5] >> (nr & 31)); -} +#define test_bit generic_test_bit +#define test_bit_acquire generic_test_bit_acquire /* * change bit

3 years, 1 month

1
0
0 0

[PATCH 4.9 2/3] wait_on_bit: add an acquire memory barrier

by Mikulas Patocka

commit 8238b4579866b7c1bb99883cfe102a43db5506ff upstream. There are several places in the kernel where wait_on_bit is not followed by a memory barrier (for example, in drivers/md/dm-bufio.c:new_read). On architectures with weak memory ordering, it may happen that memory accesses that follow wait_on_bit are reordered before wait_on_bit and they may return invalid data. Fix this class of bugs by introducing a new function "test_bit_acquire" that works like test_bit, but has acquire memory ordering semantics. Signed-off-by: Mikulas Patocka <mpatocka(a)redhat.com> Acked-by: Will Deacon <will(a)kernel.org> Cc: stable(a)vger.kernel.org Signed-off-by: Linus Torvalds <torvalds(a)linux-foundation.org> --- arch/x86/include/asm/bitops.h | 21 +++++++++++++++++++++ include/asm-generic/bitops/generic-non-atomic.h | 13 +++++++++++++ include/asm-generic/bitops/non-atomic.h | 14 ++++++++++++++ include/linux/buffer_head.h | 2 +- include/linux/wait.h | 8 ++++---- kernel/sched/wait.c | 2 +- 6 files changed, 54 insertions(+), 6 deletions(-) Index: linux-stable/arch/x86/include/asm/bitops.h =================================================================== --- linux-stable.orig/arch/x86/include/asm/bitops.h 2022-11-14 22:15:17.000000000 +0100 +++ linux-stable/arch/x86/include/asm/bitops.h 2022-11-14 22:15:17.000000000 +0100 @@ -314,6 +314,20 @@ static __always_inline bool constant_tes (addr[nr >> _BITOPS_LONG_SHIFT])) != 0; } +static __always_inline bool constant_test_bit_acquire(long nr, const volatile unsigned long *addr) +{ + bool oldbit; + + asm volatile("testb %2,%1" + CC_SET(nz) + : CC_OUT(nz) (oldbit) + : "m" (((unsigned char *)addr)[nr >> 3]), + "i" (1 << (nr & 7)) + :"memory"); + + return oldbit; +} + static __always_inline bool variable_test_bit(long nr, volatile const unsigned long *addr) { bool oldbit; @@ -340,6 +354,13 @@ static bool test_bit(int nr, const volat ? constant_test_bit((nr), (addr)) \ : variable_test_bit((nr), (addr))) +static __always_inline bool +test_bit_acquire(unsigned long nr, const volatile unsigned long *addr) +{ + return __builtin_constant_p(nr) ? constant_test_bit_acquire(nr, addr) : + variable_test_bit(nr, addr); +} + /** * __ffs - find first set bit in word * @word: The word to search Index: linux-stable/include/asm-generic/bitops/non-atomic.h =================================================================== --- linux-stable.orig/include/asm-generic/bitops/non-atomic.h 2022-11-14 22:15:17.000000000 +0100 +++ linux-stable/include/asm-generic/bitops/non-atomic.h 2022-11-14 22:15:17.000000000 +0100 @@ -2,6 +2,7 @@ #define _ASM_GENERIC_BITOPS_NON_ATOMIC_H_ #include <asm/types.h> +#include <asm/barrier.h> /** * __set_bit - Set a bit in memory @@ -105,4 +106,17 @@ static inline int test_bit(int nr, const return 1UL & (addr[BIT_WORD(nr)] >> (nr & (BITS_PER_LONG-1))); } +/** + * arch_test_bit_acquire - Determine, with acquire semantics, whether a bit is set + * @nr: bit number to test + * @addr: Address to start counting from + */ +static __always_inline bool +arch_test_bit_acquire(unsigned long nr, const volatile unsigned long *addr) +{ + unsigned long *p = ((unsigned long *)addr) + BIT_WORD(nr); + return 1UL & (smp_load_acquire(p) >> (nr & (BITS_PER_LONG-1))); +} +#define test_bit_acquire arch_test_bit_acquire + #endif /* _ASM_GENERIC_BITOPS_NON_ATOMIC_H_ */ Index: linux-stable/include/linux/buffer_head.h =================================================================== --- linux-stable.orig/include/linux/buffer_head.h 2022-11-14 22:15:17.000000000 +0100 +++ linux-stable/include/linux/buffer_head.h 2022-11-14 22:15:17.000000000 +0100 @@ -162,7 +162,7 @@ static __always_inline int buffer_uptoda * make it consistent with folio_test_uptodate * pairs with smp_mb__before_atomic in set_buffer_uptodate */ - return (smp_load_acquire(&bh->b_state) & (1UL << BH_Uptodate)) != 0; + return test_bit_acquire(BH_Uptodate, &bh->b_state); } #define bh_offset(bh) ((unsigned long)(bh)->b_data & ~PAGE_MASK) Index: linux-stable/include/linux/wait.h =================================================================== --- linux-stable.orig/include/linux/wait.h 2022-11-14 22:15:17.000000000 +0100 +++ linux-stable/include/linux/wait.h 2022-11-14 22:15:17.000000000 +0100 @@ -1066,7 +1066,7 @@ static inline int wait_on_bit(unsigned long *word, int bit, unsigned mode) { might_sleep(); - if (!test_bit(bit, word)) + if (!test_bit_acquire(bit, word)) return 0; return out_of_line_wait_on_bit(word, bit, bit_wait, @@ -1091,7 +1091,7 @@ static inline int wait_on_bit_io(unsigned long *word, int bit, unsigned mode) { might_sleep(); - if (!test_bit(bit, word)) + if (!test_bit_acquire(bit, word)) return 0; return out_of_line_wait_on_bit(word, bit, bit_wait_io, @@ -1118,7 +1118,7 @@ wait_on_bit_timeout(unsigned long *word, unsigned long timeout) { might_sleep(); - if (!test_bit(bit, word)) + if (!test_bit_acquire(bit, word)) return 0; return out_of_line_wait_on_bit_timeout(word, bit, bit_wait_timeout, @@ -1146,7 +1146,7 @@ wait_on_bit_action(unsigned long *word, unsigned mode) { might_sleep(); - if (!test_bit(bit, word)) + if (!test_bit_acquire(bit, word)) return 0; return out_of_line_wait_on_bit(word, bit, action, mode); } Index: linux-stable/kernel/sched/wait.c =================================================================== --- linux-stable.orig/kernel/sched/wait.c 2022-11-14 22:15:17.000000000 +0100 +++ linux-stable/kernel/sched/wait.c 2022-11-14 22:15:17.000000000 +0100 @@ -389,7 +389,7 @@ __wait_on_bit(wait_queue_head_t *wq, str prepare_to_wait(wq, &q->wait, mode); if (test_bit(q->key.bit_nr, q->key.flags)) ret = (*action)(&q->key, mode); - } while (test_bit(q->key.bit_nr, q->key.flags) && !ret); + } while (test_bit_acquire(q->key.bit_nr, q->key.flags) && !ret); finish_wait(wq, &q->wait); return ret; } Index: linux-stable/include/asm-generic/bitops/generic-non-atomic.h =================================================================== --- linux-stable.orig/include/asm-generic/bitops/generic-non-atomic.h 2022-11-14 22:15:17.000000000 +0100 +++ linux-stable/include/asm-generic/bitops/generic-non-atomic.h 2022-11-14 22:15:17.000000000 +0100 @@ -4,6 +4,7 @@ #define __ASM_GENERIC_BITOPS_GENERIC_NON_ATOMIC_H #include <linux/bits.h> +#include <asm/barrier.h> #ifndef _LINUX_BITOPS_H #error only <linux/bitops.h> can be included directly @@ -127,4 +128,16 @@ generic_test_bit(unsigned int nr, const return 1UL & (addr[BIT_WORD(nr)] >> (nr & (BITS_PER_LONG-1))); } +/** + * generic_test_bit_acquire - Determine, with acquire semantics, whether a bit is set + * @nr: bit number to test + * @addr: Address to start counting from + */ +static __always_inline bool +generic_test_bit_acquire(unsigned long nr, const volatile unsigned long *addr) +{ + unsigned long *p = ((unsigned long *)addr) + BIT_WORD(nr); + return 1UL & (smp_load_acquire(p) >> (nr & (BITS_PER_LONG-1))); +} + #endif /* __ASM_GENERIC_BITOPS_GENERIC_NON_ATOMIC_H */

3 years, 1 month

1
0
0 0

[PATCH 4.9 1/3] bitops: always define asm-generic non-atomic bitops

by Mikulas Patocka

commit 21bb8af513d35c005c401706030f4eb469538d1d upstream. Move generic non-atomic bitops from the asm-generic header which gets included only when there are no architecture-specific alternatives, to a separate independent file to make them always available. Almost no actual code changes, only one comment added to generic_test_bit() saying that it's an atomic operation itself and thus `volatile` must always stay there with no cast-aways. Suggested-by: Andy Shevchenko <andriy.shevchenko(a)linux.intel.com> # comment Suggested-by: Marco Elver <elver(a)google.com> # reference to kernel-doc Signed-off-by: Alexander Lobakin <alexandr.lobakin(a)intel.com> Reviewed-by: Andy Shevchenko <andriy.shevchenko(a)linux.intel.com> Reviewed-by: Marco Elver <elver(a)google.com> Signed-off-by: Yury Norov <yury.norov(a)gmail.com> --- include/asm-generic/bitops/generic-non-atomic.h | 130 ++++++++++++++++++++++++ 1 file changed, 130 insertions(+) Index: linux-stable/include/asm-generic/bitops/generic-non-atomic.h =================================================================== --- /dev/null 1970-01-01 00:00:00.000000000 +0000 +++ linux-stable/include/asm-generic/bitops/generic-non-atomic.h 2022-11-14 22:15:15.000000000 +0100 @@ -0,0 +1,130 @@ +/* SPDX-License-Identifier: GPL-2.0-only */ + +#ifndef __ASM_GENERIC_BITOPS_GENERIC_NON_ATOMIC_H +#define __ASM_GENERIC_BITOPS_GENERIC_NON_ATOMIC_H + +#include <linux/bits.h> + +#ifndef _LINUX_BITOPS_H +#error only <linux/bitops.h> can be included directly +#endif + +/* + * Generic definitions for bit operations, should not be used in regular code + * directly. + */ + +/** + * generic___set_bit - Set a bit in memory + * @nr: the bit to set + * @addr: the address to start counting from + * + * Unlike set_bit(), this function is non-atomic and may be reordered. + * If it's called on the same region of memory simultaneously, the effect + * may be that only one operation succeeds. + */ +static __always_inline void +generic___set_bit(unsigned int nr, volatile unsigned long *addr) +{ + unsigned long mask = BIT_MASK(nr); + unsigned long *p = ((unsigned long *)addr) + BIT_WORD(nr); + + *p |= mask; +} + +static __always_inline void +generic___clear_bit(unsigned int nr, volatile unsigned long *addr) +{ + unsigned long mask = BIT_MASK(nr); + unsigned long *p = ((unsigned long *)addr) + BIT_WORD(nr); + + *p &= ~mask; +} + +/** + * generic___change_bit - Toggle a bit in memory + * @nr: the bit to change + * @addr: the address to start counting from + * + * Unlike change_bit(), this function is non-atomic and may be reordered. + * If it's called on the same region of memory simultaneously, the effect + * may be that only one operation succeeds. + */ +static __always_inline +void generic___change_bit(unsigned int nr, volatile unsigned long *addr) +{ + unsigned long mask = BIT_MASK(nr); + unsigned long *p = ((unsigned long *)addr) + BIT_WORD(nr); + + *p ^= mask; +} + +/** + * generic___test_and_set_bit - Set a bit and return its old value + * @nr: Bit to set + * @addr: Address to count from + * + * This operation is non-atomic and can be reordered. + * If two examples of this operation race, one can appear to succeed + * but actually fail. You must protect multiple accesses with a lock. + */ +static __always_inline int +generic___test_and_set_bit(unsigned int nr, volatile unsigned long *addr) +{ + unsigned long mask = BIT_MASK(nr); + unsigned long *p = ((unsigned long *)addr) + BIT_WORD(nr); + unsigned long old = *p; + + *p = old | mask; + return (old & mask) != 0; +} + +/** + * generic___test_and_clear_bit - Clear a bit and return its old value + * @nr: Bit to clear + * @addr: Address to count from + * + * This operation is non-atomic and can be reordered. + * If two examples of this operation race, one can appear to succeed + * but actually fail. You must protect multiple accesses with a lock. + */ +static __always_inline int +generic___test_and_clear_bit(unsigned int nr, volatile unsigned long *addr) +{ + unsigned long mask = BIT_MASK(nr); + unsigned long *p = ((unsigned long *)addr) + BIT_WORD(nr); + unsigned long old = *p; + + *p = old & ~mask; + return (old & mask) != 0; +} + +/* WARNING: non atomic and it can be reordered! */ +static __always_inline int +generic___test_and_change_bit(unsigned int nr, volatile unsigned long *addr) +{ + unsigned long mask = BIT_MASK(nr); + unsigned long *p = ((unsigned long *)addr) + BIT_WORD(nr); + unsigned long old = *p; + + *p = old ^ mask; + return (old & mask) != 0; +} + +/** + * generic_test_bit - Determine whether a bit is set + * @nr: bit number to test + * @addr: Address to start counting from + */ +static __always_inline int +generic_test_bit(unsigned int nr, const volatile unsigned long *addr) +{ + /* + * Unlike the bitops with the '__' prefix above, this one *is* atomic, + * so `volatile` must always stay here with no cast-aways. See + * `Documentation/atomic_bitops.txt` for the details. + */ + return 1UL & (addr[BIT_WORD(nr)] >> (nr & (BITS_PER_LONG-1))); +} + +#endif /* __ASM_GENERIC_BITOPS_GENERIC_NON_ATOMIC_H */

3 years, 1 month

1
0
0 0

[PATCH 4.14 3/3] provide arch_test_bit_acquire for architectures that define test_bit

by Mikulas Patocka

commit d6ffe6067a54972564552ea45d320fb98db1ac5e upstream. Some architectures define their own arch_test_bit and they also need arch_test_bit_acquire, otherwise they won't compile. We also clean up the code by using the generic test_bit if that is equivalent to the arch-specific version. Signed-off-by: Mikulas Patocka <mpatocka(a)redhat.com> Cc: stable(a)vger.kernel.org Fixes: 8238b4579866 ("wait_on_bit: add an acquire memory barrier") Signed-off-by: Linus Torvalds <torvalds(a)linux-foundation.org> --- arch/alpha/include/asm/bitops.h | 8 +++----- arch/arc/include/asm/bitops.h | 17 +++-------------- arch/frv/include/asm/bitops.h | 26 +++----------------------- arch/h8300/include/asm/bitops.h | 3 ++- arch/hexagon/include/asm/bitops.h | 15 +++++++++++++++ arch/ia64/include/asm/bitops.h | 8 +++----- arch/m68k/include/asm/bitops.h | 8 +++----- arch/mn10300/include/asm/bitops.h | 10 +++------- arch/s390/include/asm/bitops.h | 11 +++-------- arch/sh/include/asm/bitops-op32.h | 11 ++--------- 10 files changed, 40 insertions(+), 77 deletions(-) Index: linux-stable/arch/alpha/include/asm/bitops.h =================================================================== --- linux-stable.orig/arch/alpha/include/asm/bitops.h 2022-11-14 22:15:44.000000000 +0100 +++ linux-stable/arch/alpha/include/asm/bitops.h 2022-11-14 22:15:44.000000000 +0100 @@ -8,6 +8,7 @@ #include <asm/compiler.h> #include <asm/barrier.h> +#include <asm-generic/bitops/generic-non-atomic.h> /* * Copyright 1994, Linus Torvalds. @@ -283,11 +284,8 @@ __test_and_change_bit(unsigned long nr, return (old & mask) != 0; } -static inline int -test_bit(int nr, const volatile void * addr) -{ - return (1UL & (((const int *) addr)[nr >> 5] >> (nr & 31))) != 0UL; -} +#define test_bit generic_test_bit +#define test_bit_acquire generic_test_bit_acquire /* * ffz = Find First Zero in word. Undefined if no zero exists, Index: linux-stable/arch/hexagon/include/asm/bitops.h =================================================================== --- linux-stable.orig/arch/hexagon/include/asm/bitops.h 2022-11-14 22:15:44.000000000 +0100 +++ linux-stable/arch/hexagon/include/asm/bitops.h 2022-11-14 22:15:44.000000000 +0100 @@ -186,7 +186,22 @@ static inline int __test_bit(int nr, con return retval; } +static inline int __test_bit_acquire(int nr, const volatile unsigned long *addr) +{ + int retval; + + asm volatile( + "{P0 = tstbit(%1,%2); if (P0.new) %0 = #1; if (!P0.new) %0 = #0;}\n" + : "=&r" (retval) + : "r" (addr[BIT_WORD(nr)]), "r" (nr % BITS_PER_LONG) + : "p0", "memory" + ); + + return retval; +} + #define test_bit(nr, addr) __test_bit(nr, addr) +#define test_bit_acquire(nr, addr) __test_bit_acquire(nr, addr) /* * ffz - find first zero in word. Index: linux-stable/arch/ia64/include/asm/bitops.h =================================================================== --- linux-stable.orig/arch/ia64/include/asm/bitops.h 2022-11-14 22:15:44.000000000 +0100 +++ linux-stable/arch/ia64/include/asm/bitops.h 2022-11-14 22:15:44.000000000 +0100 @@ -18,6 +18,7 @@ #include <linux/types.h> #include <asm/intrinsics.h> #include <asm/barrier.h> +#include <asm-generic/bitops/generic-non-atomic.h> /** * set_bit - Atomically set a bit in memory @@ -331,11 +332,8 @@ __test_and_change_bit (int nr, void *add return (old & bit) != 0; } -static __inline__ int -test_bit (int nr, const volatile void *addr) -{ - return 1 & (((const volatile __u32 *) addr)[nr >> 5] >> (nr & 31)); -} +#define test_bit generic_test_bit +#define test_bit_acquire generic_test_bit_acquire /** * ffz - find the first zero bit in a long word Index: linux-stable/arch/m68k/include/asm/bitops.h =================================================================== --- linux-stable.orig/arch/m68k/include/asm/bitops.h 2022-11-14 22:15:44.000000000 +0100 +++ linux-stable/arch/m68k/include/asm/bitops.h 2022-11-14 22:15:44.000000000 +0100 @@ -14,6 +14,7 @@ #include <linux/compiler.h> #include <asm/barrier.h> +#include <asm-generic/bitops/generic-non-atomic.h> /* * Bit access functions vary across the ColdFire and 68k families. @@ -148,11 +149,8 @@ static inline void bfchg_mem_change_bit( #define __change_bit(nr, vaddr) change_bit(nr, vaddr) -static inline int test_bit(int nr, const volatile unsigned long *vaddr) -{ - return (vaddr[nr >> 5] & (1UL << (nr & 31))) != 0; -} - +#define test_bit generic_test_bit +#define test_bit_acquire generic_test_bit_acquire static inline int bset_reg_test_and_set_bit(int nr, volatile unsigned long *vaddr) Index: linux-stable/arch/s390/include/asm/bitops.h =================================================================== --- linux-stable.orig/arch/s390/include/asm/bitops.h 2022-11-14 22:15:44.000000000 +0100 +++ linux-stable/arch/s390/include/asm/bitops.h 2022-11-14 22:15:44.000000000 +0100 @@ -37,6 +37,7 @@ #include <linux/compiler.h> #include <asm/atomic_ops.h> #include <asm/barrier.h> +#include <asm-generic/bitops/generic-non-atomic.h> #define __BITOPS_WORDS(bits) (((bits) + BITS_PER_LONG - 1) / BITS_PER_LONG) @@ -206,14 +207,8 @@ __test_and_change_bit(unsigned long nr, return (ch >> (nr & 7)) & 1; } -static inline int test_bit(unsigned long nr, const volatile unsigned long *ptr) -{ - const volatile unsigned char *addr; - - addr = ((const volatile unsigned char *)ptr); - addr += (nr ^ (BITS_PER_LONG - 8)) >> 3; - return (*addr >> (nr & 7)) & 1; -} +#define test_bit generic_test_bit +#define test_bit_acquire generic_test_bit_acquire static inline int test_and_set_bit_lock(unsigned long nr, volatile unsigned long *ptr) Index: linux-stable/arch/sh/include/asm/bitops-op32.h =================================================================== --- linux-stable.orig/arch/sh/include/asm/bitops-op32.h 2022-11-14 22:15:44.000000000 +0100 +++ linux-stable/arch/sh/include/asm/bitops-op32.h 2022-11-14 22:15:44.000000000 +0100 @@ -130,14 +130,7 @@ static inline int __test_and_change_bit( return (old & mask) != 0; } -/** - * test_bit - Determine whether a bit is set - * @nr: bit number to test - * @addr: Address to start counting from - */ -static inline int test_bit(int nr, const volatile unsigned long *addr) -{ - return 1UL & (addr[BIT_WORD(nr)] >> (nr & (BITS_PER_LONG-1))); -} +#define test_bit generic_test_bit +#define test_bit_acquire generic_test_bit_acquire #endif /* __ASM_SH_BITOPS_OP32_H */ Index: linux-stable/arch/h8300/include/asm/bitops.h =================================================================== --- linux-stable.orig/arch/h8300/include/asm/bitops.h 2022-11-14 22:15:44.000000000 +0100 +++ linux-stable/arch/h8300/include/asm/bitops.h 2022-11-14 22:15:44.000000000 +0100 @@ -89,7 +89,8 @@ static inline int test_bit(int nr, const return ret; } -#define __test_bit(nr, addr) test_bit(nr, addr) +#define __test_bit(nr, addr) test_bit(nr, addr) +#define test_bit_acquire(nr, addr) test_bit(nr, addr) #define H8300_GEN_TEST_BITOP(FNNAME, OP) \ static inline int FNNAME(int nr, void *addr) \ Index: linux-stable/arch/arc/include/asm/bitops.h =================================================================== --- linux-stable.orig/arch/arc/include/asm/bitops.h 2022-11-14 22:15:44.000000000 +0100 +++ linux-stable/arch/arc/include/asm/bitops.h 2022-11-14 22:15:44.000000000 +0100 @@ -21,6 +21,7 @@ #ifndef CONFIG_ARC_HAS_LLSC #include <asm/smp.h> #endif +#include <asm-generic/bitops/generic-non-atomic.h> #ifdef CONFIG_ARC_HAS_LLSC @@ -239,20 +240,8 @@ BIT_OPS(clear, & ~, CTOP_INST_AAND_DI_R2 BIT_OPS(change, ^, CTOP_INST_AXOR_DI_R2_R2_R3) #endif -/* - * This routine doesn't need to be atomic. - */ -static inline int -test_bit(unsigned int nr, const volatile unsigned long *addr) -{ - unsigned long mask; - - addr += nr >> 5; - - mask = 1UL << (nr & 0x1f); - - return ((mask & *addr) != 0); -} +#define test_bit generic_test_bit +#define test_bit_acquire generic_test_bit_acquire #ifdef CONFIG_ISA_ARCOMPACT Index: linux-stable/arch/frv/include/asm/bitops.h =================================================================== --- linux-stable.orig/arch/frv/include/asm/bitops.h 2022-11-14 22:15:44.000000000 +0100 +++ linux-stable/arch/frv/include/asm/bitops.h 2022-11-14 22:15:44.000000000 +0100 @@ -23,6 +23,7 @@ #error only <linux/bitops.h> can be included directly #endif +#include <asm-generic/bitops/generic-non-atomic.h> #include <asm-generic/bitops/ffz.h> #include <asm/atomic.h> @@ -132,29 +133,8 @@ static inline int __test_and_change_bit( return retval; } -/* - * This routine doesn't need to be atomic. - */ -static inline int -__constant_test_bit(unsigned long nr, const volatile void *addr) -{ - return ((1UL << (nr & 31)) & (((const volatile unsigned int *) addr)[nr >> 5])) != 0; -} - -static inline int __test_bit(unsigned long nr, const volatile void *addr) -{ - int * a = (int *) addr; - int mask; - - a += nr >> 5; - mask = 1 << (nr & 0x1f); - return ((mask & *a) != 0); -} - -#define test_bit(nr,addr) \ -(__builtin_constant_p(nr) ? \ - __constant_test_bit((nr),(addr)) : \ - __test_bit((nr),(addr))) +#define test_bit generic_test_bit +#define test_bit_acquire generic_test_bit_acquire #include <asm-generic/bitops/find.h> Index: linux-stable/arch/mn10300/include/asm/bitops.h =================================================================== --- linux-stable.orig/arch/mn10300/include/asm/bitops.h 2022-11-14 22:15:44.000000000 +0100 +++ linux-stable/arch/mn10300/include/asm/bitops.h 2022-11-14 22:15:44.000000000 +0100 @@ -19,6 +19,7 @@ #include <asm/cpu-regs.h> #include <asm/barrier.h> +#include <asm-generic/bitops/generic-non-atomic.h> /* * set bit @@ -65,13 +66,8 @@ static inline void __clear_bit(unsigned *a &= ~mask; } -/* - * test bit - */ -static inline int test_bit(unsigned long nr, const volatile void *addr) -{ - return 1UL & (((const volatile unsigned int *) addr)[nr >> 5] >> (nr & 31)); -} +#define test_bit generic_test_bit +#define test_bit_acquire generic_test_bit_acquire /* * change bit

3 years, 1 month

1
0
0 0

[PATCH 4.14 2/3] wait_on_bit: add an acquire memory barrier

by Mikulas Patocka

commit 8238b4579866b7c1bb99883cfe102a43db5506ff upstream. There are several places in the kernel where wait_on_bit is not followed by a memory barrier (for example, in drivers/md/dm-bufio.c:new_read). On architectures with weak memory ordering, it may happen that memory accesses that follow wait_on_bit are reordered before wait_on_bit and they may return invalid data. Fix this class of bugs by introducing a new function "test_bit_acquire" that works like test_bit, but has acquire memory ordering semantics. Signed-off-by: Mikulas Patocka <mpatocka(a)redhat.com> Acked-by: Will Deacon <will(a)kernel.org> Cc: stable(a)vger.kernel.org Signed-off-by: Linus Torvalds <torvalds(a)linux-foundation.org> --- arch/x86/include/asm/bitops.h | 21 +++++++++++++++++++++ include/asm-generic/bitops/generic-non-atomic.h | 13 +++++++++++++ include/asm-generic/bitops/non-atomic.h | 14 ++++++++++++++ include/linux/buffer_head.h | 2 +- include/linux/wait_bit.h | 8 ++++---- kernel/sched/wait_bit.c | 2 +- 6 files changed, 54 insertions(+), 6 deletions(-) Index: linux-stable/arch/x86/include/asm/bitops.h =================================================================== --- linux-stable.orig/arch/x86/include/asm/bitops.h 2022-11-14 22:15:43.000000000 +0100 +++ linux-stable/arch/x86/include/asm/bitops.h 2022-11-14 22:15:43.000000000 +0100 @@ -328,6 +328,20 @@ static __always_inline bool constant_tes (addr[nr >> _BITOPS_LONG_SHIFT])) != 0; } +static __always_inline bool constant_test_bit_acquire(long nr, const volatile unsigned long *addr) +{ + bool oldbit; + + asm volatile("testb %2,%1" + CC_SET(nz) + : CC_OUT(nz) (oldbit) + : "m" (((unsigned char *)addr)[nr >> 3]), + "i" (1 << (nr & 7)) + :"memory"); + + return oldbit; +} + static __always_inline bool variable_test_bit(long nr, volatile const unsigned long *addr) { bool oldbit; @@ -354,6 +368,13 @@ static bool test_bit(int nr, const volat ? constant_test_bit((nr), (addr)) \ : variable_test_bit((nr), (addr))) +static __always_inline bool +test_bit_acquire(unsigned long nr, const volatile unsigned long *addr) +{ + return __builtin_constant_p(nr) ? constant_test_bit_acquire(nr, addr) : + variable_test_bit(nr, addr); +} + /** * __ffs - find first set bit in word * @word: The word to search Index: linux-stable/include/asm-generic/bitops/non-atomic.h =================================================================== --- linux-stable.orig/include/asm-generic/bitops/non-atomic.h 2022-11-14 22:15:43.000000000 +0100 +++ linux-stable/include/asm-generic/bitops/non-atomic.h 2022-11-14 22:15:43.000000000 +0100 @@ -3,6 +3,7 @@ #define _ASM_GENERIC_BITOPS_NON_ATOMIC_H_ #include <asm/types.h> +#include <asm/barrier.h> /** * __set_bit - Set a bit in memory @@ -106,4 +107,17 @@ static inline int test_bit(int nr, const return 1UL & (addr[BIT_WORD(nr)] >> (nr & (BITS_PER_LONG-1))); } +/** + * arch_test_bit_acquire - Determine, with acquire semantics, whether a bit is set + * @nr: bit number to test + * @addr: Address to start counting from + */ +static __always_inline bool +arch_test_bit_acquire(unsigned long nr, const volatile unsigned long *addr) +{ + unsigned long *p = ((unsigned long *)addr) + BIT_WORD(nr); + return 1UL & (smp_load_acquire(p) >> (nr & (BITS_PER_LONG-1))); +} +#define test_bit_acquire arch_test_bit_acquire + #endif /* _ASM_GENERIC_BITOPS_NON_ATOMIC_H_ */ Index: linux-stable/include/linux/buffer_head.h =================================================================== --- linux-stable.orig/include/linux/buffer_head.h 2022-11-14 22:15:43.000000000 +0100 +++ linux-stable/include/linux/buffer_head.h 2022-11-14 22:15:43.000000000 +0100 @@ -163,7 +163,7 @@ static __always_inline int buffer_uptoda * make it consistent with folio_test_uptodate * pairs with smp_mb__before_atomic in set_buffer_uptodate */ - return (smp_load_acquire(&bh->b_state) & (1UL << BH_Uptodate)) != 0; + return test_bit_acquire(BH_Uptodate, &bh->b_state); } #define bh_offset(bh) ((unsigned long)(bh)->b_data & ~PAGE_MASK) Index: linux-stable/include/linux/wait_bit.h =================================================================== --- linux-stable.orig/include/linux/wait_bit.h 2022-11-14 22:15:43.000000000 +0100 +++ linux-stable/include/linux/wait_bit.h 2022-11-14 22:15:43.000000000 +0100 @@ -76,7 +76,7 @@ static inline int wait_on_bit(unsigned long *word, int bit, unsigned mode) { might_sleep(); - if (!test_bit(bit, word)) + if (!test_bit_acquire(bit, word)) return 0; return out_of_line_wait_on_bit(word, bit, bit_wait, @@ -101,7 +101,7 @@ static inline int wait_on_bit_io(unsigned long *word, int bit, unsigned mode) { might_sleep(); - if (!test_bit(bit, word)) + if (!test_bit_acquire(bit, word)) return 0; return out_of_line_wait_on_bit(word, bit, bit_wait_io, @@ -128,7 +128,7 @@ wait_on_bit_timeout(unsigned long *word, unsigned long timeout) { might_sleep(); - if (!test_bit(bit, word)) + if (!test_bit_acquire(bit, word)) return 0; return out_of_line_wait_on_bit_timeout(word, bit, bit_wait_timeout, @@ -156,7 +156,7 @@ wait_on_bit_action(unsigned long *word, unsigned mode) { might_sleep(); - if (!test_bit(bit, word)) + if (!test_bit_acquire(bit, word)) return 0; return out_of_line_wait_on_bit(word, bit, action, mode); } Index: linux-stable/kernel/sched/wait_bit.c =================================================================== --- linux-stable.orig/kernel/sched/wait_bit.c 2022-11-14 22:15:43.000000000 +0100 +++ linux-stable/kernel/sched/wait_bit.c 2022-11-14 22:15:43.000000000 +0100 @@ -49,7 +49,7 @@ __wait_on_bit(struct wait_queue_head *wq prepare_to_wait(wq_head, &wbq_entry->wq_entry, mode); if (test_bit(wbq_entry->key.bit_nr, wbq_entry->key.flags)) ret = (*action)(&wbq_entry->key, mode); - } while (test_bit(wbq_entry->key.bit_nr, wbq_entry->key.flags) && !ret); + } while (test_bit_acquire(wbq_entry->key.bit_nr, wbq_entry->key.flags) && !ret); finish_wait(wq_head, &wbq_entry->wq_entry); return ret; } Index: linux-stable/include/asm-generic/bitops/generic-non-atomic.h =================================================================== --- linux-stable.orig/include/asm-generic/bitops/generic-non-atomic.h 2022-11-14 22:15:43.000000000 +0100 +++ linux-stable/include/asm-generic/bitops/generic-non-atomic.h 2022-11-14 22:15:43.000000000 +0100 @@ -4,6 +4,7 @@ #define __ASM_GENERIC_BITOPS_GENERIC_NON_ATOMIC_H #include <linux/bits.h> +#include <asm/barrier.h> #ifndef _LINUX_BITOPS_H #error only <linux/bitops.h> can be included directly @@ -127,4 +128,16 @@ generic_test_bit(unsigned int nr, const return 1UL & (addr[BIT_WORD(nr)] >> (nr & (BITS_PER_LONG-1))); } +/** + * generic_test_bit_acquire - Determine, with acquire semantics, whether a bit is set + * @nr: bit number to test + * @addr: Address to start counting from + */ +static __always_inline bool +generic_test_bit_acquire(unsigned long nr, const volatile unsigned long *addr) +{ + unsigned long *p = ((unsigned long *)addr) + BIT_WORD(nr); + return 1UL & (smp_load_acquire(p) >> (nr & (BITS_PER_LONG-1))); +} + #endif /* __ASM_GENERIC_BITOPS_GENERIC_NON_ATOMIC_H */

3 years, 1 month

1
0
0 0

[PATCH 4.14 1/3] bitops: always define asm-generic non-atomic bitops

by Mikulas Patocka

commit 21bb8af513d35c005c401706030f4eb469538d1d upstream. Move generic non-atomic bitops from the asm-generic header which gets included only when there are no architecture-specific alternatives, to a separate independent file to make them always available. Almost no actual code changes, only one comment added to generic_test_bit() saying that it's an atomic operation itself and thus `volatile` must always stay there with no cast-aways. Suggested-by: Andy Shevchenko <andriy.shevchenko(a)linux.intel.com> # comment Suggested-by: Marco Elver <elver(a)google.com> # reference to kernel-doc Signed-off-by: Alexander Lobakin <alexandr.lobakin(a)intel.com> Reviewed-by: Andy Shevchenko <andriy.shevchenko(a)linux.intel.com> Reviewed-by: Marco Elver <elver(a)google.com> Signed-off-by: Yury Norov <yury.norov(a)gmail.com> --- include/asm-generic/bitops/generic-non-atomic.h | 130 ++++++++++++++++++++++++ 1 file changed, 130 insertions(+) Index: linux-stable/include/asm-generic/bitops/generic-non-atomic.h =================================================================== --- /dev/null 1970-01-01 00:00:00.000000000 +0000 +++ linux-stable/include/asm-generic/bitops/generic-non-atomic.h 2022-11-14 22:15:42.000000000 +0100 @@ -0,0 +1,130 @@ +/* SPDX-License-Identifier: GPL-2.0-only */ + +#ifndef __ASM_GENERIC_BITOPS_GENERIC_NON_ATOMIC_H +#define __ASM_GENERIC_BITOPS_GENERIC_NON_ATOMIC_H + +#include <linux/bits.h> + +#ifndef _LINUX_BITOPS_H +#error only <linux/bitops.h> can be included directly +#endif + +/* + * Generic definitions for bit operations, should not be used in regular code + * directly. + */ + +/** + * generic___set_bit - Set a bit in memory + * @nr: the bit to set + * @addr: the address to start counting from + * + * Unlike set_bit(), this function is non-atomic and may be reordered. + * If it's called on the same region of memory simultaneously, the effect + * may be that only one operation succeeds. + */ +static __always_inline void +generic___set_bit(unsigned int nr, volatile unsigned long *addr) +{ + unsigned long mask = BIT_MASK(nr); + unsigned long *p = ((unsigned long *)addr) + BIT_WORD(nr); + + *p |= mask; +} + +static __always_inline void +generic___clear_bit(unsigned int nr, volatile unsigned long *addr) +{ + unsigned long mask = BIT_MASK(nr); + unsigned long *p = ((unsigned long *)addr) + BIT_WORD(nr); + + *p &= ~mask; +} + +/** + * generic___change_bit - Toggle a bit in memory + * @nr: the bit to change + * @addr: the address to start counting from + * + * Unlike change_bit(), this function is non-atomic and may be reordered. + * If it's called on the same region of memory simultaneously, the effect + * may be that only one operation succeeds. + */ +static __always_inline +void generic___change_bit(unsigned int nr, volatile unsigned long *addr) +{ + unsigned long mask = BIT_MASK(nr); + unsigned long *p = ((unsigned long *)addr) + BIT_WORD(nr); + + *p ^= mask; +} + +/** + * generic___test_and_set_bit - Set a bit and return its old value + * @nr: Bit to set + * @addr: Address to count from + * + * This operation is non-atomic and can be reordered. + * If two examples of this operation race, one can appear to succeed + * but actually fail. You must protect multiple accesses with a lock. + */ +static __always_inline int +generic___test_and_set_bit(unsigned int nr, volatile unsigned long *addr) +{ + unsigned long mask = BIT_MASK(nr); + unsigned long *p = ((unsigned long *)addr) + BIT_WORD(nr); + unsigned long old = *p; + + *p = old | mask; + return (old & mask) != 0; +} + +/** + * generic___test_and_clear_bit - Clear a bit and return its old value + * @nr: Bit to clear + * @addr: Address to count from + * + * This operation is non-atomic and can be reordered. + * If two examples of this operation race, one can appear to succeed + * but actually fail. You must protect multiple accesses with a lock. + */ +static __always_inline int +generic___test_and_clear_bit(unsigned int nr, volatile unsigned long *addr) +{ + unsigned long mask = BIT_MASK(nr); + unsigned long *p = ((unsigned long *)addr) + BIT_WORD(nr); + unsigned long old = *p; + + *p = old & ~mask; + return (old & mask) != 0; +} + +/* WARNING: non atomic and it can be reordered! */ +static __always_inline int +generic___test_and_change_bit(unsigned int nr, volatile unsigned long *addr) +{ + unsigned long mask = BIT_MASK(nr); + unsigned long *p = ((unsigned long *)addr) + BIT_WORD(nr); + unsigned long old = *p; + + *p = old ^ mask; + return (old & mask) != 0; +} + +/** + * generic_test_bit - Determine whether a bit is set + * @nr: bit number to test + * @addr: Address to start counting from + */ +static __always_inline int +generic_test_bit(unsigned int nr, const volatile unsigned long *addr) +{ + /* + * Unlike the bitops with the '__' prefix above, this one *is* atomic, + * so `volatile` must always stay here with no cast-aways. See + * `Documentation/atomic_bitops.txt` for the details. + */ + return 1UL & (addr[BIT_WORD(nr)] >> (nr & (BITS_PER_LONG-1))); +} + +#endif /* __ASM_GENERIC_BITOPS_GENERIC_NON_ATOMIC_H */

3 years, 1 month

1
0
0 0

[PATCH 4.19 3/3] provide arch_test_bit_acquire for architectures that define test_bit

by Mikulas Patocka

commit d6ffe6067a54972564552ea45d320fb98db1ac5e upstream. Some architectures define their own arch_test_bit and they also need arch_test_bit_acquire, otherwise they won't compile. We also clean up the code by using the generic test_bit if that is equivalent to the arch-specific version. Signed-off-by: Mikulas Patocka <mpatocka(a)redhat.com> Cc: stable(a)vger.kernel.org Fixes: 8238b4579866 ("wait_on_bit: add an acquire memory barrier") Signed-off-by: Linus Torvalds <torvalds(a)linux-foundation.org> --- arch/alpha/include/asm/bitops.h | 8 +++----- arch/arc/include/asm/bitops.h | 17 +++-------------- arch/h8300/include/asm/bitops.h | 3 ++- arch/hexagon/include/asm/bitops.h | 15 +++++++++++++++ arch/ia64/include/asm/bitops.h | 8 +++----- arch/m68k/include/asm/bitops.h | 8 +++----- arch/s390/include/asm/bitops.h | 11 +++-------- arch/sh/include/asm/bitops-op32.h | 11 ++--------- 8 files changed, 34 insertions(+), 47 deletions(-) Index: linux-stable/arch/alpha/include/asm/bitops.h =================================================================== --- linux-stable.orig/arch/alpha/include/asm/bitops.h 2022-11-14 22:16:11.000000000 +0100 +++ linux-stable/arch/alpha/include/asm/bitops.h 2022-11-14 22:16:11.000000000 +0100 @@ -8,6 +8,7 @@ #include <asm/compiler.h> #include <asm/barrier.h> +#include <asm-generic/bitops/generic-non-atomic.h> /* * Copyright 1994, Linus Torvalds. @@ -283,11 +284,8 @@ __test_and_change_bit(unsigned long nr, return (old & mask) != 0; } -static inline int -test_bit(int nr, const volatile void * addr) -{ - return (1UL & (((const int *) addr)[nr >> 5] >> (nr & 31))) != 0UL; -} +#define test_bit generic_test_bit +#define test_bit_acquire generic_test_bit_acquire /* * ffz = Find First Zero in word. Undefined if no zero exists, Index: linux-stable/arch/hexagon/include/asm/bitops.h =================================================================== --- linux-stable.orig/arch/hexagon/include/asm/bitops.h 2022-11-14 22:16:11.000000000 +0100 +++ linux-stable/arch/hexagon/include/asm/bitops.h 2022-11-14 22:16:11.000000000 +0100 @@ -186,7 +186,22 @@ static inline int __test_bit(int nr, con return retval; } +static inline int __test_bit_acquire(int nr, const volatile unsigned long *addr) +{ + int retval; + + asm volatile( + "{P0 = tstbit(%1,%2); if (P0.new) %0 = #1; if (!P0.new) %0 = #0;}\n" + : "=&r" (retval) + : "r" (addr[BIT_WORD(nr)]), "r" (nr % BITS_PER_LONG) + : "p0", "memory" + ); + + return retval; +} + #define test_bit(nr, addr) __test_bit(nr, addr) +#define test_bit_acquire(nr, addr) __test_bit_acquire(nr, addr) /* * ffz - find first zero in word. Index: linux-stable/arch/ia64/include/asm/bitops.h =================================================================== --- linux-stable.orig/arch/ia64/include/asm/bitops.h 2022-11-14 22:16:11.000000000 +0100 +++ linux-stable/arch/ia64/include/asm/bitops.h 2022-11-14 22:16:11.000000000 +0100 @@ -18,6 +18,7 @@ #include <linux/types.h> #include <asm/intrinsics.h> #include <asm/barrier.h> +#include <asm-generic/bitops/generic-non-atomic.h> /** * set_bit - Atomically set a bit in memory @@ -331,11 +332,8 @@ __test_and_change_bit (int nr, void *add return (old & bit) != 0; } -static __inline__ int -test_bit (int nr, const volatile void *addr) -{ - return 1 & (((const volatile __u32 *) addr)[nr >> 5] >> (nr & 31)); -} +#define test_bit generic_test_bit +#define test_bit_acquire generic_test_bit_acquire /** * ffz - find the first zero bit in a long word Index: linux-stable/arch/m68k/include/asm/bitops.h =================================================================== --- linux-stable.orig/arch/m68k/include/asm/bitops.h 2022-11-14 22:16:11.000000000 +0100 +++ linux-stable/arch/m68k/include/asm/bitops.h 2022-11-14 22:16:11.000000000 +0100 @@ -14,6 +14,7 @@ #include <linux/compiler.h> #include <asm/barrier.h> +#include <asm-generic/bitops/generic-non-atomic.h> /* * Bit access functions vary across the ColdFire and 68k families. @@ -148,11 +149,8 @@ static inline void bfchg_mem_change_bit( #define __change_bit(nr, vaddr) change_bit(nr, vaddr) -static inline int test_bit(int nr, const volatile unsigned long *vaddr) -{ - return (vaddr[nr >> 5] & (1UL << (nr & 31))) != 0; -} - +#define test_bit generic_test_bit +#define test_bit_acquire generic_test_bit_acquire static inline int bset_reg_test_and_set_bit(int nr, volatile unsigned long *vaddr) Index: linux-stable/arch/s390/include/asm/bitops.h =================================================================== --- linux-stable.orig/arch/s390/include/asm/bitops.h 2022-11-14 22:16:11.000000000 +0100 +++ linux-stable/arch/s390/include/asm/bitops.h 2022-11-14 22:16:11.000000000 +0100 @@ -37,6 +37,7 @@ #include <linux/compiler.h> #include <asm/atomic_ops.h> #include <asm/barrier.h> +#include <asm-generic/bitops/generic-non-atomic.h> #define __BITOPS_WORDS(bits) (((bits) + BITS_PER_LONG - 1) / BITS_PER_LONG) @@ -206,14 +207,8 @@ __test_and_change_bit(unsigned long nr, return (ch >> (nr & 7)) & 1; } -static inline int test_bit(unsigned long nr, const volatile unsigned long *ptr) -{ - const volatile unsigned char *addr; - - addr = ((const volatile unsigned char *)ptr); - addr += (nr ^ (BITS_PER_LONG - 8)) >> 3; - return (*addr >> (nr & 7)) & 1; -} +#define test_bit generic_test_bit +#define test_bit_acquire generic_test_bit_acquire static inline int test_and_set_bit_lock(unsigned long nr, volatile unsigned long *ptr) Index: linux-stable/arch/sh/include/asm/bitops-op32.h =================================================================== --- linux-stable.orig/arch/sh/include/asm/bitops-op32.h 2022-11-14 22:16:11.000000000 +0100 +++ linux-stable/arch/sh/include/asm/bitops-op32.h 2022-11-14 22:16:11.000000000 +0100 @@ -130,14 +130,7 @@ static inline int __test_and_change_bit( return (old & mask) != 0; } -/** - * test_bit - Determine whether a bit is set - * @nr: bit number to test - * @addr: Address to start counting from - */ -static inline int test_bit(int nr, const volatile unsigned long *addr) -{ - return 1UL & (addr[BIT_WORD(nr)] >> (nr & (BITS_PER_LONG-1))); -} +#define test_bit generic_test_bit +#define test_bit_acquire generic_test_bit_acquire #endif /* __ASM_SH_BITOPS_OP32_H */ Index: linux-stable/arch/h8300/include/asm/bitops.h =================================================================== --- linux-stable.orig/arch/h8300/include/asm/bitops.h 2022-11-14 22:16:11.000000000 +0100 +++ linux-stable/arch/h8300/include/asm/bitops.h 2022-11-14 22:16:11.000000000 +0100 @@ -89,7 +89,8 @@ static inline int test_bit(int nr, const return ret; } -#define __test_bit(nr, addr) test_bit(nr, addr) +#define __test_bit(nr, addr) test_bit(nr, addr) +#define test_bit_acquire(nr, addr) test_bit(nr, addr) #define H8300_GEN_TEST_BITOP(FNNAME, OP) \ static inline int FNNAME(int nr, void *addr) \ Index: linux-stable/arch/arc/include/asm/bitops.h =================================================================== --- linux-stable.orig/arch/arc/include/asm/bitops.h 2022-11-14 22:16:11.000000000 +0100 +++ linux-stable/arch/arc/include/asm/bitops.h 2022-11-14 22:16:11.000000000 +0100 @@ -21,6 +21,7 @@ #ifndef CONFIG_ARC_HAS_LLSC #include <asm/smp.h> #endif +#include <asm-generic/bitops/generic-non-atomic.h> #ifdef CONFIG_ARC_HAS_LLSC @@ -239,20 +240,8 @@ BIT_OPS(clear, & ~, CTOP_INST_AAND_DI_R2 BIT_OPS(change, ^, CTOP_INST_AXOR_DI_R2_R2_R3) #endif -/* - * This routine doesn't need to be atomic. - */ -static inline int -test_bit(unsigned int nr, const volatile unsigned long *addr) -{ - unsigned long mask; - - addr += nr >> 5; - - mask = 1UL << (nr & 0x1f); - - return ((mask & *addr) != 0); -} +#define test_bit generic_test_bit +#define test_bit_acquire generic_test_bit_acquire #ifdef CONFIG_ISA_ARCOMPACT

3 years, 1 month

1
0
0 0

2025

2024

2023

2022

2021

2020

2019

2018

2017

Linux-stable-mirror November 2022