I'm unsure if this is just an LVM bug, or a BTRFS+LVM interaction bug, but LVM is definitely involved somehow. Upgrading from 5.10 to 6.1, I noticed one of my filesystems was read-only. In dmesg, I found:
BTRFS error (device dm-75): bdev /dev/mapper/lvm-brokenDisk errs: wr 0, rd 0, flush 1, corrupt 0, gen 0 BTRFS warning (device dm-75): chunk 13631488 missing 1 devices, max tolerance is 0 for writable mount BTRFS: error (device dm-75) in write_all_supers:4379: errno=-5 IO failure (errors while submitting device barriers.) BTRFS info (device dm-75: state E): forced readonly BTRFS warning (device dm-75: state E): Skipping commit of aborted transaction. BTRFS: error (device dm-75: state EA) in cleanup_transaction:1992: errno=-5 IO failure
At first I suspected a btrfs error, but a scrub found no errors, and it continued to be read-write on 5.10 kernels.
Here is my setup:
/dev/lvm/brokenDisk is a lvm-on-lvm volume. I have /dev/sd{a,b,c,d} (of varying sizes) in a lower VG, which has three LVs, all raid1 volumes. Two of the volumes are further used as PV's for an upper VGs. One of the upper VGs has no issues. The non-PV LV has no issue. The remaining one, /dev/lowerVG/lvmPool, hosting nested LVM, is used as a PV for VG "lvm", and has 3 volumes inside. Two of those volumes have no issues (and are btrfs), but the last one is /dev/lvm/brokenDisk. This volume is the only one that exhibits this behavior, so something is special.
Or described as layers: /dev/sd{a,b,c,d} => PV => VG "lowerVG" /dev/lowerVG/single (RAID1 LV) => BTRFS, works fine /dev/lowerVG/works (RAID1 LV) => PV => VG "workingUpper" /dev/workingUpper/{a,b,c} => BTRFS, works fine /dev/lowerVG/lvmPool (RAID1 LV) => PV => VG "lvm" /dev/lvm/{a,b} => BTRFS, works fine /dev/lvm/brokenDisk => BTRFS, Exhibits errors
After some investigation, here is what I've found:
1. This regression was introduced in 5.19. 5.18 and earlier kernels I can keep this filesystem rw and everything works as expected, while 5.19.0 and later the filesystem is immediately ro on any write attempt. I couldn't build rc1, but I did confirm rc2 already has this regression. 2. Passing /dev/lvm/brokenDisk to a KVM VM as /dev/vdb with an unaffected kernel inside the vm exhibits the ro barrier problem on unaffected kernels. 3. Passing /dev/lowerVG/lvmPool to a KVM VM as /dev/vdb with an affected kernel inside the VM and using LVM inside the VM exhibits correct behavior (I can keep the filesystem rw, no barrier errors on host or guest) 4. A discussion in IRC with BTRFS folks, and they think the BTRFS filesystem is fine (btrfs check and btrfs scrub also agree) 5. The dmesg error can be delayed indefinitely by not writing to the disk, or reading with noatime 6. This affects Debian, Ubuntu, NixOS, and Solus, so I'm fairly certain it's distro-agnostic, and purely a kernel issue. 7. I can't reproduce this with other LVM-on-LVM setups, so I think the asymmetric nature of the raid1 volume is potentially contributing 8. There are no new smart errors/failures on any of the disks, disks are healthy 9. I previously had raidintegrity=y and caching enabled. They didn't affect the issue
#regzbot introduced v5.18..v5.19-rc2
Patrick
On 28/02/2024 18.25, Patrick Plenefisch wrote:
I'm unsure if this is just an LVM bug, or a BTRFS+LVM interaction bug, but LVM is definitely involved somehow. Upgrading from 5.10 to 6.1, I noticed one of my filesystems was read-only. In dmesg, I found:
BTRFS error (device dm-75): bdev /dev/mapper/lvm-brokenDisk errs: wr 0, rd 0, flush 1, corrupt 0, gen 0 BTRFS warning (device dm-75): chunk 13631488 missing 1 devices, max tolerance is 0 for writable mount BTRFS: error (device dm-75) in write_all_supers:4379: errno=-5 IO failure (errors while submitting device barriers.) BTRFS info (device dm-75: state E): forced readonly BTRFS warning (device dm-75: state E): Skipping commit of aborted transaction. BTRFS: error (device dm-75: state EA) in cleanup_transaction:1992: errno=-5 IO failure
At first I suspected a btrfs error, but a scrub found no errors, and it continued to be read-write on 5.10 kernels.
Here is my setup:
/dev/lvm/brokenDisk is a lvm-on-lvm volume. I have /dev/sd{a,b,c,d} (of varying sizes) in a lower VG, which has three LVs, all raid1 volumes. Two of the volumes are further used as PV's for an upper VGs. One of the upper VGs has no issues. The non-PV LV has no issue. The remaining one, /dev/lowerVG/lvmPool, hosting nested LVM, is used as a PV for VG "lvm", and has 3 volumes inside. Two of those volumes have no issues (and are btrfs), but the last one is /dev/lvm/brokenDisk. This volume is the only one that exhibits this behavior, so something is special.
Or described as layers: /dev/sd{a,b,c,d} => PV => VG "lowerVG" /dev/lowerVG/single (RAID1 LV) => BTRFS, works fine /dev/lowerVG/works (RAID1 LV) => PV => VG "workingUpper" /dev/workingUpper/{a,b,c} => BTRFS, works fine /dev/lowerVG/lvmPool (RAID1 LV) => PV => VG "lvm" /dev/lvm/{a,b} => BTRFS, works fine /dev/lvm/brokenDisk => BTRFS, Exhibits errors
I am a bit curious about the reasons of this setup. However I understood that:
/dev/sda -+ +-- single (RAID1) -> ok +-> a ok /dev/sdb | | |-> b ok /dev/sdc +--> [lowerVG]>--+-- works (RAID1) -> [workingUpper] -+-> c ok /dev/sdd -+ | | +-> a -> ok +-- lvmPool -> [lvm] ->-| +-> b -> ok | +->brokenDisk -> fail
[xxx] means VG, the others are LVs that may act also as PV in an upper VG
So, it seems that
1) lowerVG/lvmPool/lvm/a 2) lowerVG/lvmPool/lvm/a 3) lowerVG/lvmPool/lvm/brokenDisk
are equivalent ... so I don't understand how 1) and 2) are fine but 3) is problematic.
Is my understanding of the LVM layouts correct ?
After some investigation, here is what I've found:
- This regression was introduced in 5.19. 5.18 and earlier kernels I
can keep this filesystem rw and everything works as expected, while 5.19.0 and later the filesystem is immediately ro on any write attempt. I couldn't build rc1, but I did confirm rc2 already has this regression. 2. Passing /dev/lvm/brokenDisk to a KVM VM as /dev/vdb with an unaffected kernel inside the vm exhibits the ro barrier problem on unaffected kernels.
Is /dev/lvm/brokenDisk *always* problematic with affected ( >= 5.19 ) and UNaffected ( < 5.19 ) kernel ?
- Passing /dev/lowerVG/lvmPool to a KVM VM as /dev/vdb with an
affected kernel inside the VM and using LVM inside the VM exhibits correct behavior (I can keep the filesystem rw, no barrier errors on host or guest)
Is /dev/lowerVG/lvmPool problematic with only "affected" kernel ?
[...]
On Wed, Feb 28, 2024 at 2:19 PM Goffredo Baroncelli kreijack@libero.it wrote:
On 28/02/2024 18.25, Patrick Plenefisch wrote:
I'm unsure if this is just an LVM bug, or a BTRFS+LVM interaction bug, but LVM is definitely involved somehow. Upgrading from 5.10 to 6.1, I noticed one of my filesystems was read-only. In dmesg, I found:
BTRFS error (device dm-75): bdev /dev/mapper/lvm-brokenDisk errs: wr 0, rd 0, flush 1, corrupt 0, gen 0 BTRFS warning (device dm-75): chunk 13631488 missing 1 devices, max tolerance is 0 for writable mount BTRFS: error (device dm-75) in write_all_supers:4379: errno=-5 IO failure (errors while submitting device barriers.) BTRFS info (device dm-75: state E): forced readonly BTRFS warning (device dm-75: state E): Skipping commit of aborted transaction. BTRFS: error (device dm-75: state EA) in cleanup_transaction:1992: errno=-5 IO failure
At first I suspected a btrfs error, but a scrub found no errors, and it continued to be read-write on 5.10 kernels.
Here is my setup:
/dev/lvm/brokenDisk is a lvm-on-lvm volume. I have /dev/sd{a,b,c,d} (of varying sizes) in a lower VG, which has three LVs, all raid1 volumes. Two of the volumes are further used as PV's for an upper VGs. One of the upper VGs has no issues. The non-PV LV has no issue. The remaining one, /dev/lowerVG/lvmPool, hosting nested LVM, is used as a PV for VG "lvm", and has 3 volumes inside. Two of those volumes have no issues (and are btrfs), but the last one is /dev/lvm/brokenDisk. This volume is the only one that exhibits this behavior, so something is special.
Or described as layers: /dev/sd{a,b,c,d} => PV => VG "lowerVG" /dev/lowerVG/single (RAID1 LV) => BTRFS, works fine /dev/lowerVG/works (RAID1 LV) => PV => VG "workingUpper" /dev/workingUpper/{a,b,c} => BTRFS, works fine /dev/lowerVG/lvmPool (RAID1 LV) => PV => VG "lvm" /dev/lvm/{a,b} => BTRFS, works fine /dev/lvm/brokenDisk => BTRFS, Exhibits errors
I am a bit curious about the reasons of this setup.
The lowerVG is supposed to be a pool of storage for several VM's & containers. [workingUpper] is for one VM, and [lvm] is for another VM. However right now I'm still trying to organize the files directly because I don't have all the VM's fully setup yet
However I understood that:
/dev/sda -+ +-- single (RAID1) -> ok +-> a ok /dev/sdb | | |-> b ok /dev/sdc +--> [lowerVG]>--+-- works (RAID1) -> [workingUpper] -+-> c ok /dev/sdd -+ | | +-> a -> ok +-- lvmPool -> [lvm] ->-| +-> b -> ok | +->brokenDisk -> fail
[xxx] means VG, the others are LVs that may act also as PV in an upper VG
Note that lvmPool is also RAID1, but yes
So, it seems that
- lowerVG/lvmPool/lvm/a
- lowerVG/lvmPool/lvm/a
- lowerVG/lvmPool/lvm/brokenDisk
are equivalent ... so I don't understand how 1) and 2) are fine but 3) is problematic.
I assume you meant lvm/b for 2?
Is my understanding of the LVM layouts correct ?
Your understanding is correct. The only thing that comes to my mind to cause the problem is asymmetry of the SATA devices. I have one 8TB device, plus a 1.5TB, 3TB, and 3TB drives. Doing math on the actual extents, lowerVG/single spans (3TB+3TB), and lowerVG/lvmPool/lvm/brokenDisk spans (3TB+1.5TB). Both obviously have the other leg of raid1 on the 8TB drive, but my thought was that the jump across the 1.5+3TB drive gap was at least "interesting"
After some investigation, here is what I've found:
- This regression was introduced in 5.19. 5.18 and earlier kernels I
can keep this filesystem rw and everything works as expected, while 5.19.0 and later the filesystem is immediately ro on any write attempt. I couldn't build rc1, but I did confirm rc2 already has this regression. 2. Passing /dev/lvm/brokenDisk to a KVM VM as /dev/vdb with an unaffected kernel inside the vm exhibits the ro barrier problem on unaffected kernels.
Is /dev/lvm/brokenDisk *always* problematic with affected ( >= 5.19 ) and UNaffected ( < 5.19 ) kernel ?
Yes, I didn't test it in as much depth, but 5.15 and 6.1 in the VM (and 6.1 on the host) are identically problematic
- Passing /dev/lowerVG/lvmPool to a KVM VM as /dev/vdb with an
affected kernel inside the VM and using LVM inside the VM exhibits correct behavior (I can keep the filesystem rw, no barrier errors on host or guest)
Is /dev/lowerVG/lvmPool problematic with only "affected" kernel ?
Uh, passing lvmPool directly to the VM is never problematic. I tested 5.10 and 6.1 in the VM (and 6.1 on the host), and neither setup throws barrier errors.
[...]
-- gpg @keyserver.linux.it: Goffredo Baroncelli <kreijackATinwind.it> Key fingerprint BBF5 1610 0B64 DAC6 5F7D 17B2 0EDA 9B37 8B82 E0B5
On 28/02/2024 20.37, Patrick Plenefisch wrote:
On Wed, Feb 28, 2024 at 2:19 PM Goffredo Baroncelli kreijack@libero.it wrote:
On 28/02/2024 18.25, Patrick Plenefisch wrote:
I'm unsure if this is just an LVM bug, or a BTRFS+LVM interaction bug, but LVM is definitely involved somehow. Upgrading from 5.10 to 6.1, I noticed one of my filesystems was read-only. In dmesg, I found:
BTRFS error (device dm-75): bdev /dev/mapper/lvm-brokenDisk errs: wr 0, rd 0, flush 1, corrupt 0, gen 0 BTRFS warning (device dm-75): chunk 13631488 missing 1 devices, max tolerance is 0 for writable mount BTRFS: error (device dm-75) in write_all_supers:4379: errno=-5 IO failure (errors while submitting device barriers.) BTRFS info (device dm-75: state E): forced readonly BTRFS warning (device dm-75: state E): Skipping commit of aborted transaction. BTRFS: error (device dm-75: state EA) in cleanup_transaction:1992: errno=-5 IO failure
At first I suspected a btrfs error, but a scrub found no errors, and it continued to be read-write on 5.10 kernels.
Here is my setup:
/dev/lvm/brokenDisk is a lvm-on-lvm volume. I have /dev/sd{a,b,c,d} (of varying sizes) in a lower VG, which has three LVs, all raid1 volumes. Two of the volumes are further used as PV's for an upper VGs. One of the upper VGs has no issues. The non-PV LV has no issue. The remaining one, /dev/lowerVG/lvmPool, hosting nested LVM, is used as a PV for VG "lvm", and has 3 volumes inside. Two of those volumes have no issues (and are btrfs), but the last one is /dev/lvm/brokenDisk. This volume is the only one that exhibits this behavior, so something is special.
Or described as layers: /dev/sd{a,b,c,d} => PV => VG "lowerVG" /dev/lowerVG/single (RAID1 LV) => BTRFS, works fine /dev/lowerVG/works (RAID1 LV) => PV => VG "workingUpper" /dev/workingUpper/{a,b,c} => BTRFS, works fine /dev/lowerVG/lvmPool (RAID1 LV) => PV => VG "lvm" /dev/lvm/{a,b} => BTRFS, works fine /dev/lvm/brokenDisk => BTRFS, Exhibits errors
I am a bit curious about the reasons of this setup.
The lowerVG is supposed to be a pool of storage for several VM's & containers. [workingUpper] is for one VM, and [lvm] is for another VM. However right now I'm still trying to organize the files directly because I don't have all the VM's fully setup yet
However I understood that:
/dev/sda -+ +-- single (RAID1) -> ok +-> a ok /dev/sdb | | |-> b ok /dev/sdc +--> [lowerVG]>--+-- works (RAID1) -> [workingUpper] -+-> c ok /dev/sdd -+ | | +-> a -> ok +-- lvmPool (raid1)-> [lvm] ->-| +-> b -> ok | +->brokenDisk -> fail
[xxx] means VG, the others are LVs that may act also as PV in an upper VG
Note that lvmPool is also RAID1, but yes
So, it seems that
- lowerVG/lvmPool/lvm/a
- lowerVG/lvmPool/lvm/a
- lowerVG/lvmPool/lvm/brokenDisk
are equivalent ... so I don't understand how 1) and 2) are fine but 3) is problematic.
I assume you meant lvm/b for 2?
Yes
Is my understanding of the LVM layouts correct ?
Your understanding is correct. The only thing that comes to my mind to cause the problem is asymmetry of the SATA devices. I have one 8TB device, plus a 1.5TB, 3TB, and 3TB drives. Doing math on the actual extents, lowerVG/single spans (3TB+3TB), and lowerVG/lvmPool/lvm/brokenDisk spans (3TB+1.5TB). Both obviously have the other leg of raid1 on the 8TB drive, but my thought was that the jump across the 1.5+3TB drive gap was at least "interesting"
what about lowerVG/works ?
However yes, I agree that the pair of disks involved may be the answer of the problem.
Could you show us the output of
$ sudo pvdisplay -m
After some investigation, here is what I've found:
- This regression was introduced in 5.19. 5.18 and earlier kernels I
can keep this filesystem rw and everything works as expected, while 5.19.0 and later the filesystem is immediately ro on any write attempt. I couldn't build rc1, but I did confirm rc2 already has this regression. 2. Passing /dev/lvm/brokenDisk to a KVM VM as /dev/vdb with an unaffected kernel inside the vm exhibits the ro barrier problem on unaffected kernels.
Is /dev/lvm/brokenDisk *always* problematic with affected ( >= 5.19 ) and UNaffected ( < 5.19 ) kernel ?
Yes, I didn't test it in as much depth, but 5.15 and 6.1 in the VM (and 6.1 on the host) are identically problematic
- Passing /dev/lowerVG/lvmPool to a KVM VM as /dev/vdb with an
affected kernel inside the VM and using LVM inside the VM exhibits correct behavior (I can keep the filesystem rw, no barrier errors on host or guest)
Is /dev/lowerVG/lvmPool problematic with only "affected" kernel ?
Uh, passing lvmPool directly to the VM is never problematic. I tested 5.10 and 6.1 in the VM (and 6.1 on the host), and neither setup throws barrier errors.
[...]
-- gpg @keyserver.linux.it: Goffredo Baroncelli <kreijackATinwind.it> Key fingerprint BBF5 1610 0B64 DAC6 5F7D 17B2 0EDA 9B37 8B82 E0B5
On Thu, Feb 29, 2024 at 2:56 PM Goffredo Baroncelli kreijack@inwind.it wrote:
Your understanding is correct. The only thing that comes to my mind to cause the problem is asymmetry of the SATA devices. I have one 8TB device, plus a 1.5TB, 3TB, and 3TB drives. Doing math on the actual extents, lowerVG/single spans (3TB+3TB), and lowerVG/lvmPool/lvm/brokenDisk spans (3TB+1.5TB). Both obviously have the other leg of raid1 on the 8TB drive, but my thought was that the jump across the 1.5+3TB drive gap was at least "interesting"
what about lowerVG/works ?
That one is only on two disks, it doesn't span any gaps
However yes, I agree that the pair of disks involved may be the answer of the problem.
Could you show us the output of
$ sudo pvdisplay -m
I trimmed it, but kept the relevant bits (Free PE is thus not correct):
--- Physical volume --- PV Name /dev/lowerVG/lvmPool VG Name lvm PV Size <3.00 TiB / not usable 3.00 MiB Allocatable yes PE Size 4.00 MiB Total PE 786431 Free PE 82943 Allocated PE 703488 PV UUID 7p3LSU-EAHd-xUg0-r9vT-Gzkf-tYFV-mvlU1M
--- Physical Segments --- Physical extent 0 to 159999: Logical volume /dev/lvm/brokenDisk Logical extents 0 to 159999 Physical extent 160000 to 339199: Logical volume /dev/lvm/a Logical extents 0 to 179199 Physical extent 339200 to 349439: Logical volume /dev/lvm/brokenDisk Logical extents 160000 to 170239 Physical extent 349440 to 351999: FREE Physical extent 352000 to 460026: Logical volume /dev/lvm/brokenDisk Logical extents 416261 to 524287 Physical extent 460027 to 540409: FREE Physical extent 540410 to 786430: Logical volume /dev/lvm/brokenDisk Logical extents 170240 to 416260
--- Physical volume --- PV Name /dev/sda3 VG Name lowerVG PV Size <2.70 TiB / not usable 3.00 MiB Allocatable yes PE Size 4.00 MiB Total PE 707154 Free PE 909 Allocated PE 706245 PV UUID W8gJ0P-JuMs-1y3g-b5cO-4RuA-MoFs-3zgKBn
--- Physical Segments --- Physical extent 0 to 52223: Logical volume /dev/lowerVG/single_corig_rimage_0_iorig Logical extents 629330 to 681553 Physical extent 52224 to 628940: Logical volume /dev/lowerVG/single_corig_rimage_0_iorig Logical extents 0 to 576716 Physical extent 628941 to 628941: Logical volume /dev/lowerVG/single_corig_rmeta_0 Logical extents 0 to 0 Physical extent 628942 to 628962: Logical volume /dev/lowerVG/single_corig_rimage_0_iorig Logical extents 681554 to 681574 Physical extent 628963 to 634431: Logical volume /dev/lowerVG/single_corig_rimage_0_imeta Logical extents 0 to 5468 Physical extent 634432 to 654540: FREE Physical extent 654541 to 707153: Logical volume /dev/lowerVG/single_corig_rimage_0_iorig Logical extents 576717 to 629329
--- Physical volume --- PV Name /dev/sdf2 VG Name lowerVG PV Size <7.28 TiB / not usable 4.00 MiB Allocatable yes PE Size 4.00 MiB Total PE 1907645 Free PE 414967 Allocated PE 1492678 PV UUID my0zQM-832Z-HYPD-sNfW-68ms-nddg-lMyWJM
--- Physical Segments --- Physical extent 0 to 0: Logical volume /dev/lowerVG/single_corig_rmeta_1 Logical extents 0 to 0 Physical extent 1 to 681575: Logical volume /dev/lowerVG/single_corig_rimage_1_iorig Logical extents 0 to 681574 Physical extent 681576 to 687044: Logical volume /dev/lowerVG/single_corig_rimage_1_imeta Logical extents 0 to 5468 Physical extent 687045 to 687045: Logical volume /dev/lowerVG/lvmPool_rmeta_0 Logical extents 0 to 0 Physical extent 687046 to 1049242: Logical volume /dev/lowerVG/lvmPool_rimage_0 Logical extents 0 to 362196 Physical extent 1049243 to 1056551: FREE Physical extent 1056552 to 1473477: Logical volume /dev/lowerVG/lvmPool_rimage_0 Logical extents 369506 to 786431 Physical extent 1473478 to 1480786: Logical volume /dev/lowerVG/lvmPool_rimage_0 Logical extents 362197 to 369505 Physical extent 1480787 to 1907644: FREE
--- Physical volume --- PV Name /dev/sdb3 VG Name lowerVG PV Size 1.33 TiB / not usable 3.00 MiB Allocatable yes (but full) PE Size 4.00 MiB Total PE 349398 Free PE 0 Allocated PE 349398 PV UUID Ncmgdw-ZOXS-qTYL-1jAz-w7zt-38V2-f53EpI
--- Physical Segments --- Physical extent 0 to 0: Logical volume /dev/lowerVG/lvmPool_rmeta_1 Logical extents 0 to 0 Physical extent 1 to 349397: Logical volume /dev/lowerVG/lvmPool_rimage_1 Logical extents 0 to 349396
--- Physical volume --- PV Name /dev/sde2 VG Name lowerVG PV Size 2.71 TiB / not usable 3.00 MiB Allocatable yes PE Size 4.00 MiB Total PE 711346 Free PE 255111 Allocated PE 456235 PV UUID xUG8TG-wvp0-roBo-GPo7-sbvn-aE7I-NAHU07
--- Physical Segments --- Physical extent 0 to 416925: Logical volume /dev/lowerVG/lvmPool_rimage_1 Logical extents 369506 to 786431 Physical extent 416926 to 437034: Logical volume /dev/lowerVG/lvmPool_rimage_1 Logical extents 349397 to 369505 Physical extent 437035 to 711345: FREE
Finally, I am not sure if it's relevant, but I did struggle to expand the raid1 volumes across gaps when creating this setup. I did file a bug about that, though I am not sure if it's relevant, as I removed integrity and cache for brokenDisk & lvmPool: https://gitlab.com/lvmteam/lvm2/-/issues/6
Patrick
On 29/02/2024 21.22, Patrick Plenefisch wrote:
On Thu, Feb 29, 2024 at 2:56 PM Goffredo Baroncelli kreijack@inwind.it wrote:
Your understanding is correct. The only thing that comes to my mind to cause the problem is asymmetry of the SATA devices. I have one 8TB device, plus a 1.5TB, 3TB, and 3TB drives. Doing math on the actual extents, lowerVG/single spans (3TB+3TB), and lowerVG/lvmPool/lvm/brokenDisk spans (3TB+1.5TB). Both obviously have the other leg of raid1 on the 8TB drive, but my thought was that the jump across the 1.5+3TB drive gap was at least "interesting"
what about lowerVG/works ?
That one is only on two disks, it doesn't span any gaps
Sorry, but re-reading the original email I found something that I missed before:
BTRFS error (device dm-75): bdev /dev/mapper/lvm-brokenDisk errs: wr 0, rd 0, flush 1, corrupt 0, gen 0 BTRFS warning (device dm-75): chunk 13631488 missing 1 devices, max
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
tolerance is 0 for writable mount BTRFS: error (device dm-75) in write_all_supers:4379: errno=-5 IO failure (errors while submitting device barriers.)
Looking at the code, it seems that if a FLUSH commands fails, btrfs considers that the disk is missing. The it cannot mount RW the device.
I would investigate with the LVM developers, if it properly passes the flush/barrier command through all the layers, when we have an lvm over lvm (raid1). The fact that the lvm is a raid1, is important because a flush command to be honored has to be honored by all the devices involved.
However yes, I agree that the pair of disks involved may be the answer of the problem.
Could you show us the output of
$ sudo pvdisplay -m
I trimmed it, but kept the relevant bits (Free PE is thus not correct):
--- Physical volume --- PV Name /dev/lowerVG/lvmPool VG Name lvm PV Size <3.00 TiB / not usable 3.00 MiB Allocatable yes PE Size 4.00 MiB Total PE 786431 Free PE 82943 Allocated PE 703488 PV UUID 7p3LSU-EAHd-xUg0-r9vT-Gzkf-tYFV-mvlU1M
--- Physical Segments --- Physical extent 0 to 159999: Logical volume /dev/lvm/brokenDisk Logical extents 0 to 159999 Physical extent 160000 to 339199: Logical volume /dev/lvm/a Logical extents 0 to 179199 Physical extent 339200 to 349439: Logical volume /dev/lvm/brokenDisk Logical extents 160000 to 170239 Physical extent 349440 to 351999: FREE Physical extent 352000 to 460026: Logical volume /dev/lvm/brokenDisk Logical extents 416261 to 524287 Physical extent 460027 to 540409: FREE Physical extent 540410 to 786430: Logical volume /dev/lvm/brokenDisk Logical extents 170240 to 416260
--- Physical volume --- PV Name /dev/sda3 VG Name lowerVG PV Size <2.70 TiB / not usable 3.00 MiB Allocatable yes PE Size 4.00 MiB Total PE 707154 Free PE 909 Allocated PE 706245 PV UUID W8gJ0P-JuMs-1y3g-b5cO-4RuA-MoFs-3zgKBn
--- Physical Segments --- Physical extent 0 to 52223: Logical volume /dev/lowerVG/single_corig_rimage_0_iorig Logical extents 629330 to 681553 Physical extent 52224 to 628940: Logical volume /dev/lowerVG/single_corig_rimage_0_iorig Logical extents 0 to 576716 Physical extent 628941 to 628941: Logical volume /dev/lowerVG/single_corig_rmeta_0 Logical extents 0 to 0 Physical extent 628942 to 628962: Logical volume /dev/lowerVG/single_corig_rimage_0_iorig Logical extents 681554 to 681574 Physical extent 628963 to 634431: Logical volume /dev/lowerVG/single_corig_rimage_0_imeta Logical extents 0 to 5468 Physical extent 634432 to 654540: FREE Physical extent 654541 to 707153: Logical volume /dev/lowerVG/single_corig_rimage_0_iorig Logical extents 576717 to 629329
--- Physical volume --- PV Name /dev/sdf2 VG Name lowerVG PV Size <7.28 TiB / not usable 4.00 MiB Allocatable yes PE Size 4.00 MiB Total PE 1907645 Free PE 414967 Allocated PE 1492678 PV UUID my0zQM-832Z-HYPD-sNfW-68ms-nddg-lMyWJM
--- Physical Segments --- Physical extent 0 to 0: Logical volume /dev/lowerVG/single_corig_rmeta_1 Logical extents 0 to 0 Physical extent 1 to 681575: Logical volume /dev/lowerVG/single_corig_rimage_1_iorig Logical extents 0 to 681574 Physical extent 681576 to 687044: Logical volume /dev/lowerVG/single_corig_rimage_1_imeta Logical extents 0 to 5468 Physical extent 687045 to 687045: Logical volume /dev/lowerVG/lvmPool_rmeta_0 Logical extents 0 to 0 Physical extent 687046 to 1049242: Logical volume /dev/lowerVG/lvmPool_rimage_0 Logical extents 0 to 362196 Physical extent 1049243 to 1056551: FREE Physical extent 1056552 to 1473477: Logical volume /dev/lowerVG/lvmPool_rimage_0 Logical extents 369506 to 786431 Physical extent 1473478 to 1480786: Logical volume /dev/lowerVG/lvmPool_rimage_0 Logical extents 362197 to 369505 Physical extent 1480787 to 1907644: FREE
--- Physical volume --- PV Name /dev/sdb3 VG Name lowerVG PV Size 1.33 TiB / not usable 3.00 MiB Allocatable yes (but full) PE Size 4.00 MiB Total PE 349398 Free PE 0 Allocated PE 349398 PV UUID Ncmgdw-ZOXS-qTYL-1jAz-w7zt-38V2-f53EpI
--- Physical Segments --- Physical extent 0 to 0: Logical volume /dev/lowerVG/lvmPool_rmeta_1 Logical extents 0 to 0 Physical extent 1 to 349397: Logical volume /dev/lowerVG/lvmPool_rimage_1 Logical extents 0 to 349396
--- Physical volume --- PV Name /dev/sde2 VG Name lowerVG PV Size 2.71 TiB / not usable 3.00 MiB Allocatable yes PE Size 4.00 MiB Total PE 711346 Free PE 255111 Allocated PE 456235 PV UUID xUG8TG-wvp0-roBo-GPo7-sbvn-aE7I-NAHU07
--- Physical Segments --- Physical extent 0 to 416925: Logical volume /dev/lowerVG/lvmPool_rimage_1 Logical extents 369506 to 786431 Physical extent 416926 to 437034: Logical volume /dev/lowerVG/lvmPool_rimage_1 Logical extents 349397 to 369505 Physical extent 437035 to 711345: FREE
Finally, I am not sure if it's relevant, but I did struggle to expand the raid1 volumes across gaps when creating this setup. I did file a bug about that, though I am not sure if it's relevant, as I removed integrity and cache for brokenDisk & lvmPool: https://gitlab.com/lvmteam/lvm2/-/issues/6
Patrick
linux-stable-mirror@lists.linaro.org
-
Goffredo Baroncelli -
Goffredo Baroncelli -
Patrick Plenefisch