On 29/02/2024 21.22, Patrick Plenefisch wrote:
On Thu, Feb 29, 2024 at 2:56 PM Goffredo Baroncelli kreijack@inwind.it wrote:
Your understanding is correct. The only thing that comes to my mind to cause the problem is asymmetry of the SATA devices. I have one 8TB device, plus a 1.5TB, 3TB, and 3TB drives. Doing math on the actual extents, lowerVG/single spans (3TB+3TB), and lowerVG/lvmPool/lvm/brokenDisk spans (3TB+1.5TB). Both obviously have the other leg of raid1 on the 8TB drive, but my thought was that the jump across the 1.5+3TB drive gap was at least "interesting"
what about lowerVG/works ?
That one is only on two disks, it doesn't span any gaps
Sorry, but re-reading the original email I found something that I missed before:
BTRFS error (device dm-75): bdev /dev/mapper/lvm-brokenDisk errs: wr 0, rd 0, flush 1, corrupt 0, gen 0 BTRFS warning (device dm-75): chunk 13631488 missing 1 devices, max
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
tolerance is 0 for writable mount BTRFS: error (device dm-75) in write_all_supers:4379: errno=-5 IO failure (errors while submitting device barriers.)
Looking at the code, it seems that if a FLUSH commands fails, btrfs considers that the disk is missing. The it cannot mount RW the device.
I would investigate with the LVM developers, if it properly passes the flush/barrier command through all the layers, when we have an lvm over lvm (raid1). The fact that the lvm is a raid1, is important because a flush command to be honored has to be honored by all the devices involved.
However yes, I agree that the pair of disks involved may be the answer of the problem.
Could you show us the output of
$ sudo pvdisplay -m
I trimmed it, but kept the relevant bits (Free PE is thus not correct):
--- Physical volume --- PV Name /dev/lowerVG/lvmPool VG Name lvm PV Size <3.00 TiB / not usable 3.00 MiB Allocatable yes PE Size 4.00 MiB Total PE 786431 Free PE 82943 Allocated PE 703488 PV UUID 7p3LSU-EAHd-xUg0-r9vT-Gzkf-tYFV-mvlU1M
--- Physical Segments --- Physical extent 0 to 159999: Logical volume /dev/lvm/brokenDisk Logical extents 0 to 159999 Physical extent 160000 to 339199: Logical volume /dev/lvm/a Logical extents 0 to 179199 Physical extent 339200 to 349439: Logical volume /dev/lvm/brokenDisk Logical extents 160000 to 170239 Physical extent 349440 to 351999: FREE Physical extent 352000 to 460026: Logical volume /dev/lvm/brokenDisk Logical extents 416261 to 524287 Physical extent 460027 to 540409: FREE Physical extent 540410 to 786430: Logical volume /dev/lvm/brokenDisk Logical extents 170240 to 416260
--- Physical volume --- PV Name /dev/sda3 VG Name lowerVG PV Size <2.70 TiB / not usable 3.00 MiB Allocatable yes PE Size 4.00 MiB Total PE 707154 Free PE 909 Allocated PE 706245 PV UUID W8gJ0P-JuMs-1y3g-b5cO-4RuA-MoFs-3zgKBn
--- Physical Segments --- Physical extent 0 to 52223: Logical volume /dev/lowerVG/single_corig_rimage_0_iorig Logical extents 629330 to 681553 Physical extent 52224 to 628940: Logical volume /dev/lowerVG/single_corig_rimage_0_iorig Logical extents 0 to 576716 Physical extent 628941 to 628941: Logical volume /dev/lowerVG/single_corig_rmeta_0 Logical extents 0 to 0 Physical extent 628942 to 628962: Logical volume /dev/lowerVG/single_corig_rimage_0_iorig Logical extents 681554 to 681574 Physical extent 628963 to 634431: Logical volume /dev/lowerVG/single_corig_rimage_0_imeta Logical extents 0 to 5468 Physical extent 634432 to 654540: FREE Physical extent 654541 to 707153: Logical volume /dev/lowerVG/single_corig_rimage_0_iorig Logical extents 576717 to 629329
--- Physical volume --- PV Name /dev/sdf2 VG Name lowerVG PV Size <7.28 TiB / not usable 4.00 MiB Allocatable yes PE Size 4.00 MiB Total PE 1907645 Free PE 414967 Allocated PE 1492678 PV UUID my0zQM-832Z-HYPD-sNfW-68ms-nddg-lMyWJM
--- Physical Segments --- Physical extent 0 to 0: Logical volume /dev/lowerVG/single_corig_rmeta_1 Logical extents 0 to 0 Physical extent 1 to 681575: Logical volume /dev/lowerVG/single_corig_rimage_1_iorig Logical extents 0 to 681574 Physical extent 681576 to 687044: Logical volume /dev/lowerVG/single_corig_rimage_1_imeta Logical extents 0 to 5468 Physical extent 687045 to 687045: Logical volume /dev/lowerVG/lvmPool_rmeta_0 Logical extents 0 to 0 Physical extent 687046 to 1049242: Logical volume /dev/lowerVG/lvmPool_rimage_0 Logical extents 0 to 362196 Physical extent 1049243 to 1056551: FREE Physical extent 1056552 to 1473477: Logical volume /dev/lowerVG/lvmPool_rimage_0 Logical extents 369506 to 786431 Physical extent 1473478 to 1480786: Logical volume /dev/lowerVG/lvmPool_rimage_0 Logical extents 362197 to 369505 Physical extent 1480787 to 1907644: FREE
--- Physical volume --- PV Name /dev/sdb3 VG Name lowerVG PV Size 1.33 TiB / not usable 3.00 MiB Allocatable yes (but full) PE Size 4.00 MiB Total PE 349398 Free PE 0 Allocated PE 349398 PV UUID Ncmgdw-ZOXS-qTYL-1jAz-w7zt-38V2-f53EpI
--- Physical Segments --- Physical extent 0 to 0: Logical volume /dev/lowerVG/lvmPool_rmeta_1 Logical extents 0 to 0 Physical extent 1 to 349397: Logical volume /dev/lowerVG/lvmPool_rimage_1 Logical extents 0 to 349396
--- Physical volume --- PV Name /dev/sde2 VG Name lowerVG PV Size 2.71 TiB / not usable 3.00 MiB Allocatable yes PE Size 4.00 MiB Total PE 711346 Free PE 255111 Allocated PE 456235 PV UUID xUG8TG-wvp0-roBo-GPo7-sbvn-aE7I-NAHU07
--- Physical Segments --- Physical extent 0 to 416925: Logical volume /dev/lowerVG/lvmPool_rimage_1 Logical extents 369506 to 786431 Physical extent 416926 to 437034: Logical volume /dev/lowerVG/lvmPool_rimage_1 Logical extents 349397 to 369505 Physical extent 437035 to 711345: FREE
Finally, I am not sure if it's relevant, but I did struggle to expand the raid1 volumes across gaps when creating this setup. I did file a bug about that, though I am not sure if it's relevant, as I removed integrity and cache for brokenDisk & lvmPool: https://gitlab.com/lvmteam/lvm2/-/issues/6
Patrick