Linaro-mm-sig

linaro-mm-sig@lists.linaro.org

128 participants
7046 discussions

by firmadan＠hotmail.com

Modern mimarinin vazgeçilmez yapı elemanlarından biri haline gelen demir döner merdivenler, hem estetik görünümü hem de uzun ömürlü kullanımıyla yaşam alanlarına değer katmaktadır. Özellikle dar alanlarda maksimum kullanım avantajı sunan bu merdiven sistemleri, evlerden iş yerlerine, dubleks dairelerden villa projelerine kadar birçok farklı alanda tercih edilmektedir. Dayanıklı metal yapısı sayesinde yüksek taşıma kapasitesine sahip olan demir döner merdiven modelleri, güvenli kullanım sunarken dekoratif tasarımlarıyla da dikkat çeker. Klasik, modern, endüstriyel veya minimalist dekorasyon stillerine uyum sağlayabilen bu özel merdivenler, mekânın atmosferini tamamen değiştirebilir. Demir döner merdivenlerin en önemli avantajlarından biri yer tasarrufu sağlamasıdır. Geleneksel düz merdivenlere göre daha az alan kaplayan spiral tasarım, özellikle küçük metrekareli alanlarda büyük avantaj sunar. Bu sayede hem kullanışlı hem de şık bir geçiş alanı oluşturulur. Üretim aşamasında kullanılan kaliteli demir malzeme, paslanmaya ve deformasyona karşı yüksek direnç sağlar. Elektrostatik boya uygulamaları sayesinde merdiven yüzeyi uzun yıllar ilk günkü görünümünü korur. İç mekân kullanımının yanı sıra dış mekân projelerinde de tercih edilen demir döner merdivenler, hava koşullarına dayanıklı yapısıyla güven verir. Kişiye özel ölçülerde üretilebilen modeller sayesinde her projeye uygun çözümler sunulabilir. Basamak genişliği, korkuluk tasarımı, renk seçenekleri ve merdiven çapı tamamen ihtiyaca göre şekillendirilebilir. Ahşap basamak detaylarıyla sıcak bir görünüm elde edilebilirken tamamen metal tasarımlarla modern ve güçlü bir atmosfer oluşturulabilir. Demir döner merdivenler sadece bir geçiş aracı değil, aynı zamanda dekoratif bir mimari unsur olarak da öne çıkar. Özellikle loft daireler, teras bağlantıları, çatı katları ve mağaza iç tasarımlarında estetik görünümüyle dikkat çekmektedir. Hem fonksiyonel hem de görsel açıdan güçlü bir çözüm arayanlar için ideal seçenekler arasında yer alır. Uzun ömürlü kullanım, sağlam yapı, estetik görünüm ve alan tasarrufu avantajlarını bir araya getiren demir döner merdiven sistemleri, modern yaşam alanlarının vazgeçilmez tercihleri arasında yer almaktadır. Siz de yaşam alanınıza özel tasarlanmış kaliteli ve şık bir merdiven çözümü ile mekânlarınıza değer katabilirsiniz. Demir döner merdiven https://fakrocatimerdivenleri.com/doner-merdivenler/

1 month, 2 weeks

Fakro Çatı Merdivenleri

by firmadan＠hotmail.com

Fakro çatı merdivenleri, modern yaşam alanlarında çatı katına güvenli, konforlu ve pratik erişim sağlamak için geliştirilmiş yenilikçi çözümler arasında yer almaktadır. Dayanıklı yapısı, estetik tasarımı ve uzun ömürlü kullanım avantajı ile dikkat çeken Fakro çatı merdivenleri; evler, dubleks daireler, villalar, ofisler ve depo alanları için ideal bir kullanım sunar. Kullanılmadığı zaman katlanabilir yapısı sayesinde minimum alan kaplayan bu sistemler, yaşam alanlarında ekstra yer tasarrufu sağlamaktadır. Fakro’nun geliştirdiği çatı merdiveni modelleri; ahşap, metal ve makaslı sistem seçenekleriyle farklı ihtiyaçlara hitap eder. Isı yalıtımlı kapak yapıları sayesinde enerji kaybını minimum seviyeye indirirken, özellikle kış aylarında iç mekan sıcaklığının korunmasına yardımcı olur. Kaymaz basamak yapısı ve sağlam menteşe sistemi ise kullanıcı güvenliğini üst seviyeye taşır. Böylece hem güvenli hem de ergonomik bir kullanım deneyimi elde edilir. Kurulum açısından oldukça pratik olan Fakro çatı merdivenleri, tavan boşluklarına uyum sağlayan özel ölçü seçenekleri ile üretilmektedir. Tavana tam entegre olan yapısı sayesinde dekoratif açıdan da estetik bir görünüm sunar. Açılıp kapanma mekanizmasının kolay çalışması, günlük kullanım sırasında büyük avantaj sağlar. Özellikle dar alanlarda maksimum verim almak isteyen kullanıcılar için son derece işlevsel bir çözüm oluşturur. Fakro ahşap çatı merdivenleri doğal görünümü ile yaşam alanlarına sıcak bir atmosfer kazandırırken, metal çatı merdivenleri yüksek taşıma kapasitesi ve dayanıklılığı ile öne çıkar. Makaslı alüminyum modeller ise kompakt yapıları sayesinde küçük tavan girişlerinde bile rahat kullanım imkanı sunmaktadır. Her modelde kalite standartlarının yüksek tutulması, ürünlerin uzun yıllar sorunsuz kullanılmasına katkı sağlar. Fakro çatı merdivenleri https://www.fakrocatimerdivenleri.com

1 month, 2 weeks

Top 10 Verified Recovery Firms Available Hire Primatz Guard

by harperquinn708＠gmail.com

Primatz Guard helps victims of crypto, romance, and investment scams trace funds, investigate fraud, and seek recovery with transparent consultation support.

1 month, 3 weeks

[PATCH 0/8] drm/panthor: Protected mode support for Mali CSF GPUs

by Ketil Johnsen

Hi, This is a patch series covering the support for protected mode execution in Mali Panthor CSF kernel driver. It builds on the initial RFC posted by Florent Tomasin back in January of 2025. The initial RFC can be found here: https://lore.kernel.org/lkml/cover.1738228114.git.florent.tomasin@arm.com/ The Mali CSF GPUs come with the support for protected mode execution at the HW level. This feature requires two main changes in the kernel driver: 1) Configure the GPU with a protected buffer. The system must provide a DMA heap from which the driver can allocate a protected buffer. It can be a carved-out memory or dynamically allocated protected memory region. Some system includes a trusted FW which is in charge of the protected memory. Since this problem is integration specific, the Mali Panthor CSF kernel driver must import the protected memory from a device specific exporter. 2) Handle enter and exit of the GPU HW from normal to protected mode of execution. FW sends a request for protected mode entry to the kernel driver. The acknowledgment of that request is a scheduling decision. Effectively, protected mode execution should not overrule normal mode of execution. A fair distribution of execution time will guaranty the overall performance of the device, including the UI (usually executing in normal mode), will not regress when a protected mode job is submitted by an application. Background ---------- Current Mali Panthor CSF driver does not allow a user space application to execute protected jobs on the GPU. This use case is quite common on end-user-device. A user may want to watch a video or render content that is under a "Digital Right Management" protection, or launch an application with user private data. 1) User-space: In order for an application to execute protected jobs on a Mali CSF GPU the user space application must submit jobs to the GPU within a "protected regions" (range of commands to execute in protected mode). Find here an example of a command buffer that contains protected commands: ``` <--- Normal mode ---><--- Protected mode ---><--- Normal mode ---> +-------------------------------------------------------------------------+ | ... | CMD_0 | ... | CMD_N | PROT_REGION | CMD_N+1 | ... | CMD_N+M | ... | +-------------------------------------------------------------------------+ ``` The PROT_REGION command acts as a barrier to notify the HW of upcoming protected jobs. It also defines the number of commands to execute in protected mode. The Mesa definition of the opcode can be found here: https://gitlab.freedesktop.org/mesa/mesa/-/blob/main/src/panfrost/lib/genxm… 2) Kernel-space: When loading the FW image, the Kernel driver must also load the data section of CSF FW that comes from the protected memory, in order to allow FW to execute in protected mode. Important: this memory is not owned by any process. It is a GPU device level protected memory. In addition, when a CSG (group) is created, it must have a protected suspend buffer. This memory is allocated within the kernel but bound to a specific CSG that belongs to a process. The kernel owns this allocation and does not allow user space mapping. The format of the data in this buffer is only known by the FW and does not need to be shared with other entities. The purpose of this buffer is the same as the normal suspend buffer but for protected mode. FW will use it to suspend the execution of PROT_REGION before returning to normal mode of execution. Design decisions ---------------- The Mali Panthor CSF kernel driver will allocate protected DMA buffers using a global protected DMA heap. The name of the heap can vary on the system and is integration specific. Therefore, the kernel driver will retrieve it using the DTB entry: "protected-heap-name". The Mali Panthor CSF kernel driver will handle enter/exit of protected mode with a fair consideration of the job scheduling. If the system integrator does not provide a protected DMA heap, the driver will not allow any protected mode execution. Patch series ------------ [PATCHES 1-2]: Thees patches comes from the following patch series: https://lore.kernel.org/lkml/20240720071606.27930-1-yunfei.dong@mediatek.co… These extend the DMA-buf heap API to allow other kernel drivers to Find and allocate memory from dma heaps. Note: This patch series do not include a protected DMA heap, as this is platform specific. * dma-heap: Add proper kref handling on dma-buf heaps * dma-heap: Provide accessors so that in-kernel drivers can allocate dmabufs from specific heaps [PATCHES 3, 5 and 6]: These are refactoring to aid the implementation of the protected rendering feature itself. * drm/panthor: De-duplicate FW memory section sync * drm/panthor: Minor scheduler refactoring * drm/panthor: Explicit expansion of locked VM region [Patch 4]: This introduces allocation of protected memory inside the Panthor driver. It also ensures the protected FW sections are loaded. * drm/panthor: Add support for protected memory allocation in panthor [PATCH 7]: This patch implements the logic to handle enter/exit of the GPU protected mode in Panthor CSF driver. Note: to prevent scheduler priority inversion, only a single CSG is allowed to execute while in protected mode. It must be the top priority one. * drm/panthor: Add support for entering and exiting protected mode [PATCH 8]: The final patch exposes this feature via the uAPI. * drm/panthor: Expose protected rendering features Testing ------- 1) Platform and development environment Any platform containing a Mali CSF type of GPU and a protected memory allocator that is based on DMA Heap can be used. For example, it can be a physical platform or a simulator such as Arm Total Compute FVPs platforms. Reference to the latter: https://developer.arm.com/Tools%20and%20Software/Fixed%20Virtual%20Platform… 2) Mesa: PanVK support can be found here: https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/40044 This is still work in progress. Constraints ----------- At the time of developing the feature, Linux kernel does not have a generic way of implementing protected DMA heaps. The patch series relies on previous work to expose the DMA heap API to the kernel drivers. The Mali CSF GPU requires device level allocated protected memory, which do not belong to a process. The current Linux implementation of DMA heap only allows a user space program to allocate from such heap. Having the ability to allocate this memory at the kernel level via the DMA heap API would allow support for protected mode on Mali CSF GPUs. Florent Tomasin (3): drm/panthor: Add support for protected memory allocation in panthor drm/panthor: Minor scheduler refactoring drm/panthor: Add support for entering and exiting protected mode John Stultz (2): dma-heap: Add proper kref handling on dma-buf heaps dma-heap: Provide accessors so that in-kernel drivers can allocate dmabufs from specific heaps Ketil Johnsen (3): drm/panthor: De-duplicate FW memory section sync drm/panthor: Explicit expansion of locked VM region drm/panthor: Expose protected rendering features Documentation/gpu/panthor.rst | 47 +++ drivers/dma-buf/dma-heap.c | 109 ++++++- drivers/gpu/drm/panthor/Kconfig | 1 + drivers/gpu/drm/panthor/panthor_device.c | 29 +- drivers/gpu/drm/panthor/panthor_device.h | 15 + drivers/gpu/drm/panthor/panthor_drv.c | 21 +- drivers/gpu/drm/panthor/panthor_fw.c | 137 ++++++-- drivers/gpu/drm/panthor/panthor_fw.h | 7 + drivers/gpu/drm/panthor/panthor_gem.c | 77 ++++- drivers/gpu/drm/panthor/panthor_gem.h | 16 +- drivers/gpu/drm/panthor/panthor_gpu.c | 14 +- drivers/gpu/drm/panthor/panthor_gpu.h | 6 + drivers/gpu/drm/panthor/panthor_heap.c | 2 + drivers/gpu/drm/panthor/panthor_mmu.c | 79 +++-- drivers/gpu/drm/panthor/panthor_sched.c | 387 ++++++++++++++++++----- include/linux/dma-heap.h | 8 + include/uapi/drm/panthor_drm.h | 45 ++- 17 files changed, 864 insertions(+), 136 deletions(-) -- 2.43.0

1 month, 3 weeks

Re: [RFC PATCH 04/12] vfio/pci: Allow MMIO regions to be exported through dma-buf

by Alexey Kardashevskiy

Hi! Let's reignite this topic. I've been using these patches + QEMU side hacks for 6+ months. And it's been fine until I got a device where MSIX BAR is in a middle of another BAR marked as TEE in the TDISP interface report. And no trusted MSIX yet. Every time QEMU mmaps a BAR - I request a dmabuf fd from VFIO in QEMU. Since mapping of an entire MSIX BAR is allowed by default, VFIORegion::nr_mmaps==1 and it is an entire BAR. Problem: KVM memslot mismatches the dmabuf fd size How: QEMU emulates MSIX table and PBA by adding emulated MemoryRegions on top of the mapped BARs. In the QEMU memory flatview this splits the BAR into 2 or 3 sections (2 if MSIX at the start/end, 3 if in a middle). QEMU tries registering 1 or 2 KVM memory slots for regions outside of MSIX which fails in kvm_vfio_dmabuf_bind() as regions are smaller than the exported dmabuf fd (which is the entire BAR == 32KB). Solution1: use QEMU x-msix-relocation hack to move MSIX elsewhere - end of some BAR (doubles the bar size so problematic for huge BARs like 512GB+), or another BAR (there may be no available as 3 of 64bit BARs is the limit). Solution2: modify logic in VFIO dmabuf to allow multiple KVM memory slots per dmabuf. Now it is kvm_memory_slot::dmabuf_attach with no offset into the dmabuf and one kvm_vfio_dmabuf per dma_buf. Solution3: hack QEMU into smashing a MSIX-containing BAR in vfio_pci_fixup_msix_region. Here is what it does: 0000380004000000-0000380004002fff (prio 0, ramd): 0000:41:04.0 BAR 4 mmaps[0] KVM 0000380004003000-0000380004006fff (prio 0, i/o): msix-table 0000380004007000-0000380004007fff (prio 0, ramd): 0000:41:04.0 BAR 4 mmaps[1] KVM The problem now is that the TDI report must have the same split of the BAR as the VM is going to validate TEE (==trusted) MMIO ranges and this has to match the QEMU view. Harder than it sounds as the size of MSIX table in bytes is not exactly specified anywhere except the report. Solution4: the above + modify QEMU to check the TDI report for not reporting MSIX+PBA where QEMU emulates them. The problem is that when QEMU adds emulated MRs - there is no report yet (the report is created later on, some TDISP witchery). So when the device is accepted - QEMU could reinitialize those emulated msix and pba MRs to match the report exactly so the flatview generates correct KVM memory regions and then KVM can work with dmabuf fine. Any thoughts? what is acceptable for everybody? Thanks, On 8/1/25 01:27, Xu Yilun wrote: > From: Vivek Kasireddy <vivek.kasireddy(a)intel.com> > > This is a reduced version of Vivek's series [1]. Removed the > dma_buf_ops.attach/map/unmap_dma_buf/mmap() as they are not necessary > in this series, also because of the WIP p2p dma mapping opens [2]. Just > focus on the private MMIO get PFN function in this early stage. > >>From Jason Gunthorpe: > "dma-buf has become a way to safely acquire a handle to non-struct page > memory that can still have lifetime controlled by the exporter. Notably > RDMA can now import dma-buf FDs and build them into MRs which allows for > PCI P2P operations. Extend this to allow vfio-pci to export MMIO memory > from PCI device BARs. > > The patch design loosely follows the pattern in commit > db1a8dd916aa ("habanalabs: add support for dma-buf exporter") except this > does not support pinning. > > Instead, this implements what, in the past, we've called a revocable > attachment using move. In normal situations the attachment is pinned, as a > BAR does not change physical address. However when the VFIO device is > closed, or a PCI reset is issued, access to the MMIO memory is revoked. > > Revoked means that move occurs, but an attempt to immediately re-map the > memory will fail. In the reset case a future move will be triggered when > MMIO access returns. As both close and reset are under userspace control > it is expected that userspace will suspend use of the dma-buf before doing > these operations, the revoke is purely for kernel self-defense against a > hostile userspace." > > [1] https://lore.kernel.org/kvm/20240624065552.1572580-4-vivek.kasireddy@intel.… > [2] https://lore.kernel.org/all/IA0PR11MB7185FDD56CFDD0A2B8D21468F83B2@IA0PR11M… > > Original-patch-by: Jason Gunthorpe <jgg(a)nvidia.com> > Signed-off-by: Vivek Kasireddy <vivek.kasireddy(a)intel.com> > Signed-off-by: Xu Yilun <yilun.xu(a)linux.intel.com> > --- > drivers/vfio/pci/Makefile | 1 + > drivers/vfio/pci/dma_buf.c | 223 +++++++++++++++++++++++++++++ > drivers/vfio/pci/vfio_pci_config.c | 22 ++- > drivers/vfio/pci/vfio_pci_core.c | 20 ++- > drivers/vfio/pci/vfio_pci_priv.h | 25 ++++ > include/linux/vfio_pci_core.h | 1 + > include/uapi/linux/vfio.h | 29 ++++ > 7 files changed, 316 insertions(+), 5 deletions(-) > create mode 100644 drivers/vfio/pci/dma_buf.c > > diff --git a/drivers/vfio/pci/Makefile b/drivers/vfio/pci/Makefile > index cf00c0a7e55c..0cfdc9ede82f 100644 > --- a/drivers/vfio/pci/Makefile > +++ b/drivers/vfio/pci/Makefile > @@ -2,6 +2,7 @@ > > vfio-pci-core-y := vfio_pci_core.o vfio_pci_intrs.o vfio_pci_rdwr.o vfio_pci_config.o > vfio-pci-core-$(CONFIG_VFIO_PCI_ZDEV_KVM) += vfio_pci_zdev.o > +vfio-pci-core-$(CONFIG_DMA_SHARED_BUFFER) += dma_buf.o > obj-$(CONFIG_VFIO_PCI_CORE) += vfio-pci-core.o > > vfio-pci-y := vfio_pci.o > diff --git a/drivers/vfio/pci/dma_buf.c b/drivers/vfio/pci/dma_buf.c > new file mode 100644 > index 000000000000..1d5f46744922 > --- /dev/null > +++ b/drivers/vfio/pci/dma_buf.c > @@ -0,0 +1,223 @@ > +// SPDX-License-Identifier: GPL-2.0-only > +/* Copyright (c) 2024, NVIDIA CORPORATION & AFFILIATES. > + */ > +#include <linux/dma-buf.h> > +#include <linux/dma-resv.h> > + > +#include "vfio_pci_priv.h" > + > +MODULE_IMPORT_NS("DMA_BUF"); > + > +struct vfio_pci_dma_buf { > + struct dma_buf *dmabuf; > + struct vfio_pci_core_device *vdev; > + struct list_head dmabufs_elm; > + unsigned int nr_ranges; > + struct vfio_region_dma_range *dma_ranges; > + bool revoked; > +}; > + > +static void vfio_pci_dma_buf_unpin(struct dma_buf_attachment *attachment) > +{ > +} > + > +static int vfio_pci_dma_buf_pin(struct dma_buf_attachment *attachment) > +{ > + /* > + * Uses the dynamic interface but must always allow for > + * dma_buf_move_notify() to do revoke > + */ > + return -EINVAL; > +} > + > +static int vfio_pci_dma_buf_get_pfn(struct dma_buf_attachment *attachment, > + pgoff_t pgoff, u64 *pfn, int *max_order) > +{ > + /* TODO */ > + return -EOPNOTSUPP; > +} > + > +static void vfio_pci_dma_buf_release(struct dma_buf *dmabuf) > +{ > + struct vfio_pci_dma_buf *priv = dmabuf->priv; > + > + /* > + * Either this or vfio_pci_dma_buf_cleanup() will remove from the list. > + * The refcount prevents both. > + */ > + if (priv->vdev) { > + down_write(&priv->vdev->memory_lock); > + list_del_init(&priv->dmabufs_elm); > + up_write(&priv->vdev->memory_lock); > + vfio_device_put_registration(&priv->vdev->vdev); > + } > + kfree(priv); > +} > + > +static const struct dma_buf_ops vfio_pci_dmabuf_ops = { > + .pin = vfio_pci_dma_buf_pin, > + .unpin = vfio_pci_dma_buf_unpin, > + .get_pfn = vfio_pci_dma_buf_get_pfn, > + .release = vfio_pci_dma_buf_release, > +}; > + > +static int check_dma_ranges(struct vfio_pci_dma_buf *priv, u64 *dmabuf_size) > +{ > + struct vfio_region_dma_range *dma_ranges = priv->dma_ranges; > + struct pci_dev *pdev = priv->vdev->pdev; > + resource_size_t bar_size; > + int i; > + > + for (i = 0; i < priv->nr_ranges; i++) { > + /* > + * For PCI the region_index is the BAR number like > + * everything else. > + */ > + if (dma_ranges[i].region_index >= VFIO_PCI_ROM_REGION_INDEX) > + return -EINVAL; > + > + bar_size = pci_resource_len(pdev, dma_ranges[i].region_index); > + if (!bar_size) > + return -EINVAL; > + > + if (!dma_ranges[i].offset && !dma_ranges[i].length) > + dma_ranges[i].length = bar_size; > + > + if (!IS_ALIGNED(dma_ranges[i].offset, PAGE_SIZE) || > + !IS_ALIGNED(dma_ranges[i].length, PAGE_SIZE) || > + dma_ranges[i].length > bar_size || > + dma_ranges[i].offset >= bar_size || > + dma_ranges[i].offset + dma_ranges[i].length > bar_size) > + return -EINVAL; > + > + *dmabuf_size += dma_ranges[i].length; > + } > + > + return 0; > +} > + > +int vfio_pci_core_feature_dma_buf(struct vfio_pci_core_device *vdev, u32 flags, > + struct vfio_device_feature_dma_buf __user *arg, > + size_t argsz) > +{ > + struct vfio_device_feature_dma_buf get_dma_buf; > + struct vfio_region_dma_range *dma_ranges; > + DEFINE_DMA_BUF_EXPORT_INFO(exp_info); > + struct vfio_pci_dma_buf *priv; > + u64 dmabuf_size = 0; > + int ret; > + > + ret = vfio_check_feature(flags, argsz, VFIO_DEVICE_FEATURE_GET, > + sizeof(get_dma_buf)); > + if (ret != 1) > + return ret; > + > + if (copy_from_user(&get_dma_buf, arg, sizeof(get_dma_buf))) > + return -EFAULT; > + > + dma_ranges = memdup_array_user(&arg->dma_ranges, > + get_dma_buf.nr_ranges, > + sizeof(*dma_ranges)); > + if (IS_ERR(dma_ranges)) > + return PTR_ERR(dma_ranges); > + > + priv = kzalloc(sizeof(*priv), GFP_KERNEL); > + if (!priv) { > + kfree(dma_ranges); > + return -ENOMEM; > + } > + > + priv->vdev = vdev; > + priv->nr_ranges = get_dma_buf.nr_ranges; > + priv->dma_ranges = dma_ranges; > + > + ret = check_dma_ranges(priv, &dmabuf_size); > + if (ret) > + goto err_free_priv; > + > + if (!vfio_device_try_get_registration(&vdev->vdev)) { > + ret = -ENODEV; > + goto err_free_priv; > + } > + > + exp_info.ops = &vfio_pci_dmabuf_ops; > + exp_info.size = dmabuf_size; > + exp_info.flags = get_dma_buf.open_flags; > + exp_info.priv = priv; > + > + priv->dmabuf = dma_buf_export(&exp_info); > + if (IS_ERR(priv->dmabuf)) { > + ret = PTR_ERR(priv->dmabuf); > + goto err_dev_put; > + } > + > + /* dma_buf_put() now frees priv */ > + INIT_LIST_HEAD(&priv->dmabufs_elm); > + down_write(&vdev->memory_lock); > + dma_resv_lock(priv->dmabuf->resv, NULL); > + priv->revoked = !__vfio_pci_memory_enabled(vdev); > + list_add_tail(&priv->dmabufs_elm, &vdev->dmabufs); > + dma_resv_unlock(priv->dmabuf->resv); > + up_write(&vdev->memory_lock); > + > + /* > + * dma_buf_fd() consumes the reference, when the file closes the dmabuf > + * will be released. > + */ > + return dma_buf_fd(priv->dmabuf, get_dma_buf.open_flags); > + > +err_dev_put: > + vfio_device_put_registration(&vdev->vdev); > +err_free_priv: > + kfree(dma_ranges); > + kfree(priv); > + return ret; > +} > + > +void vfio_pci_dma_buf_move(struct vfio_pci_core_device *vdev, bool revoked) > +{ > + struct vfio_pci_dma_buf *priv; > + struct vfio_pci_dma_buf *tmp; > + > + lockdep_assert_held_write(&vdev->memory_lock); > + > + list_for_each_entry_safe(priv, tmp, &vdev->dmabufs, dmabufs_elm) { > + /* > + * Returns true if a reference was successfully obtained. > + * The caller must interlock with the dmabuf's release > + * function in some way, such as RCU, to ensure that this > + * is not called on freed memory. > + */ > + if (!get_file_rcu(&priv->dmabuf->file)) > + continue; > + > + if (priv->revoked != revoked) { > + dma_resv_lock(priv->dmabuf->resv, NULL); > + priv->revoked = revoked; > + dma_buf_move_notify(priv->dmabuf); > + dma_resv_unlock(priv->dmabuf->resv); > + } > + dma_buf_put(priv->dmabuf); > + } > +} > + > +void vfio_pci_dma_buf_cleanup(struct vfio_pci_core_device *vdev) > +{ > + struct vfio_pci_dma_buf *priv; > + struct vfio_pci_dma_buf *tmp; > + > + down_write(&vdev->memory_lock); > + list_for_each_entry_safe(priv, tmp, &vdev->dmabufs, dmabufs_elm) { > + if (!get_file_rcu(&priv->dmabuf->file)) > + continue; > + dma_resv_lock(priv->dmabuf->resv, NULL); > + list_del_init(&priv->dmabufs_elm); > + priv->vdev = NULL; > + priv->revoked = true; > + dma_buf_move_notify(priv->dmabuf); > + dma_resv_unlock(priv->dmabuf->resv); > + vfio_device_put_registration(&vdev->vdev); > + dma_buf_put(priv->dmabuf); > + } > + up_write(&vdev->memory_lock); > +} > diff --git a/drivers/vfio/pci/vfio_pci_config.c b/drivers/vfio/pci/vfio_pci_config.c > index ea2745c1ac5e..5cc200e15edc 100644 > --- a/drivers/vfio/pci/vfio_pci_config.c > +++ b/drivers/vfio/pci/vfio_pci_config.c > @@ -589,10 +589,12 @@ static int vfio_basic_config_write(struct vfio_pci_core_device *vdev, int pos, > virt_mem = !!(le16_to_cpu(*virt_cmd) & PCI_COMMAND_MEMORY); > new_mem = !!(new_cmd & PCI_COMMAND_MEMORY); > > - if (!new_mem) > + if (!new_mem) { > vfio_pci_zap_and_down_write_memory_lock(vdev); > - else > + vfio_pci_dma_buf_move(vdev, true); > + } else { > down_write(&vdev->memory_lock); > + } > > /* > * If the user is writing mem/io enable (new_mem/io) and we > @@ -627,6 +629,8 @@ static int vfio_basic_config_write(struct vfio_pci_core_device *vdev, int pos, > *virt_cmd &= cpu_to_le16(~mask); > *virt_cmd |= cpu_to_le16(new_cmd & mask); > > + if (__vfio_pci_memory_enabled(vdev)) > + vfio_pci_dma_buf_move(vdev, false); > up_write(&vdev->memory_lock); > } > > @@ -707,12 +711,16 @@ static int __init init_pci_cap_basic_perm(struct perm_bits *perm) > static void vfio_lock_and_set_power_state(struct vfio_pci_core_device *vdev, > pci_power_t state) > { > - if (state >= PCI_D3hot) > + if (state >= PCI_D3hot) { > vfio_pci_zap_and_down_write_memory_lock(vdev); > - else > + vfio_pci_dma_buf_move(vdev, true); > + } else { > down_write(&vdev->memory_lock); > + } > > vfio_pci_set_power_state(vdev, state); > + if (__vfio_pci_memory_enabled(vdev)) > + vfio_pci_dma_buf_move(vdev, false); > up_write(&vdev->memory_lock); > } > > @@ -900,7 +908,10 @@ static int vfio_exp_config_write(struct vfio_pci_core_device *vdev, int pos, > > if (!ret && (cap & PCI_EXP_DEVCAP_FLR)) { > vfio_pci_zap_and_down_write_memory_lock(vdev); > + vfio_pci_dma_buf_move(vdev, true); > pci_try_reset_function(vdev->pdev); > + if (__vfio_pci_memory_enabled(vdev)) > + vfio_pci_dma_buf_move(vdev, true); > up_write(&vdev->memory_lock); > } > } > @@ -982,7 +993,10 @@ static int vfio_af_config_write(struct vfio_pci_core_device *vdev, int pos, > > if (!ret && (cap & PCI_AF_CAP_FLR) && (cap & PCI_AF_CAP_TP)) { > vfio_pci_zap_and_down_write_memory_lock(vdev); > + vfio_pci_dma_buf_move(vdev, true); > pci_try_reset_function(vdev->pdev); > + if (__vfio_pci_memory_enabled(vdev)) > + vfio_pci_dma_buf_move(vdev, true); > up_write(&vdev->memory_lock); > } > } > diff --git a/drivers/vfio/pci/vfio_pci_core.c b/drivers/vfio/pci/vfio_pci_core.c > index c3269d708411..f69eda5956ad 100644 > --- a/drivers/vfio/pci/vfio_pci_core.c > +++ b/drivers/vfio/pci/vfio_pci_core.c > @@ -287,6 +287,8 @@ static int vfio_pci_runtime_pm_entry(struct vfio_pci_core_device *vdev, > * semaphore. > */ > vfio_pci_zap_and_down_write_memory_lock(vdev); > + vfio_pci_dma_buf_move(vdev, true); > + > if (vdev->pm_runtime_engaged) { > up_write(&vdev->memory_lock); > return -EINVAL; > @@ -370,6 +372,8 @@ static void vfio_pci_runtime_pm_exit(struct vfio_pci_core_device *vdev) > */ > down_write(&vdev->memory_lock); > __vfio_pci_runtime_pm_exit(vdev); > + if (__vfio_pci_memory_enabled(vdev)) > + vfio_pci_dma_buf_move(vdev, false); > up_write(&vdev->memory_lock); > } > > @@ -690,6 +694,8 @@ void vfio_pci_core_close_device(struct vfio_device *core_vdev) > #endif > vfio_pci_core_disable(vdev); > > + vfio_pci_dma_buf_cleanup(vdev); > + > mutex_lock(&vdev->igate); > if (vdev->err_trigger) { > eventfd_ctx_put(vdev->err_trigger); > @@ -1234,7 +1240,10 @@ static int vfio_pci_ioctl_reset(struct vfio_pci_core_device *vdev, > */ > vfio_pci_set_power_state(vdev, PCI_D0); > > + vfio_pci_dma_buf_move(vdev, true); > ret = pci_try_reset_function(vdev->pdev); > + if (__vfio_pci_memory_enabled(vdev)) > + vfio_pci_dma_buf_move(vdev, false); > up_write(&vdev->memory_lock); > > return ret; > @@ -1523,6 +1532,8 @@ int vfio_pci_core_ioctl_feature(struct vfio_device *device, u32 flags, > return vfio_pci_core_pm_exit(vdev, flags, arg, argsz); > case VFIO_DEVICE_FEATURE_PCI_VF_TOKEN: > return vfio_pci_core_feature_token(vdev, flags, arg, argsz); > + case VFIO_DEVICE_FEATURE_DMA_BUF: > + return vfio_pci_core_feature_dma_buf(vdev, flags, arg, argsz); > default: > return -ENOTTY; > } > @@ -2098,6 +2109,7 @@ int vfio_pci_core_init_dev(struct vfio_device *core_vdev) > INIT_LIST_HEAD(&vdev->dummy_resources_list); > INIT_LIST_HEAD(&vdev->ioeventfds_list); > INIT_LIST_HEAD(&vdev->sriov_pfs_item); > + INIT_LIST_HEAD(&vdev->dmabufs); > init_rwsem(&vdev->memory_lock); > xa_init(&vdev->ctx); > > @@ -2480,11 +2492,17 @@ static int vfio_pci_dev_set_hot_reset(struct vfio_device_set *dev_set, > * cause the PCI config space reset without restoring the original > * state (saved locally in 'vdev->pm_save'). > */ > - list_for_each_entry(vdev, &dev_set->device_list, vdev.dev_set_list) > + list_for_each_entry(vdev, &dev_set->device_list, vdev.dev_set_list) { > + vfio_pci_dma_buf_move(vdev, true); > vfio_pci_set_power_state(vdev, PCI_D0); > + } > > ret = pci_reset_bus(pdev); > > + list_for_each_entry(vdev, &dev_set->device_list, vdev.dev_set_list) > + if (__vfio_pci_memory_enabled(vdev)) > + vfio_pci_dma_buf_move(vdev, false); > + > vdev = list_last_entry(&dev_set->device_list, > struct vfio_pci_core_device, vdev.dev_set_list); > > diff --git a/drivers/vfio/pci/vfio_pci_priv.h b/drivers/vfio/pci/vfio_pci_priv.h > index 5e4fa69aee16..d27f383f3931 100644 > --- a/drivers/vfio/pci/vfio_pci_priv.h > +++ b/drivers/vfio/pci/vfio_pci_priv.h > @@ -101,4 +101,29 @@ static inline bool vfio_pci_is_vga(struct pci_dev *pdev) > return (pdev->class >> 8) == PCI_CLASS_DISPLAY_VGA; > } > > +#ifdef CONFIG_DMA_SHARED_BUFFER > +int vfio_pci_core_feature_dma_buf(struct vfio_pci_core_device *vdev, u32 flags, > + struct vfio_device_feature_dma_buf __user *arg, > + size_t argsz); > +void vfio_pci_dma_buf_cleanup(struct vfio_pci_core_device *vdev); > +void vfio_pci_dma_buf_move(struct vfio_pci_core_device *vdev, bool revoked); > +#else > +static int > +vfio_pci_core_feature_dma_buf(struct vfio_pci_core_device *vdev, u32 flags, > + struct vfio_device_feature_dma_buf __user *arg, > + size_t argsz) > +{ > + return -ENOTTY; > +} > + > +static inline void vfio_pci_dma_buf_cleanup(struct vfio_pci_core_device *vdev) > +{ > +} > + > +static inline void vfio_pci_dma_buf_move(struct vfio_pci_core_device *vdev, > + bool revoked) > +{ > +} > +#endif > + > #endif > diff --git a/include/linux/vfio_pci_core.h b/include/linux/vfio_pci_core.h > index fbb472dd99b3..da5d8955ae56 100644 > --- a/include/linux/vfio_pci_core.h > +++ b/include/linux/vfio_pci_core.h > @@ -94,6 +94,7 @@ struct vfio_pci_core_device { > struct vfio_pci_core_device *sriov_pf_core_dev; > struct notifier_block nb; > struct rw_semaphore memory_lock; > + struct list_head dmabufs; > }; > > /* Will be exported for vfio pci drivers usage */ > diff --git a/include/uapi/linux/vfio.h b/include/uapi/linux/vfio.h > index c8dbf8219c4f..f43dfbde7352 100644 > --- a/include/uapi/linux/vfio.h > +++ b/include/uapi/linux/vfio.h > @@ -1458,6 +1458,35 @@ struct vfio_device_feature_bus_master { > }; > #define VFIO_DEVICE_FEATURE_BUS_MASTER 10 > > +/** > + * Upon VFIO_DEVICE_FEATURE_GET create a dma_buf fd for the > + * regions selected. > + * > + * For struct struct vfio_device_feature_dma_buf, open_flags are the typical > + * flags passed to open(2), eg O_RDWR, O_CLOEXEC, etc. nr_ranges is the total > + * number of dma_ranges that comprise the dmabuf. > + * > + * For struct vfio_region_dma_range, region_index/offset/length specify a slice > + * of the region to create the dmabuf from, if both offset & length are 0 then > + * the whole region is used. > + * > + * Return: The fd number on success, -1 and errno is set on failure. > + */ > +struct vfio_region_dma_range { > + __u32 region_index; > + __u32 __pad; > + __u64 offset; > + __u64 length; > +}; > + > +struct vfio_device_feature_dma_buf { > + __u32 open_flags; > + __u32 nr_ranges; > + struct vfio_region_dma_range dma_ranges[]; > +}; > + > +#define VFIO_DEVICE_FEATURE_DMA_BUF 11 > + > /* -------- API for Type1 VFIO IOMMU -------- */ > > /** -- Alexey

1 month, 3 weeks

[PATCH 0/4] Let userspace explicitly trigger memory reclaims

by Nicolas Frattaroli

RAM is not, in fact, cheap. Especially on embedded systems with a low amount of memory, but known and well-defined userspace, more explicit resource management can lead to better utilisation patterns. As an example, a resource manager process on a purpose-built device may wish to launch, and then explicitly swap out, memory of processes that are kept "warm", to improve perceived startup latency of individual full-screen applications without making the kernel figure out the usage pattern from observation alone in order to swap out the right pages. To allow for this explicit control in the context of panthor's GPU memory, add two new sysfs knobs. The first, mem_reclaim, runs an explicit priv BO reclaim cycle on the TGID written to it. The second, mem_claim, does the opposite: it swaps BOs back into active memory. Signed-off-by: Nicolas Frattaroli <nicolas.frattaroli(a)collabora.com> --- Nicolas Frattaroli (4): drm/panthor: Add freed_sz parameter to reclaim_priv_bos MAINTAINERS: Add sysfs ABI docs to list of panthor files drm/panthor: Add explicit memory reclaim sysfs knob drm/panthor: Add explicit memory claim sysfs knob Documentation/ABI/testing/sysfs-driver-panthor-mem | 34 ++++++++ MAINTAINERS | 1 + drivers/gpu/drm/panthor/panthor_drv.c | 93 ++++++++++++++++++++++ drivers/gpu/drm/panthor/panthor_gem.c | 7 +- drivers/gpu/drm/panthor/panthor_gem.h | 1 + drivers/gpu/drm/panthor/panthor_mmu.c | 70 +++++++++++++++- drivers/gpu/drm/panthor/panthor_mmu.h | 4 + 7 files changed, 205 insertions(+), 5 deletions(-) --- base-commit: 2c4b906cd135bbb44855287d0d0eff0ee0b47afe change-id: 20260506-panthor-explicit-reclaim-3dffed028d8c Best regards, -- Nicolas Frattaroli <nicolas.frattaroli(a)collabora.com>

1 month, 3 weeks

[PATCH v2 00/10] dma-bug: heaps: Add Tegra VPR support

by Thierry Reding

From: Thierry Reding <treding(a)nvidia.com> Hi, This series adds support for the video protection region (VPR) used on Tegra SoC devices. It's a special region of memory that is protected from accesses by the CPU and used to store DRM protected content (both decrypted stream data as well as decoded video frames). Patches 1 and 2 add DT binding documentation for the VPR and add the VPR to the list of memory-region items for display and host1x. Patch 3 adds bitmap_allocate(), which is like bitmap_allocate_region() but works on sizes that are not a power of two. Patch 4 introduces new APIs needed by the Tegra VPR implementation that allow CMA areas to be dynamically created at runtime rather than using the fixed, system-wide list. This is used in this driver specifically because it can use an arbitrary number of these areas (though they are currently limited to 4). Patch 5 adds some infrastructure for DMA heap implementations to provide information through debugfs. The Tegra VPR implementation is added in patch 6. See its commit message for more details about the specifics of this implementation. Finally, patches 7-10 add the VPR placeholder node on Tegra234 and hook it up to the host1x and GPU nodes so that they can make use of this region. Changes in v2: - Tegra VPR implementation is now more optimized to reduce the number of (very slow) resize operations, and allows cross-chunk allocations - dynamic CMA areas are now trackd separately from static ones, but the global number of CMA pages accounts for all areas Thierry Thierry Reding (10): dt-bindings: reserved-memory: Document Tegra VPR dt-bindings: display: tegra: Document memory regions bitmap: Add bitmap_allocate() function mm/cma: Allow dynamically creating CMA areas dma-buf: heaps: Add debugfs support dma-buf: heaps: Add support for Tegra VPR arm64: tegra: Add VPR placeholder node on Tegra234 arm64: tegra: Add GPU node on Tegra234 arm64: tegra: Hook up VPR to host1x arm64: tegra: Hook up VPR to the GPU .../display/tegra/nvidia,tegra186-dc.yaml | 10 + .../display/tegra/nvidia,tegra20-dc.yaml | 10 +- .../display/tegra/nvidia,tegra20-host1x.yaml | 7 + .../nvidia,tegra-video-protection-region.yaml | 55 + arch/arm/mm/dma-mapping.c | 2 +- arch/arm64/boot/dts/nvidia/tegra234.dtsi | 60 + arch/s390/mm/init.c | 2 +- drivers/dma-buf/dma-heap.c | 56 + drivers/dma-buf/heaps/Kconfig | 7 + drivers/dma-buf/heaps/Makefile | 1 + drivers/dma-buf/heaps/cma_heap.c | 2 +- drivers/dma-buf/heaps/tegra-vpr.c | 1265 +++++++++++++++++ include/linux/bitmap.h | 25 +- include/linux/cma.h | 7 +- include/linux/dma-heap.h | 2 + include/trace/events/tegra_vpr.h | 57 + mm/cma.c | 187 ++- mm/cma.h | 5 +- 18 files changed, 1713 insertions(+), 47 deletions(-) create mode 100644 Documentation/devicetree/bindings/reserved-memory/nvidia,tegra-video-protection-region.yaml create mode 100644 drivers/dma-buf/heaps/tegra-vpr.c create mode 100644 include/trace/events/tegra_vpr.h -- 2.52.0

1 month, 3 weeks

[PATCH] dma-buf/dma_fence_array: optimize handling v2

by Christian König

Removing the signal on any feature allows to simplfy the dma_fence_array code a lot and saves us from the need to install a callback on all fences at the same time. This results is less memory and CPU overhead. v2: fix potential double locking pointed out by Tvrtko Signed-off-by: Christian König <christian.koenig(a)amd.com> --- drivers/dma-buf/dma-fence-array.c | 134 +++++++++++++----------------- drivers/gpu/drm/xe/xe_vm.c | 2 +- include/linux/dma-fence-array.h | 22 ++--- 3 files changed, 66 insertions(+), 92 deletions(-) diff --git a/drivers/dma-buf/dma-fence-array.c b/drivers/dma-buf/dma-fence-array.c index 5e10e8df372f..8b94c6287482 100644 --- a/drivers/dma-buf/dma-fence-array.c +++ b/drivers/dma-buf/dma-fence-array.c @@ -42,97 +42,88 @@ static void dma_fence_array_clear_pending_error(struct dma_fence_array *array) cmpxchg(&array->base.error, PENDING_ERROR, 0); } -static void irq_dma_fence_array_work(struct irq_work *wrk) +static void dma_fence_array_cb_func(struct dma_fence *f, + struct dma_fence_cb *cb) { - struct dma_fence_array *array = container_of(wrk, typeof(*array), work); - - dma_fence_array_clear_pending_error(array); + struct dma_fence_array *array = + container_of(cb, struct dma_fence_array, callback); - dma_fence_signal(&array->base); - dma_fence_put(&array->base); + irq_work_queue(&array->work); } -static void dma_fence_array_cb_func(struct dma_fence *f, - struct dma_fence_cb *cb) +static bool dma_fence_array_try_add_cb(struct dma_fence_array *array) { - struct dma_fence_array_cb *array_cb = - container_of(cb, struct dma_fence_array_cb, cb); - struct dma_fence_array *array = array_cb->array; + while (array->num_pending) { + struct dma_fence *f = array->fences[array->num_pending - 1]; - dma_fence_array_set_pending_error(array, f->error); + if (!dma_fence_add_callback(f, &array->callback, + dma_fence_array_cb_func)) + return true; - if (atomic_dec_and_test(&array->num_pending)) - irq_work_queue(&array->work); - else + dma_fence_array_set_pending_error(array, f->error); + --array->num_pending; + } + return false; +} + +static void dma_fence_array_irq_work(struct irq_work *wrk) +{ + struct dma_fence_array *array = container_of(wrk, typeof(*array), work); + + --array->num_pending; + if (!dma_fence_array_try_add_cb(array)) { + dma_fence_signal(&array->base); dma_fence_put(&array->base); + } } static bool dma_fence_array_enable_signaling(struct dma_fence *fence) { struct dma_fence_array *array = to_dma_fence_array(fence); - struct dma_fence_array_cb *cb = array->callbacks; - unsigned i; - for (i = 0; i < array->num_fences; ++i) { - cb[i].array = array; + /* + * As we may report that the fence is signaled before all + * callbacks are complete, we need to take an additional + * reference count on the array so that we do not free it too + * early. The core fence handling will only hold the reference + * until we signal the array as complete (but that is now + * insufficient). + */ + dma_fence_get(&array->base); + if (!dma_fence_array_try_add_cb(array)) { /* - * As we may report that the fence is signaled before all - * callbacks are complete, we need to take an additional - * reference count on the array so that we do not free it too - * early. The core fence handling will only hold the reference - * until we signal the array as complete (but that is now - * insufficient). + * When all fences are already signaled we can drop the reference again + * and report to the caller that the array can be signaled as well. */ - dma_fence_get(&array->base); - if (dma_fence_add_callback(array->fences[i], &cb[i].cb, - dma_fence_array_cb_func)) { - int error = array->fences[i]->error; - - dma_fence_array_set_pending_error(array, error); - dma_fence_put(&array->base); - if (atomic_dec_and_test(&array->num_pending)) { - dma_fence_array_clear_pending_error(array); - return false; - } - } + dma_fence_put(&array->base); + return false; } - return true; } static bool dma_fence_array_signaled(struct dma_fence *fence) { struct dma_fence_array *array = to_dma_fence_array(fence); - int num_pending; + int num_pending, error = 0; unsigned int i; /* - * We need to read num_pending before checking the enable_signal bit - * to avoid racing with the enable_signaling() implementation, which - * might decrement the counter, and cause a partial check. - * atomic_read_acquire() pairs with atomic_dec_and_test() in - * dma_fence_array_enable_signaling() - * - * The !--num_pending check is here to account for the any_signaled case - * if we race with enable_signaling(), that means the !num_pending check - * in the is_signalling_enabled branch might be outdated (num_pending - * might have been decremented), but that's fine. The user will get the - * right value when testing again later. + * Reading num_pending without a memory barrier here is correct since + * that is only for optimization, it is perfectly acceptable to have a + * stale value for it. In all other cases num_pending is accessed by a + * single call chain. */ - num_pending = atomic_read_acquire(&array->num_pending); - if (test_bit(DMA_FENCE_FLAG_ENABLE_SIGNAL_BIT, &array->base.flags)) { - if (num_pending <= 0) - goto signal; - return false; - } + num_pending = READ_ONCE(array->num_pending); + for (i = 0; i < num_pending; ++i) { + struct dma_fence *f = array->fences[i]; - for (i = 0; i < array->num_fences; ++i) { - if (dma_fence_is_signaled(array->fences[i]) && !--num_pending) - goto signal; - } - return false; + if (!dma_fence_is_signaled(f)) + return false; -signal: + if (!error) + error = f->error; + } + dma_fence_array_set_pending_error(array, error); dma_fence_array_clear_pending_error(array); return true; } @@ -171,15 +162,12 @@ EXPORT_SYMBOL(dma_fence_array_ops); /** * dma_fence_array_alloc - Allocate a custom fence array - * @num_fences: [in] number of fences to add in the array * * Return dma fence array on success, NULL on failure */ -struct dma_fence_array *dma_fence_array_alloc(int num_fences) +struct dma_fence_array *dma_fence_array_alloc(void) { - struct dma_fence_array *array; - - return kzalloc_flex(*array, callbacks, num_fences); + return kzalloc_obj(struct dma_fence_array); } EXPORT_SYMBOL(dma_fence_array_alloc); @@ -203,10 +191,13 @@ void dma_fence_array_init(struct dma_fence_array *array, WARN_ON(!num_fences || !fences); array->num_fences = num_fences; + array->num_pending = num_fences; + array->fences = fences; + array->base.error = PENDING_ERROR; dma_fence_init(&array->base, &dma_fence_array_ops, NULL, context, seqno); - init_irq_work(&array->work, irq_dma_fence_array_work); + init_irq_work(&array->work, dma_fence_array_irq_work); /* * dma_fence_array_enable_signaling() is invoked while holding @@ -220,11 +211,6 @@ void dma_fence_array_init(struct dma_fence_array *array, */ lockdep_set_class(&array->base.inline_lock, &dma_fence_array_lock_key); - atomic_set(&array->num_pending, num_fences); - array->fences = fences; - - array->base.error = PENDING_ERROR; - /* * dma_fence_array objects should never contain any other fence * containers or otherwise we run into recursion and potential kernel @@ -265,7 +251,7 @@ struct dma_fence_array *dma_fence_array_create(int num_fences, { struct dma_fence_array *array; - array = dma_fence_array_alloc(num_fences); + array = dma_fence_array_alloc(); if (!array) return NULL; diff --git a/drivers/gpu/drm/xe/xe_vm.c b/drivers/gpu/drm/xe/xe_vm.c index 62a87a051be7..8f472911469d 100644 --- a/drivers/gpu/drm/xe/xe_vm.c +++ b/drivers/gpu/drm/xe/xe_vm.c @@ -3370,7 +3370,7 @@ static struct dma_fence *ops_execute(struct xe_vm *vm, goto err_trace; } - cf = dma_fence_array_alloc(n_fence); + cf = dma_fence_array_alloc(); if (!cf) { fence = ERR_PTR(-ENOMEM); goto err_out; diff --git a/include/linux/dma-fence-array.h b/include/linux/dma-fence-array.h index 1b1d87579c38..3ee55c0e2fa4 100644 --- a/include/linux/dma-fence-array.h +++ b/include/linux/dma-fence-array.h @@ -15,16 +15,6 @@ #include <linux/dma-fence.h> #include <linux/irq_work.h> -/** - * struct dma_fence_array_cb - callback helper for fence array - * @cb: fence callback structure for signaling - * @array: reference to the parent fence array object - */ -struct dma_fence_array_cb { - struct dma_fence_cb cb; - struct dma_fence_array *array; -}; - /** * struct dma_fence_array - fence to represent an array of fences * @base: fence base class @@ -33,18 +23,17 @@ struct dma_fence_array_cb { * @num_pending: fences in the array still pending * @fences: array of the fences * @work: internal irq_work function - * @callbacks: array of callback helpers + * @callback: callback structure for signaling */ struct dma_fence_array { struct dma_fence base; - unsigned num_fences; - atomic_t num_pending; + unsigned int num_fences; + unsigned int num_pending; struct dma_fence **fences; struct irq_work work; - - struct dma_fence_array_cb callbacks[] __counted_by(num_fences); + struct dma_fence_cb callback; }; /** @@ -78,11 +67,10 @@ to_dma_fence_array(struct dma_fence *fence) for (index = 0, fence = dma_fence_array_first(head); fence; \ ++(index), fence = dma_fence_array_next(head, index)) -struct dma_fence_array *dma_fence_array_alloc(int num_fences); +struct dma_fence_array *dma_fence_array_alloc(void); void dma_fence_array_init(struct dma_fence_array *array, int num_fences, struct dma_fence **fences, u64 context, unsigned seqno); - struct dma_fence_array *dma_fence_array_create(int num_fences, struct dma_fence **fences, u64 context, unsigned seqno); -- 2.43.0

1 month, 3 weeks

[PATCH v18 00/42] DEPT(DEPendency Tracker)

by Byungchul Park

I'm happy to see that DEPT reported real problems in practice: https://lore.kernel.org/lkml/6383cde5-cf4b-facf-6e07-1378a485657d@I-love.SA… https://lore.kernel.org/lkml/1674268856-31807-1-git-send-email-byungchul.pa… https://lore.kernel.org/all/b6e00e77-4a8c-4e05-ab79-266bf05fcc2d@igalia.com/ I’ve added documentation describing DEPT — this should help you understand what DEPT is and how it works. You can use DEPT simply by enabling CONFIG_DEPT and checking dmesg at runtime. --- Hi Linus and folks, I’ve been developing a tool to detect deadlock possibilities by tracking waits/events — rather than lock acquisition order — to cover all the synchronization mechanisms. To summarize the design rationale, starting from the problem statement, through analysis, to the solution: CURRENT STATUS -------------- Lockdep tracks lock acquisition order to identify deadlock conditions. Additionally, it tracks IRQ state changes — via {en,dis}able — to detect cases where locks are acquired unintentionally during interrupt handling. PROBLEM ------- Waits and their associated events that are never reachable can eventually lead to deadlocks. However, since Lockdep focuses solely on lock acquisition order, it has inherent limitations when handling waits and events. Moreover, by tracking only lock acquisition order, Lockdep cannot properly handle read locks or cross-event scenarios — such as wait_for_completion() and complete() — making it increasingly inadequate as a general-purpose deadlock detection tool. SOLUTION -------- Once again, waits and their associated events that are never reachable can eventually lead to deadlocks. The new solution, DEPT, focuses directly on waits and events. DEPT monitors waits and events, and reports them when any become unreachable. DEPT provides: * Correct handling of read locks. * Support for general waits and events. * Continuous operation, even after multiple reports. * Simple, intuitive annotation APIs. There are still false positives, and some are already being worked on for suppression. Especially splitting the folio class into several appropriate classes e.g. block device mapping class and regular file mapping class, is currently under active development by me and Yeoreum Yun. Anyway, these efforts will need to continue for a while, as we’ve seen with lockdep over two decades. DEPT is tagged as EXPERIMENTAL in Kconfig — meaning it’s not yet suitable for use as an automation tool. However, for those who are interested in using DEPT to analyze complex synchronization patterns and extract dependency insights, DEPT would be a great tool for the purpose. Thanks for your support and contributions to: Harry Yoo <harry.yoo(a)oracle.com> Gwan-gyeong Mun <gwan-gyeong.mun(a)intel.com> Yunseong Kim <ysk(a)kzalloc.com> Yeoreum Yun <yeoreum.yun(a)arm.com> FAQ --- Q. Is this the first attempt to solve this problem? A. No. The cross-release feature (commit b09be676e0ff2) attempted to address it — as a Lockdep extension. It was merged, but quickly reverted, because: While it uncovered valuable hidden issues, it also introduced false positives. Since these false positives mask further real problems with Lockdep — and developers strongly dislike them — the feature was rolled back. Q. Why wasn’t DEPT built as a Lockdep extension? A. Lockdep is the result of years of work by kernel developers — and is now very stable. But I chose to build DEPT separately, because: While reusing BFS(Breadth First Search) and Lockdep’s hashing is beneficial, the rest of the system must be rebuilt from scratch to align with DEPT’s wait-event model — since Lockdep was originally designed for tracking lock acquisition orders, not wait-event dependencies. Q. Do you plan to replace Lockdep entirely? A. Not at all — Lockdep still plays a vital role in validating correct lock usage. While its dependency-checking logic should eventually be superseded by DEPT, the rest of its functionality should stay. Q. Should we replace the dependency check immediately? A. Absolutely not. Lockdep’s stability is the result of years of hard work by kernel developers. Lockdep and DEPT should run side by side until DEPT matures. Q. Stronger detection often leads to more false positives — which was a major pain point when cross-release was added. Is DEPT designed to handle this? A. Yes. DEPT’s simple, generalized design enables flexible reporting — so while false positives still need fixing, they’re far less disruptive than they were under the Lockdep extension, cross-release. Q. Why not fix all false positives out-of-tree before merging? A. Since the affected subsystems span the entire kernel, like Lockdep, which has relied on annotations to avoid false positives over the last two decades, DEPT too will require the annotation efforts. Performing annotation work within the mainline will help us add annotations more appropriately and will also make DEPT a useful tool for a wider range of users more quickly. CONFIG_DEPT is marked EXPERIMENTAL, so it’s opt-in. Some users are already interested in using DEPT to analyze complex synchronization patterns and extract dependency insights. Byungchul --- Changes from v17: 1. Rebase on the mainline as of 2025 Dec 5. 2. Convert the documents' format from txt to rst. (feedbacked by Jonathan Corbet and Bagas Sanjaya) 3. Move the documents from 'Documentation/dependency' to 'Documentation/dev-tools'. (feedbakced by Jonathan Corbet) 4. Improve the documentation. (feedbacked by NeilBrown) 5. Use a common function, enter_from_user_mode(), instead of arch specific code, to notice context switch from user mode. (feedbacked by Dave Hansen, Mark Rutland, and Mark Brown) 6. Resolve the header dependency issue by using dept's internal header, instead of relocating 'struct llist_{head,node}' to another header. (feedbacked by Greg KH) 7. Improve page(or folio) usage type APIs. 8. Add rust helper for wait_for_completion(). (feedbacked by Guangbo Cui, Boqun Feng, and Danilo Krummrich) 9. Refine some commit messages. Changes from v16: 1. Rebase on v6.17. 2. Fix a false positive from rcu (by Yunseong Kim) 3. Introduce APIs to set page's usage, dept_set_page_usage() and dept_reset_page_usage() to avoid false positives. 4. Consider lock_page() as a potential wait unconditionally. 5. Consider folio_lock_killable() as a potential wait unconditionally. 6. Add support for tracking PG_writeback waits and events. 7. Fix two build errors due to the additional debug information added by dept. (by Yunseong Kim) Changes from v15: 1. Fix typo and improve comments and commit messages (feedbacked by ALOK TIWARI, Waiman Long, and kernel test robot). 2. Do not stop dept on detection of cicular dependency of recover event, allowing to keep reporting. 3. Add SK hynix to copyright. 4. Consider folio_lock() as a potential wait unconditionally. 5. Fix Kconfig dependency bug (feedbacked by kernel test rebot). 6. Do not suppress reports that involve classes even that have already involved in other reports, allowing to keep reporting. Changes from v14: 1. Rebase on the current latest, v6.15-rc6. 2. Refactor dept code. 3. With multi event sites for a single wait, even if an event forms a circular dependency, the event can be recovered by other event(or wake up) paths. Even though informing the circular dependency is worthy but it should be suppressed once informing it, if it doesn't lead an actual deadlock. So introduce APIs to annotate the relationship between event site and recover site, that are, event_site() and dept_recover_event(). 4. wait_for_completion() worked with dept map embedded in struct completion. However, it generates a few false positves since all the waits using the instance of struct completion, share the map and key. To avoid the false positves, make it not to share the map and key but each wait_for_completion() caller have its own key by default. Of course, external maps also can be used if needed. 5. Fix a bug about hardirq on/off tracing. 6. Implement basic unit test for dept. 7. Add more supports for dma fence synchronization. 8. Add emergency stop of dept e.g. on panic(). 9. Fix false positives by mmu_notifier_invalidate_*(). 10. Fix recursive call bug by DEPT_WARN_*() and DEPT_STOP(). 11. Fix trivial bugs in DEPT_WARN_*() and DEPT_STOP(). 12. Fix a bug that a spin lock, dept_pool_spin, is used in both contexts of irq disabled and enabled without irq disabled. 13. Suppress reports with classes, any of that already have been reported, even though they have different chains but being barely meaningful. 14. Print stacktrace of the wait that an event is now waking up, not only stacktrace of the event. 15. Make dept aware of lockdep_cmp_fn() that is used to avoid false positives in lockdep so that dept can also avoid them. 16. Do do_event() only if there are no ecxts have been delimited. 17. Fix a bug that was not synchronized for stage_m in struct dept_task, using a spin lock, dept_task()->stage_lock. 18. Fix a bug that dept didn't handle the case that multiple ttwus for a single waiter can be called at the same time e.i. a race issue. 19. Distinguish each kernel context from others, not only by system call but also by user oriented fault so that dept can work with more accuracy information about kernel context. That helps to avoid a few false positives. 20. Limit dept's working to x86_64 and arm64. Changes from v13: 1. Rebase on the current latest version, v6.9-rc7. 2. Add 'dept' documentation describing dept APIs. Changes from v12: 1. Refine the whole document for dept. 2. Add 'Interpret dept report' section in the document, using a deadlock report obtained in practice. Hope this version of document helps guys understand dept better. https://lore.kernel.org/lkml/6383cde5-cf4b-facf-6e07-1378a485657d@I-love.SA… https://lore.kernel.org/lkml/1674268856-31807-1-git-send-email-byungchul.pa… Changes from v11: 1. Add 'dept' documentation describing the concept of dept. 2. Rewrite the commit messages of the following commits for using weaker lockdep annotation, for better description. fs/jbd2: Use a weaker annotation in journal handling cpu/hotplug: Use a weaker annotation in AP thread (feedbacked by Thomas Gleixner) Changes from v10: 1. Fix noinstr warning when building kernel source. 2. dept has been reporting some false positives due to the folio lock's unfairness. Reflect it and make dept work based on dept annotaions instead of just wait and wake up primitives. 3. Remove the support for PG_writeback while working on 2. I will add the support later if needed. 4. dept didn't print stacktrace for [S] if the participant of a deadlock is not lock mechanism but general wait and event. However, it made hard to interpret the report in that case. So add support to print stacktrace of the requestor who asked the event context to run - usually a waiter of the event does it just before going to wait state. 5. Give up tracking raw_local_irq_{disable,enable}() since it totally messed up dept's irq tracking. So make it work in the same way as lockdep does. I will consider it once any false positives by those are observed again. 6. Change the manual rwsem_acquire_read(->j_trans_commit_map) annotation in fs/jbd2/transaction.c to the try version so that it works as much as it exactly needs. 7. Remove unnecessary 'inline' keyword in dept.c and add '__maybe_unused' to a needed place. Changes from v9: 1. Fix a bug. SDT tracking didn't work well because of my big mistake that I should've used waiter's map to indentify its class but it had been working with waker's one. FYI, PG_locked and PG_writeback weren't affected. They still worked well. (reported by YoungJun) Changes from v8: 1. Fix build error by adding EXPORT_SYMBOL(PG_locked_map) and EXPORT_SYMBOL(PG_writeback_map) for kernel module build - appologize for that. (reported by kernel test robot) 2. Fix build error by removing header file's circular dependency that was caused by "atomic.h", "kernel.h" and "irqflags.h", which I introduced - appolgize for that. (reported by kernel test robot) Changes from v7: 1. Fix a bug that cannot track rwlock dependency properly, introduced in v7. (reported by Boqun and lockdep selftest) 2. Track wait/event of PG_{locked,writeback} more aggressively assuming that when a bit of PG_{locked,writeback} is cleared there might be waits on the bit. (reported by Linus, Hillf and syzbot) 3. Fix and clean bad style code e.i. unnecessarily introduced a randome pattern and so on. (pointed out by Linux) 4. Clean code for applying dept to wait_for_completion(). Changes from v6: 1. Tie to task scheduler code to track sleep and try_to_wake_up() assuming sleeps cause waits, try_to_wake_up()s would be the events that those are waiting for, of course with proper dept annotations, sdt_might_sleep_weak(), sdt_might_sleep_strong() and so on. For these cases, class is classified at sleep entrance rather than the synchronization initialization code. Which would extremely reduce false alarms. 2. Remove the dept associated instance in each page struct for tracking dependencies by PG_locked and PG_writeback thanks to the 1. work above. 3. Introduce CONFIG_dept_AGGRESIVE_TIMEOUT_WAIT to suppress reports that waits with timeout set are involved, for those who don't like verbose reporting. 4. Add a mechanism to refill the internal memory pools on running out so that dept could keep working as long as free memory is available in the system. 5. Re-enable tracking hashed-waitqueue wait. That's going to no longer generate false positives because class is classified at sleep entrance rather than the waitqueue initailization. 6. Refactor to make it easier to port onto each new version of the kernel. 7. Apply dept to dma fence. 8. Do trivial optimizaitions. Changes from v5: 1. Use just pr_warn_once() rather than WARN_ONCE() on the lack of internal resources because WARN_*() printing stacktrace is too much for informing the lack. (feedback from Ted, Hyeonggon) 2. Fix trivial bugs like missing initializing a struct before using it. 3. Assign a different class per task when handling onstack variables for waitqueue or the like. Which makes dept distinguish between onstack variables of different tasks so as to prevent false positives. (reported by Hyeonggon) 4. Make dept aware of even raw_local_irq_*() to prevent false positives. (reported by Hyeonggon) 5. Don't consider dependencies between the events that might be triggered within __schedule() and the waits that requires __schedule(), real ones. (reported by Hyeonggon) 6. Unstage the staged wait that has prepare_to_wait_event()'ed *and* yet to get to __schedule(), if we encounter __schedule() in-between for another sleep, which is possible if e.g. a mutex_lock() exists in 'condition' of ___wait_event(). 7. Turn on CONFIG_PROVE_LOCKING when CONFIG_DEPT is on, to rely on the hardirq and softirq entrance tracing to make dept more portable for now. Changes from v4: 1. Fix some bugs that produce false alarms. 2. Distinguish each syscall context from another *for arm64*. 3. Make it not warn it but just print it in case dept ring buffer gets exhausted. (feedback from Hyeonggon) 4. Explicitely describe "EXPERIMENTAL" and "dept might produce false positive reports" in Kconfig. (feedback from Ted) Changes from v3: 1. dept shouldn't create dependencies between different depths of a class that were indicated by *_lock_nested(). dept normally doesn't but it does once another lock class comes in. So fixed it. (feedback from Hyeonggon) 2. dept considered a wait as a real wait once getting to __schedule() even if it has been set to TASK_RUNNING by wake up sources in advance. Fixed it so that dept doesn't consider the case as a real wait. (feedback from Jan Kara) 3. Stop tracking dependencies with a map once the event associated with the map has been handled. dept will start to work with the map again, on the next sleep. Changes from v2: 1. Disable dept on bit_wait_table[] in sched/wait_bit.c reporting a lot of false positives, which is my fault. Wait/event for bit_wait_table[] should've been tagged in a higher layer for better work, which is a future work. (feedback from Jan Kara) 2. Disable dept on crypto_larval's completion to prevent a false positive. Changes from v1: 1. Fix coding style and typo. (feedback from Steven) 2. Distinguish each work context from another in workqueue. 3. Skip checking lock acquisition with nest_lock, which is about correct lock usage that should be checked by lockdep. Changes from RFC(v0): 1. Prevent adding a wait tag at prepare_to_wait() but __schedule(). (feedback from Linus and Matthew) 2. Use try version at lockdep_acquire_cpus_lock() annotation. 3. Distinguish each syscall context from another. Byungchul Park (41): dept: implement DEPT(DEPendency Tracker) dept: add single event dependency tracker APIs dept: add lock dependency tracker APIs dept: tie to lockdep and IRQ tracing dept: add proc knobs to show stats and dependency graph dept: distinguish each kernel context from another dept: distinguish each work from another dept: add a mechanism to refill the internal memory pools on running out dept: record the latest one out of consecutive waits of the same class dept: apply sdt_might_sleep_{start,end}() to wait_for_completion()/complete() dept: apply sdt_might_sleep_{start,end}() to swait dept: apply sdt_might_sleep_{start,end}() to waitqueue wait dept: apply sdt_might_sleep_{start,end}() to hashed-waitqueue wait dept: apply sdt_might_sleep_{start,end}() to dma fence dept: track timeout waits separately with a new Kconfig dept: apply timeout consideration to wait_for_completion()/complete() dept: apply timeout consideration to swait dept: apply timeout consideration to waitqueue wait dept: apply timeout consideration to hashed-waitqueue wait dept: apply timeout consideration to dma fence wait dept: make dept able to work with an external wgen dept: track PG_locked with dept dept: print staged wait's stacktrace on report locking/lockdep: prevent various lockdep assertions when lockdep_off()'ed dept: add documents for dept cpu/hotplug: use a weaker annotation in AP thread dept: assign dept map to mmu notifier invalidation synchronization dept: assign unique dept_key to each distinct dma fence caller dept: make dept aware of lockdep_set_lock_cmp_fn() annotation dept: make dept stop from working on debug_locks_off() dept: assign unique dept_key to each distinct wait_for_completion() caller completion, dept: introduce init_completion_dmap() API dept: introduce a new type of dependency tracking between multi event sites dept: add module support for struct dept_event_site and dept_event_site_dep dept: introduce event_site() to disable event tracking if it's recoverable dept: implement a basic unit test for dept dept: call dept_hardirqs_off() in local_irq_*() regardless of irq state dept: introduce APIs to set page usage and use subclasses_evt for the usage dept: track PG_writeback with dept SUNRPC: relocate struct rcu_head to the first field of struct rpc_xprt mm: percpu: increase PERCPU_DYNAMIC_SIZE_SHIFT on DEPT and large PAGE_SIZE Yunseong Kim (1): rcu/update: fix same dept key collision between various types of RCU Documentation/dev-tools/dept.rst | 778 ++++++ Documentation/dev-tools/dept_api.rst | 125 + drivers/dma-buf/dma-fence.c | 23 +- include/asm-generic/vmlinux.lds.h | 13 +- include/linux/completion.h | 124 +- include/linux/dept.h | 402 +++ include/linux/dept_ldt.h | 78 + include/linux/dept_sdt.h | 68 + include/linux/dept_unit_test.h | 67 + include/linux/dma-fence.h | 74 +- include/linux/hardirq.h | 3 + include/linux/irq-entry-common.h | 4 + include/linux/irqflags.h | 21 +- include/linux/local_lock_internal.h | 1 + include/linux/lockdep.h | 105 +- include/linux/lockdep_types.h | 3 + include/linux/mm_types.h | 4 + include/linux/mmu_notifier.h | 26 + include/linux/module.h | 5 + include/linux/mutex.h | 1 + include/linux/page-flags.h | 217 +- include/linux/pagemap.h | 37 +- include/linux/percpu-rwsem.h | 2 +- include/linux/percpu.h | 4 + include/linux/rcupdate_wait.h | 13 +- include/linux/rtmutex.h | 1 + include/linux/rwlock_types.h | 1 + include/linux/rwsem.h | 1 + include/linux/sched.h | 118 + include/linux/seqlock.h | 2 +- include/linux/spinlock_types_raw.h | 3 + include/linux/srcu.h | 2 +- include/linux/sunrpc/xprt.h | 9 +- include/linux/swait.h | 3 + include/linux/wait.h | 3 + include/linux/wait_bit.h | 3 + init/init_task.c | 2 + init/main.c | 2 + kernel/Makefile | 1 + kernel/cpu.c | 2 +- kernel/dependency/Makefile | 5 + kernel/dependency/dept.c | 3499 ++++++++++++++++++++++++++ kernel/dependency/dept_hash.h | 10 + kernel/dependency/dept_internal.h | 314 +++ kernel/dependency/dept_object.h | 13 + kernel/dependency/dept_proc.c | 94 + kernel/dependency/dept_unit_test.c | 173 ++ kernel/exit.c | 1 + kernel/fork.c | 2 + kernel/locking/lockdep.c | 33 + kernel/module/main.c | 19 + kernel/rcu/rcu.h | 1 + kernel/rcu/update.c | 5 +- kernel/sched/completion.c | 62 +- kernel/sched/core.c | 9 + kernel/workqueue.c | 3 + lib/Kconfig.debug | 48 + lib/debug_locks.c | 2 + lib/locking-selftest.c | 2 + mm/filemap.c | 38 + mm/mm_init.c | 3 + mm/mmu_notifier.c | 31 +- rust/helpers/completion.c | 5 + 63 files changed, 6602 insertions(+), 121 deletions(-) create mode 100644 Documentation/dev-tools/dept.rst create mode 100644 Documentation/dev-tools/dept_api.rst create mode 100644 include/linux/dept.h create mode 100644 include/linux/dept_ldt.h create mode 100644 include/linux/dept_sdt.h create mode 100644 include/linux/dept_unit_test.h create mode 100644 kernel/dependency/Makefile create mode 100644 kernel/dependency/dept.c create mode 100644 kernel/dependency/dept_hash.h create mode 100644 kernel/dependency/dept_internal.h create mode 100644 kernel/dependency/dept_object.h create mode 100644 kernel/dependency/dept_proc.c create mode 100644 kernel/dependency/dept_unit_test.c base-commit: 43dfc13ca972988e620a6edb72956981b75ab6b0 -- 2.17.1

1 month, 3 weeks

BUY REAL PASSPORT, DRIVERS LICENSE, VISA, GREEN CARD, ID CARD, SCHOOL CERTIFICAT

by jameswalkes0987＠gmail.com

Genuine Passports, Driver's licenses And Other Documents Of All Countries For Sale. Buy High Quality Data-based Registered Machine Read-able Scan-able Driver's Licenses, ID's, Passports, And Citizenship Documents. Buy Real Passports, Driver's License, ID Cards, Visas, USA Green Card, Fake Money. We are the best producers of genuine Data-based Registered High Quality Real/Fake Passport, Driver's Licenses, ID Card's And other Citizenship Documents. Buy High Quality Data-based Registered Machine Read-able Scan-able Driver's Licenses, ID's, Passports, And Citizenship Documents. Buy Real Passports, Driver's License, ID Cards, Visas, USA Green Card, Fake Money. Email : jameswalkes0987(a)gmail.com What's app:+237 677480749 Tell:+237 677480749 We are the best producers of genuine high quality fake documents. Buy Fake Passport British(UK) For Sale Diplomatic Canadian False ID Cards Online United States(US) Fake ID Card Sell Driver's License. Buy genuine Green card Training certificates, GCSE, A-levels, High School Diploma Certificates ,GMAT, MCAT, and LSAT examination Certificates and credit cards, school diplomas, school degrees all in an entirely new name issued and registered in the government's data-base system. Welcome To David Wilson and Associates Network. Get A second Chance In Life with Wesley Hover and Associates , protect your privacy, build new credit history, by-pass criminal background checks, take back your freedom. It's a cruise, you have come to the right place for all your travel needs and guest what !!!! you're going to finally make your dream a reality. Let us help you plan the ideal trip for you if you would like to find out more about the services we have available. We are an Association responsible for the production of real genuine passports, Real Genuine Data-Base Registered Fake Passports and other Citizenship documents. I can guarantee you a new Identity starting from a clean new genuine Birth Certificate, ID card, Drivers License, Passports, Social security card with SSN, credit files can you imagine ?? WE DO OFFER LEGITIMATE'S SERVICE: .GET REGISTERED BRITISH PASSPORT. .REGISTERED CANADIAN PASSPORT. .REGISTERED FRENCH PASSPORT. .REGISTERED AMERICAN PASSPORT. .REGISTERED USA PASSPORT. .REGISTERED PASSPORT FOR COUNTRIES IN THE EUROPEAN UNION. We offer a service to help you through to meet your goals, we can help you with: Getting real government issued ID under another identity(NEW NAMES), A new social security number (verifiable with the SSA), Checking and saving accounts for your new ID. Credit cards Relocation Passports, Diplomatic passports, novelty passports. Production and obtaining new identification documents. We also do work permit and bank statements and have connections to OFFER JOBS in country like Dubai, USA , CANADA , UK ,China,Peru,Brazil,South Africa,Denmark,Sweden,Norway,France etc.. Tourist and business visa services available to residents of all 50 states and all nationalities Worldwide INTERESTED TO BUY REAL GENUINE QUALITY BANKNOTES of Euros, Dollars and Pounds ??? with security feature magnetic ink, water marks, the pen test, and the security strip that by-pass machines. Our hundreds carry "color-shifting ink," an advanced feature that gives the money an appearance of changing color when held at different angles including Intaglio. Coaching services available.... Not even an expertise custom official or machine can ever dictate the document we offer as fake, since the document is no different from Real government issued! Email : jameswalkes0987(a)gmail.com What's app:+237 677480749 Tell:+237 677480749 E-Mail Your Questions and Comments. We are looking forward to receiving your inquiries and early receipt of your first orders!

1 month, 3 weeks

Jump to page:

2026

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

Linaro-mm-sig