[ This series depends on VFIO device cdev series ]
Changelog v5: * Kept the cmd->id in the iommufd_test_create_access() so the access can be created with an ioas by default. Then, renamed the previous ioctl IOMMU_TEST_OP_ACCESS_SET_IOAS to IOMMU_TEST_OP_ACCESS_REPLACE_IOAS, so it would be used to replace an access->ioas pointer. * Added iommufd_access_replace() API after the introductions of the other two APIs iommufd_access_attach() and iommufd_access_detach(). * Since vdev->iommufd_attached is also set in emulated pathway too, call iommufd_access_update(), similar to the physical pathway. v4: https://lore.kernel.org/linux-iommu/cover.1678284812.git.nicolinc@nvidia.com... * Rebased on top of Jason's series adding replace() and hwpt_alloc() https://lore.kernel.org/linux-iommu/0-v2-51b9896e7862+8a8c-iommufd_alloc_jgg... * Rebased on top of cdev series v6 https://lore.kernel.org/kvm/20230308132903.465159-1-yi.l.liu@intel.com/ * Dropped the patch that's moved to cdev series. * Added unmap function pointer sanity before calling it. * Added "Reviewed-by" from Kevin and Yi. * Added back the VFIO change updating the ATTACH uAPI. v3: https://lore.kernel.org/linux-iommu/cover.1677288789.git.nicolinc@nvidia.com... * Rebased on top of Jason's iommufd_hwpt branch: https://lore.kernel.org/linux-iommu/0-v2-406f7ac07936+6a-iommufd_hwpt_jgg@nv... * Dropped patches from this series accordingly. There were a couple of VFIO patches that will be submitted after the VFIO cdev series. Also, renamed the series to be "emulated". * Moved dma_unmap sanity patch to the first in the series. * Moved dma_unmap sanity to cover both VFIO and IOMMUFD pathways. * Added Kevin's "Reviewed-by" to two of the patches. * Fixed a NULL pointer bug in vfio_iommufd_emulated_bind(). * Moved unmap() call to the common place in iommufd_access_set_ioas(). v2: https://lore.kernel.org/linux-iommu/cover.1675802050.git.nicolinc@nvidia.com... * Rebased on top of vfio_device cdev v2 series. * Update the kdoc and commit message of iommu_group_replace_domain(). * Dropped revert-to-core-domain part in iommu_group_replace_domain(). * Dropped !ops->dma_unmap check in vfio_iommufd_emulated_attach_ioas(). * Added missing rc value in vfio_iommufd_emulated_attach_ioas() from the iommufd_access_set_ioas() call. * Added a new patch in vfio_main to deny vfio_pin/unpin_pages() calls if vdev->ops->dma_unmap is not implemented. * Added a __iommmufd_device_detach helper and let the replace routine do a partial detach(). * Added restriction on auto_domains to use the replace feature. * Added the patch "iommufd/device: Make hwpt_list list_add/del symmetric" from the has_group removal series. v1: https://lore.kernel.org/linux-iommu/cover.1675320212.git.nicolinc@nvidia.com...
Hi all,
The existing IOMMU APIs provide a pair of functions: iommu_attach_group() for callers to attach a device from the default_domain (NULL if not being supported) to a given iommu domain, and iommu_detach_group() for callers to detach a device from a given domain to the default_domain. Internally, the detach_dev op is deprecated for the newer drivers with default_domain. This means that those drivers likely can switch an attaching domain to another one, without stagging the device at a blocking or default domain, for use cases such as: 1) vPASID mode, when a guest wants to replace a single pasid (PASID=0) table with a larger table (PASID=N) 2) Nesting mode, when switching the attaching device from an S2 domain to an S1 domain, or when switching between relevant S1 domains.
This series is rebased on top of Jason Gunthorpe's series that introduces iommu_group_replace_domain API and IOMMUFD infrastructure for the IOMMUFD "physical" devices. The IOMMUFD "emulated" deivces will need some extra steps to replace the access->ioas object and its iopt pointer.
You can also find this series on Github: https://github.com/nicolinc/iommufd/commits/iommu_group_replace_domain-v5
Thank you Nicolin Chen
Nicolin Chen (4): vfio: Do not allow !ops->dma_unmap in vfio_pin/unpin_pages() iommufd: Add iommufd_access_replace() API iommufd/selftest: Add IOMMU_TEST_OP_ACCESS_REPLACE_IOAS coverage vfio: Support IO page table replacement
drivers/iommu/iommufd/device.c | 60 +++++++++++++++---- drivers/iommu/iommufd/iommufd_test.h | 4 ++ drivers/iommu/iommufd/selftest.c | 19 ++++++ drivers/vfio/iommufd.c | 11 ++-- drivers/vfio/vfio_main.c | 4 ++ include/linux/iommufd.h | 1 + include/uapi/linux/vfio.h | 6 ++ tools/testing/selftests/iommu/iommfd*.c | 0 tools/testing/selftests/iommu/iommufd.c | 29 ++++++++- tools/testing/selftests/iommu/iommufd_utils.h | 19 ++++++ 10 files changed, 135 insertions(+), 18 deletions(-) create mode 100644 tools/testing/selftests/iommu/iommfd*.c
A driver that doesn't implement ops->dma_unmap shouldn't be allowed to do vfio_pin/unpin_pages(), though it can use vfio_dma_rw() to access an iova range. Deny !ops->dma_unmap cases in vfio_pin/unpin_pages().
Suggested-by: Kevin Tian kevin.tian@intel.com Reviewed-by: Kevin Tian kevin.tian@intel.com Reviewed-by: Yi Liu yi.l.liu@intel.com Reviewed-by: Jason Gunthorpe jgg@nvidia.com Signed-off-by: Nicolin Chen nicolinc@nvidia.com --- drivers/vfio/vfio_main.c | 4 ++++ 1 file changed, 4 insertions(+)
diff --git a/drivers/vfio/vfio_main.c b/drivers/vfio/vfio_main.c index 426ebfc0ee1c..546e1fc8d8e1 100644 --- a/drivers/vfio/vfio_main.c +++ b/drivers/vfio/vfio_main.c @@ -1482,6 +1482,8 @@ int vfio_pin_pages(struct vfio_device *device, dma_addr_t iova, /* group->container cannot change while a vfio device is open */ if (!pages || !npage || WARN_ON(!vfio_assert_device_open(device))) return -EINVAL; + if (!device->ops->dma_unmap) + return -EINVAL; if (vfio_device_has_container(device)) return vfio_device_container_pin_pages(device, iova, npage, prot, pages); @@ -1519,6 +1521,8 @@ void vfio_unpin_pages(struct vfio_device *device, dma_addr_t iova, int npage) { if (WARN_ON(!vfio_assert_device_open(device))) return; + if (WARN_ON(!device->ops->dma_unmap)) + return;
if (vfio_device_has_container(device)) { vfio_device_container_unpin_pages(device, iova, npage);
Extract all common procedure, between the iommufd_access_attach() API and the new iommufd_access_replace() API, to an iommufd_access_change_pt() helper.
Add iommufd_access_replace() function for VFIO emulated pathway to use.
Suggested-by: Jason Gunthorpe jgg@nvidia.com Signed-off-by: Nicolin Chen nicolinc@nvidia.com --- drivers/iommu/iommufd/device.c | 60 ++++++++++++++++++++----- include/linux/iommufd.h | 1 + tools/testing/selftests/iommu/iommfd*.c | 0 3 files changed, 50 insertions(+), 11 deletions(-) create mode 100644 tools/testing/selftests/iommu/iommfd*.c
diff --git a/drivers/iommu/iommufd/device.c b/drivers/iommu/iommufd/device.c index 012e85fc6d77..3b39b76fbc3b 100644 --- a/drivers/iommu/iommufd/device.c +++ b/drivers/iommu/iommufd/device.c @@ -746,39 +746,77 @@ void iommufd_access_detach(struct iommufd_access *access) } EXPORT_SYMBOL_NS_GPL(iommufd_access_detach, IOMMUFD);
-int iommufd_access_attach(struct iommufd_access *access, u32 ioas_id) +static struct iommufd_ioas * +iommufd_access_change_pt(struct iommufd_access *access, u32 ioas_id) { struct iommufd_ioas *new_ioas; struct iommufd_object *obj; int rc = 0;
+ lockdep_assert_held(&access->ioas_lock); + + obj = iommufd_get_object(access->ictx, ioas_id, IOMMUFD_OBJ_IOAS); + if (IS_ERR(obj)) + return (struct iommufd_ioas *)obj; + new_ioas = container_of(obj, struct iommufd_ioas, obj); + + rc = iopt_add_access(&new_ioas->iopt, access); + if (rc) { + iommufd_put_object(obj); + return ERR_PTR(rc); + } + iommufd_ref_to_users(obj); + return new_ioas; +} + +int iommufd_access_attach(struct iommufd_access *access, u32 ioas_id) +{ + struct iommufd_ioas *new_ioas; + mutex_lock(&access->ioas_lock); if (access->ioas != NULL && access->ioas->obj.id != ioas_id) { mutex_unlock(&access->ioas_lock); return -EINVAL; }
- obj = iommufd_get_object(access->ictx, ioas_id, IOMMUFD_OBJ_IOAS); - if (IS_ERR(obj)) { + new_ioas = iommufd_access_change_pt(access, ioas_id); + if (IS_ERR(new_ioas)) { mutex_unlock(&access->ioas_lock); - return PTR_ERR(obj); + return PTR_ERR(new_ioas); } - new_ioas = container_of(obj, struct iommufd_ioas, obj); + access->ioas = new_ioas; + access->ioas_unpin = new_ioas; + mutex_unlock(&access->ioas_lock); + return 0; +} +EXPORT_SYMBOL_NS_GPL(iommufd_access_attach, IOMMUFD);
- rc = iopt_add_access(&new_ioas->iopt, access); - if (rc) { +int iommufd_access_replace(struct iommufd_access *access, u32 ioas_id) +{ + struct iommufd_ioas *new_ioas; + + mutex_lock(&access->ioas_lock); + if (!access->ioas) { mutex_unlock(&access->ioas_lock); - iommufd_put_object(obj); - return rc; + return -ENOENT; + } + if (access->ioas->obj.id == ioas_id) { + mutex_unlock(&access->ioas_lock); + return 0; } - iommufd_ref_to_users(obj);
+ new_ioas = iommufd_access_change_pt(access, ioas_id); + if (IS_ERR(new_ioas)) { + mutex_unlock(&access->ioas_lock); + return PTR_ERR(new_ioas); + } + __iommufd_access_detach(access); access->ioas = new_ioas; access->ioas_unpin = new_ioas; mutex_unlock(&access->ioas_lock); return 0; } -EXPORT_SYMBOL_NS_GPL(iommufd_access_attach, IOMMUFD); +EXPORT_SYMBOL_NS_GPL(iommufd_access_replace, IOMMUFD);
/** * iommufd_access_notify_unmap - Notify users of an iopt to stop using it diff --git a/include/linux/iommufd.h b/include/linux/iommufd.h index 7c79f9b33246..cd7ae5a625fc 100644 --- a/include/linux/iommufd.h +++ b/include/linux/iommufd.h @@ -46,6 +46,7 @@ iommufd_access_create(struct iommufd_ctx *ictx, const struct iommufd_access_ops *ops, void *data, u32 *id); void iommufd_access_destroy(struct iommufd_access *access); int iommufd_access_attach(struct iommufd_access *access, u32 ioas_id); +int iommufd_access_replace(struct iommufd_access *access, u32 ioas_id); void iommufd_access_detach(struct iommufd_access *access);
void iommufd_ctx_get(struct iommufd_ctx *ictx); diff --git a/tools/testing/selftests/iommu/iommfd*.c b/tools/testing/selftests/iommu/iommfd*.c new file mode 100644 index 000000000000..e69de29bb2d1
From: Nicolin Chen nicolinc@nvidia.com Sent: Thursday, March 23, 2023 4:33 PM
+int iommufd_access_replace(struct iommufd_access *access, u32 ioas_id) +{
- struct iommufd_ioas *new_ioas;
- mutex_lock(&access->ioas_lock);
- if (!access->ioas) { mutex_unlock(&access->ioas_lock);
iommufd_put_object(obj);return rc;
return -ENOENT;- }
- if (access->ioas->obj.id == ioas_id) {
mutex_unlock(&access->ioas_lock);return 0;
- iommufd_ref_to_users(obj);
- new_ioas = iommufd_access_change_pt(access, ioas_id);
- if (IS_ERR(new_ioas)) {
mutex_unlock(&access->ioas_lock);return PTR_ERR(new_ioas);- }
- __iommufd_access_detach(access); access->ioas = new_ioas; access->ioas_unpin = new_ioas;
Above three lines can be moved into iommufd_access_change_pt():
If (access->ioas) __iommufd_access_detach(access); access->ioas = new_ioas; access->ioas_unpin = new_ioas;
Then both attach/replace can end by calling the common function.
On Fri, Mar 24, 2023 at 03:02:46AM +0000, Tian, Kevin wrote:
External email: Use caution opening links or attachments
From: Nicolin Chen nicolinc@nvidia.com Sent: Thursday, March 23, 2023 4:33 PM
+int iommufd_access_replace(struct iommufd_access *access, u32 ioas_id) +{
struct iommufd_ioas *new_ioas;mutex_lock(&access->ioas_lock);if (!access->ioas) { mutex_unlock(&access->ioas_lock);
iommufd_put_object(obj);return rc;
return -ENOENT;}if (access->ioas->obj.id == ioas_id) {mutex_unlock(&access->ioas_lock);return 0; }
iommufd_ref_to_users(obj);
new_ioas = iommufd_access_change_pt(access, ioas_id);if (IS_ERR(new_ioas)) {mutex_unlock(&access->ioas_lock);return PTR_ERR(new_ioas);}__iommufd_access_detach(access); access->ioas = new_ioas; access->ioas_unpin = new_ioas;Above three lines can be moved into iommufd_access_change_pt():
If (access->ioas) __iommufd_access_detach(access); access->ioas = new_ioas; access->ioas_unpin = new_ioas;Then both attach/replace can end by calling the common function.
OK. Will do that in v6.
Thanks Nicolin
Add a new IOMMU_TEST_OP_ACCESS_REPLACE_IOAS to allow replacing the access->ioas, corresponding to the iommufd_access_replace() helper.
Then add a replace coverage as a part of user_copy test case, which basically repeats the copy test after replacing the old ioas with a new one.
Reviewed-by: Kevin Tian kevin.tian@intel.com Signed-off-by: Nicolin Chen nicolinc@nvidia.com --- drivers/iommu/iommufd/iommufd_test.h | 4 +++ drivers/iommu/iommufd/selftest.c | 19 ++++++++++++ tools/testing/selftests/iommu/iommufd.c | 29 +++++++++++++++++-- tools/testing/selftests/iommu/iommufd_utils.h | 19 ++++++++++++ 4 files changed, 69 insertions(+), 2 deletions(-)
diff --git a/drivers/iommu/iommufd/iommufd_test.h b/drivers/iommu/iommufd/iommufd_test.h index dd9168a20ddf..258de2253b61 100644 --- a/drivers/iommu/iommufd/iommufd_test.h +++ b/drivers/iommu/iommufd/iommufd_test.h @@ -18,6 +18,7 @@ enum { IOMMU_TEST_OP_ACCESS_RW, IOMMU_TEST_OP_SET_TEMP_MEMORY_LIMIT, IOMMU_TEST_OP_MOCK_DOMAIN_REPLACE, + IOMMU_TEST_OP_ACCESS_REPLACE_IOAS, };
enum { @@ -91,6 +92,9 @@ struct iommu_test_cmd { struct { __u32 limit; } memory_limit; + struct { + __u32 ioas_id; + } access_replace_ioas; }; __u32 last; }; diff --git a/drivers/iommu/iommufd/selftest.c b/drivers/iommu/iommufd/selftest.c index ff4fc88640a0..9f6b2d71bc87 100644 --- a/drivers/iommu/iommufd/selftest.c +++ b/drivers/iommu/iommufd/selftest.c @@ -785,6 +785,22 @@ static int iommufd_test_create_access(struct iommufd_ucmd *ucmd, return rc; }
+static int iommufd_test_access_replace_ioas(struct iommufd_ucmd *ucmd, + unsigned int access_id, + unsigned int ioas_id) +{ + struct selftest_access *staccess; + int rc; + + staccess = iommufd_access_get(access_id); + if (IS_ERR(staccess)) + return PTR_ERR(staccess); + + rc = iommufd_access_replace(staccess->access, ioas_id); + fput(staccess->file); + return rc; +} + /* Check that the pages in a page array match the pages in the user VA */ static int iommufd_test_check_pages(void __user *uptr, struct page **pages, size_t npages) @@ -1000,6 +1016,9 @@ int iommufd_test(struct iommufd_ucmd *ucmd) case IOMMU_TEST_OP_CREATE_ACCESS: return iommufd_test_create_access(ucmd, cmd->id, cmd->create_access.flags); + case IOMMU_TEST_OP_ACCESS_REPLACE_IOAS: + return iommufd_test_access_replace_ioas( + ucmd, cmd->id, cmd->access_replace_ioas.ioas_id); case IOMMU_TEST_OP_ACCESS_PAGES: return iommufd_test_access_pages( ucmd, cmd->id, cmd->access_pages.iova, diff --git a/tools/testing/selftests/iommu/iommufd.c b/tools/testing/selftests/iommu/iommufd.c index c07252dbf62d..3c8cb9d6c22b 100644 --- a/tools/testing/selftests/iommu/iommufd.c +++ b/tools/testing/selftests/iommu/iommufd.c @@ -1252,7 +1252,13 @@ TEST_F(iommufd_mock_domain, user_copy) .dst_iova = MOCK_APERTURE_START, .length = BUFFER_SIZE, }; - unsigned int ioas_id; + struct iommu_ioas_unmap unmap_cmd = { + .size = sizeof(unmap_cmd), + .ioas_id = self->ioas_id, + .iova = MOCK_APERTURE_START, + .length = BUFFER_SIZE, + }; + unsigned int new_ioas_id, ioas_id;
/* Pin the pages in an IOAS with no domains then copy to an IOAS with domains */ test_ioctl_ioas_alloc(&ioas_id); @@ -1270,11 +1276,30 @@ TEST_F(iommufd_mock_domain, user_copy) ASSERT_EQ(0, ioctl(self->fd, IOMMU_IOAS_COPY, ©_cmd)); check_mock_iova(buffer, MOCK_APERTURE_START, BUFFER_SIZE);
+ /* Now replace the ioas with a new one */ + test_ioctl_ioas_alloc(&new_ioas_id); + test_ioctl_ioas_map_id(new_ioas_id, buffer, BUFFER_SIZE, + ©_cmd.src_iova); + test_cmd_access_replace_ioas(access_cmd.id, new_ioas_id); + + /* Destroy the old ioas and cleanup copied mapping */ + ASSERT_EQ(0, ioctl(self->fd, IOMMU_IOAS_UNMAP, &unmap_cmd)); + test_ioctl_destroy(ioas_id); + + /* Then run the same test again with the new ioas */ + access_cmd.access_pages.iova = copy_cmd.src_iova; + ASSERT_EQ(0, + ioctl(self->fd, _IOMMU_TEST_CMD(IOMMU_TEST_OP_ACCESS_PAGES), + &access_cmd)); + copy_cmd.src_ioas_id = new_ioas_id; + ASSERT_EQ(0, ioctl(self->fd, IOMMU_IOAS_COPY, ©_cmd)); + check_mock_iova(buffer, MOCK_APERTURE_START, BUFFER_SIZE); + test_cmd_destroy_access_pages( access_cmd.id, access_cmd.access_pages.out_access_pages_id); test_cmd_destroy_access(access_cmd.id);
- test_ioctl_destroy(ioas_id); + test_ioctl_destroy(new_ioas_id); }
TEST_F(iommufd_mock_domain, replace) diff --git a/tools/testing/selftests/iommu/iommufd_utils.h b/tools/testing/selftests/iommu/iommufd_utils.h index 9b6dcb921750..c31e75607355 100644 --- a/tools/testing/selftests/iommu/iommufd_utils.h +++ b/tools/testing/selftests/iommu/iommufd_utils.h @@ -117,6 +117,25 @@ static int _test_cmd_hwpt_alloc(int fd, __u32 device_id, __u32 pt_id, #define test_cmd_hwpt_alloc(device_id, pt_id, hwpt_id) \ ASSERT_EQ(0, _test_cmd_hwpt_alloc(self->fd, device_id, pt_id, hwpt_id))
+static int _test_cmd_access_replace_ioas(int fd, __u32 access_id, + unsigned int ioas_id) +{ + struct iommu_test_cmd cmd = { + .size = sizeof(cmd), + .op = IOMMU_TEST_OP_ACCESS_REPLACE_IOAS, + .id = access_id, + .access_replace_ioas = { .ioas_id = ioas_id }, + }; + int ret; + + ret = ioctl(fd, IOMMU_TEST_CMD, &cmd); + if (ret) + return ret; + return 0; +} +#define test_cmd_access_replace_ioas(access_id, ioas_id) \ + ASSERT_EQ(0, _test_cmd_access_replace_ioas(self->fd, access_id, ioas_id)) + static int _test_cmd_create_access(int fd, unsigned int ioas_id, __u32 *access_id, unsigned int flags) {
Now both the physical path and the emulated path should support an IO page table replacement. Call iommufd_device_replace/iommufd_access_replace(), when vdev->iommufd_attached is true.
Also update the VFIO_DEVICE_ATTACH_IOMMUFD_PT kdoc in the uAPI header.
Signed-off-by: Nicolin Chen nicolinc@nvidia.com --- drivers/vfio/iommufd.c | 11 ++++++----- include/uapi/linux/vfio.h | 6 ++++++ 2 files changed, 12 insertions(+), 5 deletions(-)
diff --git a/drivers/vfio/iommufd.c b/drivers/vfio/iommufd.c index f3fa03495a41..861e10a6b198 100644 --- a/drivers/vfio/iommufd.c +++ b/drivers/vfio/iommufd.c @@ -113,9 +113,9 @@ int vfio_iommufd_physical_attach_ioas(struct vfio_device *vdev, u32 *pt_id) return -EINVAL;
if (vdev->iommufd_attached) - return -EBUSY; - - rc = iommufd_device_attach(vdev->iommufd_device, pt_id); + rc = iommufd_device_replace(vdev->iommufd_device, pt_id); + else + rc = iommufd_device_attach(vdev->iommufd_device, pt_id); if (rc) return rc; vdev->iommufd_attached = true; @@ -192,8 +192,9 @@ int vfio_iommufd_emulated_attach_ioas(struct vfio_device *vdev, u32 *pt_id) if (WARN_ON(!vdev->iommufd_access)) return -ENOENT; if (vdev->iommufd_attached) - return -EBUSY; - rc = iommufd_access_attach(vdev->iommufd_access, *pt_id); + rc = iommufd_access_replace(vdev->iommufd_access, *pt_id); + else + rc = iommufd_access_attach(vdev->iommufd_access, *pt_id); if (rc) return rc; vdev->iommufd_attached = true; diff --git a/include/uapi/linux/vfio.h b/include/uapi/linux/vfio.h index 53c72e26ecd3..3f2548a5e847 100644 --- a/include/uapi/linux/vfio.h +++ b/include/uapi/linux/vfio.h @@ -243,6 +243,12 @@ struct vfio_device_bind_iommufd { * * Undo by VFIO_DEVICE_DETACH_IOMMUFD_PT or device fd close. * + * If a vfio device is currently attached to a valid hw_pagetable, without doing + * a VFIO_DEVICE_DETACH_IOMMUFD_PT, a second VFIO_DEVICE_ATTACH_IOMMUFD_PT ioctl + * passing in another hw_pagetable (hwpt) id is allowed. This action, also known + * as a hw_pagetable replacement, will replace the device's currently attached + * hw_pagetable with a new hw_pagetable corresponding to the given pt_id. + * * @argsz: user filled size of this data. * @flags: must be 0. * @pt_id: Input the target id which can represent an ioas or a hwpt
From: Nicolin Chen nicolinc@nvidia.com Sent: Thursday, March 23, 2023 4:33 PM
Now both the physical path and the emulated path should support an IO page table replacement. Call iommufd_device_replace/iommufd_access_replace(), when vdev->iommufd_attached is true.
Also update the VFIO_DEVICE_ATTACH_IOMMUFD_PT kdoc in the uAPI header.
Signed-off-by: Nicolin Chen nicolinc@nvidia.com
Reviewed-by: Kevin Tian kevin.tian@intel.com
linux-kselftest-mirror@lists.linaro.org
-
Nicolin Chen -
Tian, Kevin