This series focuses on CoreSight path power management. The changes can be divided into four parts for review:
Patches 01~07: Refactor the CPU idle flow with moving common code into the CoreSight core layer. Patches 08~15: Refactor path enabling and disabling with range, add path control during CPU idle. Patches 16~17: Support the sink (TRBE) control during CPU idle. Patches 18~20: Move the CPU hotplug flow into the coresight core layer and simplify the code.
This series is rebased on the coresight-next branch and has been verified on Juno-r2 (ETM + ETR) and FVP RevC (ETE + TRBE).
--- Changes in v7: - Added a flag in coresight_desc to indicate CPU bound device (Suzuki). - Used coresight_mutex to protect per-CPU source pointer (Suzuki). - Added a spinlock for exclusive access per-CPU source pointer (Suzuki). - Dropped .pm_is_needed() callback (Suzuki). - Supported range in path enabling / disabling (Suzuki). - Gathered test and review tags (Levi / James). - Link to v6: https://lore.kernel.org/r/20260305-arm_coresight_path_power_management_impro...
Changes in v6: - Rebase on the latest coresight-next branch. - Always save and restore TRBE context during idle (Will). - Use get_cpu() / put_cpu() when set the per CPU source pointer. - Link to v5: https://lore.kernel.org/r/20251119-arm_coresight_path_power_management_impro...
Changes in v5: - Set the per-CPU source pointer on target CPU (Suzuki). - Reused existed enable/disable buffer functions in TRBE callbacks (James). - Refactored refcount for source devices in SysFS mode. - Released path in cpu-hotplug off flow to avoid memory leak. - Updated ETMv3 driver when move common code into core layer. - Rebased on the latest coresight-next branch. - Link to v4: https://lore.kernel.org/r/20251104-arm_coresight_path_power_management_impro...
Signed-off-by: Leo Yan leo.yan@arm.com
--- Leo Yan (19): coresight: Populate CPU ID into coresight_device coresight: Remove .cpu_id() callback from source ops coresight: sysfs: Validate CPU online status for per-CPU sources coresight: Move per-CPU source pointer to core layer coresight: Register CPU PM notifier in core layer coresight: etm4x: Hook CPU PM callbacks coresight: etm4x: Remove redundant condition checks in save and restore coresight: syscfg: Use spinlock to protect active variables coresight: Introduce coresight_enable_source() helper coresight: Save active path for system tracers coresight: etm4x: Set active path on target CPU coresight: etm3x: Set active path on target CPU coresight: sysfs: Use source's path pointer for path control coresight: Control path with range coresight: Control path during CPU idle coresight: Add PM callbacks for sink device coresight: sysfs: Increment refcount only for system tracers coresight: Take hotplug lock in enable_source_store() for Sysfs mode coresight: Move CPU hotplug callbacks to core layer
Yabin Cui (1): coresight: trbe: Save and restore state across CPU low power state
drivers/hwtracing/coresight/coresight-catu.c | 2 +- drivers/hwtracing/coresight/coresight-core.c | 377 +++++++++++++++++++-- drivers/hwtracing/coresight/coresight-cti-core.c | 9 +- drivers/hwtracing/coresight/coresight-etm-perf.c | 25 +- drivers/hwtracing/coresight/coresight-etm3x-core.c | 73 +--- drivers/hwtracing/coresight/coresight-etm4x-core.c | 154 ++------- drivers/hwtracing/coresight/coresight-priv.h | 4 + drivers/hwtracing/coresight/coresight-syscfg.c | 21 +- drivers/hwtracing/coresight/coresight-syscfg.h | 2 + drivers/hwtracing/coresight/coresight-sysfs.c | 126 +++---- drivers/hwtracing/coresight/coresight-trbe.c | 61 +++- include/linux/coresight.h | 15 +- 12 files changed, 546 insertions(+), 323 deletions(-) --- base-commit: a90863095f84f6d17c49716e4e2212d28a6b25b5 change-id: 20251104-arm_coresight_path_power_management_improvement-dab4966f8280
Best regards,
Add a new flag CORESIGHT_DESC_CPU_BOUND to indicate components that are CPU bound. Populate CPU ID into the coresight_device structure; otherwise, set CPU ID to -1 for non CPU bound devices.
Use the {0} initializer to clear coresight_desc structures to avoid uninitialized values.
Signed-off-by: Leo Yan leo.yan@arm.com --- drivers/hwtracing/coresight/coresight-catu.c | 2 +- drivers/hwtracing/coresight/coresight-core.c | 5 +++++ drivers/hwtracing/coresight/coresight-cti-core.c | 9 ++++++--- drivers/hwtracing/coresight/coresight-etm3x-core.c | 2 ++ drivers/hwtracing/coresight/coresight-etm4x-core.c | 2 ++ drivers/hwtracing/coresight/coresight-trbe.c | 2 ++ include/linux/coresight.h | 8 ++++++++ 7 files changed, 26 insertions(+), 4 deletions(-)
diff --git a/drivers/hwtracing/coresight/coresight-catu.c b/drivers/hwtracing/coresight/coresight-catu.c index ce71dcddfca2558eddd625de58a709b151f2e07e..43abe13995cf3c96e70dcf97856872d70f71345a 100644 --- a/drivers/hwtracing/coresight/coresight-catu.c +++ b/drivers/hwtracing/coresight/coresight-catu.c @@ -514,7 +514,7 @@ static int __catu_probe(struct device *dev, struct resource *res) int ret = 0; u32 dma_mask; struct catu_drvdata *drvdata; - struct coresight_desc catu_desc; + struct coresight_desc catu_desc = { 0 }; struct coresight_platform_data *pdata = NULL; void __iomem *base;
diff --git a/drivers/hwtracing/coresight/coresight-core.c b/drivers/hwtracing/coresight/coresight-core.c index 46f247f73cf64a97b9353b84ba5b76b991676f5f..4a246fb6bff0a75cf12995bd00a9adefb93839ec 100644 --- a/drivers/hwtracing/coresight/coresight-core.c +++ b/drivers/hwtracing/coresight/coresight-core.c @@ -1342,6 +1342,11 @@ struct coresight_device *coresight_register(struct coresight_desc *desc) csdev->access = desc->access; csdev->orphan = true;
+ if (desc->flags & CORESIGHT_DESC_CPU_BOUND) + csdev->cpu = desc->cpu; + else + csdev->cpu = -1; + csdev->dev.type = &coresight_dev_type[desc->type]; csdev->dev.groups = desc->groups; csdev->dev.parent = desc->dev; diff --git a/drivers/hwtracing/coresight/coresight-cti-core.c b/drivers/hwtracing/coresight/coresight-cti-core.c index 2f4c9362709a90b12a1aeb5016905b7d4474b912..b2c9a4db13b4e5554bca565c17ed299fdfdb30ff 100644 --- a/drivers/hwtracing/coresight/coresight-cti-core.c +++ b/drivers/hwtracing/coresight/coresight-cti-core.c @@ -659,7 +659,7 @@ static int cti_probe(struct amba_device *adev, const struct amba_id *id) void __iomem *base; struct device *dev = &adev->dev; struct cti_drvdata *drvdata = NULL; - struct coresight_desc cti_desc; + struct coresight_desc cti_desc = { 0 }; struct coresight_platform_data *pdata = NULL; struct resource *res = &adev->res;
@@ -702,11 +702,14 @@ static int cti_probe(struct amba_device *adev, const struct amba_id *id) * eCPU ID. System CTIs will have the name cti_sys<I> where I is an * index allocated by order of discovery. */ - if (drvdata->ctidev.cpu >= 0) + if (drvdata->ctidev.cpu >= 0) { + cti_desc.cpu = drvdata->ctidev.cpu; + cti_desc.flags = CORESIGHT_DESC_CPU_BOUND; cti_desc.name = devm_kasprintf(dev, GFP_KERNEL, "cti_cpu%d", drvdata->ctidev.cpu); - else + } else { cti_desc.name = coresight_alloc_device_name("cti_sys", dev); + } if (!cti_desc.name) return -ENOMEM;
diff --git a/drivers/hwtracing/coresight/coresight-etm3x-core.c b/drivers/hwtracing/coresight/coresight-etm3x-core.c index a547a6d2e0bde84748f753e5529d316c4f5e82e2..eb665db1a37d9970f7f55395c0aa23b98a7f3118 100644 --- a/drivers/hwtracing/coresight/coresight-etm3x-core.c +++ b/drivers/hwtracing/coresight/coresight-etm3x-core.c @@ -891,6 +891,8 @@ static int etm_probe(struct amba_device *adev, const struct amba_id *id) desc.pdata = pdata; desc.dev = dev; desc.groups = coresight_etm_groups; + desc.cpu = drvdata->cpu; + desc.flags = CORESIGHT_DESC_CPU_BOUND; drvdata->csdev = coresight_register(&desc); if (IS_ERR(drvdata->csdev)) return PTR_ERR(drvdata->csdev); diff --git a/drivers/hwtracing/coresight/coresight-etm4x-core.c b/drivers/hwtracing/coresight/coresight-etm4x-core.c index d565a73f0042e3e0b21fcf9cb94009cc25834d3d..b1e0254a534027d7ede8591e56be28745d0b9974 100644 --- a/drivers/hwtracing/coresight/coresight-etm4x-core.c +++ b/drivers/hwtracing/coresight/coresight-etm4x-core.c @@ -2260,6 +2260,8 @@ static int etm4_add_coresight_dev(struct etm4_init_arg *init_arg) desc.pdata = pdata; desc.dev = dev; desc.groups = coresight_etmv4_groups; + desc.cpu = drvdata->cpu; + desc.flags = CORESIGHT_DESC_CPU_BOUND; drvdata->csdev = coresight_register(&desc); if (IS_ERR(drvdata->csdev)) return PTR_ERR(drvdata->csdev); diff --git a/drivers/hwtracing/coresight/coresight-trbe.c b/drivers/hwtracing/coresight/coresight-trbe.c index 1511f8eb95afb5b4610b8fbdacc8b174b6b08530..14e35b9660d76e47619cc6026b94929b3bb3e02b 100644 --- a/drivers/hwtracing/coresight/coresight-trbe.c +++ b/drivers/hwtracing/coresight/coresight-trbe.c @@ -1289,6 +1289,8 @@ static void arm_trbe_register_coresight_cpu(struct trbe_drvdata *drvdata, int cp desc.ops = &arm_trbe_cs_ops; desc.groups = arm_trbe_groups; desc.dev = dev; + desc.cpu = cpu; + desc.flags = CORESIGHT_DESC_CPU_BOUND; trbe_csdev = coresight_register(&desc); if (IS_ERR(trbe_csdev)) goto cpu_clear; diff --git a/include/linux/coresight.h b/include/linux/coresight.h index 2131febebee93d609df1aea8534a10898b600be2..9d85e15337bc7e339695df18bcd5dac291d44916 100644 --- a/include/linux/coresight.h +++ b/include/linux/coresight.h @@ -141,6 +141,8 @@ struct csdev_access { .base = (_addr), \ })
+#define CORESIGHT_DESC_CPU_BOUND BIT(0) + /** * struct coresight_desc - description of a component required from drivers * @type: as defined by @coresight_dev_type. @@ -153,6 +155,8 @@ struct csdev_access { * in the component's sysfs sub-directory. * @name: name for the coresight device, also shown under sysfs. * @access: Describe access to the device + * @flags: The descritpion flags. + * @cpu: The CPU this component is affined to. */ struct coresight_desc { enum coresight_dev_type type; @@ -163,6 +167,8 @@ struct coresight_desc { const struct attribute_group **groups; const char *name; struct csdev_access access; + u32 flags; + int cpu; };
/** @@ -261,6 +267,7 @@ struct coresight_trace_id_map { * CS_MODE_SYSFS. Otherwise it must be accessed from inside the * spinlock. * @orphan: true if the component has connections that haven't been linked. + * @cpu: The CPU this component is affined to (-1 for not CPU bound). * @sysfs_sink_activated: 'true' when a sink has been selected for use via sysfs * by writing a 1 to the 'enable_sink' file. A sink can be * activated but not yet enabled. Enabling for a _sink_ happens @@ -287,6 +294,7 @@ struct coresight_device { atomic_t mode; int refcnt; bool orphan; + int cpu; /* sink specific fields */ bool sysfs_sink_activated; struct dev_ext_attribute *ea;
The CPU ID can be fetched directly from the coresight_device structure, so the .cpu_id() callback is no longer needed.
Remove the .cpu_id() callback from source ops and update callers accordingly.
Signed-off-by: Leo Yan leo.yan@arm.com --- drivers/hwtracing/coresight/coresight-core.c | 8 ++++---- drivers/hwtracing/coresight/coresight-etm-perf.c | 2 +- drivers/hwtracing/coresight/coresight-etm3x-core.c | 8 -------- drivers/hwtracing/coresight/coresight-etm4x-core.c | 8 -------- drivers/hwtracing/coresight/coresight-sysfs.c | 4 ++-- include/linux/coresight.h | 3 --- 6 files changed, 7 insertions(+), 26 deletions(-)
diff --git a/drivers/hwtracing/coresight/coresight-core.c b/drivers/hwtracing/coresight/coresight-core.c index 4a246fb6bff0a75cf12995bd00a9adefb93839ec..5639c8a2124d97148525c3267dacd86510da7a20 100644 --- a/drivers/hwtracing/coresight/coresight-core.c +++ b/drivers/hwtracing/coresight/coresight-core.c @@ -787,7 +787,7 @@ static int _coresight_build_path(struct coresight_device *csdev, goto out;
if (coresight_is_percpu_source(csdev) && coresight_is_percpu_sink(sink) && - sink == per_cpu(csdev_sink, source_ops(csdev)->cpu_id(csdev))) { + sink == per_cpu(csdev_sink, csdev->cpu)) { if (_coresight_build_path(sink, source, sink, path) == 0) { found = true; goto out; @@ -1014,7 +1014,7 @@ coresight_find_default_sink(struct coresight_device *csdev) /* look for a default sink if we have not found for this device */ if (!csdev->def_sink) { if (coresight_is_percpu_source(csdev)) - csdev->def_sink = per_cpu(csdev_sink, source_ops(csdev)->cpu_id(csdev)); + csdev->def_sink = per_cpu(csdev_sink, csdev->cpu); if (!csdev->def_sink) csdev->def_sink = coresight_find_sink(csdev, &depth); } @@ -1733,10 +1733,10 @@ int coresight_etm_get_trace_id(struct coresight_device *csdev, enum cs_mode mode { int cpu, trace_id;
- if (csdev->type != CORESIGHT_DEV_TYPE_SOURCE || !source_ops(csdev)->cpu_id) + if (csdev->type != CORESIGHT_DEV_TYPE_SOURCE) return -EINVAL;
- cpu = source_ops(csdev)->cpu_id(csdev); + cpu = csdev->cpu; switch (mode) { case CS_MODE_SYSFS: trace_id = coresight_trace_id_get_cpu_id(cpu); diff --git a/drivers/hwtracing/coresight/coresight-etm-perf.c b/drivers/hwtracing/coresight/coresight-etm-perf.c index f85dedf89a3f9e85d568ca4c320fa6fa6d9059ff..cae6738e9906c35d291e4b0f3feae37306b95c06 100644 --- a/drivers/hwtracing/coresight/coresight-etm-perf.c +++ b/drivers/hwtracing/coresight/coresight-etm-perf.c @@ -824,7 +824,7 @@ static void etm_addr_filters_sync(struct perf_event *event) int etm_perf_symlink(struct coresight_device *csdev, bool link) { char entry[sizeof("cpu9999999")]; - int ret = 0, cpu = source_ops(csdev)->cpu_id(csdev); + int ret = 0, cpu = csdev->cpu; struct device *pmu_dev = etm_pmu.dev; struct device *cs_dev = &csdev->dev;
diff --git a/drivers/hwtracing/coresight/coresight-etm3x-core.c b/drivers/hwtracing/coresight/coresight-etm3x-core.c index eb665db1a37d9970f7f55395c0aa23b98a7f3118..ab47f69e923fb191b48b82367dce465c79b3a93d 100644 --- a/drivers/hwtracing/coresight/coresight-etm3x-core.c +++ b/drivers/hwtracing/coresight/coresight-etm3x-core.c @@ -466,13 +466,6 @@ static void etm_enable_sysfs_smp_call(void *info) coresight_set_mode(csdev, CS_MODE_DISABLED); }
-static int etm_cpu_id(struct coresight_device *csdev) -{ - struct etm_drvdata *drvdata = dev_get_drvdata(csdev->dev.parent); - - return drvdata->cpu; -} - void etm_release_trace_id(struct etm_drvdata *drvdata) { coresight_trace_id_put_cpu_id(drvdata->cpu); @@ -684,7 +677,6 @@ static void etm_disable(struct coresight_device *csdev, }
static const struct coresight_ops_source etm_source_ops = { - .cpu_id = etm_cpu_id, .enable = etm_enable, .disable = etm_disable, }; diff --git a/drivers/hwtracing/coresight/coresight-etm4x-core.c b/drivers/hwtracing/coresight/coresight-etm4x-core.c index b1e0254a534027d7ede8591e56be28745d0b9974..66a8058098376264d3f8c5815763a75ebffb352e 100644 --- a/drivers/hwtracing/coresight/coresight-etm4x-core.c +++ b/drivers/hwtracing/coresight/coresight-etm4x-core.c @@ -227,13 +227,6 @@ static void etm4_cs_unlock(struct etmv4_drvdata *drvdata, CS_UNLOCK(csa->base); }
-static int etm4_cpu_id(struct coresight_device *csdev) -{ - struct etmv4_drvdata *drvdata = dev_get_drvdata(csdev->dev.parent); - - return drvdata->cpu; -} - void etm4_release_trace_id(struct etmv4_drvdata *drvdata) { coresight_trace_id_put_cpu_id(drvdata->cpu); @@ -1201,7 +1194,6 @@ static void etm4_pause_perf(struct coresight_device *csdev) }
static const struct coresight_ops_source etm4_source_ops = { - .cpu_id = etm4_cpu_id, .enable = etm4_enable, .disable = etm4_disable, .resume_perf = etm4_resume_perf, diff --git a/drivers/hwtracing/coresight/coresight-sysfs.c b/drivers/hwtracing/coresight/coresight-sysfs.c index d2a6ed8bcc74d64dccc735463f14790b4e80d101..3b1e7152dd108408d837c404ce607ba511ca14a6 100644 --- a/drivers/hwtracing/coresight/coresight-sysfs.c +++ b/drivers/hwtracing/coresight/coresight-sysfs.c @@ -232,7 +232,7 @@ int coresight_enable_sysfs(struct coresight_device *csdev) * be a single session per tracer (when working from sysFS) * a per-cpu variable will do just fine. */ - cpu = source_ops(csdev)->cpu_id(csdev); + cpu = csdev->cpu; per_cpu(tracer_path, cpu) = path; break; case CORESIGHT_DEV_SUBTYPE_SOURCE_SOFTWARE: @@ -282,7 +282,7 @@ void coresight_disable_sysfs(struct coresight_device *csdev)
switch (csdev->subtype.source_subtype) { case CORESIGHT_DEV_SUBTYPE_SOURCE_PROC: - cpu = source_ops(csdev)->cpu_id(csdev); + cpu = csdev->cpu; path = per_cpu(tracer_path, cpu); per_cpu(tracer_path, cpu) = NULL; break; diff --git a/include/linux/coresight.h b/include/linux/coresight.h index 9d85e15337bc7e339695df18bcd5dac291d44916..0f53c692109964de01da06e7210b7788e7f59f36 100644 --- a/include/linux/coresight.h +++ b/include/linux/coresight.h @@ -395,15 +395,12 @@ struct coresight_ops_link { /** * struct coresight_ops_source - basic operations for a source * Operations available for sources. - * @cpu_id: returns the value of the CPU number this component - * is associated to. * @enable: enables tracing for a source. * @disable: disables tracing for a source. * @resume_perf: resumes tracing for a source in perf session. * @pause_perf: pauses tracing for a source in perf session. */ struct coresight_ops_source { - int (*cpu_id)(struct coresight_device *csdev); int (*enable)(struct coresight_device *csdev, struct perf_event *event, enum cs_mode mode, struct coresight_path *path); void (*disable)(struct coresight_device *csdev,
The current SysFS flow first enables the links and sink, then rolls back to disable them if the source fails to enable. This failure can occur if the associated CPU is offline, which causes the SMP call to fail.
Validate whether the associated CPU is online for a per-CPU tracer. If the CPU is offline, return -ENODEV and bail out.
Tested-by: James Clark james.clark@linaro.org Reviewed-by: Yeoreum Yun yeoreum.yun@arm.com Reviewed-by: James Clark james.clark@linaro.org Signed-off-by: Leo Yan leo.yan@arm.com --- drivers/hwtracing/coresight/coresight-sysfs.c | 3 +++ 1 file changed, 3 insertions(+)
diff --git a/drivers/hwtracing/coresight/coresight-sysfs.c b/drivers/hwtracing/coresight/coresight-sysfs.c index 3b1e7152dd108408d837c404ce607ba511ca14a6..2546f3ef82f810fd04a119cecd2093b8150112b6 100644 --- a/drivers/hwtracing/coresight/coresight-sysfs.c +++ b/drivers/hwtracing/coresight/coresight-sysfs.c @@ -162,6 +162,9 @@ static int coresight_validate_source_sysfs(struct coresight_device *csdev, return -EINVAL; }
+ if (coresight_is_percpu_source(csdev) && !cpu_online(csdev->cpu)) + return -ENODEV; + return 0; }
Move the per-CPU source pointer from ETM perf to the core layer, as this will be used for not only perf session and also for CPU PM notifiers.
Provides coresight_{set|clear|get}_percpu_source() helpers to access the per-CPU source pointer. Add a raw spinlock to protect exclusive access.
Device registration and unregistration call the set and clear helpers for init and teardown the pointers.
Update callers to invoke coresight_get_percpu_source() for retrieving the pointer.
Signed-off-by: Leo Yan leo.yan@arm.com --- drivers/hwtracing/coresight/coresight-core.c | 32 ++++++++++++++++++++++++ drivers/hwtracing/coresight/coresight-etm-perf.c | 21 +++++++--------- drivers/hwtracing/coresight/coresight-priv.h | 1 + 3 files changed, 42 insertions(+), 12 deletions(-)
diff --git a/drivers/hwtracing/coresight/coresight-core.c b/drivers/hwtracing/coresight/coresight-core.c index 5639c8a2124d97148525c3267dacd86510da7a20..df1ea642df7204aeb41c56c2751d0f3a00bbf7a4 100644 --- a/drivers/hwtracing/coresight/coresight-core.c +++ b/drivers/hwtracing/coresight/coresight-core.c @@ -35,6 +35,9 @@ DEFINE_MUTEX(coresight_mutex); static DEFINE_PER_CPU(struct coresight_device *, csdev_sink);
+static DEFINE_RAW_SPINLOCK(coresight_dev_lock); +static DEFINE_PER_CPU(struct coresight_device *, csdev_source); + /** * struct coresight_node - elements of a path, from source to sink * @csdev: Address of an element. @@ -82,6 +85,33 @@ struct coresight_device *coresight_get_percpu_sink(int cpu) } EXPORT_SYMBOL_GPL(coresight_get_percpu_sink);
+static void coresight_set_percpu_source(struct coresight_device *csdev) +{ + if (!csdev || !coresight_is_percpu_source(csdev)) + return; + + guard(raw_spinlock_irqsave)(&coresight_dev_lock); + per_cpu(csdev_source, csdev->cpu) = csdev; +} + +static void coresight_clear_percpu_source(struct coresight_device *csdev) +{ + if (!csdev || !coresight_is_percpu_source(csdev)) + return; + + guard(raw_spinlock_irqsave)(&coresight_dev_lock); + per_cpu(csdev_source, csdev->cpu) = NULL; +} + +struct coresight_device *coresight_get_percpu_source(int cpu) +{ + if (WARN_ON(cpu < 0)) + return NULL; + + guard(raw_spinlock_irqsave)(&coresight_dev_lock); + return per_cpu(csdev_source, cpu); +} + static struct coresight_device *coresight_get_source(struct coresight_path *path) { struct coresight_device *csdev; @@ -1403,6 +1433,7 @@ struct coresight_device *coresight_register(struct coresight_desc *desc) if (ret) goto out_unlock;
+ coresight_set_percpu_source(csdev); mutex_unlock(&coresight_mutex);
if (cti_assoc_ops && cti_assoc_ops->add) @@ -1432,6 +1463,7 @@ void coresight_unregister(struct coresight_device *csdev) cti_assoc_ops->remove(csdev);
mutex_lock(&coresight_mutex); + coresight_clear_percpu_source(csdev); etm_perf_del_symlink_sink(csdev); coresight_remove_conns(csdev); coresight_clear_default_sink(csdev); diff --git a/drivers/hwtracing/coresight/coresight-etm-perf.c b/drivers/hwtracing/coresight/coresight-etm-perf.c index cae6738e9906c35d291e4b0f3feae37306b95c06..a2c1c876099d1f5dacf039d733b78827296d9c2e 100644 --- a/drivers/hwtracing/coresight/coresight-etm-perf.c +++ b/drivers/hwtracing/coresight/coresight-etm-perf.c @@ -49,7 +49,6 @@ struct etm_ctxt { };
static DEFINE_PER_CPU(struct etm_ctxt, etm_ctxt); -static DEFINE_PER_CPU(struct coresight_device *, csdev_src);
GEN_PMU_FORMAT_ATTR(cycacc); GEN_PMU_FORMAT_ATTR(timestamp); @@ -356,7 +355,7 @@ static void *etm_setup_aux(struct perf_event *event, void **pages, struct coresight_path *path; struct coresight_device *csdev;
- csdev = per_cpu(csdev_src, cpu); + csdev = coresight_get_percpu_source(cpu); /* * If there is no ETM associated with this CPU clear it from * the mask and continue with the rest. If ever we try to trace @@ -479,10 +478,11 @@ static void etm_event_start(struct perf_event *event, int flags) struct etm_event_data *event_data; struct etm_ctxt *ctxt = this_cpu_ptr(&etm_ctxt); struct perf_output_handle *handle = &ctxt->handle; - struct coresight_device *sink, *csdev = per_cpu(csdev_src, cpu); + struct coresight_device *sink, *csdev; struct coresight_path *path; u64 hw_id;
+ csdev = coresight_get_percpu_source(cpu); if (!csdev) goto fail;
@@ -630,12 +630,14 @@ static void etm_event_stop(struct perf_event *event, int mode) { int cpu = smp_processor_id(); unsigned long size; - struct coresight_device *sink, *csdev = per_cpu(csdev_src, cpu); + struct coresight_device *sink, *csdev; struct etm_ctxt *ctxt = this_cpu_ptr(&etm_ctxt); struct perf_output_handle *handle = &ctxt->handle; struct etm_event_data *event_data; struct coresight_path *path;
+ csdev = coresight_get_percpu_source(cpu); + if (mode & PERF_EF_PAUSE) return etm_event_pause(event, csdev, ctxt);
@@ -833,17 +835,12 @@ int etm_perf_symlink(struct coresight_device *csdev, bool link) if (!etm_perf_up) return -EPROBE_DEFER;
- if (link) { + if (link) ret = sysfs_create_link(&pmu_dev->kobj, &cs_dev->kobj, entry); - if (ret) - return ret; - per_cpu(csdev_src, cpu) = csdev; - } else { + else sysfs_remove_link(&pmu_dev->kobj, entry); - per_cpu(csdev_src, cpu) = NULL; - }
- return 0; + return ret; } EXPORT_SYMBOL_GPL(etm_perf_symlink);
diff --git a/drivers/hwtracing/coresight/coresight-priv.h b/drivers/hwtracing/coresight/coresight-priv.h index 1ea882dffd703b2873e41b4ce0c2564d2ce9bbad..c96d63657f9334be890cca2320abbb8cbd807802 100644 --- a/drivers/hwtracing/coresight/coresight-priv.h +++ b/drivers/hwtracing/coresight/coresight-priv.h @@ -248,6 +248,7 @@ void coresight_add_helper(struct coresight_device *csdev,
void coresight_set_percpu_sink(int cpu, struct coresight_device *csdev); struct coresight_device *coresight_get_percpu_sink(int cpu); +struct coresight_device *coresight_get_percpu_source(int cpu); void coresight_disable_source(struct coresight_device *csdev, void *data); void coresight_pause_source(struct coresight_device *csdev); int coresight_resume_source(struct coresight_device *csdev);
The current implementation only saves and restores the context for ETM sources while ignoring the context of links. However, if funnels or replicators on a linked path resides in a CPU or cluster power domain, the hardware context for the link will be lost after resuming from low power states.
To support context management for links during CPU low power modes, a better way is to implement CPU PM callbacks in the Arm CoreSight core layer. As the core layer has sufficient information for linked paths, from tracers to links, which can be used for power management.
As a first step, this patch registers CPU PM notifier in the core layer. If a source device provides callbacks for saving and restoring context, these callbacks will be invoked in CPU suspend and resume.
With this change, there is an edge case can occur: the CPU PM notifier may access the per-CPU source pointer while the ETM module is being unloaded, which clears the per-CPU pointer and releases the pointed csdev. This race will be fixed in a later change that extends control the entire path.
Reviewed-by: James Clark james.clark@linaro.org Tested-by: James Clark james.clark@linaro.org Reviewed-by: Yeoreum Yun yeoreum.yun@arm.com Signed-off-by: Leo Yan leo.yan@arm.com --- drivers/hwtracing/coresight/coresight-core.c | 84 ++++++++++++++++++++++++++++ include/linux/coresight.h | 2 + 2 files changed, 86 insertions(+)
diff --git a/drivers/hwtracing/coresight/coresight-core.c b/drivers/hwtracing/coresight/coresight-core.c index df1ea642df7204aeb41c56c2751d0f3a00bbf7a4..7186e7d948829479dfe5ea95c8cd5a03df5fc26c 100644 --- a/drivers/hwtracing/coresight/coresight-core.c +++ b/drivers/hwtracing/coresight/coresight-core.c @@ -6,6 +6,7 @@ #include <linux/acpi.h> #include <linux/bitfield.h> #include <linux/build_bug.h> +#include <linux/cpu_pm.h> #include <linux/kernel.h> #include <linux/init.h> #include <linux/types.h> @@ -1652,6 +1653,82 @@ static void coresight_release_device_list(void) } }
+static bool coresight_pm_is_needed(struct coresight_device *csdev) +{ + if (!csdev) + return false; + + /* pm_save_disable() and pm_restore_enable() must be paired */ + if (!coresight_ops(csdev)->pm_save_disable || + !coresight_ops(csdev)->pm_restore_enable) + return false; + + /* Save and restore only if the source is active */ + if (coresight_get_mode(csdev)) + return true; + + return false; +} + +static int coresight_pm_device_save(struct coresight_device *csdev) +{ + return coresight_ops(csdev)->pm_save_disable(csdev); +} + +static void coresight_pm_device_restore(struct coresight_device *csdev) +{ + coresight_ops(csdev)->pm_restore_enable(csdev); +} + +static int coresight_pm_save(struct coresight_device *source) +{ + return coresight_pm_device_save(source); +} + +static void coresight_pm_restore(struct coresight_device *source) +{ + coresight_pm_device_restore(source); +} + +static int coresight_cpu_pm_notify(struct notifier_block *nb, unsigned long cmd, + void *v) +{ + struct coresight_device *source = + coresight_get_percpu_source(smp_processor_id()); + + if (!coresight_pm_is_needed(source)) + return NOTIFY_OK; + + switch (cmd) { + case CPU_PM_ENTER: + if (coresight_pm_save(source)) + return NOTIFY_BAD; + break; + case CPU_PM_EXIT: + case CPU_PM_ENTER_FAILED: + coresight_pm_restore(source); + break; + default: + return NOTIFY_DONE; + } + + return NOTIFY_OK; +} + +static struct notifier_block coresight_cpu_pm_nb = { + .notifier_call = coresight_cpu_pm_notify, +}; + +static int __init coresight_pm_setup(void) +{ + return cpu_pm_register_notifier(&coresight_cpu_pm_nb); +} + +static void coresight_pm_cleanup(void) +{ + cpu_pm_unregister_notifier(&coresight_cpu_pm_nb); +} + const struct bus_type coresight_bustype = { .name = "coresight", }; @@ -1706,9 +1783,15 @@ static int __init coresight_init(void)
/* initialise the coresight syscfg API */ ret = cscfg_init(); + if (ret) + goto exit_notifier; + + ret = coresight_pm_setup(); if (!ret) return 0;
+ cscfg_exit(); +exit_notifier: atomic_notifier_chain_unregister(&panic_notifier_list, &coresight_notifier); exit_perf: @@ -1720,6 +1803,7 @@ static int __init coresight_init(void)
static void __exit coresight_exit(void) { + coresight_pm_cleanup(); cscfg_exit(); atomic_notifier_chain_unregister(&panic_notifier_list, &coresight_notifier); diff --git a/include/linux/coresight.h b/include/linux/coresight.h index 0f53c692109964de01da06e7210b7788e7f59f36..f771615852b84200e37b844648a13bd18c91593d 100644 --- a/include/linux/coresight.h +++ b/include/linux/coresight.h @@ -438,6 +438,8 @@ struct coresight_ops_panic { struct coresight_ops { int (*trace_id)(struct coresight_device *csdev, enum cs_mode mode, struct coresight_device *sink); + int (*pm_save_disable)(struct coresight_device *csdev); + void (*pm_restore_enable)(struct coresight_device *csdev); const struct coresight_ops_sink *sink_ops; const struct coresight_ops_link *link_ops; const struct coresight_ops_source *source_ops;
Hook the CPU save and restore callbacks when the self-hosted state is enabled (pm_save_enable == PARAM_PM_SAVE_SELF_HOSTED) so they can be invoked by the core layer.
The CPU PM notifier in the ETMv4 driver is no longer needed, remove it along with its registration and unregistration code.
Reviewed-by: James Clark james.clark@linaro.org Tested-by: James Clark james.clark@linaro.org Reviewed-by: Yeoreum Yun yeoreum.yun@arm.com Signed-off-by: Leo Yan leo.yan@arm.com --- drivers/hwtracing/coresight/coresight-etm4x-core.c | 57 +++++----------------- 1 file changed, 11 insertions(+), 46 deletions(-)
diff --git a/drivers/hwtracing/coresight/coresight-etm4x-core.c b/drivers/hwtracing/coresight/coresight-etm4x-core.c index 66a8058098376264d3f8c5815763a75ebffb352e..6ed0ee95c78c24bcb6c636d9ebf2ae463c748b22 100644 --- a/drivers/hwtracing/coresight/coresight-etm4x-core.c +++ b/drivers/hwtracing/coresight/coresight-etm4x-core.c @@ -1200,7 +1200,7 @@ static const struct coresight_ops_source etm4_source_ops = { .pause_perf = etm4_pause_perf, };
-static const struct coresight_ops etm4_cs_ops = { +static struct coresight_ops etm4_cs_ops = { .trace_id = coresight_etm_get_trace_id, .source_ops = &etm4_source_ops, }; @@ -2000,8 +2000,9 @@ static int __etm4_cpu_save(struct etmv4_drvdata *drvdata) return ret; }
-static int etm4_cpu_save(struct etmv4_drvdata *drvdata) +static int etm4_cpu_save(struct coresight_device *csdev) { + struct etmv4_drvdata *drvdata = dev_get_drvdata(csdev->dev.parent); int ret = 0;
if (pm_save_enable != PARAM_PM_SAVE_SELF_HOSTED) @@ -2117,8 +2118,10 @@ static void __etm4_cpu_restore(struct etmv4_drvdata *drvdata) etm4_cs_lock(drvdata, csa); }
-static void etm4_cpu_restore(struct etmv4_drvdata *drvdata) +static void etm4_cpu_restore(struct coresight_device *csdev) { + struct etmv4_drvdata *drvdata = dev_get_drvdata(csdev->dev.parent); + if (pm_save_enable != PARAM_PM_SAVE_SELF_HOSTED) return;
@@ -2126,55 +2129,17 @@ static void etm4_cpu_restore(struct etmv4_drvdata *drvdata) __etm4_cpu_restore(drvdata); }
-static int etm4_cpu_pm_notify(struct notifier_block *nb, unsigned long cmd, - void *v) -{ - struct etmv4_drvdata *drvdata; - unsigned int cpu = smp_processor_id(); - - if (!etmdrvdata[cpu]) - return NOTIFY_OK; - - drvdata = etmdrvdata[cpu]; - - if (WARN_ON_ONCE(drvdata->cpu != cpu)) - return NOTIFY_BAD; - - switch (cmd) { - case CPU_PM_ENTER: - if (etm4_cpu_save(drvdata)) - return NOTIFY_BAD; - break; - case CPU_PM_EXIT: - case CPU_PM_ENTER_FAILED: - etm4_cpu_restore(drvdata); - break; - default: - return NOTIFY_DONE; - } - - return NOTIFY_OK; -} - -static struct notifier_block etm4_cpu_pm_nb = { - .notifier_call = etm4_cpu_pm_notify, -}; - /* Setup PM. Deals with error conditions and counts */ static int __init etm4_pm_setup(void) { int ret;
- ret = cpu_pm_register_notifier(&etm4_cpu_pm_nb); - if (ret) - return ret; - ret = cpuhp_setup_state_nocalls(CPUHP_AP_ARM_CORESIGHT_STARTING, "arm/coresight4:starting", etm4_starting_cpu, etm4_dying_cpu);
if (ret) - goto unregister_notifier; + return ret;
ret = cpuhp_setup_state_nocalls(CPUHP_AP_ONLINE_DYN, "arm/coresight4:online", @@ -2188,15 +2153,11 @@ static int __init etm4_pm_setup(void)
/* failed dyn state - remove others */ cpuhp_remove_state_nocalls(CPUHP_AP_ARM_CORESIGHT_STARTING); - -unregister_notifier: - cpu_pm_unregister_notifier(&etm4_cpu_pm_nb); return ret; }
static void etm4_pm_clear(void) { - cpu_pm_unregister_notifier(&etm4_cpu_pm_nb); cpuhp_remove_state_nocalls(CPUHP_AP_ARM_CORESIGHT_STARTING); if (hp_online) { cpuhp_remove_state_nocalls(hp_online); @@ -2308,6 +2269,10 @@ static int etm4_probe(struct device *dev) sizeof(struct etmv4_save_state), GFP_KERNEL); if (!drvdata->save_state) return -ENOMEM; + + /* Setup CPU PM callbacks */ + etm4_cs_ops.pm_save_disable = etm4_cpu_save; + etm4_cs_ops.pm_restore_enable = etm4_cpu_restore; }
raw_spin_lock_init(&drvdata->spinlock);
Since the core layer determines whether context save and restore operations are needed, performing the same check within the save and restore callbacks is redundant.
The save and restore flows currently use two-level functions: the first level handles the condition check, while the second level performs the low level operations. As the checks are no longer necessary, simplify the logic into single-level functions.
Reviewed-by: Mike Leach mike.leach@linaro.org Tested-by: James Clark james.clark@linaro.org Reviewed-by: Yeoreum Yun yeoreum.yun@arm.com Reviewed-by: James Clark james.clark@linaro.org Signed-off-by: Leo Yan leo.yan@arm.com --- drivers/hwtracing/coresight/coresight-etm4x-core.c | 35 +++------------------- 1 file changed, 4 insertions(+), 31 deletions(-)
diff --git a/drivers/hwtracing/coresight/coresight-etm4x-core.c b/drivers/hwtracing/coresight/coresight-etm4x-core.c index 6ed0ee95c78c24bcb6c636d9ebf2ae463c748b22..4a9b226781a5c2ab6efcadf08614de58bcd4043a 100644 --- a/drivers/hwtracing/coresight/coresight-etm4x-core.c +++ b/drivers/hwtracing/coresight/coresight-etm4x-core.c @@ -1858,11 +1858,11 @@ static int etm4_dying_cpu(unsigned int cpu) return 0; }
-static int __etm4_cpu_save(struct etmv4_drvdata *drvdata) +static int etm4_cpu_save(struct coresight_device *csdev) { int i, ret = 0; struct etmv4_save_state *state; - struct coresight_device *csdev = drvdata->csdev; + struct etmv4_drvdata *drvdata = dev_get_drvdata(csdev->dev.parent); struct csdev_access *csa; struct device *etm_dev;
@@ -2000,26 +2000,10 @@ static int __etm4_cpu_save(struct etmv4_drvdata *drvdata) return ret; }
-static int etm4_cpu_save(struct coresight_device *csdev) -{ - struct etmv4_drvdata *drvdata = dev_get_drvdata(csdev->dev.parent); - int ret = 0; - - if (pm_save_enable != PARAM_PM_SAVE_SELF_HOSTED) - return 0; - - /* - * Save and restore the ETM Trace registers only if - * the ETM is active. - */ - if (coresight_get_mode(drvdata->csdev)) - ret = __etm4_cpu_save(drvdata); - return ret; -} - -static void __etm4_cpu_restore(struct etmv4_drvdata *drvdata) +static void etm4_cpu_restore(struct coresight_device *csdev) { int i; + struct etmv4_drvdata *drvdata = dev_get_drvdata(csdev->dev.parent); struct etmv4_save_state *state = drvdata->save_state; struct csdev_access *csa = &drvdata->csdev->access;
@@ -2118,17 +2102,6 @@ static void __etm4_cpu_restore(struct etmv4_drvdata *drvdata) etm4_cs_lock(drvdata, csa); }
-static void etm4_cpu_restore(struct coresight_device *csdev) -{ - struct etmv4_drvdata *drvdata = dev_get_drvdata(csdev->dev.parent); - - if (pm_save_enable != PARAM_PM_SAVE_SELF_HOSTED) - return; - - if (coresight_get_mode(drvdata->csdev)) - __etm4_cpu_restore(drvdata); -} - /* Setup PM. Deals with error conditions and counts */ static int __init etm4_pm_setup(void) {
To support dynamically enabling and disabling the CoreSight path during CPU idle, cscfg_config_sysfs_get_active_cfg() may be called by the ETM driver in an atomic context. However, the function currently acquires a mutex, which is a sleepable lock, and this is not allowed in atomic context.
To make cscfg_config_sysfs_get_active_cfg() called in the idle flow, replace the mutex with a raw spinlock to protect the active variables.
Reviewed-by: James Clark james.clark@linaro.org Tested-by: James Clark james.clark@linaro.org Reviewed-by: Yeoreum Yun yeoreum.yun@arm.com Signed-off-by: Leo Yan leo.yan@arm.com --- drivers/hwtracing/coresight/coresight-syscfg.c | 21 ++++++++++++--------- drivers/hwtracing/coresight/coresight-syscfg.h | 2 ++ 2 files changed, 14 insertions(+), 9 deletions(-)
diff --git a/drivers/hwtracing/coresight/coresight-syscfg.c b/drivers/hwtracing/coresight/coresight-syscfg.c index d7f5037953d6ba7fb7f83a8012a1abc5ffd0a147..3b77e4da6b6b3cfc0aa687c0280c8ee5118e286b 100644 --- a/drivers/hwtracing/coresight/coresight-syscfg.c +++ b/drivers/hwtracing/coresight/coresight-syscfg.c @@ -957,15 +957,18 @@ int cscfg_config_sysfs_activate(struct cscfg_config_desc *config_desc, bool acti
cfg_hash = (unsigned long)config_desc->event_ea->var;
+ raw_spin_lock(&cscfg_mgr->spinlock); + if (activate) { /* cannot be a current active value to activate this */ if (cscfg_mgr->sysfs_active_config) { err = -EBUSY; - goto exit_unlock; + } else { + err = _cscfg_activate_config(cfg_hash); + if (!err) + cscfg_mgr->sysfs_active_config = cfg_hash; } - err = _cscfg_activate_config(cfg_hash); - if (!err) - cscfg_mgr->sysfs_active_config = cfg_hash; + } else { /* disable if matching current value */ if (cscfg_mgr->sysfs_active_config == cfg_hash) { @@ -975,7 +978,8 @@ int cscfg_config_sysfs_activate(struct cscfg_config_desc *config_desc, bool acti err = -EINVAL; }
-exit_unlock: + raw_spin_unlock(&cscfg_mgr->spinlock); + mutex_unlock(&cscfg_mutex); return err; } @@ -983,9 +987,8 @@ int cscfg_config_sysfs_activate(struct cscfg_config_desc *config_desc, bool acti /* set the sysfs preset value */ void cscfg_config_sysfs_set_preset(int preset) { - mutex_lock(&cscfg_mutex); + guard(raw_spinlock)(&cscfg_mgr->spinlock); cscfg_mgr->sysfs_active_preset = preset; - mutex_unlock(&cscfg_mutex); }
/* @@ -994,10 +997,9 @@ void cscfg_config_sysfs_set_preset(int preset) */ void cscfg_config_sysfs_get_active_cfg(unsigned long *cfg_hash, int *preset) { - mutex_lock(&cscfg_mutex); + guard(raw_spinlock)(&cscfg_mgr->spinlock); *preset = cscfg_mgr->sysfs_active_preset; *cfg_hash = cscfg_mgr->sysfs_active_config; - mutex_unlock(&cscfg_mutex); } EXPORT_SYMBOL_GPL(cscfg_config_sysfs_get_active_cfg);
@@ -1201,6 +1203,7 @@ static int cscfg_create_device(void) INIT_LIST_HEAD(&cscfg_mgr->load_order_list); atomic_set(&cscfg_mgr->sys_active_cnt, 0); cscfg_mgr->load_state = CSCFG_NONE; + raw_spin_lock_init(&cscfg_mgr->spinlock);
/* setup the device */ dev = cscfg_device(); diff --git a/drivers/hwtracing/coresight/coresight-syscfg.h b/drivers/hwtracing/coresight/coresight-syscfg.h index 66e2db890d8203853a0c3c907b48aa66dd8014e6..f52aba4608dadc1c7666e0e28c9b43dc1c33d1eb 100644 --- a/drivers/hwtracing/coresight/coresight-syscfg.h +++ b/drivers/hwtracing/coresight/coresight-syscfg.h @@ -42,6 +42,7 @@ enum cscfg_load_ops { * @sysfs_active_config:Active config hash used if CoreSight controlled from sysfs. * @sysfs_active_preset:Active preset index used if CoreSight controlled from sysfs. * @load_state: A multi-stage load/unload operation is in progress. + * @spinlock: Exclusive access sysfs_active_* variables. */ struct cscfg_manager { struct device dev; @@ -54,6 +55,7 @@ struct cscfg_manager { u32 sysfs_active_config; int sysfs_active_preset; enum cscfg_load_ops load_state; + raw_spinlock_t spinlock; };
/* get reference to dev in cscfg_manager */
Introduce the coresight_enable_source() helper for enabling source device, refine the comment for the imbalance between enable and disable pair functions.
Add validation to ensure the device is a source before proceeding with further operations.
Reviewed-by: James Clark james.clark@linaro.org Tested-by: James Clark james.clark@linaro.org Reviewed-by: Yeoreum Yun yeoreum.yun@arm.com Signed-off-by: Leo Yan leo.yan@arm.com --- drivers/hwtracing/coresight/coresight-core.c | 20 +++++++++++++++++--- drivers/hwtracing/coresight/coresight-etm-perf.c | 2 +- drivers/hwtracing/coresight/coresight-priv.h | 3 +++ drivers/hwtracing/coresight/coresight-sysfs.c | 2 +- 4 files changed, 22 insertions(+), 5 deletions(-)
diff --git a/drivers/hwtracing/coresight/coresight-core.c b/drivers/hwtracing/coresight/coresight-core.c index 7186e7d948829479dfe5ea95c8cd5a03df5fc26c..4b55c54c5fbade99404ad662aa4111d4c3c2b85b 100644 --- a/drivers/hwtracing/coresight/coresight-core.c +++ b/drivers/hwtracing/coresight/coresight-core.c @@ -416,17 +416,31 @@ static void coresight_disable_helpers(struct coresight_device *csdev, }
/* - * Helper function to call source_ops(csdev)->disable and also disable the - * helpers. + * coresight_enable_source() only enables the source but does nothing for the + * associated helpers. In contrast, coresight_disable_source() calls + * source_ops(csdev)->disable() and also disables the helpers. * * There is an imbalance between coresight_enable_path() and * coresight_disable_path(). Enabling also enables the source's helpers as part * of the path, but disabling always skips the first item in the path (which is * the source), so sources and their helpers don't get disabled as part of that - * function and we need the extra step here. + * function and we need the extra step in coresight_disable_source(). */ +int coresight_enable_source(struct coresight_device *csdev, + struct perf_event *event, enum cs_mode mode, + struct coresight_path *path) +{ + if (!coresight_is_device_source(csdev)) + return -EINVAL; + + return source_ops(csdev)->enable(csdev, event, mode, path); +} + void coresight_disable_source(struct coresight_device *csdev, void *data) { + if (!coresight_is_device_source(csdev)) + return; + source_ops(csdev)->disable(csdev, data); coresight_disable_helpers(csdev, NULL); } diff --git a/drivers/hwtracing/coresight/coresight-etm-perf.c b/drivers/hwtracing/coresight/coresight-etm-perf.c index a2c1c876099d1f5dacf039d733b78827296d9c2e..6783396635b7a62328483ad5d8837c9a59990e50 100644 --- a/drivers/hwtracing/coresight/coresight-etm-perf.c +++ b/drivers/hwtracing/coresight/coresight-etm-perf.c @@ -533,7 +533,7 @@ static void etm_event_start(struct perf_event *event, int flags) goto fail_end_stop;
/* Finally enable the tracer */ - if (source_ops(csdev)->enable(csdev, event, CS_MODE_PERF, path)) + if (coresight_enable_source(csdev, event, CS_MODE_PERF, path)) goto fail_disable_path;
/* diff --git a/drivers/hwtracing/coresight/coresight-priv.h b/drivers/hwtracing/coresight/coresight-priv.h index c96d63657f9334be890cca2320abbb8cbd807802..51ff1718502d6a5bf86b16e0723d7cdfdecc1848 100644 --- a/drivers/hwtracing/coresight/coresight-priv.h +++ b/drivers/hwtracing/coresight/coresight-priv.h @@ -249,6 +249,9 @@ void coresight_add_helper(struct coresight_device *csdev, void coresight_set_percpu_sink(int cpu, struct coresight_device *csdev); struct coresight_device *coresight_get_percpu_sink(int cpu); struct coresight_device *coresight_get_percpu_source(int cpu); +int coresight_enable_source(struct coresight_device *csdev, + struct perf_event *event, enum cs_mode mode, + struct coresight_path *path); void coresight_disable_source(struct coresight_device *csdev, void *data); void coresight_pause_source(struct coresight_device *csdev); int coresight_resume_source(struct coresight_device *csdev); diff --git a/drivers/hwtracing/coresight/coresight-sysfs.c b/drivers/hwtracing/coresight/coresight-sysfs.c index 2546f3ef82f810fd04a119cecd2093b8150112b6..de1cc05433f565eb7cb91e386f79939857dadcd5 100644 --- a/drivers/hwtracing/coresight/coresight-sysfs.c +++ b/drivers/hwtracing/coresight/coresight-sysfs.c @@ -66,7 +66,7 @@ static int coresight_enable_source_sysfs(struct coresight_device *csdev, */ lockdep_assert_held(&coresight_mutex); if (coresight_get_mode(csdev) != CS_MODE_SYSFS) { - ret = source_ops(csdev)->enable(csdev, NULL, mode, path); + ret = coresight_enable_source(csdev, NULL, mode, path); if (ret) return ret; }
Store the active path in the source device structure for system tracers (e.g. STM). Ensure the path pointer is cleared when the source is disabled.
Tested-by: James Clark james.clark@linaro.org Reviewed-by: Yeoreum Yun yeoreum.yun@arm.com Reviewed-by: James Clark james.clark@linaro.org Signed-off-by: Leo Yan leo.yan@arm.com --- drivers/hwtracing/coresight/coresight-core.c | 28 ++++++++++++++++++++++++++-- include/linux/coresight.h | 2 ++ 2 files changed, 28 insertions(+), 2 deletions(-)
diff --git a/drivers/hwtracing/coresight/coresight-core.c b/drivers/hwtracing/coresight/coresight-core.c index 4b55c54c5fbade99404ad662aa4111d4c3c2b85b..81bf4637f788f99844e839fd9dad8802ee78eb60 100644 --- a/drivers/hwtracing/coresight/coresight-core.c +++ b/drivers/hwtracing/coresight/coresight-core.c @@ -430,19 +430,43 @@ int coresight_enable_source(struct coresight_device *csdev, struct perf_event *event, enum cs_mode mode, struct coresight_path *path) { + int ret; + if (!coresight_is_device_source(csdev)) return -EINVAL;
- return source_ops(csdev)->enable(csdev, event, mode, path); + ret = source_ops(csdev)->enable(csdev, event, mode, path); + if (ret) + return ret; + + /* + * The per-CPU source has updated its path pointer in the enable() + * callback, ensuring synchronization on the target CPU. Set the + * path pointer here for non per-CPU sources. + */ + if (!coresight_is_percpu_source(csdev)) + csdev->path = path; + + return 0; }
void coresight_disable_source(struct coresight_device *csdev, void *data) { + struct coresight_path *path; + if (!coresight_is_device_source(csdev)) return;
+ /* + * The per-CPU source's disable callback may clear csdev->path, so + * save the path pointer used by disabling helper. + */ + path = csdev->path; source_ops(csdev)->disable(csdev, data); - coresight_disable_helpers(csdev, NULL); + coresight_disable_helpers(csdev, path); + + if (!coresight_is_percpu_source(csdev)) + csdev->path = NULL; } EXPORT_SYMBOL_GPL(coresight_disable_source);
diff --git a/include/linux/coresight.h b/include/linux/coresight.h index f771615852b84200e37b844648a13bd18c91593d..11d61cdfac8d4d51e831a84635d8438383665d3e 100644 --- a/include/linux/coresight.h +++ b/include/linux/coresight.h @@ -268,6 +268,7 @@ struct coresight_trace_id_map { * spinlock. * @orphan: true if the component has connections that haven't been linked. * @cpu: The CPU this component is affined to (-1 for not CPU bound). + * @path: Activated path pointer (only used for source device). * @sysfs_sink_activated: 'true' when a sink has been selected for use via sysfs * by writing a 1 to the 'enable_sink' file. A sink can be * activated but not yet enabled. Enabling for a _sink_ happens @@ -295,6 +296,7 @@ struct coresight_device { int refcnt; bool orphan; int cpu; + struct coresight_path *path; /* sink specific fields */ bool sysfs_sink_activated; struct dev_ext_attribute *ea;
Set and clear the path pointer on the target CPU during ETM enable and disable operations to avoid race conditions when retrieving the pointer in CPU PM callbacks.
Tested-by: James Clark james.clark@linaro.org Reviewed-by: Yeoreum Yun yeoreum.yun@arm.com Reviewed-by: James Clark james.clark@linaro.org Signed-off-by: Leo Yan leo.yan@arm.com --- drivers/hwtracing/coresight/coresight-etm4x-core.c | 18 +++++++++++++++--- 1 file changed, 15 insertions(+), 3 deletions(-)
diff --git a/drivers/hwtracing/coresight/coresight-etm4x-core.c b/drivers/hwtracing/coresight/coresight-etm4x-core.c index 4a9b226781a5c2ab6efcadf08614de58bcd4043a..1f100e8bbcc775a1bdc5f8b27ff15dbcaf030a63 100644 --- a/drivers/hwtracing/coresight/coresight-etm4x-core.c +++ b/drivers/hwtracing/coresight/coresight-etm4x-core.c @@ -234,6 +234,7 @@ void etm4_release_trace_id(struct etmv4_drvdata *drvdata)
struct etm4_enable_arg { struct etmv4_drvdata *drvdata; + struct coresight_path *path; int rc; };
@@ -621,8 +622,12 @@ static void etm4_enable_sysfs_smp_call(void *info) arg->rc = etm4_enable_hw(arg->drvdata);
/* The tracer didn't start */ - if (arg->rc) + if (arg->rc) { coresight_set_mode(csdev, CS_MODE_DISABLED); + return; + } + + csdev->path = arg->path; }
/* @@ -890,9 +895,13 @@ static int etm4_enable_perf(struct coresight_device *csdev,
out: /* Failed to start tracer; roll back to DISABLED mode */ - if (ret) + if (ret) { coresight_set_mode(csdev, CS_MODE_DISABLED); - return ret; + return ret; + } + + csdev->path = path; + return 0; }
static int etm4_enable_sysfs(struct coresight_device *csdev, struct coresight_path *path) @@ -922,6 +931,7 @@ static int etm4_enable_sysfs(struct coresight_device *csdev, struct coresight_pa * ensures that register writes occur when cpu is powered. */ arg.drvdata = drvdata; + arg.path = path; ret = smp_call_function_single(drvdata->cpu, etm4_enable_sysfs_smp_call, &arg, 1); if (!ret) @@ -1063,6 +1073,7 @@ static void etm4_disable_sysfs_smp_call(void *info)
etm4_disable_hw(drvdata);
+ drvdata->csdev->path = NULL; coresight_set_mode(drvdata->csdev, CS_MODE_DISABLED); }
@@ -1092,6 +1103,7 @@ static int etm4_disable_perf(struct coresight_device *csdev, /* TRCVICTLR::SSSTATUS, bit[9] */ filters->ssstatus = (control & BIT(9));
+ drvdata->csdev->path = NULL; coresight_set_mode(drvdata->csdev, CS_MODE_DISABLED);
/*
Set and clear the path pointer on the target CPU during ETM enable and disable operations.
Tested-by: James Clark james.clark@linaro.org Reviewed-by: Yeoreum Yun yeoreum.yun@arm.com Reviewed-by: James Clark james.clark@linaro.org Signed-off-by: Leo Yan leo.yan@arm.com --- drivers/hwtracing/coresight/coresight-etm3x-core.c | 17 ++++++++++++++--- 1 file changed, 14 insertions(+), 3 deletions(-)
diff --git a/drivers/hwtracing/coresight/coresight-etm3x-core.c b/drivers/hwtracing/coresight/coresight-etm3x-core.c index ab47f69e923fb191b48b82367dce465c79b3a93d..c2b8c096cbe9e9759b3d17f019a2d73f59234ac9 100644 --- a/drivers/hwtracing/coresight/coresight-etm3x-core.c +++ b/drivers/hwtracing/coresight/coresight-etm3x-core.c @@ -441,6 +441,7 @@ static int etm_enable_hw(struct etm_drvdata *drvdata)
struct etm_enable_arg { struct etm_drvdata *drvdata; + struct coresight_path *path; int rc; };
@@ -462,8 +463,12 @@ static void etm_enable_sysfs_smp_call(void *info) arg->rc = etm_enable_hw(arg->drvdata);
/* The tracer didn't start */ - if (arg->rc) + if (arg->rc) { coresight_set_mode(csdev, CS_MODE_DISABLED); + return; + } + + csdev->path = arg->path; }
void etm_release_trace_id(struct etm_drvdata *drvdata) @@ -492,10 +497,13 @@ static int etm_enable_perf(struct coresight_device *csdev, ret = etm_enable_hw(drvdata);
/* Failed to start tracer; roll back to DISABLED mode */ - if (ret) + if (ret) { coresight_set_mode(csdev, CS_MODE_DISABLED); + return ret; + }
- return ret; + csdev->path = path; + return 0; }
static int etm_enable_sysfs(struct coresight_device *csdev, struct coresight_path *path) @@ -514,6 +522,7 @@ static int etm_enable_sysfs(struct coresight_device *csdev, struct coresight_pat */ if (cpu_online(drvdata->cpu)) { arg.drvdata = drvdata; + arg.path = path; ret = smp_call_function_single(drvdata->cpu, etm_enable_sysfs_smp_call, &arg, 1); if (!ret) @@ -583,6 +592,7 @@ static void etm_disable_sysfs_smp_call(void *info)
etm_disable_hw(drvdata);
+ drvdata->csdev->path = NULL; coresight_set_mode(drvdata->csdev, CS_MODE_DISABLED); }
@@ -607,6 +617,7 @@ static void etm_disable_perf(struct coresight_device *csdev)
CS_LOCK(drvdata->csa.base);
+ drvdata->csdev->path = NULL; coresight_set_mode(drvdata->csdev, CS_MODE_DISABLED);
/*
Since the path pointer is stored in the source's structure, retrieve it directly when disabling the path.
As a result, the global variables used for caching path pointers are no longer needed. Remove them to simplify the code.
Tested-by: James Clark james.clark@linaro.org Reviewed-by: Yeoreum Yun yeoreum.yun@arm.com Reviewed-by: James Clark james.clark@linaro.org Signed-off-by: Leo Yan leo.yan@arm.com --- drivers/hwtracing/coresight/coresight-sysfs.c | 79 +++------------------------ 1 file changed, 9 insertions(+), 70 deletions(-)
diff --git a/drivers/hwtracing/coresight/coresight-sysfs.c b/drivers/hwtracing/coresight/coresight-sysfs.c index de1cc05433f565eb7cb91e386f79939857dadcd5..9e94505bccf4ac91207f2f5393ae57ae077d61d6 100644 --- a/drivers/hwtracing/coresight/coresight-sysfs.c +++ b/drivers/hwtracing/coresight/coresight-sysfs.c @@ -12,19 +12,6 @@ #include "coresight-priv.h" #include "coresight-trace-id.h"
-/* - * Use IDR to map the hash of the source's device name - * to the pointer of path for the source. The idr is for - * the sources which aren't associated with CPU. - */ -static DEFINE_IDR(path_idr); - -/* - * When operating Coresight drivers from the sysFS interface, only a single - * path can exist from a tracer (associated to a CPU) to a sink. - */ -static DEFINE_PER_CPU(struct coresight_path *, tracer_path); - ssize_t coresight_simple_show_pair(struct device *_dev, struct device_attribute *attr, char *buf) { @@ -170,11 +157,10 @@ static int coresight_validate_source_sysfs(struct coresight_device *csdev,
int coresight_enable_sysfs(struct coresight_device *csdev) { - int cpu, ret = 0; + int ret = 0; struct coresight_device *sink; struct coresight_path *path; enum coresight_dev_subtype_source subtype; - u32 hash;
subtype = csdev->subtype.source_subtype;
@@ -226,35 +212,6 @@ int coresight_enable_sysfs(struct coresight_device *csdev) if (ret) goto err_source;
- switch (subtype) { - case CORESIGHT_DEV_SUBTYPE_SOURCE_PROC: - /* - * When working from sysFS it is important to keep track - * of the paths that were created so that they can be - * undone in 'coresight_disable()'. Since there can only - * be a single session per tracer (when working from sysFS) - * a per-cpu variable will do just fine. - */ - cpu = csdev->cpu; - per_cpu(tracer_path, cpu) = path; - break; - case CORESIGHT_DEV_SUBTYPE_SOURCE_SOFTWARE: - case CORESIGHT_DEV_SUBTYPE_SOURCE_TPDM: - case CORESIGHT_DEV_SUBTYPE_SOURCE_OTHERS: - /* - * Use the hash of source's device name as ID - * and map the ID to the pointer of the path. - */ - hash = hashlen_hash(hashlen_string(NULL, dev_name(&csdev->dev))); - ret = idr_alloc_u32(&path_idr, path, &hash, hash, GFP_KERNEL); - if (ret) - goto err_source; - break; - default: - /* We can't be here */ - break; - } - out: mutex_unlock(&coresight_mutex); return ret; @@ -270,9 +227,8 @@ EXPORT_SYMBOL_GPL(coresight_enable_sysfs);
void coresight_disable_sysfs(struct coresight_device *csdev) { - int cpu, ret; - struct coresight_path *path = NULL; - u32 hash; + int ret; + struct coresight_path *path;
mutex_lock(&coresight_mutex);
@@ -280,32 +236,15 @@ void coresight_disable_sysfs(struct coresight_device *csdev) if (ret) goto out;
+ /* + * coresight_disable_source_sysfs() clears the 'csdev->path' pointer + * when disabling the source. Retrieve the path pointer here. + */ + path = csdev->path; + if (!coresight_disable_source_sysfs(csdev, NULL)) goto out;
- switch (csdev->subtype.source_subtype) { - case CORESIGHT_DEV_SUBTYPE_SOURCE_PROC: - cpu = csdev->cpu; - path = per_cpu(tracer_path, cpu); - per_cpu(tracer_path, cpu) = NULL; - break; - case CORESIGHT_DEV_SUBTYPE_SOURCE_SOFTWARE: - case CORESIGHT_DEV_SUBTYPE_SOURCE_TPDM: - case CORESIGHT_DEV_SUBTYPE_SOURCE_OTHERS: - hash = hashlen_hash(hashlen_string(NULL, dev_name(&csdev->dev))); - /* Find the path by the hash. */ - path = idr_find(&path_idr, hash); - if (path == NULL) { - pr_err("Path is not found for %s\n", dev_name(&csdev->dev)); - goto out; - } - idr_remove(&path_idr, hash); - break; - default: - /* We can't be here */ - break; - } - coresight_disable_path(path); coresight_release_path(path);
Add parameters @from and @to for path enabling and disabling, allowing finer-grained control of a path.
Disabling a path uses the range (@from..@to]. @from is exclusive to handle the case where the source is passed as the start node: the first node is skipped, as the source has its own disable function.
Enabling a path uses the range [@from..@to], where both @from and @to are inclusive. The enable path still needs to enable the source's helpers, though the source has its own enable function.
Introduce coresight_path_nodes_in_order() to validate the given range if is ordered. Update callers accordingly.
Signed-off-by: Leo Yan leo.yan@arm.com --- drivers/hwtracing/coresight/coresight-core.c | 96 ++++++++++++++++++++++++---- 1 file changed, 85 insertions(+), 11 deletions(-)
diff --git a/drivers/hwtracing/coresight/coresight-core.c b/drivers/hwtracing/coresight/coresight-core.c index 81bf4637f788f99844e839fd9dad8802ee78eb60..874bd87da2270fdbe6b32c2fa29292d7a26b8dc7 100644 --- a/drivers/hwtracing/coresight/coresight-core.c +++ b/drivers/hwtracing/coresight/coresight-core.c @@ -492,20 +492,62 @@ int coresight_resume_source(struct coresight_device *csdev) } EXPORT_SYMBOL_GPL(coresight_resume_source);
+static struct coresight_node * +coresight_path_first_node(struct coresight_path *path) +{ + return list_first_entry(&path->path_list, struct coresight_node, link); +} + +static struct coresight_node * +coresight_path_last_node(struct coresight_path *path) +{ + return list_last_entry(&path->path_list, struct coresight_node, link); +} + +static bool coresight_path_nodes_in_order(struct coresight_path *path, + struct coresight_node *from, + struct coresight_node *to) +{ + struct coresight_node *nd; + + /* Callers must fetch nodes from the path */ + if (WARN_ON_ONCE(!from || !to)) + return false; + + list_for_each_entry(nd, &path->path_list, link) { + if (nd == from) + return true; + if (nd == to) + return false; + } + + return false; +} + /* - * coresight_disable_path_from : Disable components in the given path beyond - * @nd in the list. If @nd is NULL, all the components, except the SOURCE are - * disabled. + * coresight_disable_path_from_to : Disable components in the given @path + * between @from and @to. + * + * The range excludes @from but includes @to. @from is exclusive to handle the + * case where it is the source (the first node in the path), as the source has + * its own disable function. */ -static void coresight_disable_path_from(struct coresight_path *path, - struct coresight_node *nd) +static void coresight_disable_path_from_to(struct coresight_path *path, + struct coresight_node *from, + struct coresight_node *to) { u32 type; struct coresight_device *csdev, *parent, *child; + struct coresight_node *nd;
- if (!nd) - nd = list_first_entry(&path->path_list, struct coresight_node, link); + if (!coresight_path_nodes_in_order(path, from, to)) + return;
+ /* @from is exclusive; nothing to do if @from == @to */ + if (from == to) + return; + + nd = from; list_for_each_entry_continue(nd, &path->path_list, link) { csdev = nd->csdev; type = csdev->type; @@ -545,12 +587,18 @@ static void coresight_disable_path_from(struct coresight_path *path,
/* Disable all helpers adjacent along the path last */ coresight_disable_helpers(csdev, path); + + /* Iterate up to and including @to */ + if (nd == to) + break; } }
void coresight_disable_path(struct coresight_path *path) { - coresight_disable_path_from(path, NULL); + coresight_disable_path_from_to(path, + coresight_path_first_node(path), + coresight_path_last_node(path)); } EXPORT_SYMBOL_GPL(coresight_disable_path);
@@ -574,7 +622,18 @@ static int coresight_enable_helpers(struct coresight_device *csdev, return 0; }
-int coresight_enable_path(struct coresight_path *path, enum cs_mode mode) +/* + * coresight_enable_path_from_to : Enable components in the given @path + * between @from and @to with the specified mode. + * + * The range includes both @from and @to. If @from is the source (the first + * node in the path), its helper is enabled here but the source is enabled + * in a separate function. + */ +static int coresight_enable_path_from_to(struct coresight_path *path, + enum cs_mode mode, + struct coresight_node *from, + struct coresight_node *to) { int ret = 0; u32 type; @@ -582,8 +641,12 @@ int coresight_enable_path(struct coresight_path *path, enum cs_mode mode) struct coresight_device *csdev, *parent, *child; struct coresight_device *source;
+ if (!coresight_path_nodes_in_order(path, from, to)) + return -EINVAL; + + nd = to; source = coresight_get_source(path); - list_for_each_entry_reverse(nd, &path->path_list, link) { + list_for_each_entry_from_reverse(nd, &path->path_list, link) { csdev = nd->csdev; type = csdev->type;
@@ -630,6 +693,10 @@ int coresight_enable_path(struct coresight_path *path, enum cs_mode mode) ret = -EINVAL; goto err_disable_helpers; } + + /* Iterate down to and including @from */ + if (nd == from) + break; }
out: @@ -637,10 +704,17 @@ int coresight_enable_path(struct coresight_path *path, enum cs_mode mode) err_disable_helpers: coresight_disable_helpers(csdev, path); err_disable_path: - coresight_disable_path_from(path, nd); + coresight_disable_path_from_to(path, nd, coresight_path_last_node(path)); goto out; }
+int coresight_enable_path(struct coresight_path *path, enum cs_mode mode) +{ + return coresight_enable_path_from_to(path, mode, + coresight_path_first_node(path), + coresight_path_last_node(path)); +} + struct coresight_device *coresight_get_sink(struct coresight_path *path) { struct coresight_device *csdev;
Control links and helpers on an active path during CPU idle.
The helper coresight_get_local_cpu_path() uses coresight_dev_lock to exclusive access to the per-CPU source pointer, then retrieves the path pointer (csdev->path). The path pointer is only set via an SMP call on the local CPU, which is serialized with CPU PM notifiers.
If the CPU PM notifier retrieves a non-NULL path pointer, it is safe to iterate over the path and control it up to the node before the sink to avoid latency.
Since coresight_disable_path() does not handle a source device's helpers, explicitly disable them alongside the source device.
Signed-off-by: Leo Yan leo.yan@arm.com --- drivers/hwtracing/coresight/coresight-core.c | 72 ++++++++++++++++++++++++---- 1 file changed, 63 insertions(+), 9 deletions(-)
diff --git a/drivers/hwtracing/coresight/coresight-core.c b/drivers/hwtracing/coresight/coresight-core.c index 874bd87da2270fdbe6b32c2fa29292d7a26b8dc7..b888a31ca6abffe62e2264574a6d61b43184f600 100644 --- a/drivers/hwtracing/coresight/coresight-core.c +++ b/drivers/hwtracing/coresight/coresight-core.c @@ -113,6 +113,29 @@ struct coresight_device *coresight_get_percpu_source(int cpu) return per_cpu(csdev_source, cpu); }
+/* + * The source device's csdev->path is always set on the local CPU. + * CPU PM notifiers also run on the same CPU, so accesses to + * csdev->path are serialized. + * + * When the path is constructed, each component on the path takes module + * reference counts, and the source's csdev->path is set last. Therefore, + * when the CPU PM notifier retrieves the path on the local CPU, a non-NULL + * csdev->path guarantees that the path is safe to iterate over. + */ +static struct coresight_path *coresight_get_local_cpu_path(void) +{ + struct coresight_device *csdev; + + guard(raw_spinlock_irqsave)(&coresight_dev_lock); + + csdev = this_cpu_read(csdev_source); + if (!csdev) + return NULL; + + return csdev->path; +} + static struct coresight_device *coresight_get_source(struct coresight_path *path) { struct coresight_device *csdev; @@ -1765,8 +1788,14 @@ static void coresight_release_device_list(void) } }
-static bool coresight_pm_is_needed(struct coresight_device *csdev) +static bool coresight_pm_is_needed(struct coresight_path *path) { + struct coresight_device *csdev; + + if (!path) + return false; + + csdev = coresight_get_source(path); if (!csdev) return false;
@@ -1792,33 +1821,58 @@ static void coresight_pm_device_restore(struct coresight_device *csdev) coresight_ops(csdev)->pm_restore_enable(csdev); }
-static int coresight_pm_save(struct coresight_device *source) +static int coresight_pm_save(struct coresight_path *path) { - return coresight_pm_device_save(source); + struct coresight_device *source; + struct coresight_node *from, *to; + int ret; + + source = coresight_get_source(path); + ret = coresight_pm_device_save(source); + if (ret) + return ret; + + coresight_disable_helpers(source, path); + + from = coresight_path_first_node(path); + /* Up to the node before sink to avoid latency */ + to = list_prev_entry(coresight_path_last_node(path), link); + coresight_disable_path_from_to(path, from, to); + + return 0; }
-static void coresight_pm_restore(struct coresight_device *source) +static void coresight_pm_restore(struct coresight_path *path) { + struct coresight_device *source; + struct coresight_node *from, *to; + + source = coresight_get_source(path); + from = coresight_path_first_node(path); + /* Up to the node before sink to avoid latency */ + to = list_prev_entry(coresight_path_last_node(path), link); + coresight_enable_path_from_to(path, coresight_get_mode(source), + from, to); + coresight_pm_device_restore(source); }
static int coresight_cpu_pm_notify(struct notifier_block *nb, unsigned long cmd, void *v) { - struct coresight_device *source = - coresight_get_percpu_source(smp_processor_id()); + struct coresight_path *path = coresight_get_local_cpu_path();
- if (!coresight_pm_is_needed(source)) + if (!coresight_pm_is_needed(path)) return NOTIFY_OK;
switch (cmd) { case CPU_PM_ENTER: - if (coresight_pm_save(source)) + if (coresight_pm_save(path)) return NOTIFY_BAD; break; case CPU_PM_EXIT: case CPU_PM_ENTER_FAILED: - coresight_pm_restore(source); + coresight_pm_restore(path); break; default: return NOTIFY_DONE;
Unlike a system level's sink, the per-CPU sink may lose power during CPU idle states. Currently, this refers specifically to TRBE as the sink.
This commit registers save and restore callbacks for the sink via the PM notifier.
Tested-by: James Clark james.clark@linaro.org Reviewed-by: Yeoreum Yun yeoreum.yun@arm.com Reviewed-by: James Clark james.clark@linaro.org Signed-off-by: Leo Yan leo.yan@arm.com --- drivers/hwtracing/coresight/coresight-core.c | 18 ++++++++++++++++++ 1 file changed, 18 insertions(+)
diff --git a/drivers/hwtracing/coresight/coresight-core.c b/drivers/hwtracing/coresight/coresight-core.c index b888a31ca6abffe62e2264574a6d61b43184f600..e0784e8eecedae5a5121c03d6bb29f6a3ab5026f 100644 --- a/drivers/hwtracing/coresight/coresight-core.c +++ b/drivers/hwtracing/coresight/coresight-core.c @@ -1813,11 +1813,17 @@ static bool coresight_pm_is_needed(struct coresight_path *path)
static int coresight_pm_device_save(struct coresight_device *csdev) { + if (!csdev || !coresight_ops(csdev)->pm_save_disable) + return 0; + return coresight_ops(csdev)->pm_save_disable(csdev); }
static void coresight_pm_device_restore(struct coresight_device *csdev) { + if (!csdev || !coresight_ops(csdev)->pm_restore_enable) + return; + coresight_ops(csdev)->pm_restore_enable(csdev); }
@@ -1839,7 +1845,17 @@ static int coresight_pm_save(struct coresight_path *path) to = list_prev_entry(coresight_path_last_node(path), link); coresight_disable_path_from_to(path, from, to);
+ ret = coresight_pm_device_save(coresight_get_sink(path)); + if (ret) + goto failed_out; + return 0; + +failed_out: + coresight_enable_path_from_to(path, coresight_get_mode(source), + from, to); + coresight_pm_device_restore(source); + return ret; }
static void coresight_pm_restore(struct coresight_path *path) @@ -1847,6 +1863,8 @@ static void coresight_pm_restore(struct coresight_path *path) struct coresight_device *source; struct coresight_node *from, *to;
+ coresight_pm_device_restore(coresight_get_sink(path)); + source = coresight_get_source(path); from = coresight_path_first_node(path); /* Up to the node before sink to avoid latency */
From: Yabin Cui yabinc@google.com
Similar to ETE, TRBE may lose its context when a CPU enters low power state. To make things worse, if ETE is restored without TRBE being restored, an enabled source device with no enabled sink devices can cause CPU hang on some devices (e.g., Pixel 9).
The save and restore flows are described in the section K5.5 "Context switching" of Arm ARM (ARM DDI 0487 L.a). This commit adds save and restore callbacks with following the software usages defined in the architecture manual.
Given that some bit fields may be reset to an unknown value after a CPU power cycle, this commit always save and restore all register contexts.
Signed-off-by: Yabin Cui yabinc@google.com Tested-by: James Clark james.clark@linaro.org Reviewed-by: Yeoreum Yun yeoreum.yun@arm.com Reviewed-by: James Clark james.clark@linaro.org Co-developed-by: Leo Yan leo.yan@arm.com Signed-off-by: Leo Yan leo.yan@arm.com --- drivers/hwtracing/coresight/coresight-trbe.c | 59 +++++++++++++++++++++++++++- 1 file changed, 58 insertions(+), 1 deletion(-)
diff --git a/drivers/hwtracing/coresight/coresight-trbe.c b/drivers/hwtracing/coresight/coresight-trbe.c index 14e35b9660d76e47619cc6026b94929b3bb3e02b..c7cbca45f2debd4047b93283ea9fe5dd9e1f2ebf 100644 --- a/drivers/hwtracing/coresight/coresight-trbe.c +++ b/drivers/hwtracing/coresight/coresight-trbe.c @@ -116,6 +116,20 @@ static int trbe_errata_cpucaps[] = { */ #define TRBE_WORKAROUND_OVERWRITE_FILL_MODE_SKIP_BYTES 256
+/* + * struct trbe_save_state: Register values representing TRBE state + * @trblimitr - Trace Buffer Limit Address Register value + * @trbbaser - Trace Buffer Base Register value + * @trbptr - Trace Buffer Write Pointer Register value + * @trbsr - Trace Buffer Status Register value + */ +struct trbe_save_state { + u64 trblimitr; + u64 trbbaser; + u64 trbptr; + u64 trbsr; +}; + /* * struct trbe_cpudata: TRBE instance specific data * @trbe_flag - TRBE dirty/access flag support @@ -134,6 +148,7 @@ struct trbe_cpudata { enum cs_mode mode; struct trbe_buf *buf; struct trbe_drvdata *drvdata; + struct trbe_save_state save_state; DECLARE_BITMAP(errata, TRBE_ERRATA_MAX); };
@@ -1189,6 +1204,46 @@ static irqreturn_t arm_trbe_irq_handler(int irq, void *dev) return IRQ_HANDLED; }
+static int arm_trbe_save(struct coresight_device *csdev) +{ + struct trbe_cpudata *cpudata = dev_get_drvdata(&csdev->dev); + struct trbe_save_state *state = &cpudata->save_state; + + state->trblimitr = read_sysreg_s(SYS_TRBLIMITR_EL1); + + /* Disable the unit, ensure the writes to memory are complete */ + if (state->trblimitr & TRBLIMITR_EL1_E) + trbe_drain_and_disable_local(cpudata); + + state->trbbaser = read_sysreg_s(SYS_TRBBASER_EL1); + state->trbptr = read_sysreg_s(SYS_TRBPTR_EL1); + state->trbsr = read_sysreg_s(SYS_TRBSR_EL1); + return 0; +} + +static void arm_trbe_restore(struct coresight_device *csdev) +{ + struct trbe_cpudata *cpudata = dev_get_drvdata(&csdev->dev); + struct trbe_save_state *state = &cpudata->save_state; + + write_sysreg_s(state->trbbaser, SYS_TRBBASER_EL1); + write_sysreg_s(state->trbptr, SYS_TRBPTR_EL1); + write_sysreg_s(state->trbsr, SYS_TRBSR_EL1); + + if (!(state->trblimitr & TRBLIMITR_EL1_E)) { + write_sysreg_s(state->trblimitr, SYS_TRBLIMITR_EL1); + } else { + /* + * The section K5.5 Context switching, Arm ARM (ARM DDI 0487 + * L.a), S_PKLXF requires a Context synchronization event to + * guarantee the Trace Buffer Unit will observe the new values + * of the system registers. + */ + isb(); + set_trbe_enabled(cpudata, state->trblimitr); + } +} + static const struct coresight_ops_sink arm_trbe_sink_ops = { .enable = arm_trbe_enable, .disable = arm_trbe_disable, @@ -1198,7 +1253,9 @@ static const struct coresight_ops_sink arm_trbe_sink_ops = { };
static const struct coresight_ops arm_trbe_cs_ops = { - .sink_ops = &arm_trbe_sink_ops, + .pm_save_disable = arm_trbe_save, + .pm_restore_enable = arm_trbe_restore, + .sink_ops = &arm_trbe_sink_ops, };
static ssize_t align_show(struct device *dev, struct device_attribute *attr, char *buf)
Except for system tracers (e.g. STM), other sources treat multiple enables as equivalent to a single enable. The device mode already tracks the binary state, so it is redundant to operate refcount.
Refactor to maintain the refcount only for system sources. This simplifies future CPU PM handling without refcount logic.
Tested-by: James Clark james.clark@linaro.org Reviewed-by: Yeoreum Yun yeoreum.yun@arm.com Reviewed-by: James Clark james.clark@linaro.org Signed-off-by: Leo Yan leo.yan@arm.com --- drivers/hwtracing/coresight/coresight-sysfs.c | 35 +++++++++++++++------------ 1 file changed, 19 insertions(+), 16 deletions(-)
diff --git a/drivers/hwtracing/coresight/coresight-sysfs.c b/drivers/hwtracing/coresight/coresight-sysfs.c index 9e94505bccf4ac91207f2f5393ae57ae077d61d6..04c1425eb37d69c6feb8e49446fc58ee57a5e82e 100644 --- a/drivers/hwtracing/coresight/coresight-sysfs.c +++ b/drivers/hwtracing/coresight/coresight-sysfs.c @@ -58,7 +58,17 @@ static int coresight_enable_source_sysfs(struct coresight_device *csdev, return ret; }
- csdev->refcnt++; + /* + * There could be multiple applications driving the software + * source. So keep the refcount for each such user when the + * source is already enabled. + * + * No need to increment the reference counter for other source + * types, as multiple enables are the same as a single enable. + */ + if (csdev->subtype.source_subtype == + CORESIGHT_DEV_SUBTYPE_SOURCE_SOFTWARE) + csdev->refcnt++;
return 0; } @@ -80,7 +90,10 @@ static bool coresight_disable_source_sysfs(struct coresight_device *csdev, if (coresight_get_mode(csdev) != CS_MODE_SYSFS) return false;
- csdev->refcnt--; + if (csdev->subtype.source_subtype == + CORESIGHT_DEV_SUBTYPE_SOURCE_SOFTWARE) + csdev->refcnt--; + if (csdev->refcnt == 0) { coresight_disable_source(csdev, data); return true; @@ -159,10 +172,7 @@ int coresight_enable_sysfs(struct coresight_device *csdev) { int ret = 0; struct coresight_device *sink; - struct coresight_path *path; - enum coresight_dev_subtype_source subtype; - - subtype = csdev->subtype.source_subtype; + struct coresight_path *path = NULL;
mutex_lock(&coresight_mutex);
@@ -176,16 +186,8 @@ int coresight_enable_sysfs(struct coresight_device *csdev) * coresight_enable_source() so can still race with Perf mode which * doesn't hold coresight_mutex. */ - if (coresight_get_mode(csdev) == CS_MODE_SYSFS) { - /* - * There could be multiple applications driving the software - * source. So keep the refcount for each such user when the - * source is already enabled. - */ - if (subtype == CORESIGHT_DEV_SUBTYPE_SOURCE_SOFTWARE) - csdev->refcnt++; - goto out; - } + if (coresight_get_mode(csdev) == CS_MODE_SYSFS) + goto enable_source;
sink = coresight_find_activated_sysfs_sink(csdev); if (!sink) { @@ -208,6 +210,7 @@ int coresight_enable_sysfs(struct coresight_device *csdev) if (ret) goto err_path;
+enable_source: ret = coresight_enable_source_sysfs(csdev, CS_MODE_SYSFS, path); if (ret) goto err_source;
The hotplug lock is acquired and released in etm{3|4}_disable_sysfs(), which are low-level functions. This prevents us from a new solution for hotplug.
Firstly, hotplug callbacks cannot invoke etm{3|4}_disable_sysfs() to disable the source; otherwise, a deadlock issue occurs. The reason is that, in the hotplug flow, the kernel acquires the hotplug lock before calling callbacks. Subsequently, if coresight_disable_source() is invoked and it calls etm{3|4}_disable_sysfs(), the hotplug lock will be acquired twice, leading to a double lock issue.
Secondly, when hotplugging a CPU on or off, if we want to manipulate all components on a path attached to the CPU, we need to maintain atomicity for the entire path. Otherwise, a race condition may occur with users setting the same path via the Sysfs knobs, ultimately causing mess states in CoreSight components.
This patch moves the hotplug locking from etm{3|4}_disable_sysfs() into enable_source_store(). As a result, when users control the Sysfs knobs, the whole flow is protected by hotplug locking, ensuring it is mutual exclusive with hotplug callbacks.
Note, the paired function etm{3|4}_enable_sysfs() does not use hotplug locking, which is why this patch does not modify it.
Reviewed-by: Mike Leach mike.leach@linaro.org Tested-by: James Clark james.clark@linaro.org Reviewed-by: Yeoreum Yun yeoreum.yun@arm.com Reviewed-by: James Clark james.clark@linaro.org Signed-off-by: Leo Yan leo.yan@arm.com --- drivers/hwtracing/coresight/coresight-etm3x-core.c | 8 -------- drivers/hwtracing/coresight/coresight-etm4x-core.c | 9 --------- drivers/hwtracing/coresight/coresight-sysfs.c | 7 +++++++ 3 files changed, 7 insertions(+), 17 deletions(-)
diff --git a/drivers/hwtracing/coresight/coresight-etm3x-core.c b/drivers/hwtracing/coresight/coresight-etm3x-core.c index c2b8c096cbe9e9759b3d17f019a2d73f59234ac9..c6fe8b25b855a4119110fee4162f55c0154c3d05 100644 --- a/drivers/hwtracing/coresight/coresight-etm3x-core.c +++ b/drivers/hwtracing/coresight/coresight-etm3x-core.c @@ -631,13 +631,6 @@ static void etm_disable_sysfs(struct coresight_device *csdev) { struct etm_drvdata *drvdata = dev_get_drvdata(csdev->dev.parent);
- /* - * Taking hotplug lock here protects from clocks getting disabled - * with tracing being left on (crash scenario) if user disable occurs - * after cpu online mask indicates the cpu is offline but before the - * DYING hotplug callback is serviced by the ETM driver. - */ - cpus_read_lock(); spin_lock(&drvdata->spinlock);
/* @@ -648,7 +641,6 @@ static void etm_disable_sysfs(struct coresight_device *csdev) drvdata, 1);
spin_unlock(&drvdata->spinlock); - cpus_read_unlock();
/* * we only release trace IDs when resetting sysfs. diff --git a/drivers/hwtracing/coresight/coresight-etm4x-core.c b/drivers/hwtracing/coresight/coresight-etm4x-core.c index 1f100e8bbcc775a1bdc5f8b27ff15dbcaf030a63..2870c162fdc1c119f05ee5d92c1f6c71aa1cfb5f 100644 --- a/drivers/hwtracing/coresight/coresight-etm4x-core.c +++ b/drivers/hwtracing/coresight/coresight-etm4x-core.c @@ -1118,13 +1118,6 @@ static void etm4_disable_sysfs(struct coresight_device *csdev) { struct etmv4_drvdata *drvdata = dev_get_drvdata(csdev->dev.parent);
- /* - * Taking hotplug lock here protects from clocks getting disabled - * with tracing being left on (crash scenario) if user disable occurs - * after cpu online mask indicates the cpu is offline but before the - * DYING hotplug callback is serviced by the ETM driver. - */ - cpus_read_lock(); raw_spin_lock(&drvdata->spinlock);
/* @@ -1138,8 +1131,6 @@ static void etm4_disable_sysfs(struct coresight_device *csdev)
cscfg_csdev_disable_active_config(csdev);
- cpus_read_unlock(); - /* * we only release trace IDs when resetting sysfs. * This permits sysfs users to read the trace ID after the trace diff --git a/drivers/hwtracing/coresight/coresight-sysfs.c b/drivers/hwtracing/coresight/coresight-sysfs.c index 04c1425eb37d69c6feb8e49446fc58ee57a5e82e..c8d45f69072546bc5ae61e22646000cd47530dba 100644 --- a/drivers/hwtracing/coresight/coresight-sysfs.c +++ b/drivers/hwtracing/coresight/coresight-sysfs.c @@ -305,6 +305,13 @@ static ssize_t enable_source_store(struct device *dev, if (ret) return ret;
+ /* + * CoreSight hotplug callbacks in core layer control a activated path + * from its source to sink. Taking hotplug lock here protects a race + * condition with hotplug callbacks. + */ + guard(cpus_read_lock)(); + if (val) { ret = coresight_enable_sysfs(csdev); if (ret)
This commit moves CPU hotplug callbacks from ETMv4 driver to core layer. The motivation is the core layer can control all components on an activated path rather but not only managing tracer in ETMv4 driver.
The perf event layer will disable CoreSight PMU event 'cs_etm' when hotplug off a CPU. That means a perf mode will be always converted to disabled mode in CPU hotplug. Arm CoreSight CPU hotplug callbacks only need to handle the Sysfs mode and ignore the perf mode.
The core layer disables and releases the active path when hotplug-off a CPU. When hotplug-in a CPU, it does not re-enable the path, in this case, users need to re-enable the trace via Sysfs interface.
Tested-by: James Clark james.clark@linaro.org Reviewed-by: Yeoreum Yun yeoreum.yun@arm.com Reviewed-by: James Clark james.clark@linaro.org Signed-off-by: Leo Yan leo.yan@arm.com --- drivers/hwtracing/coresight/coresight-core.c | 36 ++++++++++++++++++- drivers/hwtracing/coresight/coresight-etm3x-core.c | 40 ---------------------- drivers/hwtracing/coresight/coresight-etm4x-core.c | 37 -------------------- 3 files changed, 35 insertions(+), 78 deletions(-)
diff --git a/drivers/hwtracing/coresight/coresight-core.c b/drivers/hwtracing/coresight/coresight-core.c index e0784e8eecedae5a5121c03d6bb29f6a3ab5026f..fd982c4eafc62b10c01b70e0ec3efe4a4add0afe 100644 --- a/drivers/hwtracing/coresight/coresight-core.c +++ b/drivers/hwtracing/coresight/coresight-core.c @@ -1903,13 +1903,47 @@ static struct notifier_block coresight_cpu_pm_nb = { .notifier_call = coresight_cpu_pm_notify, };
+static int coresight_dying_cpu(unsigned int cpu) +{ + struct coresight_path *path = coresight_get_local_cpu_path(); + struct coresight_device *source = coresight_get_source(path); + + if (!path || !source) + return 0; + + /* + * The perf event layer will disable PMU events in the CPU hotplug. + * CoreSight driver should never handle the CS_MODE_PERF case. + */ + if (coresight_get_mode(source) != CS_MODE_SYSFS) + return 0; + + coresight_disable_source(source, NULL); + coresight_disable_path(path); + coresight_release_path(path); + return 0; +} + static int __init coresight_pm_setup(void) { - return cpu_pm_register_notifier(&coresight_cpu_pm_nb); + int ret; + + ret = cpu_pm_register_notifier(&coresight_cpu_pm_nb); + if (ret) + return ret; + + ret = cpuhp_setup_state_nocalls(CPUHP_AP_ARM_CORESIGHT_STARTING, + "arm/coresight-core:starting", + NULL, coresight_dying_cpu); + if (ret) + cpu_pm_unregister_notifier(&coresight_cpu_pm_nb); + + return ret; }
static void coresight_pm_cleanup(void) { + cpuhp_remove_state_nocalls(CPUHP_AP_ARM_CORESIGHT_STARTING); cpu_pm_unregister_notifier(&coresight_cpu_pm_nb); }
diff --git a/drivers/hwtracing/coresight/coresight-etm3x-core.c b/drivers/hwtracing/coresight/coresight-etm3x-core.c index c6fe8b25b855a4119110fee4162f55c0154c3d05..862ad0786699c41433eae8683406b3c1340a6cb6 100644 --- a/drivers/hwtracing/coresight/coresight-etm3x-core.c +++ b/drivers/hwtracing/coresight/coresight-etm3x-core.c @@ -699,35 +699,6 @@ static int etm_online_cpu(unsigned int cpu) return 0; }
-static int etm_starting_cpu(unsigned int cpu) -{ - if (!etmdrvdata[cpu]) - return 0; - - spin_lock(&etmdrvdata[cpu]->spinlock); - if (!etmdrvdata[cpu]->os_unlock) { - etm_os_unlock(etmdrvdata[cpu]); - etmdrvdata[cpu]->os_unlock = true; - } - - if (coresight_get_mode(etmdrvdata[cpu]->csdev)) - etm_enable_hw(etmdrvdata[cpu]); - spin_unlock(&etmdrvdata[cpu]->spinlock); - return 0; -} - -static int etm_dying_cpu(unsigned int cpu) -{ - if (!etmdrvdata[cpu]) - return 0; - - spin_lock(&etmdrvdata[cpu]->spinlock); - if (coresight_get_mode(etmdrvdata[cpu]->csdev)) - etm_disable_hw(etmdrvdata[cpu]); - spin_unlock(&etmdrvdata[cpu]->spinlock); - return 0; -} - static bool etm_arch_supported(u8 arch) { switch (arch) { @@ -795,13 +766,6 @@ static int __init etm_hp_setup(void) { int ret;
- ret = cpuhp_setup_state_nocalls(CPUHP_AP_ARM_CORESIGHT_STARTING, - "arm/coresight:starting", - etm_starting_cpu, etm_dying_cpu); - - if (ret) - return ret; - ret = cpuhp_setup_state_nocalls(CPUHP_AP_ONLINE_DYN, "arm/coresight:online", etm_online_cpu, NULL); @@ -812,15 +776,11 @@ static int __init etm_hp_setup(void) return 0; }
- /* failed dyn state - remove others */ - cpuhp_remove_state_nocalls(CPUHP_AP_ARM_CORESIGHT_STARTING); - return ret; }
static void etm_hp_clear(void) { - cpuhp_remove_state_nocalls(CPUHP_AP_ARM_CORESIGHT_STARTING); if (hp_online) { cpuhp_remove_state_nocalls(hp_online); hp_online = 0; diff --git a/drivers/hwtracing/coresight/coresight-etm4x-core.c b/drivers/hwtracing/coresight/coresight-etm4x-core.c index 2870c162fdc1c119f05ee5d92c1f6c71aa1cfb5f..1d7bfa21d7c205ae1b903bd62bc6d675d0fa949d 100644 --- a/drivers/hwtracing/coresight/coresight-etm4x-core.c +++ b/drivers/hwtracing/coresight/coresight-etm4x-core.c @@ -1834,33 +1834,6 @@ static int etm4_online_cpu(unsigned int cpu) return 0; }
-static int etm4_starting_cpu(unsigned int cpu) -{ - if (!etmdrvdata[cpu]) - return 0; - - raw_spin_lock(&etmdrvdata[cpu]->spinlock); - if (!etmdrvdata[cpu]->os_unlock) - etm4_os_unlock(etmdrvdata[cpu]); - - if (coresight_get_mode(etmdrvdata[cpu]->csdev)) - etm4_enable_hw(etmdrvdata[cpu]); - raw_spin_unlock(&etmdrvdata[cpu]->spinlock); - return 0; -} - -static int etm4_dying_cpu(unsigned int cpu) -{ - if (!etmdrvdata[cpu]) - return 0; - - raw_spin_lock(&etmdrvdata[cpu]->spinlock); - if (coresight_get_mode(etmdrvdata[cpu]->csdev)) - etm4_disable_hw(etmdrvdata[cpu]); - raw_spin_unlock(&etmdrvdata[cpu]->spinlock); - return 0; -} - static int etm4_cpu_save(struct coresight_device *csdev) { int i, ret = 0; @@ -2110,13 +2083,6 @@ static int __init etm4_pm_setup(void) { int ret;
- ret = cpuhp_setup_state_nocalls(CPUHP_AP_ARM_CORESIGHT_STARTING, - "arm/coresight4:starting", - etm4_starting_cpu, etm4_dying_cpu); - - if (ret) - return ret; - ret = cpuhp_setup_state_nocalls(CPUHP_AP_ONLINE_DYN, "arm/coresight4:online", etm4_online_cpu, NULL); @@ -2127,14 +2093,11 @@ static int __init etm4_pm_setup(void) return 0; }
- /* failed dyn state - remove others */ - cpuhp_remove_state_nocalls(CPUHP_AP_ARM_CORESIGHT_STARTING); return ret; }
static void etm4_pm_clear(void) { - cpuhp_remove_state_nocalls(CPUHP_AP_ARM_CORESIGHT_STARTING); if (hp_online) { cpuhp_remove_state_nocalls(hp_online); hp_online = 0;
-
Leo Yan