CoreSight

coresight@lists.linaro.org

7 participants
2453 discussions

Re: [PATCH] coresight-etm4x: add isb() before reading the TRCSTATR

by Suzuki K Poulose

Hi On 04/12/2024 10:13, yuanfang zhang wrote: > From: Yuanfang Zhang <quic_yuanfang(a)quicinc.com> > > As recommended by section 4.3.7 ("Synchronization when using system > instructions to progrom the trace unit") of ARM IHI 0064H.b, the > self-hosted trace analyzer must perform a Context synchronization > event between writing to the TRCPRGCTLR and reading the TRCSTATR. > Thanks for the patch. Please could you add the above in a comment in the changes below ? Also, add Fixes tag ? (It may go all the way back to initial support) Otherwise looks sensible to me. Suzuki > Signed-off-by: Yuanfang Zhang <quic_yuanfang(a)quicinc.com> > --- > drivers/hwtracing/coresight/coresight-etm4x-core.c | 1 + > 1 file changed, 1 insertion(+) > > diff --git a/drivers/hwtracing/coresight/coresight-etm4x-core.c b/drivers/hwtracing/coresight/coresight-etm4x-core.c > index 66d44a404ad0..5da2c523c30a 100644 > --- a/drivers/hwtracing/coresight/coresight-etm4x-core.c > +++ b/drivers/hwtracing/coresight/coresight-etm4x-core.c > @@ -906,6 +906,7 @@ static void etm4_disable_hw(void *info) > tsb_csync(); > etm4x_relaxed_write32(csa, control, TRCPRGCTLR); > > + isb(); > /* wait for TRCSTATR.PMSTABLE to go to '1' */ > if (coresight_timeout(csa, TRCSTATR, TRCSTATR_PMSTABLE_BIT, 1)) > dev_err(etm_dev,

8 months, 3 weeks

Re: [PATCH 3/4] coresight: tmc-etr: Fix race condition between sysfs and perf mode

by Suzuki K Poulose

Hi Yicong On 02/12/2024 09:24, Yicong Yang wrote: > From: Yicong Yang <yangyicong(a)hisilicon.com> > > When trying to run perf and sysfs mode simultaneously, the WARN_ON() > in tmc_etr_enable_hw() is triggered sometimes: > > WARNING: CPU: 42 PID: 3911571 at drivers/hwtracing/coresight/coresight-tmc-etr.c:1060 tmc_etr_enable_hw+0xc0/0xd8 [coresight_tmc] > [..snip..] > Call trace: > tmc_etr_enable_hw+0xc0/0xd8 [coresight_tmc] (P) > tmc_enable_etr_sink+0x11c/0x250 [coresight_tmc] (L) > tmc_enable_etr_sink+0x11c/0x250 [coresight_tmc] > coresight_enable_path+0x1c8/0x218 [coresight] > coresight_enable_sysfs+0xa4/0x228 [coresight] > enable_source_store+0x58/0xa8 [coresight] > dev_attr_store+0x20/0x40 > sysfs_kf_write+0x4c/0x68 > kernfs_fop_write_iter+0x120/0x1b8 > vfs_write+0x2c8/0x388 > ksys_write+0x74/0x108 > __arm64_sys_write+0x24/0x38 > el0_svc_common.constprop.0+0x64/0x148 > do_el0_svc+0x24/0x38 > el0_svc+0x3c/0x130 > el0t_64_sync_handler+0xc8/0xd0 > el0t_64_sync+0x1ac/0x1b0 > ---[ end trace 0000000000000000 ]--- > > Since the enablement of sysfs mode is separeted into two critical regions, > one for sysfs buffer allocation and another for hardware enablement, it's > possible to race with the perf mode. Fix this by double check whether > the perf mode's been used before enabling the hardware in sysfs mode. Thanks for the fix. Some minor comments below. It needs a Fixes tag. > > Signed-off-by: Yicong Yang <yangyicong(a)hisilicon.com> > --- > .../hwtracing/coresight/coresight-tmc-etr.c | 30 +++++++++++++++++++ > 1 file changed, 30 insertions(+) > > diff --git a/drivers/hwtracing/coresight/coresight-tmc-etr.c b/drivers/hwtracing/coresight/coresight-tmc-etr.c > index ad83714ca4dc..d382d95da5ff 100644 > --- a/drivers/hwtracing/coresight/coresight-tmc-etr.c > +++ b/drivers/hwtracing/coresight/coresight-tmc-etr.c > @@ -1230,6 +1230,36 @@ static int tmc_enable_etr_sink_sysfs(struct coresight_device *csdev) > > spin_lock_irqsave(&drvdata->spinlock, flags); > > + /* > + * Since the sysfs buffer allocation and the hardware enablement is not > + * in the same critical region, it's possible to race with the perf > + * mode: > + * [sysfs mode] [perf mode] > + * tmc_etr_get_sysfs_buffer() > + * spin_lock(&drvdata->spinlock) > + * [sysfs buffer allocation] > + * spin_unlock(&drvdata->spinlock) > + * spin_lock(&drvdata->spinlock) > + * tmc_etr_enable_hw() > + * drvdata->etr_buf = etr_perf->etr_buf > + * spin_unlock(&drvdata->spinlock) > + * spin_lock(&drvdata->spinlock) > + * tmc_etr_enable_hw() > + * WARN_ON(drvdata->etr_buf) // WARN sicne etr_buf initialized at > + * the perf side > + * spin_unlock(&drvdata->spinlock) > + * > + * So check here before continue. > + */ > + if (coresight_get_mode(csdev) == CS_MODE_PERF) { > + drvdata->sysfs_buf = NULL; > + spin_unlock_irqrestore(&drvdata->spinlock, flags); > + > + /* Free allocated memory out side of the spinlock */ > + tmc_etr_free_sysfs_buf(sysfs_buf); > + return -EBUSY; > + } With this in place, I think we should remove the check for CS_MODE_PERF in get_etr_sysfs_buf() to avoid confusion (which I believe opened up this race) Suzuki > + > /* > * In sysFS mode we can have multiple writers per sink. Since this > * sink is already enabled no memory is needed and the HW need not be

8 months, 4 weeks

Re: [PATCH 0/9] coresight: change some driver' spinlock type to raw_spinlock_t

by James Clark

On 25/11/2024 9:48 am, Yeoreum Yun wrote: > In some coresight drivers, drvdata->spinlock can be held during __schedule() > by perf_event_task_sched_out()/in(). > > Since drvdata->spinlock type is spinlock_t and > perf_event_task_sched_out()/in() is called after acquiring rq_lock, > which is raw_spinlock_t (an unsleepable lock), > this poses an issue in PREEMPT_RT kernel where spinlock_t is sleepable. > > To address this,change type drvdata->spinlock in some coresight drivers, > which can be called by perf_event_task_sched_out()/in(), > from spinlock_t to raw_spinlock_t. > > Levi Yun (9): > coresight: change coresight_device lock type to raw_spinlock_t > coresight-etm4x: change etmv4_drvdata spinlock type to raw_spinlock_t > coresight: change coresight_trace_id_map's lock type to raw_spinlock_t > coresight-cti: change cti_drvdata spinlock's type to raw_spinlock_t > coresight-etb10: change etb_drvdata spinlock's type to raw_spinlock_t > coresight-funnel: change funnel_drvdata spinlock's type to > raw_spinlock_t > coresight-replicator: change replicator_drvdata spinlock's type to > raw_spinlock_t > coresight-tmc: change tmc_drvdata spinlock's type to raw_spinlock_t > coresight/ultrasoc: change cti_drvdata spinlock's type to > raw_spinlock_t > > .../hwtracing/coresight/coresight-config.c | 21 +- > .../hwtracing/coresight/coresight-config.h | 2 +- > drivers/hwtracing/coresight/coresight-core.c | 2 +- > .../hwtracing/coresight/coresight-cti-core.c | 65 +- > .../hwtracing/coresight/coresight-cti-sysfs.c | 135 +++-- > drivers/hwtracing/coresight/coresight-cti.h | 2 +- > drivers/hwtracing/coresight/coresight-etb10.c | 62 +- > .../coresight/coresight-etm4x-core.c | 71 ++- > .../coresight/coresight-etm4x-sysfs.c | 566 +++++++++--------- > drivers/hwtracing/coresight/coresight-etm4x.h | 2 +- > .../hwtracing/coresight/coresight-funnel.c | 34 +- > .../coresight/coresight-replicator.c | 36 +- > .../hwtracing/coresight/coresight-syscfg.c | 75 ++- > .../hwtracing/coresight/coresight-tmc-core.c | 9 +- > .../hwtracing/coresight/coresight-tmc-etf.c | 195 +++--- > .../hwtracing/coresight/coresight-tmc-etr.c | 199 +++--- > drivers/hwtracing/coresight/coresight-tmc.h | 2 +- > .../hwtracing/coresight/coresight-trace-id.c | 93 ++- > drivers/hwtracing/coresight/ultrasoc-smb.c | 12 +- > drivers/hwtracing/coresight/ultrasoc-smb.h | 2 +- > include/linux/coresight.h | 4 +- > 21 files changed, 751 insertions(+), 838 deletions(-) > > -- > LEVI:{C3F47F37-75D8-414A-A8BA-3980EC8A46D7} > Can TPIU be used as a sink with Perf? If so that one is missing the update. Makes me wonder if something like a "perf_component" flag in coresight_device or a #define would make sense. Then the lock type gets auto picked. Seems like in a few years we will have completely forgotten why some of the coresight locks are raw and others aren't.

9 months

Re: [PATCH 0/9] coresight: change some driver' spinlock type to raw_spinlock_t

by James Clark

On 25/11/2024 9:48 am, Yeoreum Yun wrote: > In some coresight drivers, drvdata->spinlock can be held during __schedule() > by perf_event_task_sched_out()/in(). > > Since drvdata->spinlock type is spinlock_t and > perf_event_task_sched_out()/in() is called after acquiring rq_lock, > which is raw_spinlock_t (an unsleepable lock), > this poses an issue in PREEMPT_RT kernel where spinlock_t is sleepable. > > To address this,change type drvdata->spinlock in some coresight drivers, > which can be called by perf_event_task_sched_out()/in(), > from spinlock_t to raw_spinlock_t. > > Levi Yun (9): > coresight: change coresight_device lock type to raw_spinlock_t > coresight-etm4x: change etmv4_drvdata spinlock type to raw_spinlock_t > coresight: change coresight_trace_id_map's lock type to raw_spinlock_t > coresight-cti: change cti_drvdata spinlock's type to raw_spinlock_t > coresight-etb10: change etb_drvdata spinlock's type to raw_spinlock_t > coresight-funnel: change funnel_drvdata spinlock's type to > raw_spinlock_t > coresight-replicator: change replicator_drvdata spinlock's type to > raw_spinlock_t > coresight-tmc: change tmc_drvdata spinlock's type to raw_spinlock_t > coresight/ultrasoc: change cti_drvdata spinlock's type to > raw_spinlock_t > > .../hwtracing/coresight/coresight-config.c | 21 +- > .../hwtracing/coresight/coresight-config.h | 2 +- > drivers/hwtracing/coresight/coresight-core.c | 2 +- > .../hwtracing/coresight/coresight-cti-core.c | 65 +- > .../hwtracing/coresight/coresight-cti-sysfs.c | 135 +++-- > drivers/hwtracing/coresight/coresight-cti.h | 2 +- > drivers/hwtracing/coresight/coresight-etb10.c | 62 +- > .../coresight/coresight-etm4x-core.c | 71 ++- > .../coresight/coresight-etm4x-sysfs.c | 566 +++++++++--------- > drivers/hwtracing/coresight/coresight-etm4x.h | 2 +- > .../hwtracing/coresight/coresight-funnel.c | 34 +- > .../coresight/coresight-replicator.c | 36 +- > .../hwtracing/coresight/coresight-syscfg.c | 75 ++- > .../hwtracing/coresight/coresight-tmc-core.c | 9 +- > .../hwtracing/coresight/coresight-tmc-etf.c | 195 +++--- > .../hwtracing/coresight/coresight-tmc-etr.c | 199 +++--- > drivers/hwtracing/coresight/coresight-tmc.h | 2 +- > .../hwtracing/coresight/coresight-trace-id.c | 93 ++- > drivers/hwtracing/coresight/ultrasoc-smb.c | 12 +- > drivers/hwtracing/coresight/ultrasoc-smb.h | 2 +- > include/linux/coresight.h | 4 +- > 21 files changed, 751 insertions(+), 838 deletions(-) > > -- > LEVI:{C3F47F37-75D8-414A-A8BA-3980EC8A46D7} > Reviewed-by: James Clark <james.clark(a)linaro.org> Somewhat related to this change, but not something that needs to be done now: All of the spinlocks for the syfs store/read functions don't need to be raw spinlocks, or spinlocks at all. They're never called from the scheduling code and can be locked by coresight_mutex instead. coresight_mutex is held on the sysfs enable/disable path when those values are actually used. Same point as here: https://lore.kernel.org/linux-arm-kernel/9a637e74-d81d-405c-bad0-c97ec1aa4b… Probably needs a review of which values in each device might be shared between perf mode and sysfs mode when they shouldn't be, as there was one in the link above. I thought the only thing shared between sysfs and perf should be the 'mode' which is taken with a compare and swap without locking anyway.

9 months

Re: [PATCH 2/9] coresight-etm4x: change etmv4_drvdata spinlock type to raw_spinlock_t

by James Clark

On 25/11/2024 9:48 am, Yeoreum Yun wrote: > From: Levi Yun <yeoreum.yun(a)arm.com> > > In coresight-etm4x drivers, etmv4_drvdata->spinlock can be held during > __schedule() by perf_event_task_sched_out()/in(). > > Since etmv4_drvdata->spinlock type is spinlock_t and > perf_event_task_sched_out()/in() is called after acquiring rq_lock, > which is raw_spinlock_t (an unsleepable lock), > this poses an issue in PREEMPT_RT kernel where spinlock_t is sleepable. > > To address this, change type etmv4_drvdata->spinlock > in coresight-etm4x drivers, which can be called > by perf_event_task_sched_out()/in(), from spinlock_t to raw_spinlock_t. > > Signed-off-by: Yeoreum Yun <yeoreum.yun(a)arm.com> > --- [...] > diff --git a/drivers/hwtracing/coresight/coresight-etm4x-sysfs.c b/drivers/hwtracing/coresight/coresight-etm4x-sysfs.c > index a9f19629f3f8..2e6b79c37f87 100644 > --- a/drivers/hwtracing/coresight/coresight-etm4x-sysfs.c > +++ b/drivers/hwtracing/coresight/coresight-etm4x-sysfs.c > @@ -174,99 +174,99 @@ static ssize_t reset_store(struct device *dev, > if (kstrtoul(buf, 16, &val)) > return -EINVAL; > > - spin_lock(&drvdata->spinlock); > - if (val) > - config->mode = 0x0; > + scoped_guard(raw_spinlock, &drvdata->spinlock) { > + if (val) > + config->mode = 0x0; > > - /* Disable data tracing: do not trace load and store data transfers */ > - config->mode &= ~(ETM_MODE_LOAD | ETM_MODE_STORE); > - config->cfg &= ~(TRCCONFIGR_INSTP0_LOAD | TRCCONFIGR_INSTP0_STORE); > + /* Disable data tracing: do not trace load and store data transfers */ > + config->mode &= ~(ETM_MODE_LOAD | ETM_MODE_STORE); > + config->cfg &= ~(TRCCONFIGR_INSTP0_LOAD | TRCCONFIGR_INSTP0_STORE); > > - /* Disable data value and data address tracing */ > - config->mode &= ~(ETM_MODE_DATA_TRACE_ADDR | > - ETM_MODE_DATA_TRACE_VAL); > - config->cfg &= ~(TRCCONFIGR_DA | TRCCONFIGR_DV); > + /* Disable data value and data address tracing */ > + config->mode &= ~(ETM_MODE_DATA_TRACE_ADDR | > + ETM_MODE_DATA_TRACE_VAL); > + config->cfg &= ~(TRCCONFIGR_DA | TRCCONFIGR_DV); > > - /* Disable all events tracing */ > - config->eventctrl0 = 0x0; > - config->eventctrl1 = 0x0; > + /* Disable all events tracing */ > + config->eventctrl0 = 0x0; > + config->eventctrl1 = 0x0; > > - /* Disable timestamp event */ > - config->ts_ctrl = 0x0; > + /* Disable timestamp event */ > + config->ts_ctrl = 0x0; > > - /* Disable stalling */ > - config->stall_ctrl = 0x0; > + /* Disable stalling */ > + config->stall_ctrl = 0x0; > > - /* Reset trace synchronization period to 2^8 = 256 bytes*/ > - if (drvdata->syncpr == false) > - config->syncfreq = 0x8; > + /* Reset trace synchronization period to 2^8 = 256 bytes*/ > + if (drvdata->syncpr == false) > + config->syncfreq = 0x8; > > - /* > - * Enable ViewInst to trace everything with start-stop logic in > - * started state. ARM recommends start-stop logic is set before > - * each trace run. > - */ > - config->vinst_ctrl = FIELD_PREP(TRCVICTLR_EVENT_MASK, 0x01); > - if (drvdata->nr_addr_cmp > 0) { > - config->mode |= ETM_MODE_VIEWINST_STARTSTOP; > - /* SSSTATUS, bit[9] */ > - config->vinst_ctrl |= TRCVICTLR_SSSTATUS; > - } > + /* > + * Enable ViewInst to trace everything with start-stop logic in > + * started state. ARM recommends start-stop logic is set before > + * each trace run. > + */ > + config->vinst_ctrl = FIELD_PREP(TRCVICTLR_EVENT_MASK, 0x01); > + if (drvdata->nr_addr_cmp > 0) { > + config->mode |= ETM_MODE_VIEWINST_STARTSTOP; > + /* SSSTATUS, bit[9] */ > + config->vinst_ctrl |= TRCVICTLR_SSSTATUS; > + } > > - /* No address range filtering for ViewInst */ > - config->viiectlr = 0x0; > + /* No address range filtering for ViewInst */ > + config->viiectlr = 0x0; > > - /* No start-stop filtering for ViewInst */ > - config->vissctlr = 0x0; > - config->vipcssctlr = 0x0; > + /* No start-stop filtering for ViewInst */ > + config->vissctlr = 0x0; > + config->vipcssctlr = 0x0; > > - /* Disable seq events */ > - for (i = 0; i < drvdata->nrseqstate-1; i++) > - config->seq_ctrl[i] = 0x0; > - config->seq_rst = 0x0; > - config->seq_state = 0x0; > + /* Disable seq events */ > + for (i = 0; i < drvdata->nrseqstate-1; i++) > + config->seq_ctrl[i] = 0x0; > + config->seq_rst = 0x0; > + config->seq_state = 0x0; > > - /* Disable external input events */ > - config->ext_inp = 0x0; > + /* Disable external input events */ > + config->ext_inp = 0x0; > > - config->cntr_idx = 0x0; > - for (i = 0; i < drvdata->nr_cntr; i++) { > - config->cntrldvr[i] = 0x0; > - config->cntr_ctrl[i] = 0x0; > - config->cntr_val[i] = 0x0; > - } > + config->cntr_idx = 0x0; > + for (i = 0; i < drvdata->nr_cntr; i++) { > + config->cntrldvr[i] = 0x0; > + config->cntr_ctrl[i] = 0x0; > + config->cntr_val[i] = 0x0; > + } > > - config->res_idx = 0x0; > - for (i = 2; i < 2 * drvdata->nr_resource; i++) > - config->res_ctrl[i] = 0x0; > + config->res_idx = 0x0; > + for (i = 2; i < 2 * drvdata->nr_resource; i++) > + config->res_ctrl[i] = 0x0; > > - config->ss_idx = 0x0; > - for (i = 0; i < drvdata->nr_ss_cmp; i++) { > - config->ss_ctrl[i] = 0x0; > - config->ss_pe_cmp[i] = 0x0; > - } > + config->ss_idx = 0x0; > + for (i = 0; i < drvdata->nr_ss_cmp; i++) { > + config->ss_ctrl[i] = 0x0; > + config->ss_pe_cmp[i] = 0x0; > + } > > - config->addr_idx = 0x0; > - for (i = 0; i < drvdata->nr_addr_cmp * 2; i++) { > - config->addr_val[i] = 0x0; > - config->addr_acc[i] = 0x0; > - config->addr_type[i] = ETM_ADDR_TYPE_NONE; > - } > + config->addr_idx = 0x0; > + for (i = 0; i < drvdata->nr_addr_cmp * 2; i++) { > + config->addr_val[i] = 0x0; > + config->addr_acc[i] = 0x0; > + config->addr_type[i] = ETM_ADDR_TYPE_NONE; > + } > > - config->ctxid_idx = 0x0; > - for (i = 0; i < drvdata->numcidc; i++) > - config->ctxid_pid[i] = 0x0; > + config->ctxid_idx = 0x0; > + for (i = 0; i < drvdata->numcidc; i++) > + config->ctxid_pid[i] = 0x0; > > - config->ctxid_mask0 = 0x0; > - config->ctxid_mask1 = 0x0; > + config->ctxid_mask0 = 0x0; > + config->ctxid_mask1 = 0x0; > > - config->vmid_idx = 0x0; > - for (i = 0; i < drvdata->numvmidc; i++) > - config->vmid_val[i] = 0x0; > - config->vmid_mask0 = 0x0; > - config->vmid_mask1 = 0x0; > + config->vmid_idx = 0x0; > + for (i = 0; i < drvdata->numvmidc; i++) > + config->vmid_val[i] = 0x0; > > - spin_unlock(&drvdata->spinlock); > + config->vmid_mask0 = 0x0; > + config->vmid_mask1 = 0x0; > + } There's a lot of churn in this function just to use the new scoped guard, but there was only one lock and unlock anyway. There are a few other cases of this in the whole set. The scoped guards make it easier to write correct code, but I'm not sure we gain anything here other than a bigger diff and more to review by changing already working code. Having said that I did review all the changes and I'm pretty sure they're all ok, so I'm on the fence about whether it's worth going back and undoing all of them. Surely updating to raw spinlocks is a search and replace to fix a specific problem, and the scoped guard stuff can be done on a case by case basis when anything is re-written in the future? I don't know if we're considering a fixes tag, if PREEMPT_RT is 6.12? It's probably not necessary but if so we definitely want to minimise the change.

9 months

Re: [PATCH v7 11/12] KVM: arm64: Swap TRFCR on guest switch

by James Clark

On 26/11/2024 4:23 pm, Oliver Upton wrote: > On Thu, Nov 21, 2024 at 12:50:10PM +0000, James Clark wrote: >> >> >> On 20/11/2024 5:31 pm, Oliver Upton wrote: >>> On Tue, Nov 12, 2024 at 10:37:10AM +0000, James Clark wrote: >>>> +void kvm_set_trfcr(u64 host_trfcr, u64 guest_trfcr) >>>> +{ >>>> + if (kvm_arm_skip_trace_state()) >>>> + return; >>>> + >>>> + if (has_vhe()) >>>> + write_sysreg_s(guest_trfcr, SYS_TRFCR_EL12); >>>> + else >>>> + if (host_trfcr != guest_trfcr) { >>>> + *host_data_ptr(host_debug_state.trfcr_el1) = guest_trfcr; >>> >>> Huh? That's going into host_debug_state, which is the dumping grounds >>> for *host* context when entering a guest. >>> >>> Not sure why we'd stick a *guest* value in there... >>> >> >> Only to save a 3rd storage place for trfcr when just the register and one >> place is technically enough. But yes if it's more readable to have >> guest_trfcr_el1 separately then that makes sense. > > Yeah, since this is all per-cpu data at this point rather than per-vCPU, > it isn't the end of the world to use a few extra bytes. > >> That works, it would be nice to have it consistent and have it that way for >> filtering, like kvm_set_guest_trace_filters(bool kernel, bool user). But I >> suppose we can justify not doing it there because we're not really >> interpreting the TRFCR value just writing it whole. > > Agreed, the biggest thing I'd want to see in the exported interfaces > like this is to have enable/disable helpers to tell KVM when a driver > wants KVM to start/stop managing a piece of state while in a guest. > > Then the hypervisor code can blindly save/restore some opaque values to > whatever registers it needs to update. > >>> What if trace is disabled in the guest or in the host? Do we need to >>> synchronize when transitioning from an enabled -> disabled state like we >>> do today? >>> >> >> By synchronize do you mean the tsb_csync()? I can only see it being >> necessary for the TRBE case because then writing to the buffer is fatal. >> Without TRBE the trace sinks still work and the boundary of when exactly >> tracing is disabled in the kernel isn't critical. > > Ack, I had the blinders on that we cared only about TRBE here. > >>> I took a stab at this, completely untested of course && punts on >>> protected mode. But this is _generally_ how I'd like to see everything >>> fit together. >>> >> >> Would you expect to see the protected mode stuff ignored if I sent another >> version more like yours below? Or was that just skipped to keep the example >> shorter? > > Skipped since I slapped this together in a hurry. > >> I think I'm a bit uncertain on that one because removing HAS_TRBE means you >> can't check if TRBE is enabled or not in protected mode and it will go wrong >> if it is. > > The protected mode hypervisor will need two bits of information. > Detecting that the feature is present can be done in the kernel so long > as the corresponding static key / cpucap is toggled before we drop > privileges. > > Whether or not it is programmable + enabled is a decision that must be > made by observing hardware state from the hypervisor before entering a > guest. > > [...] > >>> +void kvm_enable_trbe(u64 guest_trfcr) >>> +{ >>> + if (WARN_ON_ONCE(preemptible())) >>> + return; >>> + >>> + if (has_vhe()) { >>> + write_sysreg_s(guest_trfcr, SYS_TRFCR_EL12); >>> + return; >>> + } >>> + >>> + *host_data_ptr(guest_trfcr_el1) = guest_trfcr; >>> + host_data_set_flag(HOST_TRBE_ENABLED); >> >> FWIW TRBE and TRF are separate features, so this wouldn't do the filtering >> correctly if TRBE wasn't in use, but I can split it out into >> separate kvm_enable_trbe(void) and kvm_set_guest_filters(u64 guest_trfcr). > > KVM manages the same piece of state (TRFCR_EL1) either way though right? > > The expectation I had is that KVM is informed any time a trace session > (TRBE or otherwise) is enabled/disabled on a CPU, likely with a TRFCR_EL1 > of 0 if guest mode is excluded. > > The function names might need massaging, but I was hoping to have a > single set of enable/disable knobs to cover all bases here. > I sent another version, it did come out much simpler and still does all the same things as before. I didn't manage to make a single enable/disable knob though. The thing is the filtering is set on the source side of the driver and trbe is a sink thing. I would have to couple them together and add knowledge of the sink type to the source to make it work. That would then open up the possibility for anyone adding a new source to get the trbe bit wrong in the future. Having KVM override the filter setting when trbe is in use seems a lot safer and easier to understand.

9 months

[PATCH v8 0/8] kvm/coresight: Support exclude guest and exclude host

by James Clark

FEAT_TRF is a Coresight feature that allows trace capture to be completely filtered at different exception levels, unlike the existing TRCVICTLR controls which may still emit target addresses of branches, even if the following trace is filtered. Without FEAT_TRF, it was possible to start a trace session on a host and also collect trace from the guest as TRCVICTLR was never programmed to exclude guests (and it could still emit target addresses even if it was). With FEAT_TRF, the current behavior of trace in guests exists depends on whether nVHE or VHE are being used. Both of the examples below are from the host's point of view, as Coresight isn't accessible from guests. This patchset is only relevant to when FEAT_TRF exists, otherwise there is no change. Current behavior: nVHE/pKVM: Because the host and the guest are both using TRFCR_EL1, trace will be generated in guests depending on the same filter rules the host is using. For example if the host is tracing userspace only, then guest userspace trace will also be collected. (This is further limited by whether TRBE is used because an issue with TRBE means that it's completely disabled in nVHE guests, but it's possible to have other tracing components.) VHE: With VHE, the host filters will be in TRFCR_EL2, but the filters in TRFCR_EL1 will be active when the guest is running. Because we don't write to TRFCR_EL1, guest trace will be completely disabled. New behavior: The guest filtering rules from the Perf session are now honored for both nVHE and VHE modes. This is done by either writing to TRFCR_EL12 at the start of the Perf session and doing nothing else further, or caching the guest value and writing it at guest switch for nVHE. In pKVM, trace is now be disabled for both protected and unprotected guests. There is also an optimization where the Coresight drivers pass their enabled state to KVM. This means in the common case KVM doesn't have to touch any sysregs when the feature isn't in use. Applies to kvmarm/next (60ad25e14a) but includes two commits from Oliver for a conflicting change to move TRBE and SPE flags to host data [7]. --- Changes since V7 [6]: * Drop SPE changes * Change the interface to be based on intent, i.e kvm_enable_trbe() rather than passing the raw register value * Drop change to re-use vcpu_flags mechanism in favour of [7] * Simplify by using the same switch function to and from guest Changes since V6 [5]: * Implement a better "do nothing" case where both the SPE and Coresight drivers give the enabled state to KVM, allowing some register reads to be dropped. * Move the state and feature flags out of the vCPU into the per-CPU host_debug_state. * Simplify the switch logic by adding a new flag HOST_STATE_SWAP_TRFCR and only storing a single TRFCR value. * Rename vcpu flag macros to a more generic kvm_flag... Changes since V5 [4]: * Sort new sysreg entries by encoding * Add a comment about sorting arch/arm64/tools/sysreg * Warn on preemptible() before calling smp_processor_id() * Pickup tags * Change TRFCR_EL2 from SysregFields to Sysreg because it was only used once Changes since V4 [3]: * Remove all V3 changes that made it work in pKVM and just disable trace there instead * Restore PMU host/hyp state sharing back to how it was (kvm_pmu_update_vcpu_events()) * Simplify some of the duplication in the comments and function docs * Add a WARN_ON_ONCE() if kvm_etm_set_guest_trfcr() is called when the trace filtering feature doesn't exist. * Split sysreg change into a tools update followed by the new register addition Changes since V3: * Create a new shared area to store the host state instead of copying it before each VCPU run * Drop commit that moved SPE and trace registers from host_debug_state into the kvm sysregs array because the guest values were never used * Document kvm_etm_set_guest_trfcr() * Guard kvm_etm_set_guest_trfcr() with a feature check * Drop Mark B and Suzuki's review tags on the sysreg patch because it turned out that broke the Perf build and needed some unconventional changes to fix it (as in: to update the tools copy of the headers in the same commit as the kernel changes) Changes since V2: * Add a new iflag to signify presence of FEAT_TRF and keep the existing TRBE iflag. This fixes the issue where TRBLIMITR_EL1 was being accessed even if TRBE didn't exist * Reword a commit message Changes since V1: * Squashed all the arm64/tools/sysreg changes into the first commit * Add a new commit to move SPE and TRBE regs into the kvm sysreg array * Add a comment above the TRFCR global that it's per host CPU rather than vcpu Changes since nVHE RFC [1]: * Re-write just in terms of the register value to be written for the host and the guest. This removes some logic from the hyp code and a value of kvm_vcpu_arch:trfcr_el1 = 0 no longer means "don't restore". * Remove all the conditional compilation and new files. * Change the kvm_etm_update_vcpu_events macro to a function. * Re-use DEBUG_STATE_SAVE_TRFCR so iflags don't need to be expanded anymore. * Expand the cover letter. Changes since VHE v3 [2]: * Use the same interface as nVHE mode so TRFCR_EL12 is now written by kvm. [1]: https://lore.kernel.org/kvmarm/20230804101317.460697-1-james.clark@arm.com/ [2]: https://lore.kernel.org/kvmarm/20230905102117.2011094-1-james.clark@arm.com/ [3]: https://lore.kernel.org/linux-arm-kernel/20240104162714.1062610-1-james.cla… [4]: https://lore.kernel.org/all/20240220100924.2761706-1-james.clark@arm.com/ [5]: https://lore.kernel.org/linux-arm-kernel/20240226113044.228403-1-james.clar… [6]: https://lore.kernel.org/kvmarm/20241112103717.589952-1-james.clark@linaro.o… [7]: https://lore.kernel.org/kvmarm/20241115224924.2132364-4-oliver.upton@linux.… James Clark (6): arm64/sysreg: Add a comment that the sysreg file should be sorted tools: arm64: Update sysreg.h header files arm64/sysreg/tools: Move TRFCR definitions to sysreg KVM: arm64: coresight: Give TRBE enabled state to KVM KVM: arm64: Support trace filtering for guests coresight: Pass guest TRFCR value to KVM Oliver Upton (2): KVM: arm64: Get rid of __kvm_get_mdcr_el2() and related warts KVM: arm64: Track presence of SPE/TRBE in kvm_host_data instead of vCPU arch/arm64/include/asm/kvm_asm.h | 5 +- arch/arm64/include/asm/kvm_host.h | 39 +- arch/arm64/include/asm/sysreg.h | 12 - arch/arm64/kvm/arm.c | 5 +- arch/arm64/kvm/debug.c | 92 ++-- arch/arm64/kvm/hyp/nvhe/debug-sr.c | 61 +-- arch/arm64/kvm/hyp/nvhe/hyp-main.c | 6 - arch/arm64/kvm/hyp/vhe/debug-sr.c | 5 - arch/arm64/tools/sysreg | 38 ++ .../coresight/coresight-etm4x-core.c | 43 +- drivers/hwtracing/coresight/coresight-etm4x.h | 2 +- drivers/hwtracing/coresight/coresight-priv.h | 3 + drivers/hwtracing/coresight/coresight-trbe.c | 5 + tools/arch/arm64/include/asm/sysreg.h | 410 +++++++++++++++++- tools/include/linux/kasan-tags.h | 15 + 15 files changed, 609 insertions(+), 132 deletions(-) create mode 100644 tools/include/linux/kasan-tags.h -- 2.34.1

9 months

Re: [PATCH] coresight: Fix dsb_mode_store() unsigned val is never less than zero

by James Clark

On 21/11/2024 2:40 am, Pei Xiao wrote: > dsb_mode_store() warn: unsigned 'val' is never less than zero. > > Reported-by: kernel test robot <lkp(a)intel.com> > Closes: https://lore.kernel.org/oe-kbuild-all/202410150702.UaZ7kvet-lkp@intel.com/ > Fixes: 018e43ad1eee ("coresight-tpdm: Add node to set dsb programming mode") > Signed-off-by: Pei Xiao <xiaopei01(a)kylinos.cn> > --- > drivers/hwtracing/coresight/coresight-tpdm.c | 3 +-- > 1 file changed, 1 insertion(+), 2 deletions(-) > > diff --git a/drivers/hwtracing/coresight/coresight-tpdm.c b/drivers/hwtracing/coresight/coresight-tpdm.c > index b7d99e91ab84..158950243d83 100644 > --- a/drivers/hwtracing/coresight/coresight-tpdm.c > +++ b/drivers/hwtracing/coresight/coresight-tpdm.c > @@ -640,8 +640,7 @@ static ssize_t dsb_mode_store(struct device *dev, > struct tpdm_drvdata *drvdata = dev_get_drvdata(dev->parent); > unsigned long val; > > - if ((kstrtoul(buf, 0, &val)) || (val < 0) || > - (val & ~TPDM_DSB_MODE_MASK)) > + if ((kstrtoul(buf, 0, &val)) || (val & ~TPDM_DSB_MODE_MASK)) > return -EINVAL; > > spin_lock(&drvdata->spinlock); Reviewed-by: James Clark <james.clark(a)linaro.org>

9 months, 1 week

Re: [PATCH v7 11/12] KVM: arm64: Swap TRFCR on guest switch

by James Clark

On 20/11/2024 5:31 pm, Oliver Upton wrote: > On Tue, Nov 12, 2024 at 10:37:10AM +0000, James Clark wrote: >> +void kvm_set_trfcr(u64 host_trfcr, u64 guest_trfcr) >> +{ >> + if (kvm_arm_skip_trace_state()) >> + return; >> + >> + if (has_vhe()) >> + write_sysreg_s(guest_trfcr, SYS_TRFCR_EL12); >> + else >> + if (host_trfcr != guest_trfcr) { >> + *host_data_ptr(host_debug_state.trfcr_el1) = guest_trfcr; > > Huh? That's going into host_debug_state, which is the dumping grounds > for *host* context when entering a guest. > > Not sure why we'd stick a *guest* value in there... > Only to save a 3rd storage place for trfcr when just the register and one place is technically enough. But yes if it's more readable to have guest_trfcr_el1 separately then that makes sense. >> + host_data_set_flag(HOST_STATE_SWAP_TRFCR); >> + } else >> + host_data_clear_flag(HOST_STATE_SWAP_TRFCR); >> +} >> +EXPORT_SYMBOL_GPL(kvm_set_trfcr); > > I have a rather strong distaste for this interface, both with the > coresight driver and internally with the hypervisor. It'd be better if > the driver actually told KVM what the *intent* is rather than throwing a > pile of bits over the fence and forcing KVM to interpret what that > configuration means. > That works, it would be nice to have it consistent and have it that way for filtering, like kvm_set_guest_trace_filters(bool kernel, bool user). But I suppose we can justify not doing it there because we're not really interpreting the TRFCR value just writing it whole. >> +static void __debug_swap_trace(void) >> +{ >> + u64 trfcr = read_sysreg_el1(SYS_TRFCR); >> + >> + write_sysreg_el1(*host_data_ptr(host_debug_state.trfcr_el1), SYS_TRFCR); >> + *host_data_ptr(host_debug_state.trfcr_el1) = trfcr; >> + host_data_set_flag(HOST_STATE_RESTORE_TRFCR); >> +} >> + > > What if trace is disabled in the guest or in the host? Do we need to > synchronize when transitioning from an enabled -> disabled state like we > do today? > By synchronize do you mean the tsb_csync()? I can only see it being necessary for the TRBE case because then writing to the buffer is fatal. Without TRBE the trace sinks still work and the boundary of when exactly tracing is disabled in the kernel isn't critical. > I took a stab at this, completely untested of course && punts on > protected mode. But this is _generally_ how I'd like to see everything > fit together. > Would you expect to see the protected mode stuff ignored if I sent another version more like yours below? Or was that just skipped to keep the example shorter? I think I'm a bit uncertain on that one because removing HAS_TRBE means you can't check if TRBE is enabled or not in protected mode and it will go wrong if it is. But other than that I think I get the general point of what you mean: * Add an explicit guest_trfcr variable rather than cheating and using the host one * kvm_enable_trbe() rather than interpreting the TRBLIMITR value * Some code reuse by calling __trace_do_switch() with flipped arguments on both entry and exit And see below but I think it requires one minor change to support filtering without TRBE > diff --git a/arch/arm64/include/asm/kvm_host.h b/arch/arm64/include/asm/kvm_host.h > index 8bc0ec151684..b4714cece5f0 100644 > --- a/arch/arm64/include/asm/kvm_host.h > +++ b/arch/arm64/include/asm/kvm_host.h > @@ -611,7 +611,7 @@ struct cpu_sve_state { > */ > struct kvm_host_data { > #define KVM_HOST_DATA_FLAG_HAS_SPE 0 > -#define KVM_HOST_DATA_FLAG_HAS_TRBE 1 > +#define KVM_HOST_DATA_FLAG_HOST_TRBE_ENABLED 1 > #define KVM_HOST_DATA_FLAG_HOST_SVE_ENABLED 2 > #define KVM_HOST_DATA_FLAG_HOST_SME_ENABLED 3 > unsigned long flags; > @@ -659,6 +659,9 @@ struct kvm_host_data { > u64 mdcr_el2; > } host_debug_state; > > + /* Guest trace filter value */ > + u64 guest_trfcr_el1; > + > /* Number of programmable event counters (PMCR_EL0.N) for this CPU */ > unsigned int nr_event_counters; > > @@ -1381,6 +1384,8 @@ static inline bool kvm_pmu_counter_deferred(struct perf_event_attr *attr) > void kvm_set_pmu_events(u64 set, struct perf_event_attr *attr); > void kvm_clr_pmu_events(u64 clr); > bool kvm_set_pmuserenr(u64 val); > +void kvm_enable_trbe(u64 guest_trfcr); > +void kvm_disable_trbe(void); > #else > static inline void kvm_set_pmu_events(u64 set, struct perf_event_attr *attr) {} > static inline void kvm_clr_pmu_events(u64 clr) {} > @@ -1388,6 +1393,8 @@ static inline bool kvm_set_pmuserenr(u64 val) > { > return false; > } > +void kvm_enable_trbe(u64 guest_trfcr) {} > +void kvm_disable_trbe(void) {} > #endif > > void kvm_vcpu_load_vhe(struct kvm_vcpu *vcpu); > diff --git a/arch/arm64/kvm/debug.c b/arch/arm64/kvm/debug.c > index 46dbeabd6833..6ef8d8f4b452 100644 > --- a/arch/arm64/kvm/debug.c > +++ b/arch/arm64/kvm/debug.c > @@ -72,10 +72,6 @@ void kvm_init_host_debug_data(void) > if (cpuid_feature_extract_unsigned_field(dfr0, ID_AA64DFR0_EL1_PMSVer_SHIFT) && > !(read_sysreg_s(SYS_PMBIDR_EL1) & PMBIDR_EL1_P)) > host_data_set_flag(HAS_SPE); > - > - if (cpuid_feature_extract_unsigned_field(dfr0, ID_AA64DFR0_EL1_TraceBuffer_SHIFT) && > - !(read_sysreg_s(SYS_TRBIDR_EL1) & TRBIDR_EL1_P)) > - host_data_set_flag(HAS_TRBE); > } > > /* > @@ -215,3 +211,27 @@ void kvm_debug_handle_oslar(struct kvm_vcpu *vcpu, u64 val) > kvm_arch_vcpu_load(vcpu, smp_processor_id()); > preempt_enable(); > } > + > +void kvm_enable_trbe(u64 guest_trfcr) > +{ > + if (WARN_ON_ONCE(preemptible())) > + return; > + > + if (has_vhe()) { > + write_sysreg_s(guest_trfcr, SYS_TRFCR_EL12); > + return; > + } > + > + *host_data_ptr(guest_trfcr_el1) = guest_trfcr; > + host_data_set_flag(HOST_TRBE_ENABLED); FWIW TRBE and TRF are separate features, so this wouldn't do the filtering correctly if TRBE wasn't in use, but I can split it out into separate kvm_enable_trbe(void) and kvm_set_guest_filters(u64 guest_trfcr). > +} > +EXPORT_SYMBOL_GPL(kvm_enable_trbe); > + > +void kvm_disable_trbe(void) > +{ > + if (has_vhe() || WARN_ON_ONCE(preemptible())) > + return; > + > + host_data_clear_flag(HOST_TRBE_ENABLED); > +} > +EXPORT_SYMBOL_GPL(kvm_disable_trbe); > diff --git a/arch/arm64/kvm/hyp/nvhe/debug-sr.c b/arch/arm64/kvm/hyp/nvhe/debug-sr.c > index 858bb38e273f..d36cbce75bee 100644 > --- a/arch/arm64/kvm/hyp/nvhe/debug-sr.c > +++ b/arch/arm64/kvm/hyp/nvhe/debug-sr.c > @@ -51,32 +51,33 @@ static void __debug_restore_spe(u64 pmscr_el1) > write_sysreg_el1(pmscr_el1, SYS_PMSCR); > } > > -static void __debug_save_trace(u64 *trfcr_el1) > +static void __trace_do_switch(u64 *saved_trfcr, u64 new_trfcr) > { > - *trfcr_el1 = 0; > + *saved_trfcr = read_sysreg_el1(SYS_TRFCR); > + write_sysreg_el1(new_trfcr, SYS_TRFCR); > > - /* Check if the TRBE is enabled */ > - if (!(read_sysreg_s(SYS_TRBLIMITR_EL1) & TRBLIMITR_EL1_E)) > + /* Nothing left to do if going to an enabled state */ > + if (new_trfcr) > return; > + > /* > - * Prohibit trace generation while we are in guest. > - * Since access to TRFCR_EL1 is trapped, the guest can't > - * modify the filtering set by the host. > + * Switching to a context with trace generation disabled. Drain the > + * trace buffer to memory. > */ > - *trfcr_el1 = read_sysreg_el1(SYS_TRFCR); > - write_sysreg_el1(0, SYS_TRFCR); > isb(); > - /* Drain the trace buffer to memory */ > tsb_csync(); > } > > -static void __debug_restore_trace(u64 trfcr_el1) > +static void __trace_switch_to_guest(void) > { > - if (!trfcr_el1) > - return; > + __trace_do_switch(host_data_ptr(host_debug_state.trfcr_el1), > + *host_data_ptr(guest_trfcr_el1)); > +} > > - /* Restore trace filter controls */ > - write_sysreg_el1(trfcr_el1, SYS_TRFCR); > +static void __trace_switch_to_host(void) > +{ > + __trace_do_switch(host_data_ptr(guest_trfcr_el1), > + *host_data_ptr(host_debug_state.trfcr_el1)); > } > > void __debug_save_host_buffers_nvhe(struct kvm_vcpu *vcpu) > @@ -84,9 +85,13 @@ void __debug_save_host_buffers_nvhe(struct kvm_vcpu *vcpu) > /* Disable and flush SPE data generation */ > if (host_data_test_flag(HAS_SPE)) > __debug_save_spe(host_data_ptr(host_debug_state.pmscr_el1)); > - /* Disable and flush Self-Hosted Trace generation */ > - if (host_data_test_flag(HAS_TRBE)) > - __debug_save_trace(host_data_ptr(host_debug_state.trfcr_el1)); > + > + /* > + * Switch the trace filter, potentially disabling and flushing trace > + * data generation > + */ > + if (host_data_test_flag(HOST_TRBE_ENABLED)) > + __trace_switch_to_guest(); > } > > void __debug_switch_to_guest(struct kvm_vcpu *vcpu) > @@ -98,8 +103,8 @@ void __debug_restore_host_buffers_nvhe(struct kvm_vcpu *vcpu) > { > if (host_data_test_flag(HAS_SPE)) > __debug_restore_spe(*host_data_ptr(host_debug_state.pmscr_el1)); > - if (host_data_test_flag(HAS_TRBE)) > - __debug_restore_trace(*host_data_ptr(host_debug_state.trfcr_el1)); > + if (host_data_test_flag(HOST_TRBE_ENABLED)) > + __trace_switch_to_host(); > } > > void __debug_switch_to_host(struct kvm_vcpu *vcpu) >

9 months, 1 week

Re: [PATCH v7 07/12] KVM: arm64: arm_spe: Give SPE enabled state to KVM

by James Clark

On 20/11/2024 9:16 am, Oliver Upton wrote: > Hi James, > > On Tue, Nov 12, 2024 at 10:37:06AM +0000, James Clark wrote: >> Currently in nVHE, KVM has to check if SPE is enabled on every guest >> switch even if it was never used. Because it's a debug feature and is >> more likely to not be used than used, give KVM the SPE buffer status to >> allow a much simpler and faster do-nothing path in the hyp. >> >> This is always called with preemption disabled except for probe/hotplug >> which gets wrapped with preempt_disable(). > > Unless the performance penalty of checking if SPE is measurably bad, I'd > rather we keep things as-is. > > Folks that want to go fast are probably using VHE to begin with. As you > note below, we need the hypervisor to decide if SPE is enabled based on > hardware in protected mode anyway. Using a common flow for protected and > non-protected configs keeps complexity down and increases the likelihood > SPE save/restore code actually gets tested. > I'm not sure if there is any measurable difference. This change was actually in response to this review from Marc here [1]: > Why do we need to save anything if nothing was enabled, which is > *all the time*? I'm sorry to break it to you, but nobody uses these > features. So I'd like them to have zero cost when not in use. > Surely there is something there that should say "yup, tracing" or > not (such as the enable bits), which would avoid hitting the sysreg > pointlessly? I suppose I could have taken the "zero cost" bit a bit too literally and maybe there were some simpler optimizations that didn't involve strongly coupling the driver to KVM. At least for enable/disable, for filtering it would still be required. I'm trying to think if there is some middle ground where there is a systemwide flag or static key that gets set on the very first SPE or trace session. In theory it could be simpler than this per-cpu enable disable stuff, but in the end it pretty much ends up needing the same info from the driver (and has the same protected mode issue). So you might as well do it as fine grained as this or not at all like you suggest. [1]: https://lore.kernel.org/linux-arm-kernel/86bk832jza.wl-maz@kernel.org/

9 months, 1 week

Jump to page:

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

CoreSight