Hi,
Attending the Hardware Trace on Linux talk at Linaro Connect the topic of ARMv7 support for trace decoding came up.
What is the current state and are there patches available somewhere?
--
Stefan
This patch set is to explore Coresight tracing data for postmortem
debugging. When kernel panic happens, the Coresight panic kdump can
help to save on-chip tracing data and tracer metadata into DRAM, later
relies on kdump and crash/perf tools to recovery tracing data for
"offline" analysis.
Comparing the patch series v4 and previous series, this patch series
has heavily refactored the implementation after investigated Intel PT
for kdump support. Intel PT calls one function for emergency stopping
trace when kernel panic occurs, in the function it reuses perf operation
to dump trace data into ring buffer, later crash tool extracts trace
data from perf ring buffer.
This patch series takes Intel PT as an example to use the same way to
stop ETM trace with perf mode. So far the related work is primarily
to focus on to support Coresight kdump with perf mode and we can add
support SysFS mode if later there have more clear requirement.
Comparing to previous series, this patch series also simplifies the
handling for tracer metadata. The old series introduced extra data
structure and two double link lists to maintain CoreSignt kdump
components; in the old implementation, one list was used to track tracer
metadata and another list was used to trace dump buffers, later these
two lists can be used to retrieve metadata and trace data buffer from
vmcore file. In this patch series it directly relies on CoreSight
driver global variables to retrieve related info, e.g. for perf mode we
can rely on per CPU pointer 'ctx_handle' to get perf ring buffer related
info and 'csdev_src' is for per CPU tracer device structure for metadata.
The crash extension program now has been enhanced to parse the data
structures in the kernel and use them to extract metadata and dump trace
data [1]; the crash extension program is updated to build with OpenCSD
decoder so this can simplize the decoding process, rather than before
needs to use perf to help decoding trace data.
This patch series has been verified on 96boards DB410c with below steps,
the 'long_loop' is a pretty simple program to only execute big number
loops so can generate big amount number of branch instructions.
Enable trace on the target board:
$ perf record -e cs_etm/(a)825000.etf/ --per-thread ./long_loop &
$ sleep 3
$ echo c > /proc/sysrq-trigger
Use crash tool for post analysis:
$ crash vmcore vmlinux
crash> extend arm_cs_dump.so
crash> arm_cs_dump -o out
[1] https://git.linaro.org/people/leo.yan/crash.git/log/?h=arm_cs_dump_etm_perf
Changes from v4:
* Support for CoreSight ETM with perf mode;
* Add API for crash stop;
* Simplized implementation with removing kdump dedicated data structures
and functions;
Changes from v3:
* Following Mathieu suggestion, reworked the panic kdump framework,
used kdump array to maintain source and sink device handlers;
* According to Mathieu suggestion, optimized panic notifier to
firstly dump panic CPU tracing data and then dump other CPUs tracing
data;
* Refined doc to reflect these implementation changes;
* Changed ETMv4 driver to add source device handler at probe phase;
* Refactored crash extension program to reflect kernel changes.
Changes from v2:
* Add the two patches for documentation.
* Following Mathieu suggestion, reworked the panic kdump framework,
removed the useless flag "PRE_PANIC".
* According to comment, changed to add and delete kdump node operations
in sink enable/disable functions;
* According to Mathieu suggestion, handle kdump node
addition/deletion/updating separately for sysFS interface and perf
method.
Changes from v1:
* Add support to dump ETMv4 meta data.
* Wrote 'crash' extension csdump.so so rely on it to generate 'perf'
format compatible file.
* Refactored panic dump driver to support pre & post panic dump.
Changes from RFC:
* Follow Mathieu's suggestion, use general framework to support dump
functionality.
* Changed to use perf to analyse trace data.
Leo Yan (6):
doc: Add Coresight documentation directory
doc: Add documentation for Coresight panic kdump
coresight: etm4x: Save ID values in config structure
coresight: tmc: Update latest value for page index and offset
coresight: etm-perf: Add interface to stop etm trace
arm64: smp: Stop CoreSight trace for kdump
.../trace/{ => coresight}/coresight-cpu-debug.txt | 0
.../trace/coresight/coresight-panic-kdump.txt | 99 ++++++++++++++++++++++
Documentation/trace/{ => coresight}/coresight.txt | 0
MAINTAINERS | 5 +-
arch/arm64/kernel/smp.c | 5 ++
drivers/hwtracing/coresight/Kconfig | 10 +++
drivers/hwtracing/coresight/coresight-etm-perf.c | 10 +++
drivers/hwtracing/coresight/coresight-etm4x.c | 7 ++
drivers/hwtracing/coresight/coresight-etm4x.h | 8 ++
drivers/hwtracing/coresight/coresight-tmc-etf.c | 8 ++
include/linux/coresight.h | 6 ++
11 files changed, 156 insertions(+), 2 deletions(-)
rename Documentation/trace/{ => coresight}/coresight-cpu-debug.txt (100%)
create mode 100644 Documentation/trace/coresight/coresight-panic-kdump.txt
rename Documentation/trace/{ => coresight}/coresight.txt (100%)
--
2.7.4
Coresight architecture defines CLAIM tags for a device to negotiate
control of the components (external agent vs self-hosted). Each device
has a pair of registers (CLAIMSET & CLAIMCLR) for managing the CLAIM
tags. However, the protocol for the CLAIM tags is IMPLEMENTATION DEFINED.
PSCI has recommendations for the use of the CLAIM tags to negotiate
controls for external agent vs self-hosted use, as defined in
ARM DEN 0022D, Section "6.8.1 Debug and Trace save and restore".
This series implements the recommended protocol by PSCI.
There were two options for the implementation.
1) Have the claim/disclaim operations performed from the coresight
generic driver - This doesn't work unfortunately for ETM devices
as the need cross-CPU calls to access the CLAIM registers. Also,
makes it complex for error recovery and reference counting.
2) Have the claim/disclaim operations performed from the device
specific drivers. The disadvantage is that the calls are sprinkled
in each driver, but this makes the operation much simpler.
This series implements the method (2). The first part of the series
prepares different drivers to handle errors from the lower layer
and clean up the state. The second part of the series updates the
existing drivers to claim/disclaim the devices as necessary.
Tested with a hacked coresight driver which modifies the external
claim tag via sysfs handle.
Applies on coresight/next in Mathieu's tree.
Changes since V1:
- Handle errors is enabling path and disable only the components
that were enabled in the iteration.
- Fix build break on arm32 (etm3x)
- Update commit description for "coresight: Add support for CLAIM tag protocol"
Suzuki K Poulose (14):
coresight: Handle failures in enabling a trace path
coresight: tmc-etr: Refactor for handling errors
coresight: tmc-etr: Handle errors enabling CATU
coresight: tmc-etb/etf: Prepare to handle errors enabling
coresight: etm4x: Add support for handling errors
coresight: etm3: Add support for handling errors
coresight: etb10: Handle errors enabling the device
coresight: dynamic-replicator: Handle multiple connections
coresight: Add support for CLAIM tag protocol
coresight: etmx: Claim devices before use
coresight: funnel: Claim devices before use
coresight: catu: Claim device before use
coresight: dynamic-replicator: Claim device for use
coreisght: tmc: Claim device before use
drivers/hwtracing/coresight/coresight-catu.c | 6 ++
.../coresight/coresight-dynamic-replicator.c | 79 ++++++++++----
drivers/hwtracing/coresight/coresight-etb10.c | 18 +++-
drivers/hwtracing/coresight/coresight-etm3x.c | 56 +++++++---
drivers/hwtracing/coresight/coresight-etm4x.c | 51 ++++++---
drivers/hwtracing/coresight/coresight-funnel.c | 26 ++++-
drivers/hwtracing/coresight/coresight-priv.h | 7 ++
drivers/hwtracing/coresight/coresight-tmc-etf.c | 95 +++++++++++------
drivers/hwtracing/coresight/coresight-tmc-etr.c | 80 +++++++++-----
drivers/hwtracing/coresight/coresight.c | 118 +++++++++++++++++++--
include/linux/coresight.h | 20 ++++
11 files changed, 434 insertions(+), 122 deletions(-)
--
2.7.4
This patch series is to two fixing of updating ring buffer in tmc-etf
driver. The first patch is to fix alignment setting for RRP; the second
patch tries to fix discarding trace data issue caused by filling
barrier packets in the same place, the patch keeps complete trace data
with inserting extra barrier packets.
This patch series has been rebased on CoreSight next branch:
https://git.linaro.org/kernel/coresight.git/log/?h=next with latest
commit 3733ca5a6578 ("coresight: tmc: Refactor loops in etb dump").
Changes from v1:
* Rebased on CoreSight next branch (Sept 11th, 2018);
* Added checking 'lost || to_read > handle->size' to set 'barrier_sz'.
Leo Yan (2):
coresight: tmc: Fix byte-address alignment for RRP
coresight: tmc: Fix writing barrier packets for ring buffer
drivers/hwtracing/coresight/coresight-tmc-etf.c | 41 +++++++++++++++++--------
1 file changed, 29 insertions(+), 12 deletions(-)
--
2.7.4
We do not enable scatter-gather mode in the TMC-ETR by default
to prevent malfunctioning of systems where the ETR may not be
properly connected to the memory subsystem to allow for simultaneous
READ/WRITE transactions when used in SG mode. Instead we whitelist
the platforms where we know that it is safe to use the mode.
All revisions of Juno have a proper ETR connection and hence
white list them.
Cc: Mathieu Poirier <mathieu.poirier(a)linaro.org>
Cc: Mike Leach <mike.leach(a)linaro.org>
Cc: Sudeep Holla <sudeep.holla(a)arm.com>
Cc: Liviu Dudau <liviu.dudau(a)arm.com>
Cc: Lorenzo Pierlisi <lorenzo.pieralisi(a)arm.com>
Signed-off-by: Suzuki K Poulose <suzuki.poulose(a)arm.com>
---
arch/arm64/boot/dts/arm/juno-base.dtsi | 1 +
1 file changed, 1 insertion(+)
diff --git a/arch/arm64/boot/dts/arm/juno-base.dtsi b/arch/arm64/boot/dts/arm/juno-base.dtsi
index ce56a4a..3596e5d 100644
--- a/arch/arm64/boot/dts/arm/juno-base.dtsi
+++ b/arch/arm64/boot/dts/arm/juno-base.dtsi
@@ -199,6 +199,7 @@
clocks = <&soc_smc50mhz>;
clock-names = "apb_pclk";
power-domains = <&scpi_devpd 0>;
+ arm,scatter-gather;
port {
etr_in_port: endpoint {
slave-mode;
--
2.7.4
>From the comment in the code, it claims the requirement for byte-address
alignment for RRP register: 'for 32-bit, 64-bit and 128-bit wide trace
memory, the four LSBs must be 0s. For 256-bit wide trace memory, the
five LSBs must be 0s'. This isn't consistent with the program, the
program sets five LSBs as zeros for 32/64/128-bit wide trace memory and
set six LSBs zeros for 256-bit wide trace memory.
After checking with the CoreSight Trace Memory Controller technical
reference manual (ARM DDI 0461B, section 3.3.4 RAM Read Pointer
Register), it proves the comment is right and the program does wrong
setting.
This patch fixes byte-address alignment for RRP by following correct
definition in the technical reference manual.
Cc: Mathieu Poirier <mathieu.poirier(a)linaro.org>
Cc: Mike Leach <mike.leach(a)linaro.org>
Signed-off-by: Leo Yan <leo.yan(a)linaro.org>
---
drivers/hwtracing/coresight/coresight-tmc-etf.c | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git a/drivers/hwtracing/coresight/coresight-tmc-etf.c b/drivers/hwtracing/coresight/coresight-tmc-etf.c
index 0549249..e310613 100644
--- a/drivers/hwtracing/coresight/coresight-tmc-etf.c
+++ b/drivers/hwtracing/coresight/coresight-tmc-etf.c
@@ -438,10 +438,10 @@ static void tmc_update_etf_buffer(struct coresight_device *csdev,
case TMC_MEM_INTF_WIDTH_32BITS:
case TMC_MEM_INTF_WIDTH_64BITS:
case TMC_MEM_INTF_WIDTH_128BITS:
- mask = GENMASK(31, 5);
+ mask = GENMASK(31, 4);
break;
case TMC_MEM_INTF_WIDTH_256BITS:
- mask = GENMASK(31, 6);
+ mask = GENMASK(31, 5);
break;
}
--
2.7.4
In ETB dump function tmc_etb_dump_hw() it has nested loops. The second
level loop is to iterate index in the range [0 .. drvdata->memwidth);
but the index isn't really used in the code, thus the second level
loop is useless.
This patch is to remove the second level loop; the refactor also reduces
indentation and we can use 'break' to replace 'goto' tag.
Cc: Mathieu Poirier <mathieu.poirier(a)linaro.org>
Signed-off-by: Leo Yan <leo.yan(a)linaro.org>
---
drivers/hwtracing/coresight/coresight-tmc-etf.c | 17 +++++++----------
1 file changed, 7 insertions(+), 10 deletions(-)
diff --git a/drivers/hwtracing/coresight/coresight-tmc-etf.c b/drivers/hwtracing/coresight/coresight-tmc-etf.c
index 9c599c9..8b34161 100644
--- a/drivers/hwtracing/coresight/coresight-tmc-etf.c
+++ b/drivers/hwtracing/coresight/coresight-tmc-etf.c
@@ -34,23 +34,20 @@ static void tmc_etb_dump_hw(struct tmc_drvdata *drvdata)
{
char *bufp;
u32 read_data, lost;
- int i;
/* Check if the buffer wrapped around. */
lost = readl_relaxed(drvdata->base + TMC_STS) & TMC_STS_FULL;
bufp = drvdata->buf;
drvdata->len = 0;
while (1) {
- for (i = 0; i < drvdata->memwidth; i++) {
- read_data = readl_relaxed(drvdata->base + TMC_RRD);
- if (read_data == 0xFFFFFFFF)
- goto done;
- memcpy(bufp, &read_data, 4);
- bufp += 4;
- drvdata->len += 4;
- }
+ read_data = readl_relaxed(drvdata->base + TMC_RRD);
+ if (read_data == 0xFFFFFFFF)
+ break;
+ memcpy(bufp, &read_data, 4);
+ bufp += 4;
+ drvdata->len += 4;
}
-done:
+
if (lost)
coresight_insert_barrier_packet(drvdata->buf);
return;
--
2.7.4
Coresight uses DT graph bindings to describe the connections of the
components. However we have some undocumented usage of the bindings
to describe some of the properties of the connections.
The coresight driver needs to know the hardware ports invovled
in the connection and the direction of data flow to effectively
manage the trace sessions. So far we have relied on the "port"
address (as described by the generic graph bindings) to represent
the hardware port of the component for a connection.
The hardware uses separate numbering scheme for input and output
ports, which implies, we could have two different (input and output)
ports with the same port number. This could create problems in the
graph bindings where the label of the port wouldn't match the address.
e.g, with the existing bindings we get :
port@0{ // Output port 0
reg = <0>;
...
};
port@1{
reg = <0>; // Input port 0
endpoint {
slave-mode;
...
};
};
With the new enforcement in the DT rules, mismatches in label and address
are not allowed (as see in the case for port@1). So, we need a new mechanism
to describe the hardware port number reliably.
Also, we relied on an undocumented "slave-mode" property (see the above
example) to indicate if the port is an input port. Let us formalise and
switch to a new property to describe the direction of data flow.
There were three options considered for the hardware port number scheme:
1) Use natural ordering in the DT to infer the hardware port number.
i.e, Mandate that the all ports are listed in the DT and in the ascending
order for each class (input and output respectively).
Pros :
- We don't need new properties and if the existing DTS list them in
order (which most of them do), they work out of the box.
Cons :
- We must list all the ports even if the system cannot/shouldn't use
it.
- It is prone to human errors (if the order is not kept).
2) Use an explicit property to list both the direction and the hw port
number and direction. Define "coresight,hwid" as 2 member array of u32,
where the members are port number and the direction respectively.
e.g
port@0{
reg = <0>;
endpoint {
coresight,hwid = <0 1>; // Port # 0, Output
}
};
port@1{
reg = <1>;
endpoint {
coresight,hwid = <0 0>; // Port # 0, Input
};
};
Pros:
- The bindings are formal but not so reader friendly and could
potentially lead to human errors.
Cons:
- Backward compatiblity is lost.
3) Use explicit properties (implemented in the series) for the hardware
port id and direction. We define a new property "coresight,hwid" for
each endpoint in coresight devices to specify the hardware port number
explicitly. Also use a separate property "direction" to specify the
direction of the data flow.
e.g,
port@0{
reg = <0>;
endpoint {
direction = <1>; // Output
coresight,hwid = <0>; // Port # 0
}
};
port@1{
reg = <1>;
endpoint {
direction = <0>; // Input
coresight,hwid = <0>; // Port # 0
};
};
Pros:
- The bindings are formal and reader friendly, and less prone to errors.
Cons:
- Backward compatibility is lost.
After a round of discussions [1], the following option (4) is adopted :
4) Group ports based on the directions under a dedicated node. This has been
checked with the upstream DTC tool to resolve the "address mismatch" issue.
e.g,
out-ports { // Output ports for this component
port@0 { // Outport 0
reg = 0;
endpoint { ... };
};
port@1 { // Outport 1
reg = 1;
endpoint { ... };
};
};
in-ports { // Input ports for this component
port@0 { // Inport 0
reg = 0;
endpoint { ... };
};
port@1 { // Inport 1
reg = 1;
endpoint { ... };
};
};
This series implements Option (4) listed above and falls back to the old
bindings if the new bindings are not available. This allows the systems
with old bindings work with the new driver. The driver now issues a warning
(once) when it encounters the old bindings. The series contains DT update
for Juno platform. The remaining in-kernel sources could be updated once
we are fine with the proposal.
It also cleans up the platform parsing code to reduce the memory usage by
reusing the platform description.
Applies on coresight/next
Changes since V2:
- Clean of_coresight_parse_endpoint() to return 1 to indicate a connection
record was updated.
- Drop documentation for old bindings
Changes since V1:
- Implement the proposal by Rob.
- Drop the DTS updates for all platforms except Juno
- Drop the incorrect fix in coresight_register. Instead document the code
to prevent people trying to un-fix it again.
- Add a patch to drop remote device references in DT graph parsing
- Split of_node refcount fixing patch, fix a typo in the comment.
- Add Reviewed-by tags from Mathieu.
- Drop patches picked up for 4.18-rc series
Changes since RFC:
- Fixed style issues
- Fix an existing memory leak coresight_register (Found in code update)
- Fix missing of_node_put() in the existing driver (Reported-by Mathieu)
- Update the existing dts in kernel tree.
Suzuki K Poulose (9):
coresight: Document error handling in coresight_register
coresight: platform: Refactor graph endpoint parsing
coresight: platform: Fix refcounting for graph nodes
coresight: platform: Fix leaking device reference
coresight: Fix remote endpoint parsing
coresight: Add helper to check if the endpoint is input
coresight: platform: Cleanup coresight connection handling
coresight: Cleanup coresight DT bindings
dts: juno: Update coresight bindings
.../devicetree/bindings/arm/coresight.txt | 95 +++++---
arch/arm64/boot/dts/arm/juno-base.dtsi | 161 ++++++------
arch/arm64/boot/dts/arm/juno-cs-r1r2.dtsi | 52 ++--
arch/arm64/boot/dts/arm/juno.dts | 13 +-
drivers/hwtracing/coresight/coresight.c | 35 +--
drivers/hwtracing/coresight/of_coresight.c | 269 ++++++++++++++-------
include/linux/coresight.h | 9 +-
7 files changed, 359 insertions(+), 275 deletions(-)
--
2.7.4