Good afternoon,
GDB has a valuable feature consisting of process record and replay. In
fact, GDB can record a log of process execution and save it. This record
can be loaded later on, and used for debugging. This is called offline
debugging. it offers the advantage that you can catch the issue once, and
replay it as much as needed to find the root cause and fix it. this is
extremely valuable for non reproduce-able or hard to reproduce bugs. you
can replay the record either forwards or backwards, which is very convivial
for observing and analyzing the software.
To realize this functionality, GDB is in fact executing the software, one
assembly instruction after another and recording relevant registers and
memory locations. This is a slow operation that can drastically change the
timing of process execution, and thus change the conditions that raise the
bug. To overcome this limitation, GDB can use available SoC IPs to
accelerate the operation. As per today, GDB has support for "Intel
Processor Trace" and " Branch Trace Store" IP on Intel processors.
ARM based SoCs have also IPs that can be used to assist process record,
namely CoreSight trace sources (ETM, PTM ..), trace links ( Funnels ...)
and trace sinks (ETB, ETR, TPIU...). They are now supported in Linux
kernel, through corresponding drivers and the extension of perf. A library
for decoding ETM traces (OpenCSD) is also available. The way is now paved
to bring acceleration of process record for ARM based SoC to GDB.
I am re-sending RFC and making it available as basis for discussions for
implementing this feature. it is also attached as text file
B.R.
Zied Guermazi
Non intrusive execution recording
for GDB using ARM CoreSight
Status of this Memo
This memo provides information for Linaro coresight and toolchain
communities.
Distribution of this memo is unlimited.
Abstract
A method of realizing execution recording in GDB in a non-intrusive way.
This method is based on the use of CoreSight hardware tracing, available on
ARM Cortex devices.
Table of Contents
1 Introduction
2 State of the art
3 Use cases
3.1 Self hosted debug monitor
3.2 Remote debug monitor
3.3 External debugger
4 Implementation needs
4.1 Self hosted debug monitor
4.2 Remote Debug monitor
4.3 External debugger
5 Remote protocol execution sequence
6 Remote protocol extensions
7 Solutions and alternatives .
7.1 Scope definition
7.2 CoreSight infrastructure exposure to the user
7.3 Parameters needed for parsing traces
1. Introduction
CoreSight technology offers a toolset for tracing the execution of a
program on a CPU, as well as routing the traces to an external trace port
analyzer or storing it in a dedicated internal memory. Those traces do not
affects system performance, and can be used as a record for program
execution.
GDB offers reverse debugging by recording program execution and storing
it. GDB offers either full record or program flow (branch) record. Records
can be replayed later-on for forwards or backwards debugging.
This request for comments is about realizing GDB record and replay
functionality using CoreSight technology. it presents typical use cases
and discuss different alternatives for realizing above mentioned feature.
2. State of the art
GDB currently supports two execution recording variants:
- full record: where registers as well as memory are recorded for each
instruction. in this case GDB collects the registers as well as involved
memory area after each instruction. currently this has no support for
hardware accelerators
- branch record: where only program flow is recorded. In this case GDB
collects program execution flow. currently branch record is implemented
either with or without hardware acceleration by using Intel branch trace
store "bts" and Intel processor trace "pt" hardware accelerator on
supported cpus.
3. Use cases
Programs running on ARM processors can be be debugged in many
configurations. three of them are selected in this RFC as base for
discussion :
3.1. Self hosted debug monitor
Those are systems where the debugger program runs on the same cpu as the
debugged program and monitors it. user interacts with the debugging session
on the target host itself.
Linux GDB is an example of such systems. in such a system following
setup is considered
- Target: a process running on an ARM cortex A
- Debugger: gnu GDB via ptrace API (arm-linux-gnueabihf-gdb)
+-----------------------------------+
| Target |
| +------------+ |
| +------+ | Coresight | |
| | | | components:| |
| | GDB |<--------->| | |
| | | ^ | DWT, ETM, | |
| +------+ | | ITM, TPIU | |
| ^ | | TMC, ETB | |
| | | +------------+ |
+----|---------|--------------------+
| |
| |
arm-linux- |
gnueabihf- |
gdb |
debug: ptrace
trace: perf/CoreSight drivers
3.2. Remote debug monitor
Those are systems where the debugger program runs on the same cpu as the
debugged program and monitors it. user interacts with the debugging session
remotely from a PC
Linux GDB is an example of such systems. in such a system following
setup is considered
- Target: a process running on an ARM cortex A
- GDB server: gnu gdbserver (arm-linux-gnueabihf-gdbserver)
- GDB client: gnu GDB (arm-linux-gnueabihf-gdb)
- UI: eclipse with needed plugins, MI interface is used.
+--------------------------+ +---------------------------------------+
| Host | | Target |
| | | +------------+ |
| +-----+ +------+ | | +------+ | Coresight | |
| | | | GDB | | | | GDB | | components:| |
| | UI |<--->| |<--->|<--->|<--->| |<--------->| | |
| | | ^ |Client| ^ | ^ | |Server| ^ | DWT, ETM, | |
| +-----+ | +------+ | | | | +------+ | | ITM, TPIU | |
| ^ | ^ | | | | ^ | | TMC, ETB | |
| | | | | | | | | | +------------+ |
+----|-----|-----|------|--+ | +--------|---------|--------------------+
| | | | | | |
| | | | | | |
Eclipse | arm-linux- | | arm-linux- |
| gnueabihf- | TCP/IP gnueabihf- |
| gdb | UART gdbserver |
GDB MI GDB remote debug: ptrace
protocol trace: perf/CoreSight drivers
3.3. External debugger
Those are systems where an external debugger is used. It accesses the
target using JTAG or SWD. Target is usually a bare metal embedded systems
or systems with an rtos.
as an example, following setup is considered:
- Target: firmware running on ARM cortex M.
- Debugger: external debug and trace device.
- GDB server: OpenOcd.
- GDB Client: arm-none-eabi-gdb.
- UI: eclipse with needed plugins, MI interface is used.
+--------------------------------------+ +-------+ +-------------+
| Host | | dbggr | | Target |
| | | | | |
| +-----+ +------+ +------+ | | | | Coresight |
| | | | GDB | | GDB | | | Debug | | components: |
| | UI |<--->| |<--->| |<-->|<--->| + |<--->| |
| | | ^ |Client| ^ |Server| | ^ | Trace | ^ | DWT, ETM, |
| +-----+ | +------+ | +------+ | | | | | | ITM, TPIU |
| ^ | ^ | ^ | | | | | | |
| | | | | | | | | | | | |
+----|-----|-----|------|-----|--------+ | +-------+ | +-------------+
| | | | | | |
| | | | | | |
Eclipse | arm-none- | OpenOcd | |
| eabi-gdb | PyOcd | |
| | | |
GDB MI GDB remote Ethernet debug: JTAG/SWD
protocol USB trace: Serial/Parallel
4. Implementation needs
4.1 Self hosted debug monitor
GDB : arm-linux-gnueabihf-gdb
the interface defined in btrace.h for capturing and processing traces
has to be implemented for arm CoreSight.
needed actions:
- in btrace-common.h: add needed structures for capturing and
handling etm traces
- in linux-btrace.h:
- add btrace_tinfo_etm
- amend btrace_target_info
- in linux-btrace.c: change following functions to support etm
traces
- linux_enable_btrace
- linux_disable_btrace
- linux_read_btrace
- linux_btrace_conf
- in arm-linux-nat.c:add an api to
- configure btrace
- enable btrace
- disable btrace
- read btrace
- in btrace.c
- btrace_add_pc btrace_fetch has to be implemented for
Coresight this means using opencsd library to parse etms and then
reconstruct executed instructions accordingly (btrace_compute_ftrace_1)
- in record-btrace.c
- add command for showing record btrace etm options
- add command for starting tracing with CoreSight and its
handler (cmd_record_btrace_etm_start)
- adapt cmd_show_record_btrace_cpu
...
perf:
needed actions:
- make sure that perf can start/stop tracing a process with its
threads, collect etm traces and deliver them to the user
4.2 Remote Debug monitor
changes described in 7.1 are needed. in addition, and to support remote
protocol following changes are needed
GDB server: arm-linux-gnueabihf-gdbserver
needed actions:
- in linux-low
- linux_low_read_btrace: add support for etm traces formatting.
- linux_low_btrace_conf: :add support for etm configuration
formatting.
GDB client: arm-linux-gnueabihf-gdb
needed actions:
- in remote.c
- adapt enable_btrace
- adapt disable_btrace
- in btrace.c
- parse_xml_btrace: update btrace.dtd [2] and related data
structures btrace_xxx
- parse_xml_btrace_conf: update btrace-conf.dtd [3] and related
data structures btrace_conf_xxx
- extend Remote protocol handling to support coresight etm traces
UI: eclipse
needed actions
make sure that the plugin for recoding execution and replaying it
is coping well in case of arm-linux
Remote protocol needs to be extended by
-1- Adding Qbtrace:CoreSight (or etm) to start collecting etm traces
-2- Amending 'Branch Trace Format' xml specification to consider etm
traces transfer
-3- Amending 'Branch Trace Configuration Format' xml specification to
consider parameters needed for etm
4.3 External debugger
changes described in 7.2 are needed. in addition, and to support tracing
a remote dealing with an external debugger (bare metal embedded system)
following changes are needed
GDB server: OpenOcd
needed actions:
- rework etm driver to make it up to date.
- add a driver for configuring trace interconnect IPs
- rework the driver for TPIU.
- integrate support for a Trace port analyzer.
- extend remote protocol implementation to support recording
Coresight infrastructure of the SoC is to be set in OpenOcd through
configuration files. Parameters that are not relevant for GDB are also
specified in configuration files (trace sink, trace protocol, port size,
trace synch frequency, cycle accurate tracing etc ...)
GDB client: arm-none-eabi-gdb
needed actions:
- extend Remote protocol to support coresight etm traces
- integrate etm trace parsing library
- interface the parser to record_btrace_target
Remote protocol needs -in addition to 7.2- to be extended by
- Adding Qbtrace-conf:CoreSight:core=value to support multicore SoC
- Adding btrace-conf:CoreSight:id=value to support demultiplexing
multiple trace sources
- Adding Qbtrace-conf:CoreSight:filter:context=value to support
filtering traces belonging to a given process/thread
- Adding Qbtrace-conf:CoreSight:filter:start-address=value
and Qbtrace-conf:CoreSight:filter:end-address=value to
support filtering traces for given functions/blocks/lib
- Adding Qbtrace-conf:CoreSight:trigger:on-address=value
and Qbtrace-conf:CoreSight:trigger:off-address=value to
support triggering tracing or stop tracing if a certain function/block/lib
is executed
alternatively some of configurations related to filtering and
triggering can be delegated to the GDB server.
UI: eclipse
test and verify that existing plugins cope well with GDB extensions
5. Remote protocol execution sequence
GDB and gdbserver are communicating using the GDB remote protocol.
on a semantic level a tracing session runs though following sequence
(1) GDB client queries gdb server support for branch trace
(2) GDB server answers with
- qXfer:btrace:read
- qXfer:btrace-conf:read
- Qbtrace:off
- Qbtrace:CoreSight
- Qbtrace-conf:CoreSight:xxx where xxx is the parameter name
(3) GDB client sends command to let start emitting and collecting
traces (Qbtrace:CoreSight)
(4) GDB server executes the commands
(5) GDB client sends command to stop emitting and collecting traces
(Qbtrace:off)
(6) GDB server exectues the command
(7) GDB client sends command to get collected traces from trace sink
(qXfer:btrace:read:annex:offset,length)
(8) GDB server executes the command and sends back collected traces
(9) GDB client parses the traces and reconstructs target states
6. Remote protocol extensions
the remote protocol needs be extended with following primitives to
support CoreSight tracing
- start tracing and traces capture using CoreSight (Qbtrace:CoreSight)
the remote protocol can be extended with following primitives to take
advantages of etm functionalities.
- select the core to trace on in the case of a multicore system
GDB client sends command to select the core to trace
(Qbtrace-conf:CoreSight:core=value)
- set the trace id for the traces
GDB client sends command to set trace id
(Qbtrace-conf:CoreSight:id=value)
- select the context to trace
GDB client sends command to select the context to trace
(Qbtrace-conf:CoreSight:filter:context=value)
- select the address range to trace
GDB client sends command to select the address range to trace
(Qbtrace-conf:CoreSight:filter:start-address=value)
(Qbtrace-conf:CoreSight:filter:end-address=value)
- set triggers for starting and stopping tracing
GDB client sends command to select the address to trigger tracing
(Qbtrace-conf:CoreSight:trigger:on-address=value)
(Qbtrace-conf:CoreSight:trigger:off-address=value)
7. alternatives and discussions
7.1. Scope definition
Coresight ETM IP comes in many versions and many implementations.
According to its capabilities, it can trace instructions only or
instructions and involved data/data address. All ETMs variants support
instructions tracing and can therefore be used for for branch tracing.
7.2. CoreSight infrastructure exposure to the user
it is here about assigning the responsibility of configuring Coresight
infrastructure to generate and route traces. two alternatives are possible:
- coresight infrastructure exposed to GDB client (and UI):
in this alternative the user or the UI is responsible for
configuring coresight IPs in the SoC, by accessing their registers
directly or via coresigh driver. Remote protocol is used to configure trace
sink (ETB or TPA) to start/stop collecting traces
- coresight infrastructure is not exposed outside of gdbserver.
in this case high level commands can be provided by gdbserver
remote protocol to setup and configure coresight IPs in the SoC.
My recommendation is to extend remote protocol to provide high level
commands to setup and configure coresight IPs in the SoC, or to use a
different channel to pass configuration parameters to GDB server
7.3. Parameters needed for parsing traces
Some configuration parameters like etm version, trace id ... (content of
registers ETMCR, ETMIDR, ETMCCER, ETMTRACEIDR) are needed for extracting
and parsing etm trace, those parameters needs to be exchanged between GDB
server and client. following alternatives are possible:
- extend the remote protocol to get those params with explicit queries
- add them to the content of the response to qXfer:btrace-conf:read
- add them to the content of the response to qXfer:btrace:read
[VIRT-263 # ARMv8.1-VHE Virtual Host Extensions ]
Got to "kvm enabled with vhe" kernel message, but then
the kernel hangs there. Irritatingly works with the
kernel I built myself, but not a distro supplied kernel.
Need to track down the config difference so I can continue
using gdbstub.
[VIRT-344 # ARMv8.5-MemTag, Memory Tagging Extension ]
Regenerated an mte+linux-user branch for Google engineers
to use to develop llvm. This is code previously posted,
but my current branch striped out linux-user for ease of
review of the system code.
[VIRT-327 # Richard's upstream QEMU work ]
Fixed mmap assert, signal handler method.
Fixed constant folding of extract2.
Fixed aarch64 host output of extract2.
Posted pull request for those.
Started reviewing the GSoC risugen patches, v3 for avx.
r~
== This Week ==
* PR90723
- Committed fix to trunk in r273466.
* PR90724
- Posted patch upstream, waiting for feedback.
* Validation
- SPEC2k6 with SVE seems to compile, but spotted couple of infra issues.
- Sent abe patch to store sum files.
* Misc
- Meetings
== Progress ==
* GCC:
- FDPIC: received feedback on generic patches, will address after holidays
- noinit attribute: iterated on generic attribute patch, not approved yet
* GNU-583 (Fix Linux kernel built for Thumb-2 with GCC using LTO)
- no progress this week
* GCC upstream validation:
- reported a few regressions.
* Binutils:
- Non-contiguous memory regions support in the BFD linker: proposal
looks OK, will start implementation after holidays
* misc:
- infra fixes / troubleshooting
- reported several regressions in QEMU, promptly fixed by our awesome
team members
== Next ==
GCC:
- handle feedback on FDPIC and noinit patches
- binutils/linker support for non-contiguous memory regions
- GNU-583
- GCC upstream validation: Add a config for cortex-m33 (v8-m)
== Holidays ==
July 13-27
Aug 2-11
QEMU Tooling ([VIRT-252])
=========================
QEMU plugin support ([VIRT-280])
- posted {PATCH for semihosting-tests} semihosting tests: add v7m
tests Message-Id: <20190711135726.14191-1-alex.bennee(a)linaro.org>
- semihosting re-factor now in v4 branch
- cleaned up translator_ld stuff for arm
- posted {PATCH for 4.1?} includes: remove stale {smp|max}_cpus
externs Message-Id: <20190711130546.18578-1-alex.bennee(a)linaro.org>
- fixed up code needing smp/max_cpus
[VIRT-280] https://projects.linaro.org/browse/VIRT-280
[v4 branch] https://github.com/stsquad/qemu/tree/plugins/plugins-v4
GSoC Mentoring ([VIRT-348])
- starting to look quite workable
- looks like chunks of CONFIG_PROFILER can be made runtime
select-able
Upstream Work ([VIRT-109])
==========================
- more regression hunting for 4.1 release
- looked at bugs [1834496] and [1836078]
- posted {PATCH v2 for 4.1} target/arm: report ARMv8-A FP support
for AArch32 -cpu max Message-Id:
<20190711103737.10017-1-alex.bennee(a)linaro.org>
- rth and pm215 also posted various fixes
- ieee_6 test looks like a [fortran/gcc runtime issue]
- posted {PATCH for 4.1? v1 0/7} testing/next (docker, win-cross)
Message-Id: <20190712111849.9006-1-alex.bennee(a)linaro.org>
[VIRT-109] https://projects.linaro.org/browse/VIRT-109
[1834496] https://bugs.launchpad.net/bugs/1834496
[1836078] https://bugs.launchpad.net/bugs/1836078
[fortran/gcc runtime issue]
https://gcc.gnu.org/bugzilla/show_bug.cgi?id=78314
Completed Reviews [5/5]
=======================
{PATCH 0/5} tcg: Fix mmap_lock assertion failure, take 2
Message-Id: <87zhlned2x.fsf(a)zen.linaroharston>
{PATCH for-4.1} target/arm: Set VFP-related MVFR0 fields for arm926 and arm1026
Message-Id: <20190711121231.3601-1-peter.maydell(a)linaro.org>
{PATCH for-4.1 0/2} Compatibility fixes for nettle 2.7 vs 3.0 vs 3.5
Message-Id: <20190712101849.8993-3-berrange(a)redhat.com>
{PATCH v2 0/5} tests/docker: add podman support
Message-Id: <20190709194330.837-1-marcandre.lureau(a)redhat.com>
- CLOSING NOTE [2019-07-12 Fri 18:07]
Looks ok - need to get a podman system up for testing
{RISU PATCH v3 00/18} Support for generating x86 SIMD test images
Message-Id: <20190711223300.6061-5-jan.bobek(a)gmail.com>
Absences
========
- 18-19th July
Current Review Queue
====================
* {PATCH 0/2} tests/acceptance: Add test of NeXTcube framebuffer using OCR
Message-Id: <20190629150056.9071-1-f4bug(a)amsat.org>
* {Qemu-devel} {PATCH 0/4} Introduce the microvm machine type
Message-Id: <20190628115349.60293-1-slp(a)redhat.com>
* {PATCH 0/3} tests/acceptance: Add tests for the Leon3 board
Message-Id: <20190627115331.2373-1-f4bug(a)amsat.org>
* {PATCH 0/5} tests/acceptance: Add bFLT loader linux-user test
Message-Id: <20190625101524.13447-1-philmd(a)redhat.com>
* {PATCH v2 0/9} KVM: arm/arm64: vgic: ITS translation cache
Message-Id: <20190611170336.121706-1-marc.zyngier(a)arm.com>
* {Qemu-devel} {RFC PATCH 0/7} Proof of concept for Meson integration
Message-Id: <1560165301-39026-1-git-send-email-pbonzini(a)redhat.com>
--
Alex Bennée
Progress:
* VIRT-65 [QEMU upstream maintainership]
+ QEMU 4.1 rc0 sent out of the door
+ investigated and sent fix for a regression with arm926 and arm1020
emulation: we accidentally turned off VFP double-precision support
on these cores with the recent VFP refactoring
+ helped track down a booting failure Beata ran into on aarch64 hosts
to a regression in the TCG backend, which RTH has now sent a patch for
+ sent a cleanup for some dead code spotted by Coverity in the imx6ul SoC
thanks
-- PMM
== Progress ==
* Investigate running benchmarks in containers [TCWG-1513]
- Faffing about with our benchmarking scripts, not sure how to test
changes without disrupting our infrastructure
- Cooked up some viz scripts so I can easily look at the noise
levels in benchmark results with/without containers
* Started LLVM 8.0.1 rc4 build
- In progress on ARM, infrastructure issues on AArch64
== Plan ==
* Upcoming vacation: 6 - 13 August
[LLVM-583] LLVM Code Size reduction ideas from Zephyr and CMSIS
- Started a ticket to record areas of improvement where GCC does
better than LLVM.
- Upstream defaults to -mno-unaligned-access for clang which needs to
be corrected for.
- Much of the difference goes away when inlining is disabled, implying
that different inlining strategies could be most significant
difference.
- Sent in Linaro Connect presentation submission to cover all of TCWGs
code-size improvement work.
Planned absences
- Rest of this week, back in the office on the 15th July
[Code size investigation]
Results (clang 2% larger than gcc) replicated on cortex-m0 and
cortex-m4 on Zephyr.
- Clang optimisation to use BLX rather than BL when same function
called multiple times is a pessimisation on Zephyr, especially on M0.
- GCC register allocation seems to result in fewer spills
TODO: Get an estimate of how much code-size difference is down to
different inlining decisions.
On CMSIS DSP cortex-m4f clang appears to be producing smaller than
GCC, not measured averages yet.
[LLD]
- Quite a few upstream reviews, PRs and investigations surrounding them.
- Likely that LLD will be converting to the new variable naming convention.
- Received a request to add cortex-a8 erratum fix for Google Android team.
[Linaro Connect]
Registered and contacted travel.
Drafted a submission for presentation, will submit next week.
Planned Absences:
On holiday Wednesday, Thursday, Friday next week
