[PATCHv13,0/9] lib/rwmmio/arm64: Add support to trace register reads/writes

Generic MMIO read/write i.e., __raw_{read,write}{b,l,w,q} accessors
are typically used to read/write from/to memory mapped registers
and can cause hangs or some undefined behaviour in following cases,

* If the access to the register space is unclocked, for example: if
  there is an access to multimedia(MM) block registers without MM
  clocks.

* If the register space is protected and not set to be accessible from
  non-secure world, for example: only EL3 (EL: Exception level) access
  is allowed and any EL2/EL1 access is forbidden.

* If xPU(memory/register protection units) is controlling access to
  certain memory/register space for specific clients.

and more...

Such cases usually results in instant reboot/SErrors/NOC or interconnect
hangs and tracing these register accesses can be very helpful to debug
such issues during initial development stages and also in later stages.

So use ftrace trace events to log such MMIO register accesses which
provides rich feature set such as early enablement of trace events,
filtering capability, dumping ftrace logs on console and many more.

Sample output:

rwmmio_write: __qcom_geni_serial_console_write+0x160/0x1e0 width=32 val=0xa0d5d addr=0xfffffbfffdbff700
rwmmio_post_write: __qcom_geni_serial_console_write+0x160/0x1e0 width=32 val=0xa0d5d addr=0xfffffbfffdbff700
rwmmio_read: qcom_geni_serial_poll_bit+0x94/0x138 width=32 addr=0xfffffbfffdbff610
rwmmio_post_read: qcom_geni_serial_poll_bit+0x94/0x138 width=32 val=0x0 addr=0xfffffbfffdbff610

This series is a follow-up for the series [1] and a recent series [2] making use
of both.

[1] https://lore.kernel.org/lkml/cover.1536430404.git.saiprakash.ranjan@codeaurora.org/
[2] https://lore.kernel.org/lkml/1604631386-178312-1-git-send-email-psodagud@codeaurora.org/

Note in v4 version, Arnd suggested to benchmark and compare size with callback
based implementation, please see [3] for more details on that with brief comparison below.


**Inline version with CONFIG_FTRACE=y and CONFIG_TRACE_MMIO_ACCESS=y**
$ size vmlinux
   text           data             bss     dec             hex         filename
 23884219        14284468         532568 38701255        24e88c7        vmlinux

**Callback version with CONFIG_FTRACE=y and CONFIG_TRACE_MMIO_ACCESS=y**
$ size vmlinux
    text          data             bss     dec             hex        filename
 24108179        14279596         532568 38920343        251e097       vmlinux

$ ./scripts/bloat-o-meter inline-vmlinux callback-vmlinux
add/remove: 8/3 grow/shrink: 4889/89 up/down: 242244/-11564 (230680)
Total: Before=25812612, After=26043292, chg +0.89%

[3] https://lore.kernel.org/lkml/466449a1-36da-aaa9-7e4f-477f36b52c9e@quicinc.com/

Changes in v13:
 * Remove the copyright update as one line change doesn't warrant it (Greg and Lawyers :))
 * Update the comment about disabling MMIO traces in geni se and uart drivers.
 * Add description for the build time flag to asm-generic/io.h.

Changes in v12:
 * Split the generic flag addition patch (Greg).
 * Move the flag from makefile to driver .c file (Greg).

Changes in v11:
 * Use unsigned long for caller ip and current ip addr (Steven Rostedt).
 * Include review tags from Arnd.

Changes in v10:
 * Use GENMASK(31, 0) for -Woverflow warning in irqchip tegra driver (Marc).
 * Convert ETM4x ARM64 driver to use asm-generic IO memory barriers (Catalin).
 * Collect ack from Catalin for arm64 change.

Changes in v9:
 * Use TRACE_EVENT_CLASS for rwmmio_write and post_write (Steven Rostedt).

Changes in v8:
 * Fix build error reported by kernel test robot.

Changes in v7:
 * Use lib/ instead of kernel/trace/ based on review comment by Steven Rostedt.

Changes in v6:
 * Implemented suggestions by Arnd Bergmann:
   - Use arch independent IO barriers in arm64/asm
   - Add ARCH_HAVE_TRACE_MMIO_ACCESS
   - Add post read and post write logging support
   - Remove tracepoint_active check
 * Fix build error reported by kernel test robot.

Changes in v5:
 * Move arm64 to use asm-generic provided high level MMIO accessors (Arnd).
 * Add inline logging for MMIO relaxed and non-relaxed accessors.
 * Move nVHE KVM comment to makefile (Marc).
 * Fix overflow warning due to switch to inline accessors instead of macro.
 * Modify trace event field to include caller and parent details for more detailed logs.

Changes in v4:
 * Drop dynamic debug based filter support since that will be developed later with
   the help from Steven (Ftrace maintainer).
 * Drop value passed to writel as it is causing hangs when tracing is enabled.
 * Code cleanup for trace event as suggested by Steven for earlier version.
 * Fixed some build errors reported by 0-day bot.

Changes in v3:
 * Create a generic mmio header for instrumented version (Earlier suggested in [1]
   by Will Deacon and recently [2] by Greg to have a generic version first).
 * Add dynamic debug support to filter out traces which can be very useful for targeted
   debugging specific to subsystems or drivers.
 * Few modifications to the rwmmio trace event fields to include the mmio width and print
   addresses in hex.
 * Rewrote commit msg to explain some more about usecases.

Prasad Sodagudi (1):
  lib: Add register read/write tracing support

Sai Prakash Ranjan (8):
  arm64: io: Use asm-generic high level MMIO accessors
  coresight: etm4x: Use asm-generic IO memory barriers
  irqchip/tegra: Fix overflow implicit truncation warnings
  drm/meson: Fix overflow implicit truncation warnings
  KVM: arm64: Add a flag to disable MMIO trace for nVHE KVM
  asm-generic/io: Add logging support for MMIO accessors
  serial: qcom_geni_serial: Disable MMIO tracing for geni serial
  soc: qcom: geni: Disable MMIO tracing for GENI SE

 arch/Kconfig                                  |  3 +
 arch/arm64/Kconfig                            |  1 +
 arch/arm64/include/asm/io.h                   | 41 ++------
 arch/arm64/kvm/hyp/nvhe/Makefile              |  7 +-
 drivers/gpu/drm/meson/meson_viu.c             | 22 ++---
 .../coresight/coresight-etm4x-core.c          |  8 +-
 drivers/hwtracing/coresight/coresight-etm4x.h |  8 +-
 drivers/irqchip/irq-tegra.c                   | 10 +-
 drivers/soc/qcom/qcom-geni-se.c               |  3 +
 drivers/tty/serial/qcom_geni_serial.c         |  3 +
 include/asm-generic/io.h                      | 91 ++++++++++++++++-
 include/trace/events/rwmmio.h                 | 97 +++++++++++++++++++
 lib/Kconfig                                   |  7 ++
 lib/Makefile                                  |  2 +
 lib/trace_readwrite.c                         | 47 +++++++++
 15 files changed, 288 insertions(+), 62 deletions(-)
 create mode 100644 include/trace/events/rwmmio.h
 create mode 100644 lib/trace_readwrite.c

On Thu, 28 Apr 2022 18:14:12 +0530
Sai Prakash Ranjan <quic_saipraka@quicinc.com> wrote:

> Disable MMIO tracing for geni serial engine driver as it is a
> high frequency operation with many register reads/writes and
> not very useful to log all MMIO traces and prevent excessive
> logging.

This states what it does but does not really state why. Are you using MMIO
tracing in other locations and this is causing too much noise?

What is the real issue. Just saying "excessive logging" is not sufficient.
That would be a reason to disable function tracing ;-)

-- Steve


> 
> Cc: Bjorn Andersson <bjorn.andersson@linaro.org>
> Signed-off-by: Sai Prakash Ranjan <quic_saipraka@quicinc.com>
> ---
>  drivers/soc/qcom/qcom-geni-se.c | 3 +++
>  1 file changed, 3 insertions(+)
> 
> diff --git a/drivers/soc/qcom/qcom-geni-se.c b/drivers/soc/qcom/qcom-geni-se.c
> index 28a8c0dda66c..a0ceeede450f 100644
> --- a/drivers/soc/qcom/qcom-geni-se.c
> +++ b/drivers/soc/qcom/qcom-geni-se.c
> @@ -1,6 +1,9 @@
>  // SPDX-License-Identifier: GPL-2.0
>  // Copyright (c) 2017-2018, The Linux Foundation. All rights reserved.
>  
> +/* Disable MMIO tracing to prevent excessive logging of unwanted MMIO traces */
> +#define __DISABLE_TRACE_MMIO__
> +
>  #include <linux/acpi.h>
>  #include <linux/clk.h>
>  #include <linux/slab.h>

Hi Steve,

On 4/28/2022 6:59 PM, Steven Rostedt wrote:
> On Thu, 28 Apr 2022 18:14:12 +0530
> Sai Prakash Ranjan <quic_saipraka@quicinc.com> wrote:
>
>> Disable MMIO tracing for geni serial engine driver as it is a
>> high frequency operation with many register reads/writes and
>> not very useful to log all MMIO traces and prevent excessive
>> logging.
> This states what it does but does not really state why. Are you using MMIO
> tracing in other locations and this is causing too much noise?
> What is the real issue. Just saying "excessive logging" is not sufficient.
> That would be a reason to disable function tracing ;-)
>
> -- Steve
>

Disabling MMIO trace is what it does and why it does is to prevent excessive logging.
I will add more details on why preventing excessive logging is required here.

It is quite known that any access over serial console would involve a lot of TX and RX
register accesses (and few others), so these MMIO read/write trace events in these
drivers cause a lot of unwanted noise because of the high frequency of such operations
and is not very useful tracing these events for such driver. And we want to enable these
trace events on development devices(maybe not production devices) where performance
also really matters since we want to debug and track any crashes with such register
accesses on these devices and not just something which runs with every debug option
out there (meaning we enable this MMIO tracing with almost on par production kernels).
And these traces of such driver add additional overhead both in terms of memory and
CPU cycles.

And also one more reason is that if we try to display trace buffer containing these register
trace events onto serial console, then it would also involve register accesses from these
drivers which would lead to recursive tracing.

I will update the commit text adding these details and since we got your attention, can you
also please look at patch 5 [1] which was updated based on your previous review comments.
If you could review/ack them, then Arnd can take this series into his tree once I post the next
version with these updated commit text.

[1] https://lore.kernel.org/lkml/9827bae40f6f319f294d06859c9e3c7442f067f2.1651149615.git.quic_saipraka@quicinc.com/

Thanks,
Sai

>> Cc: Bjorn Andersson <bjorn.andersson@linaro.org>
>> Signed-off-by: Sai Prakash Ranjan <quic_saipraka@quicinc.com>
>> ---
>>   drivers/soc/qcom/qcom-geni-se.c | 3 +++
>>   1 file changed, 3 insertions(+)
>>
>> diff --git a/drivers/soc/qcom/qcom-geni-se.c b/drivers/soc/qcom/qcom-geni-se.c
>> index 28a8c0dda66c..a0ceeede450f 100644
>> --- a/drivers/soc/qcom/qcom-geni-se.c
>> +++ b/drivers/soc/qcom/qcom-geni-se.c
>> @@ -1,6 +1,9 @@
>>   // SPDX-License-Identifier: GPL-2.0
>>   // Copyright (c) 2017-2018, The Linux Foundation. All rights reserved.
>>   
>> +/* Disable MMIO tracing to prevent excessive logging of unwanted MMIO traces */
>> +#define __DISABLE_TRACE_MMIO__
>> +
>>   #include <linux/acpi.h>
>>   #include <linux/clk.h>
>>   #include <linux/slab.h>