Troubleshooting interrupt delivery

Fri Oct 15 01:06:26 CEST 2021

Hi again,

I'm completely without ideas and need to ask for help.

I've been able to go a little further with tracing what is going on that
causes 'rpi_usb_host_drv' to stop working. Like I wrote earlier I
believed that interrupt is not properly passed. Now I know that I was
wrong and have some more details but I still don't know how to fix the
problem.

I'm compiling and running `usb_hid_raw.run` and `ping.run` on rpi1, rpi2
and rpi3 (64bit). I know exactly where the processing stops but I don't
know why.

Generally the problem is somewhere after enabling interrupts in
`rpi_usb_host_drv` which causes log:

  dev_dbg: Calling enable_global_interrupts

and expected log:

  Task::run: irq_20

which in some cases does not happen.

`usb_hid_raw` scenario works similarly on all rpi variants producing
generally the same following logs (they differ only in some addresses:

  dev_dbg: Calling enable_global_interrupts
  DWC_MODIFY_REG32: 10008
  dev_dbg: Done
  device_add(): Probe return 0
  Task::block: linux
  Task::schedule: linux
  Task::run: device_worker
  Task::block: device_worker
  Task::schedule: device_worker
  MG_acquire: Signal_receiver::pending_signal 0xc33b4
  MG_acquired: Signal_receiver::pending_signal 0xc33b4
  MG_release: Signal_receiver::pending_signal 0xc33b4
  MG_released: Signal_receiver::pending_signal 0xc33b4
  MG_acquire: Signal_receiver::pending_signal 0xc33b4
  MG_acquired: Signal_receiver::pending_signal 0xc33b4
  MG_release: Signal_receiver::pending_signal 0xc33b4
  MG_released: Signal_receiver::pending_signal 0xc33b4
  unblock /projects/genode/genode/repos/dde_linux/src/lib/legacy/lx_kit/irq.cc:149
  Task::run: irq_20
  handle_irq /projects/genode/genode/repos/dde_linux/src/lib/legacy/lx_kit/irq.cc:161

So everything is ok in this area.

But when I run `ping` scenario produced logs look like below (logs are
caught using tracing and contain also information about rpc):

  dev_dbg: Calling enable_global_interrupts
  DWC_MODIFY_REG32: 20008
  dev_dbg: Done
  device_add(): Probe return 0
  +update
  -update
  Task::block: linux
  Task::schedule: linux
  Task::run: device_worker
  +alloc
  -alloc
  +attach
  -attach
  Task::block: device_worker
  Task::schedule: device_worker
  <signal
  *session
  +alloc
  -alloc
  +attach
  -attach
  +size
  -size
  Signal_receiver::constructor(0x1400f10c)/projects/genode/genode/depot/tomga/src/base-hw-rpi/2021-10-14/src/lib/base/signal_receiver.cc:58 0x1400f134
  Signal_receiver::constructor(1, 0x1400f130)
  Signal_receiver::constructor(2, 0x1400f14c)
  MG_acquire: Signal_receiver::Signal_receiver 0x1400f134

    On Raspberry Pi 2 and 3 there are no other logs from this component.

    On Raspberry Pi 3 there is however a message that will probably
    allow someone to explain me what is going on but unfortunately I
    failed to explain it by myself for quite some time. This message is:

      Kernel: MMU-fault not handled ESR=0x92000035
      Kernel: init -> drivers -> rpi_usb_host_drv -> ep raised unhandled MMU fault ip=0x70e80 fault-addr=0x1400c200 type=unknown

    and the fault-addr is exactly an address of the mutex that is being
    locked. I've added two additional fields in this class (one before
    the mutex and one after it) to verify that I can access (and modify)
    their values and I can. But there is some problem with locking the
    mutex that causes silent hang on Raspberry Pi 2 (arm_v7a) MMU-fault
    on Raspberry Pi 3 (arm_v8a) and works without problems on Raspberry
    Pi 1 (arm_v6).

    All later logs appear only on Raspberry Pi 1 where this driver seems
    to work.

  MG_acquired: Signal_receiver::Signal_receiver 0x1400f134
  MG_release: Signal_receiver::Signal_receiver 0x1400f134
  MG_released: Signal_receiver::Signal_receiver 0x1400f134
  +alloc_signal_source
  -alloc_signal_source
  Signal_receiver::manage(0x1400f10c)/projects/genode/genode/depot/tomga/src/base-hw-rpi/2021-10-14/src/lib/base/signal_receiver.cc:112 0x1400f134
  MG_acquire: Signal_receiver::manage 0x1400f134
  MG_acquired: Signal_receiver::manage 0x1400f134
  Signal_receiver::manage(0x1400f10c)/projects/genode/genode/depot/tomga/src/base-hw-rpi/2021-10-14/src/lib/base/signal_receiver.cc:118
  Signal_receiver::manage(0x1400f10c)/projects/genode/genode/depot/tomga/src/base-hw-rpi/2021-10-14/src/lib/base/signal_receiver.cc:120
  Signal_receiver::manage(0x1400f10c)/projects/genode/genode/depot/tomga/src/base-hw-rpi/2021-10-14/src/lib/base/signal_receiver.cc:123
  Signal_receiver::manage(0x1400f10c)/projects/genode/genode/depot/tomga/src/base-hw-rpi/2021-10-14/src/lib/base/signal_receiver.cc:132
  +alloc_context
  -alloc_context
  Signal_receiver::manage(0x1400f10c)/projects/genode/genode/depot/tomga/src/base-hw-rpi/2021-10-14/src/lib/base/signal_receiver.cc:134
  MG_release: Signal_receiver::manage 0x1400f134
  MG_released: Signal_receiver::manage 0x1400f134
  Signal_receiver::constructor(0x1400f1d4)/projects/genode/genode/depot/tomga/src/base-hw-rpi/2021-10-14/src/lib/base/signal_receiver.cc:58 0x1400f1fc
  Signal_receiver::constructor(1, 0x1400f1f8)
  Signal_receiver::constructor(2, 0x1400f214)
  MG_acquire: Signal_receiver::Signal_receiver 0x1400f1fc
  MG_acquired: Signal_receiver::Signal_receiver 0x1400f1fc
  MG_release: Signal_receiver::Signal_receiver 0x1400f1fc
  MG_released: Signal_receiver::Signal_receiver 0x1400f1fc
  +alloc_signal_source
  -alloc_signal_source
  Signal_receiver::manage(0x1400f1d4)/projects/genode/genode/depot/tomga/src/base-hw-rpi/2021-10-14/src/lib/base/signal_receiver.cc:112 0x1400f1fc
  MG_acquire: Signal_receiver::manage 0x1400f1fc
  MG_acquired: Signal_receiver::manage 0x1400f1fc
  Signal_receiver::manage(0x1400f1d4)/projects/genode/genode/depot/tomga/src/base-hw-rpi/2021-10-14/src/lib/base/signal_receiver.cc:118
  Signal_receiver::manage(0x1400f1d4)/projects/genode/genode/depot/tomga/src/base-hw-rpi/2021-10-14/src/lib/base/signal_receiver.cc:120
  Signal_receiver::manage(0x1400f1d4)/projects/genode/genode/depot/tomga/src/base-hw-rpi/2021-10-14/src/lib/base/signal_receiver.cc:123
  Signal_receiver::manage(0x1400f1d4)/projects/genode/genode/depot/tomga/src/base-hw-rpi/2021-10-14/src/lib/base/signal_receiver.cc:132
  +alloc_context
  -alloc_context
  Signal_receiver::manage(0x1400f1d4)/projects/genode/genode/depot/tomga/src/base-hw-rpi/2021-10-14/src/lib/base/signal_receiver.cc:134
  MG_release: Signal_receiver::manage 0x1400f1fc
  MG_released: Signal_receiver::manage 0x1400f1fc
  +alloc_rpc_cap
  -alloc_rpc_cap
  Signal_receiver::manage(0xc338c)/projects/genode/genode/depot/tomga/src/base-hw-rpi/2021-10-14/src/lib/base/signal_receiver.cc:112 0xc33b4
  MG_acquire: Signal_receiver::manage 0xc33b4
  MG_acquired: Signal_receiver::manage 0xc33b4
  Signal_receiver::manage(0xc338c)/projects/genode/genode/depot/tomga/src/base-hw-rpi/2021-10-14/src/lib/base/signal_receiver.cc:118
  Signal_receiver::manage(0xc338c)/projects/genode/genode/depot/tomga/src/base-hw-rpi/2021-10-14/src/lib/base/signal_receiver.cc:120
  Signal_receiver::manage(0xc338c)/projects/genode/genode/depot/tomga/src/base-hw-rpi/2021-10-14/src/lib/base/signal_receiver.cc:123
  Signal_receiver::manage(0xc338c)/projects/genode/genode/depot/tomga/src/base-hw-rpi/2021-10-14/src/lib/base/signal_receiver.cc:132
  +alloc_context
  -alloc_context
  Signal_receiver::manage(0xc338c)/projects/genode/genode/depot/tomga/src/base-hw-rpi/2021-10-14/src/lib/base/signal_receiver.cc:134
  MG_release: Signal_receiver::manage 0xc33b4
  MG_released: Signal_receiver::manage 0xc33b4
  Signal_receiver::manage(0xc338c)/projects/genode/genode/depot/tomga/src/base-hw-rpi/2021-10-14/src/lib/base/signal_receiver.cc:112 0xc33b4
  MG_acquire: Signal_receiver::manage 0xc33b4
  MG_acquired: Signal_receiver::manage 0xc33b4
  Signal_receiver::manage(0xc338c)/projects/genode/genode/depot/tomga/src/base-hw-rpi/2021-10-14/src/lib/base/signal_receiver.cc:118
  Signal_receiver::manage(0xc338c)/projects/genode/genode/depot/tomga/src/base-hw-rpi/2021-10-14/src/lib/base/signal_receiver.cc:120
  Signal_receiver::manage(0xc338c)/projects/genode/genode/depot/tomga/src/base-hw-rpi/2021-10-14/src/lib/base/signal_receiver.cc:123
  Signal_receiver::manage(0xc338c)/projects/genode/genode/depot/tomga/src/base-hw-rpi/2021-10-14/src/lib/base/signal_receiver.cc:132
  +alloc_context
  -alloc_context
  Signal_receiver::manage(0xc338c)/projects/genode/genode/depot/tomga/src/base-hw-rpi/2021-10-14/src/lib/base/signal_receiver.cc:134
  MG_release: Signal_receiver::manage 0xc33b4
  MG_released: Signal_receiver::manage 0xc33b4
  +alloc_rpc_cap
  -alloc_rpc_cap
  <session
  *signal
  MG_acquire: Signal_receiver::pending_signal 0xc33b4
  MG_acquired: Signal_receiver::pending_signal 0xc33b4
  MG_release: Signal_receiver::pending_signal 0xc33b4
  MG_released: Signal_receiver::pending_signal 0xc33b4
  +elapsed_us
  -elapsed_us
  +elapsed_us
  -elapsed_us
  +elapsed_us
  -elapsed_us
  +elapsed_us
  -elapsed_us
  +elapsed_us
  -elapsed_us
  +elapsed_us
  -elapsed_us
  +trigger_once
  -trigger_once
  <signal
  *signal
  MG_acquire: Signal_receiver::pending_signal 0xc33b4
  MG_acquired: Signal_receiver::pending_signal 0xc33b4
  MG_release: Signal_receiver::pending_signal 0xc33b4
  MG_released: Signal_receiver::pending_signal 0xc33b4
  unblock /projects/genode/genode/depot/tomga/src/usb_host_drv/2021-10-14/src/lib/legacy/lx_kit/irq.cc:149
  Task::run: irq_20
  handle_irq /projects/genode/genode/depot/tomga/src/usb_host_drv/2021-10-14/src/lib/legacy/lx_kit/irq.cc:161

I'm really out of ideas now and I'm asking for help.

My `rpi_master_20210815` branch (with full of debugging code added) is
in [1] in case someone would like to look but I mostly ask for some
ideas what can explain what can cause such behavior and advice how to
fix it.

Regards
Tomasz Gajewski

[1] https://github.com/tomga/genode/tree/rpi_master_20210815