Hi again,
I'm completely without ideas and need to ask for help.
I've been able to go a little further with tracing what is going on that causes 'rpi_usb_host_drv' to stop working. Like I wrote earlier I believed that interrupt is not properly passed. Now I know that I was wrong and have some more details but I still don't know how to fix the problem.
I'm compiling and running `usb_hid_raw.run` and `ping.run` on rpi1, rpi2 and rpi3 (64bit). I know exactly where the processing stops but I don't know why.
Generally the problem is somewhere after enabling interrupts in `rpi_usb_host_drv` which causes log:
dev_dbg: Calling enable_global_interrupts
and expected log:
Task::run: irq_20
which in some cases does not happen.
`usb_hid_raw` scenario works similarly on all rpi variants producing generally the same following logs (they differ only in some addresses:
dev_dbg: Calling enable_global_interrupts DWC_MODIFY_REG32: 10008 dev_dbg: Done device_add(): Probe return 0 Task::block: linux Task::schedule: linux Task::run: device_worker Task::block: device_worker Task::schedule: device_worker MG_acquire: Signal_receiver::pending_signal 0xc33b4 MG_acquired: Signal_receiver::pending_signal 0xc33b4 MG_release: Signal_receiver::pending_signal 0xc33b4 MG_released: Signal_receiver::pending_signal 0xc33b4 MG_acquire: Signal_receiver::pending_signal 0xc33b4 MG_acquired: Signal_receiver::pending_signal 0xc33b4 MG_release: Signal_receiver::pending_signal 0xc33b4 MG_released: Signal_receiver::pending_signal 0xc33b4 unblock /projects/genode/genode/repos/dde_linux/src/lib/legacy/lx_kit/irq.cc:149 Task::run: irq_20 handle_irq /projects/genode/genode/repos/dde_linux/src/lib/legacy/lx_kit/irq.cc:161
So everything is ok in this area.
But when I run `ping` scenario produced logs look like below (logs are caught using tracing and contain also information about rpc):
dev_dbg: Calling enable_global_interrupts DWC_MODIFY_REG32: 20008 dev_dbg: Done device_add(): Probe return 0 +update -update Task::block: linux Task::schedule: linux Task::run: device_worker +alloc -alloc +attach -attach Task::block: device_worker Task::schedule: device_worker <signal *session +alloc -alloc +attach -attach +size -size Signal_receiver::constructor(0x1400f10c)/projects/genode/genode/depot/tomga/src/base-hw-rpi/2021-10-14/src/lib/base/signal_receiver.cc:58 0x1400f134 Signal_receiver::constructor(1, 0x1400f130) Signal_receiver::constructor(2, 0x1400f14c) MG_acquire: Signal_receiver::Signal_receiver 0x1400f134
On Raspberry Pi 2 and 3 there are no other logs from this component.
On Raspberry Pi 3 there is however a message that will probably allow someone to explain me what is going on but unfortunately I failed to explain it by myself for quite some time. This message is:
Kernel: MMU-fault not handled ESR=0x92000035 Kernel: init -> drivers -> rpi_usb_host_drv -> ep raised unhandled MMU fault ip=0x70e80 fault-addr=0x1400c200 type=unknown
and the fault-addr is exactly an address of the mutex that is being locked. I've added two additional fields in this class (one before the mutex and one after it) to verify that I can access (and modify) their values and I can. But there is some problem with locking the mutex that causes silent hang on Raspberry Pi 2 (arm_v7a) MMU-fault on Raspberry Pi 3 (arm_v8a) and works without problems on Raspberry Pi 1 (arm_v6).
All later logs appear only on Raspberry Pi 1 where this driver seems to work.
MG_acquired: Signal_receiver::Signal_receiver 0x1400f134 MG_release: Signal_receiver::Signal_receiver 0x1400f134 MG_released: Signal_receiver::Signal_receiver 0x1400f134 +alloc_signal_source -alloc_signal_source Signal_receiver::manage(0x1400f10c)/projects/genode/genode/depot/tomga/src/base-hw-rpi/2021-10-14/src/lib/base/signal_receiver.cc:112 0x1400f134 MG_acquire: Signal_receiver::manage 0x1400f134 MG_acquired: Signal_receiver::manage 0x1400f134 Signal_receiver::manage(0x1400f10c)/projects/genode/genode/depot/tomga/src/base-hw-rpi/2021-10-14/src/lib/base/signal_receiver.cc:118 Signal_receiver::manage(0x1400f10c)/projects/genode/genode/depot/tomga/src/base-hw-rpi/2021-10-14/src/lib/base/signal_receiver.cc:120 Signal_receiver::manage(0x1400f10c)/projects/genode/genode/depot/tomga/src/base-hw-rpi/2021-10-14/src/lib/base/signal_receiver.cc:123 Signal_receiver::manage(0x1400f10c)/projects/genode/genode/depot/tomga/src/base-hw-rpi/2021-10-14/src/lib/base/signal_receiver.cc:132 +alloc_context -alloc_context Signal_receiver::manage(0x1400f10c)/projects/genode/genode/depot/tomga/src/base-hw-rpi/2021-10-14/src/lib/base/signal_receiver.cc:134 MG_release: Signal_receiver::manage 0x1400f134 MG_released: Signal_receiver::manage 0x1400f134 Signal_receiver::constructor(0x1400f1d4)/projects/genode/genode/depot/tomga/src/base-hw-rpi/2021-10-14/src/lib/base/signal_receiver.cc:58 0x1400f1fc Signal_receiver::constructor(1, 0x1400f1f8) Signal_receiver::constructor(2, 0x1400f214) MG_acquire: Signal_receiver::Signal_receiver 0x1400f1fc MG_acquired: Signal_receiver::Signal_receiver 0x1400f1fc MG_release: Signal_receiver::Signal_receiver 0x1400f1fc MG_released: Signal_receiver::Signal_receiver 0x1400f1fc +alloc_signal_source -alloc_signal_source Signal_receiver::manage(0x1400f1d4)/projects/genode/genode/depot/tomga/src/base-hw-rpi/2021-10-14/src/lib/base/signal_receiver.cc:112 0x1400f1fc MG_acquire: Signal_receiver::manage 0x1400f1fc MG_acquired: Signal_receiver::manage 0x1400f1fc Signal_receiver::manage(0x1400f1d4)/projects/genode/genode/depot/tomga/src/base-hw-rpi/2021-10-14/src/lib/base/signal_receiver.cc:118 Signal_receiver::manage(0x1400f1d4)/projects/genode/genode/depot/tomga/src/base-hw-rpi/2021-10-14/src/lib/base/signal_receiver.cc:120 Signal_receiver::manage(0x1400f1d4)/projects/genode/genode/depot/tomga/src/base-hw-rpi/2021-10-14/src/lib/base/signal_receiver.cc:123 Signal_receiver::manage(0x1400f1d4)/projects/genode/genode/depot/tomga/src/base-hw-rpi/2021-10-14/src/lib/base/signal_receiver.cc:132 +alloc_context -alloc_context Signal_receiver::manage(0x1400f1d4)/projects/genode/genode/depot/tomga/src/base-hw-rpi/2021-10-14/src/lib/base/signal_receiver.cc:134 MG_release: Signal_receiver::manage 0x1400f1fc MG_released: Signal_receiver::manage 0x1400f1fc +alloc_rpc_cap -alloc_rpc_cap Signal_receiver::manage(0xc338c)/projects/genode/genode/depot/tomga/src/base-hw-rpi/2021-10-14/src/lib/base/signal_receiver.cc:112 0xc33b4 MG_acquire: Signal_receiver::manage 0xc33b4 MG_acquired: Signal_receiver::manage 0xc33b4 Signal_receiver::manage(0xc338c)/projects/genode/genode/depot/tomga/src/base-hw-rpi/2021-10-14/src/lib/base/signal_receiver.cc:118 Signal_receiver::manage(0xc338c)/projects/genode/genode/depot/tomga/src/base-hw-rpi/2021-10-14/src/lib/base/signal_receiver.cc:120 Signal_receiver::manage(0xc338c)/projects/genode/genode/depot/tomga/src/base-hw-rpi/2021-10-14/src/lib/base/signal_receiver.cc:123 Signal_receiver::manage(0xc338c)/projects/genode/genode/depot/tomga/src/base-hw-rpi/2021-10-14/src/lib/base/signal_receiver.cc:132 +alloc_context -alloc_context Signal_receiver::manage(0xc338c)/projects/genode/genode/depot/tomga/src/base-hw-rpi/2021-10-14/src/lib/base/signal_receiver.cc:134 MG_release: Signal_receiver::manage 0xc33b4 MG_released: Signal_receiver::manage 0xc33b4 Signal_receiver::manage(0xc338c)/projects/genode/genode/depot/tomga/src/base-hw-rpi/2021-10-14/src/lib/base/signal_receiver.cc:112 0xc33b4 MG_acquire: Signal_receiver::manage 0xc33b4 MG_acquired: Signal_receiver::manage 0xc33b4 Signal_receiver::manage(0xc338c)/projects/genode/genode/depot/tomga/src/base-hw-rpi/2021-10-14/src/lib/base/signal_receiver.cc:118 Signal_receiver::manage(0xc338c)/projects/genode/genode/depot/tomga/src/base-hw-rpi/2021-10-14/src/lib/base/signal_receiver.cc:120 Signal_receiver::manage(0xc338c)/projects/genode/genode/depot/tomga/src/base-hw-rpi/2021-10-14/src/lib/base/signal_receiver.cc:123 Signal_receiver::manage(0xc338c)/projects/genode/genode/depot/tomga/src/base-hw-rpi/2021-10-14/src/lib/base/signal_receiver.cc:132 +alloc_context -alloc_context Signal_receiver::manage(0xc338c)/projects/genode/genode/depot/tomga/src/base-hw-rpi/2021-10-14/src/lib/base/signal_receiver.cc:134 MG_release: Signal_receiver::manage 0xc33b4 MG_released: Signal_receiver::manage 0xc33b4 +alloc_rpc_cap -alloc_rpc_cap <session *signal MG_acquire: Signal_receiver::pending_signal 0xc33b4 MG_acquired: Signal_receiver::pending_signal 0xc33b4 MG_release: Signal_receiver::pending_signal 0xc33b4 MG_released: Signal_receiver::pending_signal 0xc33b4 +elapsed_us -elapsed_us +elapsed_us -elapsed_us +elapsed_us -elapsed_us +elapsed_us -elapsed_us +elapsed_us -elapsed_us +elapsed_us -elapsed_us +trigger_once -trigger_once <signal *signal MG_acquire: Signal_receiver::pending_signal 0xc33b4 MG_acquired: Signal_receiver::pending_signal 0xc33b4 MG_release: Signal_receiver::pending_signal 0xc33b4 MG_released: Signal_receiver::pending_signal 0xc33b4 unblock /projects/genode/genode/depot/tomga/src/usb_host_drv/2021-10-14/src/lib/legacy/lx_kit/irq.cc:149 Task::run: irq_20 handle_irq /projects/genode/genode/depot/tomga/src/usb_host_drv/2021-10-14/src/lib/legacy/lx_kit/irq.cc:161
I'm really out of ideas now and I'm asking for help.
My `rpi_master_20210815` branch (with full of debugging code added) is in [1] in case someone would like to look but I mostly ask for some ideas what can explain what can cause such behavior and advice how to fix it.
Regards Tomasz Gajewski
[1] https://github.com/tomga/genode/tree/rpi_master_20210815