Hi Josef:

On Mon, Jan 9, 2023 at 5:48 AM Josef Söntgen josef.soentgen@genode-labs.com wrote:

Hello Colin,

I am glad you managed to get your USB wifi dongle working.

Thanks! It was a fun project.

I only briefly skimmed your commit series but I think it would be worthwhile to integrate your additions. As we want to use the driver component also on other platforms besides the PC the code should be restructured (i.e. move the common parts from the 'pc' into the 'dde_linux' repo) anyway.

This would be fantastic. The reason I got stuck in the 'pc' repo is because I was working from the example of wifi_drv. In order to run dde_linux I assumed it was necessary to have the directory repos/pc/src/lib/pc/lx_emul or some analogous version. I completely agree 'dde_linux' is more appropriate. Indeed, the driver should also work on ARM systems with USB provided the lx_emul/kit framework can be used the same way.

At this point splitting the PCI and adding the

USB backend appears to be reasonable.

Yes. Regarding splitting up the components, a thought I had was to make the 'wifi_supplicant' a separate Genode component. Since it already runs as a separate thread using a shared buffer and semaphores, it shouldn't be too difficult to split it as its own component. The advantages would be that all interfacing of sculpt_manager with the supplicant would be common across drivers, making it easier to port more drivers, and each driver would inherit the same feature set of the supplicant (e.g. if ad-hoc mode were enabled it would not require much, if any, change to the drivers).

As you probably have noticed we have some ported USB device drivers, albeit using the legacy lx_kit/lx_emul implementation, that share the glue-code. Eventually we want to update them as well - so by all means, if you have some ideas formed by your current experience feel free to share them.

Yes, I did see these drivers. My experience with the legacy lx_kit/lx_emul implementation has been much less positive than with the current version, sorry to say. My thoughts on porting USB drivers:

1) My 'wlan_usb' and 'lx_usb_wrapper' don't conform to the lx_emul/lx_kit distinction so would need to be refactored as you said, but the basic idea is the same, to have a layer of separation written in c between c++ code and linux-kernel code. Roughly 'wlab_usb' is intended to be something like lx_kit, and lx_usb_wrapper to be something like lx_emul. Ideally the author of a port should be required only to instantiate something like a 'Lx_kit::Usb' and make sure that the initcall is configured to call 'usb_register_driver'.

2) lx_kit requests a 'Platform' session to enumerate the devices. This is completely unnecessary in the case of a USB driver, since no devices will be used (Genode provides a separate USB interface). However, the lx_kit will loop forever if the device list is empty. As a workaround my driver asks for platform information about the USB host controller itself, just to get out of this loop by having a non-empty list. Ideally there would be a way to start a lx_kit session with the understanding that no devices are needed.

3) The ath9k driver on Linux opens a lot of simultaneous inbound interrupt URBs (something like 64, I'd have to check). I suppose this is to handle cases of heavy host workload so that the device-side queue doesn't fill. Unfortunately, Genode's USB interface doesn't seem capable of having so many URBs queued at once. Hence I had to implement a secondary queue, which is done a bit ad-hoc at the moment. Potentially Genode's limit could be relaxed and this would become unnecessary, or else it could be done more generically (i.e. applied to all endpoints).

4) Not relevant to making things work, but to performance, the model for preparing URBs is different between Linux and Genode. In Linux the memory needs to be allocated generically as a "socket buffer" (aka skbuff aka skb) and then attached to a URB, whereas in Genode one needs to allocate the space with the intention ahead of time that it's destined for USB (Usb::Interface::alloc). Hence any emulation of Linux kernel behavior by Genode is forced to bounce-buffer out of the Linux-allocated skb into a Genode-allocated packet. Perhaps this is an architectural distinction that cannot be overcome? My understanding of Genode IPC is not complete, but as I understand it the Genode 'packet_stream' interface is somewhat similar to Linux skbuff, except that each packet belongs to an assigned stream (e.g. network or USB). Is it possible to set up a common pool of stream packets and send them to network or USB as a later decision? In this case alloc_skb could allocate from the stream packet pool and save a bounce buffer transaction which occurs on 100% of wifi data in the current model (and probably happens on the network side too).

Regards, Colin

Hi Josef,

I added the ath9k driver [0] to the 'pc_wifi_drv' driver where it worked

fine in my limited testing (with the one device I have at hand and the GSI/MSI quick-fix).

Good to hear. Was this done with the changes I made to the supplicant interface? I'm curious if those break the existing wifi functionality, which I can't test other than to say it compiles successfully.

I also started to look at your commits in more detail (I have split them up somewhat to get a better grasp of the changes [1] - for the commit in world I diff'ed the driver in pc and yours to get your adjustments and additions). I noticed that the 'pc_lx_extras' library code seems to be missing from your commit.

At some point I believe this was re-named by me to pc_lx_ath9k_extras. You should find it in world/lib/mk and the import file in world/lib/import. The only point as I remember is to add the necessary files from the linux contrib dir. Since the ath9k driver sits in world, rather than in "pc", the corresponding source.list and dep.list files are not imported automatically by pc_linux API, hence the need for the pc_lx_ath9k_extras API for depot builds, and the import-pc_lx_ath9k_extras file to correctly locate the imported files. But, I am a genode build/depot build newbie, so perhaps I made this too complicated? If you want to build in the "pc" repo itself I suppose it isn't needed.

I would not go as far as splitting the 'wpa_supplicant' from the driver for now. From Sculpt's perspective the wifi-management-layer ('Wifi::Frontend') is the interface to talk to. If ad-hoc mode or even host-ap functionality becomes a desired feature I would integrate support for that in this interface as well.

That being said, I entertain the idea of changing the inner workings of the wifi-management ↔ supplicant ↔ driver chain to get rid of the internal socket call interface or rather clean it up a bit.

Ok, understood. I'm not sure how much cleanup it even needs. As is, I think it will not be as difficult as I thought to update drivers in the face of changes to Wifi::Frontend, since basically one just needs to link to a different version of a shared library. Admittedly I was also thinking of some unusual scenarios, such as multiple wifi adapters where it makes sense to have one supplicant and multiple drivers. But that's strange enough maybe it doesn't need to be supported.

BTW, were you able to build the wifi_ath9k_drv component? I hope that I have a version that is suitable for others to try. Well, if anyone has the hardware, which maybe is not as common as I thought since i bought it a while ago.

Josef,

[…] Admittedly I was also thinking of some unusual scenarios, such as

...

multiple wifi adapters where it makes sense to have one supplicant and multiple drivers. But that's strange enough maybe it doesn't need to be supported.

At least from my perspective I would rather treat the supplicant as a “minor” detail of the driver itself. So each instance has its own supplicant and a management component could take care of the overall configuration on the system level for multiple wireless devices.

Ok, I guess that makes sense.

However, I enabled the RT2800USB driver [0], added a 'wifi_ath9k.run' run script based on 'pc/run/wifi.run' to ease testing and will use this for further testing and development before tackling the Sculpt integration.

Ok, I may give this a look with the ath9k that I have already, it could be a great help in reducing compile time versus the depot builds.

I also had to adapt the test-bed somewhat [1] as the driver still requires the Platform session - removing this hard dependency is one of the tasks I implied by the PCI/USB split.

I'm not quite sure why this is for RT2800, it was not necessary for ath9k?

So far the RT2800USB driver is non-working and the next step is figuring out why. For that I will take a closer look at your USB back-end work and its integration into the kernel.

Bummer. I may pick one up too (i.e. RT2800-based device), they don't look pricey.

(Bare with me as this is done in my spare time, it might take a while.)

Sure, same.

-CVP

Hi Colin,

...

...

out why. For that I will take a closer look at your USB back-end work and its integration into the kernel.

I have done some investigation into this (see [1]), there is a simple issue and a harder issue.

The simple issue is that for ath9k all control transfers were outgoing, so I simply neglected to support incoming control transfers. That was easily fixed, but exposed the other issue.

The harder issue is that Usb::Packet_handler::submit can block, allowing IO (including timer) signals to be handled. Timer signals in turn can use Lx_kit::Scheduler::schedule to try to run the kernel. Hence, it is a risk to submit any USB transaction from the kernel emulation stack, since this will be a re-entrant call to schedule if it does block and a timer fires. For whatever reason this never happened with ath9k, but seems to happen with rt2800usb. […]

I stumpled across those issues as well ([1], [2]), sorry for not bringing those to your immediate attention - I hope you did not spend too much time chasing them down. To be honest this USB API ('usb/usb.h') should not be used anymore, especially as it contains such stumpling blocks under the hood. You took care to use it a non-blocking fashion but the preferred way is Using the Usb session directly.

(I currently don't have the capacity to give you well-grounded help regarding changing the flow of your LX USB connection which is why I omitted that part of your post.)

Regards Josef

I was able to re-factor the usb emulation to prevent re-entrant scheduling. However, now I encounter some timer-related issues that I struggle with for not understanding the lx_emul/lx_kit's handling of linux ticks and timing (nor am I an expert in how native Linux does this, it seems to be quite involved). The basic problem is that the driver wants to call "msleep" in Linux to wait between various register writes and reads to the USB hardware. This function is not emulated, so the native Linux implementation is called, which sets a timer.

Now, if I understand it right, the timer setting operation does not update the "hardware" timer until the next schedule (I can trace Genode's clocksource emulation and the msleep call doesn't reset it). In a Linux schedule, I guess this would be handled somehow, but in the emulated schedule, the timer code (tick_nohz_enter/exit) only gets called if (task_flags & PF_WQ_WORKER) is true, which apparently it isn't when msleep is called. If I disable that check, meaning force a tick_nohz_enter/exit on every schedule, then the clocksource will reset after scheduling and the driver can at least initialize the hardware. But as is, it leaves the clocksource scheduled with some default timeout that takes essentially forever (2000 seconds or something), meaning *every* msleep call waits 20+ minutes unless there was a timer event already scheduled.

So, it would be nice to know what's the way msleep emulation in lx_emul/lx_kit is "supposed" to work? Do I have to make sure the calling function always has PF_WQ_WORKER context? Have I disabled some critical code path that makes it run? Do I need to change kernel CONFIG_NO_HZ settings?

To Josef's points about the USP API:

To be honest this USB API ('usb/usb.h') should not be used anymore,

especially as it contains such stumpling blocks under the hood. You took care to use it a non-blocking fashion but the preferred way is Using the Usb session directly.

Ok, I understand that Usb::Session may be better, but even using it directly I am not sure it's possible to avoid having to maintain a secondary packet queue. The problem (I think) is this line: enum { TX_QUEUE_SIZE = 64 }; If there were a way to declare a session with larger TX_QUEUE_SIZE it might be possible, but 64 is simply too small for the ath9k usb driver, which leaves 64 inbound interrupt and 8 inbound bulk URBs open at all times, in addition to requiring at least a few spaces for outbound URBs. So for the moment to keep things simple I am still using the USB API.

(I currently don't have the capacity to give you well-grounded help regarding changing the flow of your LX USB connection which is why I omitted that part of your post.)

Sure, for the moment I think my approach is at least possible for a proof-of-concept driver, even if the Usb::Session might ultimately be better. It works for ath9k and I think it is close to working for rt2800, if I can figure this timer thing out.

Regards, CP

Hi Colin,

[…]

Josef I am curious if this version will also work for you?

[1] https://github.com/cvparker/genode-world/tree/ath9k_driver_support-2023-01-2... [2] https://github.com/cvparker/genode/tree/ath9k_driver_support-2023-01-22

Thanks for your continous effort, I'll give your branches a try.

Regards Josef

Hi Colin,

On Mon, May 22, 2023 at 09:53:31PM -0400, Colin Parker wrote:

As an update, I think I understand why the rt2x00 version cannot recover from the rfkill block. In addition to flushing the USBs tx queues, it also flushes the rx queue, which calls usb_kill_urb. I have emulated that function to simply wait for the packet to complete. For tx queues that's fine, either the transmission goes through or it errors and eventually it will clear. But rx urbs can sit indefinitely in the queue, and looking through the Usb::Connection interface and Packet_stream_tx I don't know if I can see a way to pull something out of the queue after it's been added. If I understand it right, that would require re-doing things in a pretty major way. But, on the bright side, if you don't use the rfkill feature (and you can always physically remove a USB device anyway), it's useable for me.

it looks like you nailed it. In the current setup, I would change an USB client to close the USB session, and later re-open it again if rfkill is triggered. However, if it is not a pressing problem for you, I would postpone such a change a bit, because we plan to re-work the USB session anyway. With the result that a client can better handle dynamic changes of its own (future) virtual USB bus, where it can claim and release an USB interface, and different interface of one USB device can get shared in between different drivers. This change will most propably be implemented right this year, and of course we'll have to adapt the USB-clients accordingly. We'll keep this use-case in our heads when re-working the USB-session protocol.

Best regards Stefan

On Thu, Apr 27, 2023 at 11:40 AM Josef Söntgen < josef.soentgen@genode-labs.com> wrote:

...

Hi Colin,

...

[…]

Josef I am curious if this version will also work for you?

[1]

https://github.com/cvparker/genode-world/tree/ath9k_driver_support-2023-01-2...

...

[2]

https://github.com/cvparker/genode/tree/ath9k_driver_support-2023-01-22

Thanks for your continous effort, I'll give your branches a try.

Regards Josef

-- Josef Söntgen Genode Labs

http://www.genode-labs.com/ · http://genode.org/ _______________________________________________ Genode users mailing list users@lists.genode.org https://lists.genode.org/listinfo/users

---

Genode users mailing list users@lists.genode.org https://lists.genode.org/listinfo/users

Hi Colin,

Thanks for your continous effort, I'll give your branches a try.

as you may have noticed, besides briefly looking into your branches and pushing simple changes to make it easier to test the driver in Qemu I unfortunately did not find the time to dig deeper why it apparently was not working properly for me.

That being said, yesterday I rebased [1] your changes to add hot-plugging support to the 'wpa_supplicant' (they compile but I did not test whether they work or not) to the staging branch containg the wifi stack changes for the upcoming release. I also sketched using the 'virt_linux' Linux target as basis for a USB wifi driver - it will not link as 'virt_linux' is not yet properly prepared for that scenario - as I believe that is beneficial for an USB driver where DMA memory (and to that extend the usage of the Platform service) is not necessary.

My (somewhat uninformed) guess is that it could be as simple as using your 'lx_emul_usb.c' back end instead of, for example, 'lx_emul_pci.c' (omitting the gory details/obstacles lurking in the dark…).

Maybe you want to give that a try?

[1] https://github.com/cnuke/genode/commits/usb_wifi-2023-05-24

Regards Josef

Hi Colin,

Did you try with a recent branch? I've been having success with rt2x00 and ath9k both lately.

No, I have not found the time yet but there should be some time this weekend.

...

That being said, yesterday I rebased [1] your changes to add hot-plugging support to the 'wpa_supplicant' (they compile but I did not test whether they work or not) to the staging branch containg the wifi stack changes for the upcoming release.

I was able to get this to work with my "world" version [1]. You can see the small changes I needed to make to the main repo in [2]. Commit 7f5c967 fixes a small issue that wasn't revealed earlier where wpa_supplicant_ctrl_iface_deinit is called with a null pointer for the second argument. Commit 0b24c1c adds some switches to the linux compile without which the usb version of rt2x00 won't build, and 76c138d adds usb_report_filter to the config scenario and is not strictly necessary unless you want to switch between two types of devices (ath9k and rt2x00) on-the-fly.

Thanks, I'll take a look.

...

Maybe you want to give that a try?

Ok. It may not be so completely simple. Likely there would need to be at least small changes to wifi_drv, for example compare "frontend.h" between the two, you will see some additional features like _remove_all_aps(), and IFACE-DEINIT had to be added to support hot-plug.

Uh sorry, I missed those changes - naturally these should be included as well.

Regards Josef

Josef, Fantastic!

I am happy to confirm that your driver also works with my Ralink-based

D-Link dongle (running via USB passthrough in QEMU and flood pinging it from the out-site). I encountered the I/O signal handling issue once but since its a known problem I mostly ignored that for now.

I had chased that away in "normal" use, but I'm not surprised you could trigger it with a ping flood.

I added the missing feature to the generic driver binary and changed the 'wifi_ath9k.run' run-script to use that instead of the specific one.

(I pushed the corresponding branches called 'usb_wifi-2023-06-11' for the genode and genode-world repo.)

I switched to those branches, it works for both my dongles (I suppose the correct branch name is cnuke/ath9k_driver_support-2023-06-11 for genode-world).

If I understand it right, the remaining suggestions are to switch to virt_linux and to re-implement USB with the "Connection" object. For the former I may take a look at it this weekend if I have time. I may also try to build a Sculpt configuration around usb_report_filter and supporting both dongles with the same configuration. Regarding the latter there is supposed to be a change to Genode's USB structures pending so maybe I'll just wait to see what happens with that?

Best, CP

Hi Josef,

We looked at the problem and could you give this commit [1] a try (it is

still WIP and not finalized). It might be helpful to add a diagnostic message to 'alloc' in 'usb/packet_handler.h' to see when the situation occurs.

[1] https://github.com/cnuke/genode/commit/8b4896e7

I think I understand the logic of this commit, and in general it seems to me a better way of handling the scheduling. I used cherry-pick to add it to my branch [1], and it seemed to work the same as before. That said, I am unable to trigger the signal handler issue with my hardware, even using 'sudo ping -f'.

1. Adapt your current version (that is based on the PC platform) by using the PC wifi driver as a template to the way the other drivers are by now implemented. That could involve making additions to the wireless stack but that's fine as merging your changes eventually would be the preferred way.

Ok, I think there won't need to be many changes to the wireless stack, but the USB emulation code needs to be somewhere. So I suppose I should add a lx_emul_usb.c to the dde_linux/lib/wifi directory, and besides that it will require the code from wlan_usb.cc and wlan_usb.h to be copied somewhere. As a most convenient option that can be in the same directory. Alternatively such code could live somewhere in lx_kit/lx_emul independent of wifi, if someone wanted to use it for porting another type of USB device driver, but for now that might be too optimistic.

-CP

[1] https://github.com/cvparker/genode/tree/usb_wifi-2023-06-18