Hi everyone,
First of all thanks for the great work on the new release, keep it up!
I'm experimenting with the new iwlwifi driver on a device with 7260 card. I enabled support for this card as mentioned in the release notes and added the corresponding firmware file iwlwifi-7260-8.ucode. Unfortunately the driver is not working out of the box for this card.
The problem breaks down to a failure during DMA memory allocation. Here an excerpt from wifi_drv's log:
Slab too large 131072 reqested 86032 cached 0 dma alloc 81920 failed Slab too large 131072 reqested 105428 cached 0 dma alloc 101316 failed .... Slab too large 131072 reqested 86032 cached 0 dma alloc 81920 failed Slab too large 262144 reqested 175072 cached 0 dma alloc 170960 failed
I naively increased maximum slab size by setting SLAB_STOP_LOG2 in lxc_emul.cc to 20. Then failure changes to:
Failed to get slab for 262144 dma alloc: 170960 failed
So as it appears such a large slab cannot be allocated. Why is that? What would be the right way (tm) to resolve this issue?
Thanks Christian
Hallo Christian,
On 12/01/14 16:32, Christian Menard wrote:
I'm experimenting with the new iwlwifi driver on a device with 7260 card. I enabled support for this card as mentioned in the release notes and added the corresponding firmware file iwlwifi-7260-8.ucode. Unfortunately the driver is not working out of the box for this card.
The problem breaks down to a failure during DMA memory allocation. Here an excerpt from wifi_drv's log:
Slab too large 131072 reqested 86032 cached 0 dma alloc 81920 failed Slab too large 131072 reqested 105428 cached 0 dma alloc 101316 failed .... Slab too large 131072 reqested 86032 cached 0 dma alloc 81920 failed Slab too large 262144 reqested 175072 cached 0 dma alloc 170960 failed
I naively increased maximum slab size by setting SLAB_STOP_LOG2 in lxc_emul.cc to 20. Then failure changes to:
Failed to get slab for 262144 dma alloc: 170960 failed
So as it appears such a large slab cannot be allocated. Why is that? What would be the right way (tm) to resolve this issue?
Could you please attach the whole log file? These error messages are normally harmless — the driver tries to load the firmware all at once and falls back to loading the it in smaller chunks if that is not possible. I suspect the issue lies elsewhere.
Regards Josef
Hello Josef,
unfortunately I still have no possibility to get LOG output via serial, I only can print the LOG to screen and I'm too lazy to type the full log ;). However, I did some investigation into the problem. The device is stopped because of this error:
dma_map_single: virtual address b008a78 not registered for DMA, called from:
1070bfc
def_error: Error sending BT_CONFIG: enqueu_hcmd failed: -12
Where dma_map_single is called by enqueu_hcmd. So the problem is related to DMA,
Btw, I also tried to increase the memory quota to ensure there is enough memory.
Ask if you need any more info from the logs, then I will type or do a photo)
Cheers Christian
On Monday 01 December 2014 16:50:14 Josef Söntgen wrote:
Hallo Christian,
On 12/01/14 16:32, Christian Menard wrote:
I'm experimenting with the new iwlwifi driver on a device with 7260 card. I enabled support for this card as mentioned in the release notes and added the corresponding firmware file iwlwifi-7260-8.ucode. Unfortunately the driver is not working out of the box for this card.
The problem breaks down to a failure during DMA memory allocation. Here an
excerpt from wifi_drv's log:
Slab too large 131072 reqested 86032 cached 0 dma alloc 81920 failed Slab too large 131072 reqested 105428 cached 0 dma alloc 101316 failed .... Slab too large 131072 reqested 86032 cached 0 dma alloc 81920 failed Slab too large 262144 reqested 175072 cached 0 dma alloc 170960 failed
I naively increased maximum slab size by setting SLAB_STOP_LOG2 in lxc_emul.cc> to 20. Then failure changes to:
Failed to get slab for 262144 dma alloc: 170960 failed
So as it appears such a large slab cannot be allocated. Why is that? What would be the right way (tm) to resolve this issue?
Could you please attach the whole log file? These error messages are normally harmless — the driver tries to load the firmware all at once and falls back to loading the it in smaller chunks if that is not possible. I suspect the issue lies elsewhere.
Regards Josef
-- Download BIRT iHub F-Type - The Free Enterprise-Grade BIRT Server from Actuate! Instantly Supercharge Your Business Reports and Dashboards with Interactivity, Sharing, Native Excel Exports, App Integration & more Get technology previously reserved for billion-dollar corporations, FREE http://pubads.g.doubleclick.net/gampad/clk?id=157005751&iu=/4140/ostg.cl... _______________________________________________ genode-main mailing list genode-main@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/genode-main
Hallo Christian,
On 12/01/14 17:11, Christian Menard wrote:
unfortunately I still have no possibility to get LOG output via serial, I only- can print the LOG to screen and I'm too lazy to type the full log ;).
I see — in this case you might want to try using the tcp_terminal in combination with terminal_log and route the LOG session of the wifi_drv accordingly. This atleast enables you to get the wifi_drv messages via TCP :-)
However, I did some investigation into the problem. The device is stopped because of this error:
dma_map_single: virtual address b008a78 not registered for DMA, called from:-
1070bfc
def_error: Error sending BT_CONFIG: enqueu_hcmd failed: -12
Where dma_map_single is called by enqueu_hcmd. So the problem is related to- DMA,
Yeah, the problem is related to how DMA memory is handled in Linux and with how we deal with it in dde_linux. Almost all memory allocations in the vanilla source, even the ones that are used later on for DMA, are done with GFP_KERNEL. In dde_linux we employ two memory pools. One for “normal” memory and one for memory which may be used for DMA. We introduced a flag (GFP_LX_DMA) to distinguish between the two. Unfortunatly, you have to patch the vanilla sources for this to work (you may take a look at the wifi_mem.patch). Since the wifi_drv was only tested on 6xxx cards only the dvm source is patched. To enable the 7260 card you would have to trace all memory allocations done by this part of the iwlwifi driver and change the flags on all allocations that might end up being used in an DMA transaction.
I am afraid that at the moment I do not have the time to look any further into this matter but maybe you get away by changing all allocations to use the DMA allocator for now.
Regards Josef
Hi,
I see — in this case you might want to try using the tcp_terminal in combination with terminal_log and route the LOG session of the wifi_drv accordingly. This atleast enables you to get the wifi_drv messages via TCP :-)
I used this configuration on a different device, but on my target device (Tablet) all I have for the moment is USB.... I also have an usb-nic but it is not detected by Genode's driver.
Yeah, the problem is related to how DMA memory is handled in Linux and with how we deal with it in dde_linux. Almost all memory allocations in the vanilla source, even the ones that are used later on for DMA, are done with GFP_KERNEL. In dde_linux we employ two memory pools. One for “normal” memory and one for memory which may be used for DMA. We introduced a flag (GFP_LX_DMA) to distinguish between the two. Unfortunatly, you have to patch the vanilla sources for this to work (you may take a look at the wifi_mem.patch). Since the wifi_drv was only tested on 6xxx cards only the dvm source is patched. To enable the 7260 card you would have to trace all memory allocations done by this part of the iwlwifi driver and change the flags on all allocations that might end up being used in an DMA transaction.
Thank you for the explanation. I traced down two such allocs and changed the flag accordingly. Now the log is error free (besides the "slab too large" messages mentioned before). However, as it appears the driver is not working. I get the message "iwl driver loaded" but then nothing happens anymore. I guess there should be more output for searching the network, establishing connection etc?
I tried to track down the problem, but so far I only found that wpa_supplicant blocks on it's socket in netlink_init(). The _handle() function which would unblock wpa_supplicant is never called. Oddly, this part seems to work fine without my changes to the DMA memory allocations. Do you have any ideas what might be wrong?
Cheers Christian
Hi Christian,
* Christian Menard <christian.menard@...104...> [2014-12-02 21:07:48 +0300]:
Thank you for the explanation. I traced down two such allocs and changed the flag accordingly. Now the log is error free (besides the "slab too large" messages mentioned before). However, as it appears the driver is not working. I get the message "iwl driver loaded" but then nothing happens anymore. I guess there should be more output for searching the network, establishing connection etc?
Well, it is somewhat unfortunate that even if the driver fails to load properly there is still âiwl driver loadedâ printedâ¦
I tried to track down the problem, but so far I only found that wpa_supplicant blocks on it's socket in netlink_init(). The _handle() function which would unblock wpa_supplicant is never called. Oddly, this part seems to work fine without my changes to the DMA memory allocations. Do you have any ideas what might be wrong?
My guess is that the wifi_drv_ep is currently blocked or rather there is a deadlock somewhere involving the task scheduling. A signal is send if the the socket_call interface is used and if the signal is not handled by the entrypoint wpa_supplicant will hang indefinitely as you have already observed.
It is probably worth a shot to look at the task scheduling by setting DEBUG_SCHEDULE to 1 in lx.h. You can also edit Scheduler::schedule() directly and add a print statement on line 175. Since the scheduling of tasks is done periodicly you should see some output in a good case scenario. If not, the wifi_drv_ep is not running properly which is the case if some task blocks w/o invoking the scheduling of another task.
It might work fine w/o your DMA changes because w/o them another code path in the driver code is taken because the DMA operations failed early.
Regards Josef
Hi Christian,
* Christian Menard <christian.menard@...104...> [2014-12-02 21:07:48 +0300]:
Thank you for the explanation. I traced down two such allocs and changed the flag accordingly. Now the log is error free (besides the "slab too large" messages mentioned before). However, as it appears the driver is not working. I get the message "iwl driver loaded" but then nothing happens anymore. I guess there should be more output for searching the network, establishing connection etc?
Well, it is somewhat unfortunate that even if the driver fails to load properly there is still “iwl driver loaded” printed…
I tried to track down the problem, but so far I only found that wpa_supplicant blocks on it's socket in netlink_init(). The _handle() function which would unblock wpa_supplicant is never called. Oddly, this part seems to work fine without my changes to the DMA memory allocations. Do you have any ideas what might be wrong?
My guess is that the wifi_drv_ep is currently blocked or rather there is a deadlock somewhere involving the task scheduling. A signal is send if the the socket_call interface is used and if the signal is not handled by the entrypoint wpa_supplicant will hang indefinitely as you have already observed.
It is probably worth a shot to look at the task scheduling by setting DEBUG_SCHEDULE to 1 in lx.h. You can also edit Scheduler::schedule() directly and add a print statement on line 175. Since the scheduling of tasks is done periodicly you should see some output in a good case scenario. If not, the wifi_drv_ep is not running properly which is the case if some task blocks w/o invoking the scheduling of another task.
It might work fine w/o your DMA changes because w/o them another code path in the driver code is taken because the DMA operations failed early.
Regards Josef
Hello,
I got the driver running! Thanks for the help. The problem turned out to be a page fault due to nullpointer access, witch I did not see as I was only focused on the wifi_drv log... Furthermore I had to fix more DMA memory allocations and another page fault. However, now it's working fine and netperf shows very reasonable performance :)
I will clean up my patches and create a pull request on github, so you can have a look at my changes and decide if you want to merge them in.
Cheers Christian
On Tuesday 02 December 2014 23:24:49 Josef Söntgen wrote:
Hi Christian,
- Christian Menard <christian.menard@...104...> [2014-12-02 21:07:48
+0300]:
Thank you for the explanation. I traced down two such allocs and changed the flag accordingly. Now the log is error free (besides the "slab too large" messages mentioned before). However, as it appears the driver is not working. I get the message "iwl driver loaded" but then nothing happens anymore. I guess there should be more output for searching the network, establishing connection etc?
Well, it is somewhat unfortunate that even if the driver fails to load properly there is still “iwl driver loaded” printed…
I tried to track down the problem, but so far I only found that wpa_supplicant blocks on it's socket in netlink_init(). The _handle() function which would unblock wpa_supplicant is never called. Oddly, this part seems to work fine without my changes to the DMA memory allocations. Do you have any ideas what might be wrong?
My guess is that the wifi_drv_ep is currently blocked or rather there is a deadlock somewhere involving the task scheduling. A signal is send if the the socket_call interface is used and if the signal is not handled by the entrypoint wpa_supplicant will hang indefinitely as you have already observed.
It is probably worth a shot to look at the task scheduling by setting DEBUG_SCHEDULE to 1 in lx.h. You can also edit Scheduler::schedule() directly and add a print statement on line 175. Since the scheduling of tasks is done periodicly you should see some output in a good case scenario. If not, the wifi_drv_ep is not running properly which is the case if some task blocks w/o invoking the scheduling of another task.
It might work fine w/o your DMA changes because w/o them another code path in the driver code is taken because the DMA operations failed early.
Regards Josef
-- Download BIRT iHub F-Type - The Free Enterprise-Grade BIRT Server from Actuate! Instantly Supercharge Your Business Reports and Dashboards with Interactivity, Sharing, Native Excel Exports, App Integration & more Get technology previously reserved for billion-dollar corporations, FREE http://pubads.g.doubleclick.net/gampad/clk?id=164703151&iu=/4140/ostg.cl... _______________________________________________ genode-main mailing list genode-main@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/genode-main
Hello Christian,
On 12/04/14 14:58, Christian Menard wrote:
I got the driver running! Thanks for the help. The problem turned out to be a- page fault due to nullpointer access, witch I did not see as I was only- focused on the wifi_drv log... Furthermore I had to fix more DMA memory- allocations and another page fault. However, now it's working fine and netperf- shows very reasonable performance :)
congratulation, that is great indeed! May I ask what “very reasonable performance” means? We are currently experiencing some performance issues when transmitting frames on 6300 cards while receiving frames is okayish.
I will clean up my patches and create a pull request on github, so you can- have a look at my changes and decide if you want to merge them in.
I am looking forward to the pull request :-)
Cheers Josef
Hi Josef,
congratulation, that is great indeed! May I ask what “very reasonable performance” means? We are currently experiencing some performance issues when transmitting frames on 6300 cards while receiving frames is okayish.
I just noticed that I only looked on the speed for receiving (~8 Mbit/s). I did the test again for sending and got 700 - 800 kbit/s. The numbers also appear different after I did a test with Linux for comparison, it showed up to 90 Mbit/s receiving.
I don't know if it has a relevant meaning, but I also get a lot of messages "dev_error: Current Rate is not valid"
Cheers Christian
Hi Christian,
On 12/04/14 16:34, Christian Menard wrote:
I just noticed that I only looked on the speed for receiving (~8 Mbit/s). I- did the test again for sending and got 700 - 800 kbit/s. The numbers also- appear different after I did a test with Linux for comparison, it showed up to- 90 Mbit/s receiving.
That is about the same performance we are currently getting on the 6300 cards. It is certainly useable if you use the driver for i.e., browsing the WWW as a client but it would be desirable to match the receiving performance also when sending data.
I don't know if it has a relevant meaning, but I also get a lot of messages- "dev_error: Current Rate is not valid"
It is related because in this case the driver only uses the lowest rate possibly to transmit frames which limits the overall bandwidth. To be honest I do not comprehend the whole rate scaling mechanism so far but I am looking into that it at the moment.
All in all we are currently working on getting the driver to perform better so bear with us :-)
Regards Josef
-
Christian Menard -
Josef Söntgen