libc blocking functions/with_libc inside thread

Thu Jun 29 16:43:44 CEST 2017

Hi,

Christian, thanks a lot for your speedy refactoring of the OpenVPN port to run the OpenVPN code in the main thread. The OpenVPN code no longer blocks on opening a socket and now tries to setup a VPN connection with the configured server. Unfortunately we are now stumbling upon two new problems.

With OpenVPN configured to use UDP, the OpenVPN component starts the TLS handshake but fails. After some debugging we noticed a pattern of retransmissions by the OpenVPN client. It appears to us that the OpenVPN client cannot read incoming packets from the socket until after (again) writing to the socket (which happens due to retransmission after timeout). If you are interested, take a look at the attached pcap in Wireshark and notice the duplication of messages. For reference I also added a pcap of the OpenVPN port on 16.05.

Also we notice that OpenVPN reads on the socket are non-blocking, proven by the massive amount of READ (len -1) debug messages. This was previously not the case.

With OpenVPN configured to use TCP, the TLS handshake and key-exchange passes successfully, yielding an OpenVPN connection between both client and server. We would now expect the corresponding Nic session to become available for the Genode client that issued the Nic session request, but this is not the case. Instead the client blocks on the creation of the Nic::Connection indefinitely. In the OpenVPN server Root::_create_session returns and the Root calls _ep.manage(..) etc.. What could keep the constructor of Nic::Connection blocking? Is this somehow related to the new asynchronous session creation process?

Met vriendelijke groet / kind regards,

Martijn Verschoor

Cyber Security Labs B.V. | Gooimeer 6-31 | 1411 DD Naarden | The Netherlands
+31 35 631 3253 (office) |  +31 616 014 087 (mobile)

-------------- next part --------------
A non-text attachment was scrubbed...
Name: Genode_openvpn_udp_retransmissions.pcapng
Type: application/octet-stream
Size: 14528 bytes
Desc: not available
URL: <http://lists.genode.org/pipermail/users/attachments/20170629/03fc6dde/attachment.obj>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: Genode_openvpn-16.05.pcapng
Type: application/octet-stream
Size: 18736 bytes
Desc: not available
URL: <http://lists.genode.org/pipermail/users/attachments/20170629/03fc6dde/attachment-0001.obj>
-------------- next part --------------

On 26 Jun 2017, at 16:17, Christian Helmuth <Christian.Helmuth at ...1...> wrote:

> Hello Boris,
> 
> On Mon, Jun 26, 2017 at 02:46:08PM +0200, Boris Mulder wrote:
>> Actually the OpenVPN code hangs once it calls the libc socket()
>> function. Internally, this function calls a blocking write(), and this
>> write() is handled by Libc::Kernel.
> 
> thanks to your provided test case and the hint with "blocking write" I
> was able to validate my suspicion about the blocker in your scenario.
> A rough sketch of my solution can be found here
> 
>  https://github.com/chelmuth/genode/commits/openvpn_17.05.
> 
> The issue is the unfortunate interplay of I/O-signal handling in the
> initial entrypoint and the current implementation of the VFS plugin,
> which interfaces with our file-system session. In the case of
> "blocking write" the VFS plugin calls
> wait_and_dispatch_one_io_signal() directly on the initial entrypoint.
> In your scenario this results in the initial-entrypoint thread and the
> OpenVPN thread racing on the handling of first I/O signal. As the
> entrypoint always wins, the OpenVPN thread is blocked until another
> I/O signal occurs (which may never happen in the startup phase).
> 
> The sketched solution just reverses the roles of the first and second
> application thread. Now, the initial entrypoint implements OpenVPN
> (handling its own I/O signals) and the additional entrypoint
> implements the NIC server (with root and session component).
> 
> I hope this helps.
> 
> Greets
> -- 
> Christian Helmuth
> Genode Labs
> 
> https://www.genode-labs.com/ · https://genode.org/
> https://twitter.com/GenodeLabs · /ˈdʒiː.nəʊd/
> 
> Genode Labs GmbH · Amtsgericht Dresden · HRB 28424 · Sitz Dresden
> Geschäftsführer: Dr.-Ing. Norman Feske, Christian Helmuth
> 
>> So openVPN does not send or receive any packet yet as it is blocked at
>> socket().
>> 
>> Earlier, we have used lwip as a socket library. When we did that,
>> socket() (and connect() in TCP mode) did work, but it failed to send any
>> initial data to the server, likewise blocking on some function.
>> 
>> We are reaching the limit of our knowledge of genode libc and the
>> side-effects of the asynchronous entrypoint. At this point our debugging
>> went down into the libc kernel and there is a limit how deep we can go.
>> Help on this topic would be appreciated.
>> 
>> We uploaded the new 17.05 ready code of openVPN (including a run script
>> which can be run through make run/openvpn) onto
>> https://github.com/nlcsl/genode/tree/openvpn_17.05 .
>> 
>> If you have the time, could you try to run it and see if it is possible
>> to let it produce a single UDP packet? For this, it is not necessary to
>> setup a server. From there, we could pick it up again.
>> 
>> We appreciate it,
>> 
>> Boris
>> 
>> 
>> On 26-06-17 10:57, Christian Helmuth wrote:
>>> Hello Boris,
>>> 
>>> On Fri, Jun 23, 2017 at 03:59:53PM +0200, Boris Mulder wrote:
>>>> The entrypoint creates the root component, spawns the thread and
>>>> returns. It will then handle RPC requests, as entrypoints do IIRC.
>>>> 
>>>> The program acts as a server (serving Nic sessions asynchronously) and
>>>> as a client to lxip vfs with libc. the code can be found in [1].
>>>> 
>>>> How can I have the entrypoint handle I/O signals in libc while also
>>>> being able to serve clients in Genode?
>>> This should happen automatically under the hood as libc processes
>>> signals in ordinary I/O signal handlers in the entrypoint.
>>> 
>>> Are you able to run the scenario under linux and inspect the
>>> processing of both threads via GDB? I fear that I cannot help with
>>> specifics of OpenVPN, but may guide with more details about the
>>> blocking situation. It may be interesting to know if any network
>>> packets reach the OpenVPN code.
>>> 
>>> Greets
>> 
>> -- 
>> 
>> Met vriendelijke groet / kind regards,
>> 
>> Boris Mulder
>> 
>> Cyber Security Labs B.V. | Gooimeer 6-31 | 1411 DD Naarden | The Netherlands
>> +31 35 631 3253 (office)
>> 
>> 
>> 
>> ------------------------------------------------------------------------------
>> Check out the vibrant tech community on one of the world's most
>> engaging tech sites, Slashdot.org! http://sdm.link/slashdot
>> _______________________________________________
>> genode-main mailing list
>> genode-main at lists.sourceforge.net
>> https://lists.sourceforge.net/lists/listinfo/genode-main
> 
> ------------------------------------------------------------------------------
> Check out the vibrant tech community on one of the world's most
> engaging tech sites, Slashdot.org! http://sdm.link/slashdot
> _______________________________________________
> genode-main mailing list
> genode-main at lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/genode-main