process execution from libc

[Nobody III]

I definitely like the idea of integrating noux functionality into libc. The
most obvious barrier that I see is Genode's recursive structure and the
comparatively flat POSIX structure. In particular, there is no way (AFAIK)
for a process to create a sibling or sibling-like process without the
parent process's support. How can we overcome this barrier without a shared
parent (noux)?

Also, what is your rough plan so far, Norman?

On Sat, Feb 10, 2018 at 6:59 AM, Norman Feske <norman.feske at ...1...>
wrote:

> Hi Ben,
>
> On 07.02.2018 16:55, Nobody III wrote:
> > Many programs and libraries call other processes. For example,
> > libarchive uses external programs for most of its compression filters,
> > including gzip, bzip2, and xz. Any IDE (e.g. Qt Creator) needs to call
> > an external compiler and the program being written. Genode doesn't seem
> > to have this feature, except in noux, making it unavailable to
> > multi-threaded programs. How can we make this work?
>
> there are two directions:
>
> In the short term, the only reasonable way to spawn external programs in
> non-noux applications is the use of the Genode 'Child' API (as used by
> the slave examples). This requires one to change third party software
> that depends on such a mechanism. In practice, so far, I hardly
> encountered a situation where this problem cannot be worked around.
> E.g., in the case of libarchive, the dependency from external programs
> is an option. Libarchive can use libraries for almost all archive types
> directly. Complex programs like QtCreator use a higher-level application
> framework (i.e., Qt), which abstract-away fork/execve anyway. This gives
> us the opportunity to add a Genode-specific process-creation mechanism
> in the backend of such an application framework.
>
> As the second direction, we should strive to remove possible points of
> friction when using 3rd-party software directly on Genode. Fork/execve
> is a major point of friction right now. Hence, we should definitely
> overcome the current limitation. I have a rough plan for implementing
> this features in our libc. But since this is not easy and there are more
> pressing topics, this feature has to wait for now.
>
> > Option 1 (extend noux):
> > 1. modify noux to support pthreads
> > 2. modify noux to provide access to Nitpicker (and probably other
> > external services)
> >
> > Option 2 (extend libc):
> > 1. add functions to run libc programs as child processes with argument
> > handling, stdin/stdout redirection, process control via PIDs (much like
> > in noux)
> > 2. patch ported programs and libraries (e.g. Qt, libarchive) to use that
> > function
> >
> > For option 2, we could create our own functions, or we could implement
> > posix_spawn() or vfork() and execve().
> >
> > I'm leaning toward option 2 with vfork() and execve(), because it should
> > work as a libc plugin, and posix_spawn() and popen() use those functions
> > under the hood. Here's my pseudocode:
>
> I agree. Actually, when speaking about the longer term, I'd like to
> remove noux. Historically (and shamelessly over-simplifying), Noux
> provided us with two important functionalities: POSIX-like process
> creation (execve/fork) and the VFS. Since the latter has found its way
> into our libc, only the execve/fork mechanism justifies the existence of
> noux today. If we manage to move those to the libc, we won't need noux
> anymore.
>
> The posix_spawn method looks particularly attractive since it avoids
> fork. However, many programs like bash depend on fork/execve. So I
> regard posix_spawn not as an alternative, but rather an optimization. We
> still need to provide execve/fork.
>
> > Am I missing something? Any thoughts or suggestions?
>
> Yes! You are missing the elephant in the room, which is the
> implementation of execve/fork in a way that works across all kernels
> supported by Genode. The mechanism implemented in noux is almost black
> magic to even the most regular Genode developers - and it does not even
> work on all kernels (Linux is not supported). To give you an impression,
> when speaking about adding execve/fork in the libc, one has to solve the
> following problems:
>
> * How to mirror the forking virtual address space into a new protection
>   domains? Microkernels do not support copy-on-write address spaces.
>
> * How to share the file descriptors of the forking program with the new
>   one? Genode does not even have the notion of file descriptors.
>
> * How to account the resources consumed by the child? On Unix, the
>   kernel pays for everything. On Genode, resources are traded between
>   components at user level.
>
> These are quite fundamental contraditions. Noux provides an answer to
> those questions. But without noux, we have to find new answers. As I
> hinted above, I have a rough idea in the back of my mind. But it is far
> too blurry to call it a plan.
>
> Cheers
> Norman
>
> --
> Dr.-Ing. Norman Feske
> Genode Labs
>
> https://www.genode-labs.com · https://genode.org
>
> Genode Labs GmbH · Amtsgericht Dresden · HRB 28424 · Sitz Dresden
> Geschäftsführer: Dr.-Ing. Norman Feske, Christian Helmuth
>
> ------------------------------------------------------------
> ------------------
> Check out the vibrant tech community on one of the world's most
> engaging tech sites, Slashdot.org! http://sdm.link/slashdot
> _______________________________________________
> genode-main mailing list
> genode-main at lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/genode-main
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.genode.org/pipermail/users/attachments/20180210/f8c209c3/attachment.html>