process execution from libc

[Norman Feske]

Hi Ben,

On 07.02.2018 16:55, Nobody III wrote:
> Many programs and libraries call other processes. For example,
> libarchive uses external programs for most of its compression filters,
> including gzip, bzip2, and xz. Any IDE (e.g. Qt Creator) needs to call
> an external compiler and the program being written. Genode doesn't seem
> to have this feature, except in noux, making it unavailable to
> multi-threaded programs. How can we make this work?

there are two directions:

In the short term, the only reasonable way to spawn external programs in
non-noux applications is the use of the Genode 'Child' API (as used by
the slave examples). This requires one to change third party software
that depends on such a mechanism. In practice, so far, I hardly
encountered a situation where this problem cannot be worked around.
E.g., in the case of libarchive, the dependency from external programs
is an option. Libarchive can use libraries for almost all archive types
directly. Complex programs like QtCreator use a higher-level application
framework (i.e., Qt), which abstract-away fork/execve anyway. This gives
us the opportunity to add a Genode-specific process-creation mechanism
in the backend of such an application framework.

As the second direction, we should strive to remove possible points of
friction when using 3rd-party software directly on Genode. Fork/execve
is a major point of friction right now. Hence, we should definitely
overcome the current limitation. I have a rough plan for implementing
this features in our libc. But since this is not easy and there are more
pressing topics, this feature has to wait for now.

> Option 1 (extend noux):
> 1. modify noux to support pthreads
> 2. modify noux to provide access to Nitpicker (and probably other
> external services)
> 
> Option 2 (extend libc):
> 1. add functions to run libc programs as child processes with argument
> handling, stdin/stdout redirection, process control via PIDs (much like
> in noux)
> 2. patch ported programs and libraries (e.g. Qt, libarchive) to use that
> function
> 
> For option 2, we could create our own functions, or we could implement
> posix_spawn() or vfork() and execve().
> 
> I'm leaning toward option 2 with vfork() and execve(), because it should
> work as a libc plugin, and posix_spawn() and popen() use those functions
> under the hood. Here's my pseudocode:

I agree. Actually, when speaking about the longer term, I'd like to
remove noux. Historically (and shamelessly over-simplifying), Noux
provided us with two important functionalities: POSIX-like process
creation (execve/fork) and the VFS. Since the latter has found its way
into our libc, only the execve/fork mechanism justifies the existence of
noux today. If we manage to move those to the libc, we won't need noux
anymore.

The posix_spawn method looks particularly attractive since it avoids
fork. However, many programs like bash depend on fork/execve. So I
regard posix_spawn not as an alternative, but rather an optimization. We
still need to provide execve/fork.

> Am I missing something? Any thoughts or suggestions?

Yes! You are missing the elephant in the room, which is the
implementation of execve/fork in a way that works across all kernels
supported by Genode. The mechanism implemented in noux is almost black
magic to even the most regular Genode developers - and it does not even
work on all kernels (Linux is not supported). To give you an impression,
when speaking about adding execve/fork in the libc, one has to solve the
following problems:

* How to mirror the forking virtual address space into a new protection
  domains? Microkernels do not support copy-on-write address spaces.

* How to share the file descriptors of the forking program with the new
  one? Genode does not even have the notion of file descriptors.

* How to account the resources consumed by the child? On Unix, the
  kernel pays for everything. On Genode, resources are traded between
  components at user level.

These are quite fundamental contraditions. Noux provides an answer to
those questions. But without noux, we have to find new answers. As I
hinted above, I have a rough idea in the back of my mind. But it is far
too blurry to call it a plan.

Cheers
Norman

-- 
Dr.-Ing. Norman Feske
Genode Labs

https://www.genode-labs.com · https://genode.org

Genode Labs GmbH · Amtsgericht Dresden · HRB 28424 · Sitz Dresden
Geschäftsführer: Dr.-Ing. Norman Feske, Christian Helmuth