Hi Ben,
On 07.02.2018 16:55, Nobody III wrote:
Many programs and libraries call other processes. For example, libarchive uses external programs for most of its compression filters, including gzip, bzip2, and xz. Any IDE (e.g. Qt Creator) needs to call an external compiler and the program being written. Genode doesn't seem to have this feature, except in noux, making it unavailable to multi-threaded programs. How can we make this work?
there are two directions:
In the short term, the only reasonable way to spawn external programs in non-noux applications is the use of the Genode 'Child' API (as used by the slave examples). This requires one to change third party software that depends on such a mechanism. In practice, so far, I hardly encountered a situation where this problem cannot be worked around. E.g., in the case of libarchive, the dependency from external programs is an option. Libarchive can use libraries for almost all archive types directly. Complex programs like QtCreator use a higher-level application framework (i.e., Qt), which abstract-away fork/execve anyway. This gives us the opportunity to add a Genode-specific process-creation mechanism in the backend of such an application framework.
As the second direction, we should strive to remove possible points of friction when using 3rd-party software directly on Genode. Fork/execve is a major point of friction right now. Hence, we should definitely overcome the current limitation. I have a rough plan for implementing this features in our libc. But since this is not easy and there are more pressing topics, this feature has to wait for now.
Option 1 (extend noux):
- modify noux to support pthreads
- modify noux to provide access to Nitpicker (and probably other
external services)
Option 2 (extend libc): 1. add functions to run libc programs as child processes with argument handling, stdin/stdout redirection, process control via PIDs (much like in noux) 2. patch ported programs and libraries (e.g. Qt, libarchive) to use that function
For option 2, we could create our own functions, or we could implement posix_spawn() or vfork() and execve().
I'm leaning toward option 2 with vfork() and execve(), because it should work as a libc plugin, and posix_spawn() and popen() use those functions under the hood. Here's my pseudocode:
I agree. Actually, when speaking about the longer term, I'd like to remove noux. Historically (and shamelessly over-simplifying), Noux provided us with two important functionalities: POSIX-like process creation (execve/fork) and the VFS. Since the latter has found its way into our libc, only the execve/fork mechanism justifies the existence of noux today. If we manage to move those to the libc, we won't need noux anymore.
The posix_spawn method looks particularly attractive since it avoids fork. However, many programs like bash depend on fork/execve. So I regard posix_spawn not as an alternative, but rather an optimization. We still need to provide execve/fork.
Am I missing something? Any thoughts or suggestions?
Yes! You are missing the elephant in the room, which is the implementation of execve/fork in a way that works across all kernels supported by Genode. The mechanism implemented in noux is almost black magic to even the most regular Genode developers - and it does not even work on all kernels (Linux is not supported). To give you an impression, when speaking about adding execve/fork in the libc, one has to solve the following problems:
* How to mirror the forking virtual address space into a new protection domains? Microkernels do not support copy-on-write address spaces.
* How to share the file descriptors of the forking program with the new one? Genode does not even have the notion of file descriptors.
* How to account the resources consumed by the child? On Unix, the kernel pays for everything. On Genode, resources are traded between components at user level.
These are quite fundamental contraditions. Noux provides an answer to those questions. But without noux, we have to find new answers. As I hinted above, I have a rough idea in the back of my mind. But it is far too blurry to call it a plan.
Cheers Norman