process execution from libc

[Nobody III]

Many programs and libraries call other processes. For example, libarchive
uses external programs for most of its compression filters, including gzip,
bzip2, and xz. Any IDE (e.g. Qt Creator) needs to call an external compiler
and the program being written. Genode doesn't seem to have this feature,
except in noux, making it unavailable to multi-threaded programs. How can
we make this work?

Here are the main options I see:
1. extend noux with the features needed (primarily pthreads and access to
Genode services such as Nitpicker) to support modern posix applications
2. extend libc with noux-like functionality, including a way to create
child processes

In issue #1837 (https://github.com/genodelabs/genode/issues/1837), Norman
Feske voiced his opposition to the first option, with the following
argument:

"In the context of Genode, the only benefit we would get from adding
pthread support to Noux would be the support of programs that rely on both
fork and pthreads, and that are not portable enough to support a non-POSIX
backend. I argue that this applies to very few programs. Should we
sacrifice the simplicity of Noux to the few programs that fall in this
category? Personally, I would not open this can of worms."

However, fork() followed by execve() seems to be the standard method of
executing external processes on POSIX systems. This is the method used in
libarchive, and more importantly in Qt 5. Multithreading is required in Qt
applications (unless you compile Qt with QT_NO_THREAD, and multithreading
isn't used anywhere in the application code), so in order for a Qt-based
IDE or archive manager to work, we need fork() and execve(), or else
another method of running and monitoring external libc-based programs.

As far as I can tell, here are the required tasks for each option:

Option 1 (extend noux):
1. modify noux to support pthreads
2. modify noux to provide access to Nitpicker (and probably other external
services)

Option 2 (extend libc):
1. add functions to run libc programs as child processes with argument
handling, stdin/stdout redirection, process control via PIDs (much like in
noux)
2. patch ported programs and libraries (e.g. Qt, libarchive) to use that
function

For option 2, we could create our own functions, or we could implement
posix_spawn() or vfork() and execve().

I'm leaning toward option 2 with vfork() and execve(), because it should
work as a libc plugin, and posix_spawn() and popen() use those functions
under the hood. Here's my pseudocode:

/* simple vfork()/execve() pseudocode that doesn't actually create a new
process when vfork() is called */
/* this excludes the required PID system and create_child() function */
int vfork()
{
enter_vfork();
if (in_vfork)
return 0;
else
return pid;
}

int execve(...) {
if (!in_vfork)
fail;

create_child(pid, ...); /* possibly under noux */
exit_vfork();
}

void enter_vfork() {
suspend_other_threads();
save_pid();
in_vfork = 1;
generate_pid();
save_program_counter();
}

void exit_vfork() {
in_vfork = 0;
restore_pid();
resume_other_threads();
restore_program_counter();
}

Am I missing something? Any thoughts or suggestions?
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.genode.org/pipermail/users/attachments/20180207/3ad841b4/attachment.html>