Many programs and libraries call other processes. For example, libarchive uses external programs for most of its compression filters, including gzip, bzip2, and xz. Any IDE (e.g. Qt Creator) needs to call an external compiler and the program being written. Genode doesn't seem to have this feature, except in noux, making it unavailable to multi-threaded programs. How can we make this work?
Here are the main options I see: 1. extend noux with the features needed (primarily pthreads and access to Genode services such as Nitpicker) to support modern posix applications 2. extend libc with noux-like functionality, including a way to create child processes
In issue #1837 (https://github.com/genodelabs/genode/issues/1837), Norman Feske voiced his opposition to the first option, with the following argument:
"In the context of Genode, the only benefit we would get from adding pthread support to Noux would be the support of programs that rely on both fork and pthreads, and that are not portable enough to support a non-POSIX backend. I argue that this applies to very few programs. Should we sacrifice the simplicity of Noux to the few programs that fall in this category? Personally, I would not open this can of worms."
However, fork() followed by execve() seems to be the standard method of executing external processes on POSIX systems. This is the method used in libarchive, and more importantly in Qt 5. Multithreading is required in Qt applications (unless you compile Qt with QT_NO_THREAD, and multithreading isn't used anywhere in the application code), so in order for a Qt-based IDE or archive manager to work, we need fork() and execve(), or else another method of running and monitoring external libc-based programs.
As far as I can tell, here are the required tasks for each option:
Option 1 (extend noux): 1. modify noux to support pthreads 2. modify noux to provide access to Nitpicker (and probably other external services)
Option 2 (extend libc): 1. add functions to run libc programs as child processes with argument handling, stdin/stdout redirection, process control via PIDs (much like in noux) 2. patch ported programs and libraries (e.g. Qt, libarchive) to use that function
For option 2, we could create our own functions, or we could implement posix_spawn() or vfork() and execve().
I'm leaning toward option 2 with vfork() and execve(), because it should work as a libc plugin, and posix_spawn() and popen() use those functions under the hood. Here's my pseudocode:
/* simple vfork()/execve() pseudocode that doesn't actually create a new process when vfork() is called */ /* this excludes the required PID system and create_child() function */ int vfork() { enter_vfork(); if (in_vfork) return 0; else return pid; }
int execve(...) { if (!in_vfork) fail;
create_child(pid, ...); /* possibly under noux */ exit_vfork(); }
void enter_vfork() { suspend_other_threads(); save_pid(); in_vfork = 1; generate_pid(); save_program_counter(); }
void exit_vfork() { in_vfork = 0; restore_pid(); resume_other_threads(); restore_program_counter(); }
Am I missing something? Any thoughts or suggestions?