Hello Valery,
I have the impression that your undertaking goes very much against the grain of our regular work flow. Frankly speaking, I miss the point of bypassing our tooling and instead manually working with boot modules. We deliberately moved away from relying on the boot loader to load individual ROM modules, for a number of reasons:
1. Not all kernels provide the a way for roottask (Genode's core) to access individual boot modules. In particular seL4 or OKL4 do not.
2. On ARM there is not such concept. Boot loaders on ARM load a kernel image only.
3. For the reasons above, Genode has to provide a way to include the initial ROM modules in the boot image. Using this mechanism across all base platforms reduces Genode's complexity and ensures that the solution is well tested. In contrast, the prior used kernel-specific code was more fragile.
4. We hit limitations of several multi-boot loaders. E.g., I am thinking of the GRUB's maximum number of boot modules. With iPXE, we hit different surprises such as a slightly different convention of how boot modules are named. By using one unified mechanism, we rule out these sources of trouble.
5. Shuffling boot modules manually is bug prone. By letting a run script generate one image that contains all needed ingredients in their current version eliminates the chance for inconsistencies between the modules.
In short, we went into the deep end, experienced the limits of the multi-boot approach, and decided for a more robust approach. Our tooling reflects that. It makes it arguably difficult to edit init's configuration on the fly - as you noted - and requires you to execute the run script after each change. You present this as a limitation. But I regard the approach of mutating the boot image manually as misguided. It is not only bug prone but also evades the versioning of the individual modifications. In contrast, if you embrace the work with run scripts, you can always reproduce your scenario and naturally track modifications using Git.
PS: A feature request for Genode build system: May be, it would be good to add options for etc/build.conf, to disable removing the "genode" subdirectory, containing all the binaries, plus optionally generate "core" together with "image.elf", so that it will generate a core image without embedded modules. It would be more convenient if a developer wants to copy modules to some GRUB installation manually. So, one can run the run script to generate both the "image.elf" and separate binaries. This would allow both to 1) deploy the scenario on test machine automatically and 2) copy the scenario manually to GRUB config file. For manual copying, it would be more convenient to have separate binaries, so they are not duplicated in each scenario image.elf images, which both take much space on disk, and the same binaries can be reused in many scenarios. Also, with all modules built into the "image.elf", it is not very comfortable to edit the "config" files without regenerating the "image.elf".
The latter point was indeed a concern we had when unifying the boot-module handling. However, on closer inspection, we found three possible scenarios where the mechanism is used:
1. The majority of run scripts are test cases. They are small and are executed ad-hoc (or automatically) but are never permanently installed. So sharing binaries across scenarios would not give any benefit.
2. Run scripts that describe self-sustaining systems, like the Turmvilla scenario. Here we have a large base system consisting of many boot modules, maybe even including virtual disk images. This situation likely corresponds to your's. It would be nice to reuse selected boot modules (like a virtual disk image) between scenarios. The single-image approach is clearly limiting.
3. Run scripts that create the boot image of a multi-stage scenario, like the Sculpt scenario. Here, the boot image contains merely the components needed to bootstrap a second stage from within Genode. The initial boot image features a block-device driver, file system, and fs-rom server. The interesting part happens at a second stage where the Genode system can access information from the disk directly.
Of these three cases, only the second one would really benefit from loading individual ROM modules as multi-boot modules. Based on our experience with Turmvilla, we figured that scenarios of this type - where a complex system is bootstrapped by the boot loader only - do not scale well. Since the direction is inherently limiting, we should stop pursing it but instead embrace systems of the third type.
With Sculpt as the most prominent example of third type, one can already see the benefits. Thanks to fetching all ingredients from the depot, executing the run script is fast. The resulting boot image is quite small (less than 20 MiB) and will stay small even when the second-stage system grows. The potential benefit of sharing parts of it between multiple scenario is negligible.
I hope that this background sheds some light on our line of thoughts that went into the boot-module handling of Genode. Please understand that we won't like to go back to supporting earlier approaches that haven't worked out for us.
Cheers Norman