Hi Tomasz,
thank you very much for your elaborate posting. Your insightful reflections are an interesting read.
In my roadmap for 2020 I stated two areas in which I'm going to work. First was continuing work for supporting different rpi devices. Unfortunately this work did not progress much as the blocker issue with usb handling is still unresolved. At some point in time I felt powerless and left it. At least until now noone else had motivation to fight with it so this work is stalled.
Even though there was a brief period of activity when Stefan made some progress, the topic [1] became stale again. I understand the slip of motivation. Even though I have enabled the Rpi (including working USB) originally, I would not consider picking it up again in my spare time. I'm sorry that the support from our side was insufficient for you to carry on. Unfortunately, I doubt that 2021 will look much different.
[1] https://github.com/genodelabs/genode/issues/3247
During this process once again I realized why I don't like build system based on make and especially after I found out that recursive make in case of Genode build system can be written with capitalized "R" in recursive :-).
Please let me address the statement regarding "recursive make" in Genode.
What is commonly referred to as a unholy practice "recursive make" is the use of Makefiles scattered throughout a directory structure where the Makefile of the parent directory triggers the Makefiles of the child directories. Genode's build system does not do that. The instances of make spawned by the build system serves different and orthogonal purposes.
- The frontend (Makefile in the build directory) consumes the build configuration and triggers lib-dependency stage, the builds, and the invocation of run scripts.
- The library-dependency stage traverses target description files without building anythin, to construct a library-dependency graph. This could strictly be taken as an example of "recursive make" if we decided to split hairs.
- Orchestration of the library and target builds (executing the library-dependency graph, which happens to be a Makefile).
- Library build, building a single library.
- Target build, building a single target.
If you add the depot tools to the picture, this story continues because the depot tools are, well, based on make and invoke Genode's make-based build system.
So yes, Genode relies on make in a nested fashion. But it does not operate in the way commonly described as "Recursive Make Considered Harmful".
Maybe I'm just incompatible with make but trying to understand details of it was so hard for me, that one more time I started thinking if it would be possible have this build system implemented/converted to scons, in what ways it would be better and what would be worse.
It has to come down to different tastes. Build systems - like programming languages, coding styles, and editors - tend to invoke emotional responses in programmers. After all, the personal background, familiarity, aesthetic sense vary from person to person.
I'm not in the position to question anyone's taste. But I feel very strongly about consistency. In my (strong) opinion, the true mastering of a few orthogonal tools is preferable to large tool box of casually used tools. It takes time to reach the level of craftsmanship with a tool. Even though I repulsed against make for a long time, I once took the deliberate decision to *embrace* it despite its shortcomings. Since I embraced it, I had a chance to master it, and now I apply it consistently and happily.
It is not unreasonable to prefer Scons over Make, or Python over Tcl, or JSON over XML. But regardless of the quality of a tool, adding a non-orthogonal tool to the toolbox spoils the consistency of the whole unless the new tool replaces an existing tool completely. I don't see that coming in the case of make for Genode.
The current tool set of Coreutils, Make, Tcl, XML, C++ may not be perfect. But we apply them consistently and competently. To justify the huge investment of retraining our tool proficiency towards the consistent use of Python replacing make, there ought to be hard business arguments.
Please don't take the words above critique. I admire that you actually implement the improvements, instead of solely expressing your discomfort with Genode's tools. I just hope that you won't be let down by me not sharing your enthusiasm with switching away from make for Genode's build system.
In consequence I started my own project to create a scons based build system for Genode. My plan was to create one and use it myself for rpi development, fix all issues that I really dislike about current build and verify if there are some problems that I don't see that would reject the whole idea. Given the simplicity of most of makefiles in Genode I did not expect that this small part will be complex enough that implementation will take me half of the year.
The most important things I do not like in current build system are:
- big recompilations after touching makefiles and after switching branches
I have to admit that my use of ccache made me blind to see this pain point. We completely missed to cover ccache in our documentation. It may be worthwhile to streamline the use of ccache with Genode's build system similar to how one can enable parallel builds in the build.conf file.
- cases when leftovers in build directory caused hard to diagnose build problems
I sometimes hit this problem, in particular when moving header files around. It is inherent to the use of dependency files. Scons notably goes a different way.
In practice, I would not dramatize these corner cases. In relation to the developer time sunk into all kinds of other issues, I perceive it a negligible inconvenience.
- lack of easy ways to debug what is going on inside make (or maybe it's just my inability to comprehend it)
Make is a can of worms, no question. Please excuse my polemics, but this argument can go towards all kinds of power tools [2]. ;-)
[2] https://github.com/satwikkansal/wtfpython
My main goals were:
implement build in a way that would be able to use current build description with as little changes in Genode repository as possible (currently only two symlinks have to be created and one "marker" file in each build directory to specify that this build directory is built with scons)
implement a verification mechanisms to be sure that scons build is 100% correct replacement of current build (I build database of commands from both types of build and compare them list of targets and that commands are identical - after some normalization)
remove all absolute paths in compilation commands - they are used for all paths in current build - and they are bad for reproducible builds (which should be a goal for a project at some point in time), bad for using ccache or similar tools and bad for defining a project in IDE
be able to understand and retrieve all informations about build to be able to use it in external tools ide and possibly other like static analyzers
If I understand these points correctly, you are actually not after replacing Genode's build system but rather supplementing scons as another option? That would side-step the friction I feared, which would be pretty cool.
Current state and missing parts:
above goals are achieved for build of almost all target on linux build on sources from release 20.08 with repositories enabled by default (base, base-linux, os, demo). First attempts for arm6 shown only some minor inconsistencies in command flags so it should quickly go forward
missing any installation and usage documentation
need to verify on 20.11 release and current master
I hope to finish those parts quickly and make it public this year.
I'm looking forward to try it out.
Plans for next year
I plan to further develop build system based on scons at least until it fullfills my own whishes about it or until I find some blocker that I won't be able to overcome. As originally I plan to use it for development for rpi. I know that I'll have to add some support for ports (I plan only wrapper for current descriptions), depots and possibly something more.
I wonder, have you considered adding scons support to the Goa tool [3]? Goa is completely independent from Genode's build system and already supports make, cmake, autoconf, and qmake.
[3] https://github.com/nfeske/goa
Depending on results of this experiment and reception of Genode team and community I will throw it away, use it myself or maybe develop further features and fixes - I'm curious how it will be.
On the rpi front I still plan to work on it. I plan some work on rpi4 (which is not affected by current usb blocker) and hopefully other variants too.
Thanks for sharing these plans. Let's cross fingers about USB on rpi4.
Comment on Norman's plans
All of your plans are really great.
Plan on supporting PinePhone is definitely appealing for broader audience. I'll probably buy one in case of success and see how it works.
Optimization attempts also seem to be a great target. Improving efficiency will probably further improve code quality and will require more tooling that will allow to constantly test for more kinds of regresssions - especially those related to efficiency.
But for me ideas around impremovements of DDE seem to be most interesting. Especially that I had my own ideas that I queued to explore for myself. During driver porting attempts I felt that amount of code that has to be duplicated (even though it was just dummy stubs) was too big. If I refresh this knowledge during work on rpi I'll definitely like to participate in discussion about it. Porting device drivers from Linux seem to me a very important aspect.
I'm happy to see your resonance to my plans. It's especially cool that you want to join the DDE-Linux line of work!
Cheers Norman