Roadmap 2021

[Norman Feske]

Hi Tomasz,

thank you very much for your elaborate posting. Your insightful
reflections are an interesting read.

> In my roadmap for 2020 I stated two areas in which I'm going to
> work. First was continuing work for supporting different rpi
> devices. Unfortunately this work did not progress much as the blocker
> issue with usb handling is still unresolved. At some point in time I
> felt powerless and left it. At least until now noone else had motivation
> to fight with it so this work is stalled.

Even though there was a brief period of activity when Stefan made some
progress, the topic [1] became stale again. I understand the slip of
motivation. Even though I have enabled the Rpi (including working USB)
originally, I would not consider picking it up again in my spare time.
I'm sorry that the support from our side was insufficient for you to
carry on. Unfortunately, I doubt that 2021 will look much different.

[1] https://github.com/genodelabs/genode/issues/3247

> During this process once again I realized why I don't like build system
> based on make and especially after I found out that recursive make in
> case of Genode build system can be written with capitalized "R" in
> recursive :-).

Please let me address the statement regarding "recursive make" in Genode.

What is commonly referred to as a unholy practice "recursive make" is
the use of Makefiles scattered throughout a directory structure where
the Makefile of the parent directory triggers the Makefiles of the child
directories. Genode's build system does not do that. The instances of
make spawned by the build system serves different and orthogonal purposes.

- The frontend (Makefile in the build directory) consumes the build
  configuration and triggers lib-dependency stage, the builds, and the
  invocation of run scripts.

- The library-dependency stage traverses target description files
  without building anythin, to construct a library-dependency graph.
  This could strictly be taken as an example of "recursive make" if
  we decided to split hairs.

- Orchestration of the library and target builds (executing the
  library-dependency graph, which happens to be a Makefile).

- Library build, building a single library.

- Target build, building a single target.

If you add the depot tools to the picture, this story continues because
the depot tools are, well, based on make and invoke Genode's make-based
build system.

So yes, Genode relies on make in a nested fashion. But it does not
operate in the way commonly described as "Recursive Make Considered
Harmful".

> Maybe I'm just incompatible with make but trying to
> understand details of it was so hard for me, that one more time I
> started thinking if it would be possible have this build system
> implemented/converted to scons, in what ways it would be better and
> what would be worse.

It has to come down to different tastes. Build systems - like
programming languages, coding styles, and editors - tend to invoke
emotional responses in programmers. After all, the personal background,
familiarity, aesthetic sense vary from person to person.

I'm not in the position to question anyone's taste. But I feel very
strongly about consistency. In my (strong) opinion, the true mastering
of a few orthogonal tools is preferable to large tool box of casually
used tools. It takes time to reach the level of craftsmanship with a
tool. Even though I repulsed against make for a long time, I once took
the deliberate decision to *embrace* it despite its shortcomings. Since
I embraced it, I had a chance to master it, and now I apply it
consistently and happily.

It is not unreasonable to prefer Scons over Make, or Python over Tcl, or
JSON over XML. But regardless of the quality of a tool, adding a
non-orthogonal tool to the toolbox spoils the consistency of the whole
unless the new tool replaces an existing tool completely. I don't see
that coming in the case of make for Genode.

The current tool set of Coreutils, Make, Tcl, XML, C++ may not be
perfect. But we apply them consistently and competently. To justify the
huge investment of retraining our tool proficiency towards the
consistent use of Python replacing make, there ought to be hard business
arguments.

Please don't take the words above critique. I admire that you actually
implement the improvements, instead of solely expressing your discomfort
with Genode's tools. I just hope that you won't be let down by me not
sharing your enthusiasm with switching away from make for Genode's build
system.

> In consequence I started my own project to create a scons based build
> system for Genode. My plan was to create one and use it myself for rpi
> development, fix all issues that I really dislike about current build
> and verify if there are some problems that I don't see that would reject
> the whole idea. Given the simplicity of most of makefiles in Genode I
> did not expect that this small part will be complex enough that
> implementation will take me half of the year.
> 
> The most important things I do not like in current build system are:
> 
>  * big recompilations after touching makefiles and after switching
>    branches

I have to admit that my use of ccache made me blind to see this pain
point. We completely missed to cover ccache in our documentation. It may
be worthwhile to streamline the use of ccache with Genode's build system
similar to how one can enable parallel builds in the build.conf file.

>  * cases when leftovers in build directory caused hard to diagnose build
>    problems

I sometimes hit this problem, in particular when moving header files
around. It is inherent to the use of dependency files. Scons notably
goes a different way.

In practice, I would not dramatize these corner cases. In relation to
the developer time sunk into all kinds of other issues, I perceive it a
negligible inconvenience.

>  * lack of easy ways to debug what is going on inside make (or maybe
>    it's just my inability to comprehend it)

Make is a can of worms, no question. Please excuse my polemics, but this
argument can go towards all kinds of power tools [2]. ;-)

[2] https://github.com/satwikkansal/wtfpython

> My main goals were:
> 
>  * implement build in a way that would be able to use current build
>    description with as little changes in Genode repository as possible
>    (currently only two symlinks have to be created and one "marker" file
>    in each build directory to specify that this build directory is built
>    with scons)
> 
>  * implement a verification mechanisms to be sure that scons build is
>    100% correct replacement of current build (I build database of
>    commands from both types of build and compare them list of targets
>    and that commands are identical - after some normalization)
> 
>  * remove all absolute paths in compilation commands - they are used for
>    all paths in current build - and they are bad for reproducible builds
>    (which should be a goal for a project at some point in time), bad for
>    using ccache or similar tools and bad for defining a project in IDE
> 
>  * be able to understand and retrieve all informations about build to be
>    able to use it in external tools ide and possibly other like static
>    analyzers

If I understand these points correctly, you are actually not after
replacing Genode's build system but rather supplementing scons as
another option? That would side-step the friction I feared, which would
be pretty cool.

> Current state and missing parts:
> 
>  * above goals are achieved for build of almost all target on linux
>    build on sources from release 20.08 with repositories enabled by
>    default (base, base-linux, os, demo). First attempts for arm6 shown
>    only some minor inconsistencies in command flags so it should quickly
>    go forward
> 
>  * missing any installation and usage documentation
> 
>  * need to verify on 20.11 release and current master
> 
> I hope to finish those parts quickly and make it public this year.

I'm looking forward to try it out.

> Plans for next year
> 
> I plan to further develop build system based on scons at least until it
> fullfills my own whishes about it or until I find some blocker that I
> won't be able to overcome. As originally I plan to use it for
> development for rpi. I know that I'll have to add some support for ports
> (I plan only wrapper for current descriptions), depots and possibly
> something more.

I wonder, have you considered adding scons support to the Goa tool [3]?
Goa is completely independent from Genode's build system and already
supports make, cmake, autoconf, and qmake.

[3] https://github.com/nfeske/goa

> Depending on results of this experiment and reception of Genode team and
> community I will throw it away, use it myself or maybe develop further
> features and fixes - I'm curious how it will be.
> 
> On the rpi front I still plan to work on it. I plan some work on rpi4
> (which is not affected by current usb blocker) and hopefully other
> variants too.

Thanks for sharing these plans. Let's cross fingers about USB on rpi4.

> Comment on Norman's plans
> 
> All of your plans are really great.
> 
> Plan on supporting PinePhone is definitely appealing for broader
> audience. I'll probably buy one in case of success and see how it works.
> 
> Optimization attempts also seem to be a great target. Improving
> efficiency will probably further improve code quality and will require
> more tooling that will allow to constantly test for more kinds of
> regresssions - especially those related to efficiency.
> 
> But for me ideas around impremovements of DDE seem to be most
> interesting. Especially that I had my own ideas that I queued to explore
> for myself. During driver porting attempts I felt that amount of code
> that has to be duplicated (even though it was just dummy stubs) was too
> big. If I refresh this knowledge during work on rpi I'll definitely like
> to participate in discussion about it. Porting device drivers from Linux
> seem to me a very important aspect.

I'm happy to see your resonance to my plans. It's especially cool that
you want to join the DDE-Linux line of work!

Cheers
Norman

-- 
Dr.-Ing. Norman Feske
Genode Labs

https://www.genode-labs.com · https://genode.org

Genode Labs GmbH · Amtsgericht Dresden · HRB 28424 · Sitz Dresden
Geschäftsführer: Dr.-Ing. Norman Feske, Christian Helmuth