Building Genode with Nix

[Norman Feske]

Hi Emery,

> In what started as an effort to fix some simple problems when building
> Genode on NixOS, I have gone ahead and attempted to replace the Make
> build system with one implemented in Nix.

that is a seriously cool experiment - thanks for sharing it!

> To start out with, here are some problems with a Nix build system:
> 
>  * Nix must be ported to Genode if Genode shall be self hosting.

I was planning to investigate the use of Nix as a package-management
solution on Genode. Porting Nix to Noux (sounds a bit funny) will be
needed then anyway. So I don't see this as a problem.

>  * Each Make target will probably have to be re-expressed in Nix.
>  * Nix produces more arcane output than Make.

Admittedly, I haven't considered Nix as a build-system replacement
because the current make-based build system works quite well. I could
not see the added benefits that Nix would bring as a build tool. Maybe,
your experiment will be eye-opening in this respect. :-)

One point not to forget is that Nix adds a new concept to the whole
picture, which is alien to must people but must be understood by
everyone exploring Genode then. Genode is already quite difficult to
approach. I am a bit afraid that the mandatory use of Nix would further
raise the bar. Maybe my fear is unfounded and Nix would actually
simplify the status quo?

> Some benefits as I see it:
> 
>  * Nix is the only build system dependency.

To see this value, I'd first need to understand the dependencies of Nix
itself. But coreutils + findutils + bash + make is not a very
complicated software stack either, don't you agree?

>  * More build parallelism.

Is this really true? When enabling parallel builds in the Genode build
system (by adding 'MAKE += -j4' to the 'etc/build.conf' file) the actual
build stage gets executed in parallel already, managed by make. Are you
sure that Nix can squeeze more performance out of the build process? I
am a bit skeptical because of the following blog posting (the section
about work):


http://allsoftwaresucks.blogspot.de/2014/06/minor-notes-about-recent-stuff.html

>  * Lazy dependency evaluation means faster rebuilding.
>  * Binary component packages.
>  * NixOS can be nested in the Genode build system.
>  * Genode can be nested in the NixOS build system.

Those points sound very good, especially the possible integration with
NixOS.

I'd like to note that the Genode build system actually employs an
optimization to accelerate rebuilding. At the first stage, the build
systems discovers the subset of libraries needed for the actual build
and generates a makefile for those (the file is generated at
'var/libdeps'). The second stage executes all the targets contained in
the libdeps file but omits everything else. Effectively, this mechanism
prunes the dependency graph to those portions that are really needed. To
accelerate the rebuild time, the first (lib-dependency-generating) stage
can even be omitted by issuing 'make again'.

>  * Less boilerplate.
>  * Less user interaction.

Can you please substantiate those points a little? Are you referring to
the installation of the tool-chain?

> That said, my conversion is far from complete, my focus has been on
> getting the Linux and Nova implemenations working. I have core and init
> building for Linux and Nova as well as a number of tests. The Expect
> framework for executing Genode has been wrapped and works for Linux, but
> I will have to express the Nova kernel before moving on to Qemu support.
> I've been bypassing ARM to avoid getting into the details of the spec
> managment for now.
> 
> 
> If anyone would like to try it out I'm working on the 'nix' branch at
> https://github.com/ehmry/genode.git
> 
> The only depency is Nix which you can get here: http://nixos.org/nix/
> 
> There is a script named 'try_me.sh' that will run the tests that have
> been defined.
> 
> If you wish to explore further the command that you will need to get
> aquainted with is 'nix-build'. For example, to build some components:
>   nix-build ~/genode/specs/x86_64-nova.nix -A base.core
>   nix-build ~/genode/specs/x86_64-nova.nix -A os.tests
>   nix-build ~/genode/specs/x86_64-linux.nix -A run.thread
> 
> Nix-build will leave a symlink named 'result' in your current directory
> that contains the output of the Nix expression you call.
> 
> To remove the outputs that Nix has built, delete the result symlink and
> run 'nix-store -gc' to invoke the garbarge collector.
> 
> 
> If you first want to know what Nix will do when building a run from
> scratch, here is a summary:
>  
>   Install dependencies for the Nix build environment, Nix will likely
>   not use anything in your current PATH.
> 
>   Download and patch the Genode toolchain for the Genode build environment.
> 
>   Compile the sources of each dependency into object files.
> 
>   Merge objects into libraries.
> 
>   Link libraries into applications.
> 
>   Execute Genode and leave the build output and a log of the run in
>   a directory in the Nix store.
> 
> If there are any questions I will be lurking as 'emery' in the IRC
> channel on Freenode.

Thank you for having taking the time to document the steps in such detail.

Right now, I'm busy with the upcoming Genode release. Once it is out,
I'm looking forward to experiment with your branch to learn more about
the possibilities of combining Nix with Genode.

Cheers
Norman

-- 
Dr.-Ing. Norman Feske
Genode Labs

http://www.genode-labs.com · http://genode.org

Genode Labs GmbH · Amtsgericht Dresden · HRB 28424 · Sitz Dresden
Geschäftsführer: Dr.-Ing. Norman Feske, Christian Helmuth