Unified build directories / adaptation to the new API

[Norman Feske]

Hello everyone,

I am happy to report that we wrapped up the first phase of our big API
adaptation work today and introduced the unified build directories into
the master branch. This posting is both a heads-up warning and an
invitation to experiment with the new mechanism.

First, the warning part (quite literally):

With the commit "Warn about the use of deprecated env() function" [1], a
compiler warning will appear each time the legacy 'env()' function is
used, either explicitly or implicitly. This makes it easy to spot the
places that need our attention because they still rely on global side
effects. Since New Year, we have been very busy to overhaul our code
base and made huge progress. Thanks to everyone involved! However, as
you will see, there are still many places left unattended. Therefore,
the warnings somehow spoil the impression when building Genode right
now. Their main downside is that build errors and important warnings are
more difficult to spot. Please bear with us. ;-) Don't worry though, the
"deprecated" warnings are merely cosmetic and will gradually disappear.

[1]
https://github.com/genodelabs/genode/commit/a7f40b24cad0a34f0ac0bd9256a5285a6bf7cd94

Second, I proudly present you the fruit of my labor with respect to the
Genode ABI. Genode has now become able to use different kernels from
within the same build directory. Of course, this change comes with a
sight change of the tooling, in particular to the 'create_builddir'
tool, the build system, and the autopilot tool. Let's take the new
version for a quick test drive.

1. Checkout the current master branch of Genode.

   git clone https://github.com/genodelabs/genode.git

2. Invoke the 'create_builddir' tool to see the new options of
   build directories.

   cd genode
   ./tool/create_builddir

   As you can see, the original kernel-specific platform parameters
   are still there but they are marked as "deprecated". They will
   be removed shortly. The new unified build directories are the new
   way to go now. There is one option for each supported hardware
   platform, e.g., 'x86_64', or 'usb_armory'. Note that the kernel
   is left unspecified. Assuming you have a 64-bit Linux host system,
   let us create a build directory that matches your Linux system:

   ./tool/create_builddir x86_64

   (if you are using a 32-bit Linux system, specify 'x86_32' instead)
   Now, enter the build directory and look at the build configuration:

   cd build/x86_64
   <editor> etc/build.conf

   You may notice the changed "Run-tool configuration" section. Parts
   this configuration are dependent of a new 'KERNEL' variable.
   The variable has the following effects:

   * It selects kernel-specific run-tool arguments 'KERNEL_RUN_OPT',
   * It selects kernel-specific Qemu arguments, e.g. 'QEMU_OPT(nova)'
   * It adds the kernel-specific 'base-<kernel>' directory to the
     list of source-code 'REPOSITORIES'.

   Once you have your build.conf open in your editor, you may enable
   parallel build by uncommenting the corresponding line at the
   beginning of the file.

3. Let's build a Genode component, e.g., init:

   make init

   Wow, that was quick! The build system created the init binary
   without even building any parts of the Genode base system. How
   is that possible? The answer lies at 'repos/base/lib/symbols/ld',
   which is the binary interface provided by Genode's dynamic linker.
   The build system uses this information to build a pseudo shared
   object that is linked against init instead of the real dynamic
   linker. From the perspective of the init executable (just put
   yourself into the shoes of an ELF executable to follow me), it
   feels like being linked against Genode's dynamic linker that
   provides the base framework. But in reality, you got presented
   just an empty hull of Genode - a bunch of symbols. In case you
   are curious, you can find this ABI stub library at
   var/libcache/ld/ld.abi.so. If you look closely into the build
   subdirectory of init/, you will see that the used 'ld.lib.so'
   is actually a symlink to the stub.

   At this point, the build directory is clear from any kernel-
   specific peculiarities.

4. Building a kernel

   To build a kernel, the build system needs to know the kernel-
   specific parts of Genode, i.e., the corresponding 'base-<kernel>'
   source code repository must be added to the 'REPOSITORIES'. The
   default build configuration does this automatically if you
   specify a 'KERNEL' argument:

   make kernel KERNEL=nova

   At this point, the build system may point out that you will need
   to download the kernel source code. Just follow the proposed
   command to do these preparatory steps. If done, execute the
   above command again. In the particular case of NOVA, the result of
   the kernel build can be found at 'kernel/nova/'.

5. Executing a scenario on Linux

   Let's build and execute Genode's graphical demo scenario on
   your Linux host system:

   make run/demo KERNEL=linux

   The build will take a while. Most of the produced components are
   kernel-independent. Only a few, such as core, fb_sdl, and the timer
   driver are Linux-specific. Also a Linux-specific version of
   Genode's dynamic linker will be built. It is named 'ld-linux.lib.so'.

   Once the build completes, the scenario is executed. So you should
   see a window with the demo scenario popping up.

6. Switching to another kernel

   Now, let's execute the same scenario on, let's say, NOVA. To use this
   kernel instead of Linux, a few actual drivers are needed. So let us
   enable the 'libports' and 'dde_linux' repositories in the build
   configuration.

   At this point, you may have guessed the command to use:

   make run/demo KERNEL=nova

   You will be prompted to prepare a few ports of 3rd-party software,
   in particular, the libc, dde_linux (USB driver), x86emu (for VESA).
   After installing those ports, issue the above command again.

   Note that the build process will build the additional drivers and
   NOVA-specific parts of Genode. But all the components that
   we built before when executing the scenario on Linux remain
   untouched now.

   At the end of the build process, the scenario is stared in Qemu.

To sum it up, switching from one kernel to another has become just a
matter of the 'KERNEL' argument. This has a number of benefits:

* When using multiple kernels, there is no need to have multiple
  build directories. You waited an hour to build a Qt5-based scenario
  on NOVA but want to run it on Linux now? This is now possible in
  a few seconds.

* If an executable has a bug, the bug will be there regardless of the
  kernel being used. To debug the problem, you can use the kernel
  with the most appropriate debugging instruments available.

* The packaging of kernel-agnostic binary packages has become within
  close reach now.

* The new ABI feature that works so well for Genode's dynamic linker
  can actually be leveraged by arbitrary libraries. For example, I just
  introduced an ABI for the libc (libports/lib/symbols/libc).
  So in the future we can even have multiple levels of ABI stability.

I hope that you will like the unified build directories. This being a
very recent development, there may still be rough edges. Your feedback
is very welcome. I hope that we will have smoothened those edges and
removed most of the "deprecated" warnings by the time of releasing
version 17.02 at the end of February.

Cheers
Norman

-- 
Dr.-Ing. Norman Feske
Genode Labs

http://www.genode-labs.com · http://genode.org

Genode Labs GmbH · Amtsgericht Dresden · HRB 28424 · Sitz Dresden
Geschäftsführer: Dr.-Ing. Norman Feske, Christian Helmuth