Unified build directories / adaptation to the new API

Fri Jan 13 20:49:22 CET 2017

Hi
    Norman-

    I gota hand it to you, for the heavy weight Genode ABI work your
pulling together!
I'm sure its not easy keeping track of everything staying coherent in the
context of Genode's dynamic linker.   Then tis is exactly the kind of work
that provides the foundation for a production strength OS.. and that seems
to be Genode's future..

    you mention: "** The packaging of kernel-agnostic binary packages has
become within **close reach now.*"
     it would be great to hear you current reflection as to how a unified,
repeatable build src / binary package management system might help keep the
Genode OS package versioning well organized..

   -Peter

On Fri, Jan 13, 2017 at 7:30 AM, Norman Feske <norman.feske at ...1...>
wrote:

> Hello everyone,
>
> I am happy to report that we wrapped up the first phase of our big API
> adaptation work today and introduced the unified build directories into
> the master branch. This posting is both a heads-up warning and an
> invitation to experiment with the new mechanism.
>
> First, the warning part (quite literally):
>
> With the commit "Warn about the use of deprecated env() function" [1], a
> compiler warning will appear each time the legacy 'env()' function is
> used, either explicitly or implicitly. This makes it easy to spot the
> places that need our attention because they still rely on global side
> effects. Since New Year, we have been very busy to overhaul our code
> base and made huge progress. Thanks to everyone involved! However, as
> you will see, there are still many places left unattended. Therefore,
> the warnings somehow spoil the impression when building Genode right
> now. Their main downside is that build errors and important warnings are
> more difficult to spot. Please bear with us. ;-) Don't worry though, the
> "deprecated" warnings are merely cosmetic and will gradually disappear.
>
> [1]
> https://github.com/genodelabs/genode/commit/a7f40b24cad0a34f0ac0bd9256a528
> 5a6bf7cd94
>
> Second, I proudly present you the fruit of my labor with respect to the
> Genode ABI. Genode has now become able to use different kernels from
> within the same build directory. Of course, this change comes with a
> sight change of the tooling, in particular to the 'create_builddir'
> tool, the build system, and the autopilot tool. Let's take the new
> version for a quick test drive.
>
> 1. Checkout the current master branch of Genode.
>
>    git clone https://github.com/genodelabs/genode.git
>
> 2. Invoke the 'create_builddir' tool to see the new options of
>    build directories.
>
>    cd genode
>    ./tool/create_builddir
>
>    As you can see, the original kernel-specific platform parameters
>    are still there but they are marked as "deprecated". They will
>    be removed shortly. The new unified build directories are the new
>    way to go now. There is one option for each supported hardware
>    platform, e.g., 'x86_64', or 'usb_armory'. Note that the kernel
>    is left unspecified. Assuming you have a 64-bit Linux host system,
>    let us create a build directory that matches your Linux system:
>
>    ./tool/create_builddir x86_64
>
>    (if you are using a 32-bit Linux system, specify 'x86_32' instead)
>    Now, enter the build directory and look at the build configuration:
>
>    cd build/x86_64
>    <editor> etc/build.conf
>
>    You may notice the changed "Run-tool configuration" section. Parts
>    this configuration are dependent of a new 'KERNEL' variable.
>    The variable has the following effects:
>
>    * It selects kernel-specific run-tool arguments 'KERNEL_RUN_OPT',
>    * It selects kernel-specific Qemu arguments, e.g. 'QEMU_OPT(nova)'
>    * It adds the kernel-specific 'base-<kernel>' directory to the
>      list of source-code 'REPOSITORIES'.
>
>    Once you have your build.conf open in your editor, you may enable
>    parallel build by uncommenting the corresponding line at the
>    beginning of the file.
>
> 3. Let's build a Genode component, e.g., init:
>
>    make init
>
>    Wow, that was quick! The build system created the init binary
>    without even building any parts of the Genode base system. How
>    is that possible? The answer lies at 'repos/base/lib/symbols/ld',
>    which is the binary interface provided by Genode's dynamic linker.
>    The build system uses this information to build a pseudo shared
>    object that is linked against init instead of the real dynamic
>    linker. From the perspective of the init executable (just put
>    yourself into the shoes of an ELF executable to follow me), it
>    feels like being linked against Genode's dynamic linker that
>    provides the base framework. But in reality, you got presented
>    just an empty hull of Genode - a bunch of symbols. In case you
>    are curious, you can find this ABI stub library at
>    var/libcache/ld/ld.abi.so. If you look closely into the build
>    subdirectory of init/, you will see that the used 'ld.lib.so'
>    is actually a symlink to the stub.
>
>    At this point, the build directory is clear from any kernel-
>    specific peculiarities.
>
> 4. Building a kernel
>
>    To build a kernel, the build system needs to know the kernel-
>    specific parts of Genode, i.e., the corresponding 'base-<kernel>'
>    source code repository must be added to the 'REPOSITORIES'. The
>    default build configuration does this automatically if you
>    specify a 'KERNEL' argument:
>
>    make kernel KERNEL=nova
>
>    At this point, the build system may point out that you will need
>    to download the kernel source code. Just follow the proposed
>    command to do these preparatory steps. If done, execute the
>    above command again. In the particular case of NOVA, the result of
>    the kernel build can be found at 'kernel/nova/'.
>
> 5. Executing a scenario on Linux
>
>    Let's build and execute Genode's graphical demo scenario on
>    your Linux host system:
>
>    make run/demo KERNEL=linux
>
>    The build will take a while. Most of the produced components are
>    kernel-independent. Only a few, such as core, fb_sdl, and the timer
>    driver are Linux-specific. Also a Linux-specific version of
>    Genode's dynamic linker will be built. It is named 'ld-linux.lib.so'.
>
>    Once the build completes, the scenario is executed. So you should
>    see a window with the demo scenario popping up.
>
> 6. Switching to another kernel
>
>    Now, let's execute the same scenario on, let's say, NOVA. To use this
>    kernel instead of Linux, a few actual drivers are needed. So let us
>    enable the 'libports' and 'dde_linux' repositories in the build
>    configuration.
>
>    At this point, you may have guessed the command to use:
>
>    make run/demo KERNEL=nova
>
>    You will be prompted to prepare a few ports of 3rd-party software,
>    in particular, the libc, dde_linux (USB driver), x86emu (for VESA).
>    After installing those ports, issue the above command again.
>
>    Note that the build process will build the additional drivers and
>    NOVA-specific parts of Genode. But all the components that
>    we built before when executing the scenario on Linux remain
>    untouched now.
>
>    At the end of the build process, the scenario is stared in Qemu.
>
> To sum it up, switching from one kernel to another has become just a
> matter of the 'KERNEL' argument. This has a number of benefits:
>
> * When using multiple kernels, there is no need to have multiple
>   build directories. You waited an hour to build a Qt5-based scenario
>   on NOVA but want to run it on Linux now? This is now possible in
>   a few seconds.
>
> * If an executable has a bug, the bug will be there regardless of the
>   kernel being used. To debug the problem, you can use the kernel
>   with the most appropriate debugging instruments available.
>
> * The packaging of kernel-agnostic binary packages has become within
>   close reach now.
>
> * The new ABI feature that works so well for Genode's dynamic linker
>   can actually be leveraged by arbitrary libraries. For example, I just
>   introduced an ABI for the libc (libports/lib/symbols/libc).
>   So in the future we can even have multiple levels of ABI stability.
>
> I hope that you will like the unified build directories. This being a
> very recent development, there may still be rough edges. Your feedback
> is very welcome. I hope that we will have smoothened those edges and
> removed most of the "deprecated" warnings by the time of releasing
> version 17.02 at the end of February.
>
> Cheers
> Norman
>
> --
> Dr.-Ing. Norman Feske
> Genode Labs
>
> http://www.genode-labs.com · http://genode.org
>
> Genode Labs GmbH · Amtsgericht Dresden · HRB 28424 · Sitz Dresden
> Geschäftsführer: Dr.-Ing. Norman Feske, Christian Helmuth
>
> ------------------------------------------------------------
> ------------------
> Developer Access Program for Intel Xeon Phi Processors
> Access to Intel Xeon Phi processor-based developer platforms.
> With one year of Intel Parallel Studio XE.
> Training and support from Colfax.
> Order your platform today. http://sdm.link/xeonphi
> _______________________________________________
> genode-main mailing list
> genode-main at lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/genode-main
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.genode.org/pipermail/users/attachments/20170113/76e7fec5/attachment.html>