Management of ported 3rd-party software

[Norman Feske]

Hello,

now that Genode's new directory structure is in place, let's take the
next step towards a bullet-proof solution for integrating 3rd-party code
with Genode (see [1] for the corresponding issue). The new solution,
which is very much inspired by the fabulous Nix package manager [2]
comes in the form of new tools to be found at 'tool/ports/' on the
staging branch [3].

[1] https://github.com/genodelabs/genode/issues/1082
[2] http://nixos.org/nix/
[3] https://github.com/genodelabs/genode/commits/staging

Hereby, I'd like to briefly explain how the new solution works from the
viewpoint of a Genode user, describe the steps needed to add a new port
to Genode, and outline how we will evolutionary move from the old 'make
prepare' mechanism to the new concept.

Note that even though the port mechanism described herein looks a bit
like "package management", it covers a different problem. The problem
covered here is the integration of existing 3rd-party source code with
the Genode source tree. Packaging, on the other hand, would provide a
means to distribute self-contained portions of the Genode source tree
including their respective 3rd-party counterparts as separate packages.
Package management is not addressed yet.


The use of the new mechanism
----------------------------

Genode's source-code repositories used to come with Makefiles in their
respective base directories. Those makefile provided the rules 'prepare'
and 'clean'. The 'prepare' rule allowed for the automated downloading
and installation of 3rd-party code, whereas the 'clean' rule reverted
the installation. Source-code archives were downloaded at
<rep-dir>/download/ whereas the extracted 3rd-party code usually resided
at <rep-dir>/contrib/. In the case of ported libraries, the 'make
prepare' step also used to create a bunch of symlinks within
<rep-dir/include/ that pointed to the respective header files within
<rep-dir>/contrib/.

The old 'make prepare' approach was implemented for each individual
repository. In contrast, the new solution unifies the procedure across
all the repositories located at <genode-dir>/repos/. To install a port
provided by any repository, just invoke the tool
<genode-dir>/tool/port/prepare_port with the name of the port as
argument. The tool will scan all repositories for the specified port and
install the port to <genode-dir>/contrib/. Each version of an installed
port resides in a dedicated subdirectory within the contrib/ directory.
The port-specific directory is called port directory. It is named
<port-fingerprint>-<port-name>. The <fingerprint> uniquely identifies
the version of the port (it is a SHA1 hash of the ingredients of the
port). If two versions of the same port are installed, each of them will
have a different fingerprint. So they and up in different directories.

Within the Genode source tree, a port is represented by two files, a
<port-name>.port and a <port-name>.hash file. Both files reside at the
<rep-dir>/ports/ subdirectory of the corresponding repository. The
<port-name>.port file is the port description, which declares the
ingredients of the port, e.g., archives to download, patches to apply.
The <port-name>.hash file contains the fingerprint of the corresponding
port description, thereby uniquely identifying a version of the port.

To see, which ports are available, look out for <rep-dir>/port/*.port files:

  find <genode-dir> -mindepth 4 -maxdepth 4 -name "*.port"

So how does Genode's build system find the source codes for the right
port directory to use? If the build system encounters a target that
incorporates ported source code, it looks up the respective
<port-name>.hash file in the repositories as specified in the build
configuration. The fingerprint found in the hash file is used to
construct the path to the port directory under contrib/.


Adding a port
-------------

The basic steps of adding a new port to Genode are as follows. Let us
assume the 3rd-party source codes comes in the form of a tar.gz archive.

1) Create '<rep-dir>/ports/<port-name>.port' file:

  LICENSE := unknown
  VERSION := <version>
  DOWNLOADS := <archive-name>.archive
  URL(<archive-name>) := http://the-url-of-the.archive.tar.gz

The 'DOWNLOADS' declaration contains a list of items to download. Each
item is suffixed with the type of the download. Supported types are
'file' (a plain file), 'archive' (an archive of the types tar.gz,
tar.xz, tgz, tar.bz2, or zip), 'git' (a Git repository), or 'svn' (a
Subversion repository). For each item, there have to be a few additional
declarations, in particular the URL where to to download it from.

2) Create 'ports/<port-name>.hash' file with the content 'dummy'.

This one will be used during the porting work and replaced once the port
if finished. Because the hash file contains the string "dummy", the port
directory will be located at <genode-dir>/contrib/dummy-<port-name>/.

3) Declaring the hash sum of a downloaded archive

Try to execute the new port file to download the archive.

  <genode-dir>/tool/ports/prepare_port <port-name>

This step will create the port directory and download the archive to the
base of this directory. The prepare_port tool tries to validate the
correct version of the downloaded file using an SHA1 hash sum. Because
the port description lack the known-good SHA1 sum, the check will fail:

  Error: Hash sum check for <port-name> failed

Calculate the SHA1 hash sum of the archive:

  sha1sum <port-dir>/<archive-name>.tar.gz

Declare the hash sum of the archive in the <port-name>.port file:

  SHA(<archive-name>) := <hash-value>

When executing the prepare_port step again, the integrity check for the
downloaded archive should succeed. However, we get the following error
message:

  Error: <rep-dir>/ports/<port-name>.port is out of date, expected
<fingerprint>

We get this message because we had specified the "dummy" hash value in
the <port-name>.hash file. The prepare_port tool computes a fingerprint
of the actual version of the port and compares this fingerprint with the
hash value specified in <port-name>.hash. The computed fingerprint can
be found at <port-dir>/<port-name>.hash. In the final step of the port,
we will replace the dummy fingerprint with the actual fingerprint of the
port. But before finalizing the porting work, it is practical to keep
using the dummy hash and suppress the fingerprint check. This can be
done by adding 'CHECK_HASH=no' as argument to the prepare_port tool:

  <genode-dir>/tool/ports/prepare-port <port-name> CHECK_HASH=no

4) Extracting the source code

When executing the prepare-step now, the tool will present you with the
following message:

  Error: Missing definition of DIR(<port-name>) in <port-file>

We need to declare where to extract the downloaded archive. E.g.,

  DIR(<port-name>) := src/lib/<port-name>

Each port directory is principally organized like a Genode source-code
repository. So it is good practice to place the extracted code at the
location where it would reside if hosted within a repository. Most
source packages distributed as tar archives contain a directory named
after the package and the version number in the top-level directory of
the archive. By default, the prepare tool will strip this directory when
extracting the archive. So the actual content will be installed at the
directory specified via the 'DIR(<archive-name>)' declaration. You can
override the default extraction argument by specifying a custom
'TAR_OPT(<archive-name>)' declaration.

5) Declaration of the license

Now that you have downloaded and extracted the 3rd-party source code
within the port directory, revisit the code for its license. Update the
LICENSE declaration in the <port-name>.port file accordingly.

6) Assembling the include directory exported by the port

Define include files to be presented to the Genode build system, e.g.:

  DIRS := include
  DIR_CONTENT(include) src/lib/<port-name>/include/*.h

This declaration tells the prepare_port tool to copy the files found in
the include/ directory of the extracted archive to the
<port-dir>/include/ directory. Using this mechanism, arbitrary directory
structures can be constructed out of the downloaded content. Validate
that the DIRS declarations work as expected by executing the
prepare_port step again and revisiting the content of the port directory.

7) Using the ported code from Genode compilation targets

Now the preparation step is complete. The final piece of the puzzle is
telling the Genode build system to use the port. Within any library
description file, target.mk file, or import-*.mk file, you can use the
function 'select_from_ports' to query a port directory using the port
name as argument. E.g., assuming <port-name> refers to a library, the
corresponding import-<port-name>.mk file may contain the following
declaration:

  INC_DIR += $(call select_from_ports,<port-name>)/include

This declaration will add the <port-name>/include directory of the port
to the include-search path of every target that uses the library.
Similarly, the library description file may use the 'select_from_ports'
function to define the vpath of the 3rd-party source codes.

Under the hood, the 'select_from_ports' function looks up the
fingerprint of the specified port by reading the corresponding
<port-name>.hash file. It then uses this hash value to construct the
directory path within the 'contrib/' directory that belongs to the
matching version of the port. If there is no hash file that matches the
port name, or if the port directory does not exist, the build system
will present you with an error message.

Finally, after having tested that both the preparation-step and the
build of the ported source code works as expected, it is time to
finalize the fingerprint stored in the <rep-dir>/ports/<port-name>.hash
file. This can be done by copying the content of the
<port-dir>/<port-name>.hash file. The next time, you invoke the
prepare_port tool, do not specify the 'CHECK_HASH=no' argument. So the
fingerprint check will validate that the <port-name>.hash file
corresponds to your <port-name>.port file. From now on, the
contrib/dummy-<port-name> directory will no longer be used because the
<port-name>.hash file points to the port directory named after the real
fingerprint.


Transition to the new mechanism
-------------------------------

In the last couple of days, I have reworked more than 60 ports to use
the new mechanism. The ports not covered so far are:

* libports; qt5
* base-codezero
* base-fiasco
* base-pistachio
* base-foc
* dde_rump
* dde_linux
* dde_ipxe
* ports-foc
* ports: gcc
* ports: gdb
* ports: seoul
* ports: virtualbox

During the transition phase (the next release cycle), we will keep the
original 'make prepare' mechanism as a front end. So the 'make prepare'
instructions as found in many tutorials will still work. But under the
hood, 'make prepare' will just invoke the new _tool/ports/prepare_port_
tool.

In the current version at the staging branch, the hash sum check is
disabled for all ports installed via the old 'make prepare' front end.
Nearly all hash files contain "dummy". This is because the prepare_port
tool is still in flux. Each change of the tool would require new
fingerprints for all packages, which is inconvenient while working on
the tool. However, once the development of the tools settles, the dummy
hash sums will be replaced by the real hash sums.


I hope that the new mechanism will make Genode more convenient to use.
Hopefully, the errors caused by missing or outdated 'make prepare' steps
will be a thing of the past. However, the current change is just another
step. Once we have fully adopted the prepare_port tool, we can fairly
easily add further utilities to work with ports, e.g., adding dependency
information between ports, garbage collecting stale versions, updating
all installed ports, etc. Also, the new way of how ports are organized
(having a layout similar to Genode repositories) will greatly help us to
implement proper package management for Genode. For the latter, I'd love
to embrace the Nix package manager.

As immediate steps, we should rework the remaining ports (as mentioned
above) to use the new port mechanism and update the documentation, in
particular the porting guide [4]

[4] https://github.com/genodelabs/genode/blob/master/doc/porting_guide.txt

Cheers
Norman

-- 
Dr.-Ing. Norman Feske
Genode Labs

http://www.genode-labs.com · http://genode.org

Genode Labs GmbH · Amtsgericht Dresden · HRB 28424 · Sitz Dresden
Geschäftsführer: Dr.-Ing. Norman Feske, Christian Helmuth