Hello,
now that Genode's new directory structure is in place, let's take the next step towards a bullet-proof solution for integrating 3rd-party code with Genode (see [1] for the corresponding issue). The new solution, which is very much inspired by the fabulous Nix package manager [2] comes in the form of new tools to be found at 'tool/ports/' on the staging branch [3].
[1] https://github.com/genodelabs/genode/issues/1082 [2] http://nixos.org/nix/ [3] https://github.com/genodelabs/genode/commits/staging
Hereby, I'd like to briefly explain how the new solution works from the viewpoint of a Genode user, describe the steps needed to add a new port to Genode, and outline how we will evolutionary move from the old 'make prepare' mechanism to the new concept.
Note that even though the port mechanism described herein looks a bit like "package management", it covers a different problem. The problem covered here is the integration of existing 3rd-party source code with the Genode source tree. Packaging, on the other hand, would provide a means to distribute self-contained portions of the Genode source tree including their respective 3rd-party counterparts as separate packages. Package management is not addressed yet.
The use of the new mechanism ----------------------------
Genode's source-code repositories used to come with Makefiles in their respective base directories. Those makefile provided the rules 'prepare' and 'clean'. The 'prepare' rule allowed for the automated downloading and installation of 3rd-party code, whereas the 'clean' rule reverted the installation. Source-code archives were downloaded at <rep-dir>/download/ whereas the extracted 3rd-party code usually resided at <rep-dir>/contrib/. In the case of ported libraries, the 'make prepare' step also used to create a bunch of symlinks within <rep-dir/include/ that pointed to the respective header files within <rep-dir>/contrib/.
The old 'make prepare' approach was implemented for each individual repository. In contrast, the new solution unifies the procedure across all the repositories located at <genode-dir>/repos/. To install a port provided by any repository, just invoke the tool <genode-dir>/tool/port/prepare_port with the name of the port as argument. The tool will scan all repositories for the specified port and install the port to <genode-dir>/contrib/. Each version of an installed port resides in a dedicated subdirectory within the contrib/ directory. The port-specific directory is called port directory. It is named <port-fingerprint>-<port-name>. The <fingerprint> uniquely identifies the version of the port (it is a SHA1 hash of the ingredients of the port). If two versions of the same port are installed, each of them will have a different fingerprint. So they and up in different directories.
Within the Genode source tree, a port is represented by two files, a <port-name>.port and a <port-name>.hash file. Both files reside at the <rep-dir>/ports/ subdirectory of the corresponding repository. The <port-name>.port file is the port description, which declares the ingredients of the port, e.g., archives to download, patches to apply. The <port-name>.hash file contains the fingerprint of the corresponding port description, thereby uniquely identifying a version of the port.
To see, which ports are available, look out for <rep-dir>/port/*.port files:
find <genode-dir> -mindepth 4 -maxdepth 4 -name "*.port"
So how does Genode's build system find the source codes for the right port directory to use? If the build system encounters a target that incorporates ported source code, it looks up the respective <port-name>.hash file in the repositories as specified in the build configuration. The fingerprint found in the hash file is used to construct the path to the port directory under contrib/.
Adding a port -------------
The basic steps of adding a new port to Genode are as follows. Let us assume the 3rd-party source codes comes in the form of a tar.gz archive.
1) Create '<rep-dir>/ports/<port-name>.port' file:
LICENSE := unknown VERSION := <version> DOWNLOADS := <archive-name>.archive URL(<archive-name>) := http://the-url-of-the.archive.tar.gz
The 'DOWNLOADS' declaration contains a list of items to download. Each item is suffixed with the type of the download. Supported types are 'file' (a plain file), 'archive' (an archive of the types tar.gz, tar.xz, tgz, tar.bz2, or zip), 'git' (a Git repository), or 'svn' (a Subversion repository). For each item, there have to be a few additional declarations, in particular the URL where to to download it from.
2) Create 'ports/<port-name>.hash' file with the content 'dummy'.
This one will be used during the porting work and replaced once the port if finished. Because the hash file contains the string "dummy", the port directory will be located at <genode-dir>/contrib/dummy-<port-name>/.
3) Declaring the hash sum of a downloaded archive
Try to execute the new port file to download the archive.
<genode-dir>/tool/ports/prepare_port <port-name>
This step will create the port directory and download the archive to the base of this directory. The prepare_port tool tries to validate the correct version of the downloaded file using an SHA1 hash sum. Because the port description lack the known-good SHA1 sum, the check will fail:
Error: Hash sum check for <port-name> failed
Calculate the SHA1 hash sum of the archive:
sha1sum <port-dir>/<archive-name>.tar.gz
Declare the hash sum of the archive in the <port-name>.port file:
SHA(<archive-name>) := <hash-value>
When executing the prepare_port step again, the integrity check for the downloaded archive should succeed. However, we get the following error message:
Error: <rep-dir>/ports/<port-name>.port is out of date, expected <fingerprint>
We get this message because we had specified the "dummy" hash value in the <port-name>.hash file. The prepare_port tool computes a fingerprint of the actual version of the port and compares this fingerprint with the hash value specified in <port-name>.hash. The computed fingerprint can be found at <port-dir>/<port-name>.hash. In the final step of the port, we will replace the dummy fingerprint with the actual fingerprint of the port. But before finalizing the porting work, it is practical to keep using the dummy hash and suppress the fingerprint check. This can be done by adding 'CHECK_HASH=no' as argument to the prepare_port tool:
<genode-dir>/tool/ports/prepare-port <port-name> CHECK_HASH=no
4) Extracting the source code
When executing the prepare-step now, the tool will present you with the following message:
Error: Missing definition of DIR(<port-name>) in <port-file>
We need to declare where to extract the downloaded archive. E.g.,
DIR(<port-name>) := src/lib/<port-name>
Each port directory is principally organized like a Genode source-code repository. So it is good practice to place the extracted code at the location where it would reside if hosted within a repository. Most source packages distributed as tar archives contain a directory named after the package and the version number in the top-level directory of the archive. By default, the prepare tool will strip this directory when extracting the archive. So the actual content will be installed at the directory specified via the 'DIR(<archive-name>)' declaration. You can override the default extraction argument by specifying a custom 'TAR_OPT(<archive-name>)' declaration.
5) Declaration of the license
Now that you have downloaded and extracted the 3rd-party source code within the port directory, revisit the code for its license. Update the LICENSE declaration in the <port-name>.port file accordingly.
6) Assembling the include directory exported by the port
Define include files to be presented to the Genode build system, e.g.:
DIRS := include DIR_CONTENT(include) src/lib/<port-name>/include/*.h
This declaration tells the prepare_port tool to copy the files found in the include/ directory of the extracted archive to the <port-dir>/include/ directory. Using this mechanism, arbitrary directory structures can be constructed out of the downloaded content. Validate that the DIRS declarations work as expected by executing the prepare_port step again and revisiting the content of the port directory.
7) Using the ported code from Genode compilation targets
Now the preparation step is complete. The final piece of the puzzle is telling the Genode build system to use the port. Within any library description file, target.mk file, or import-*.mk file, you can use the function 'select_from_ports' to query a port directory using the port name as argument. E.g., assuming <port-name> refers to a library, the corresponding import-<port-name>.mk file may contain the following declaration:
INC_DIR += $(call select_from_ports,<port-name>)/include
This declaration will add the <port-name>/include directory of the port to the include-search path of every target that uses the library. Similarly, the library description file may use the 'select_from_ports' function to define the vpath of the 3rd-party source codes.
Under the hood, the 'select_from_ports' function looks up the fingerprint of the specified port by reading the corresponding <port-name>.hash file. It then uses this hash value to construct the directory path within the 'contrib/' directory that belongs to the matching version of the port. If there is no hash file that matches the port name, or if the port directory does not exist, the build system will present you with an error message.
Finally, after having tested that both the preparation-step and the build of the ported source code works as expected, it is time to finalize the fingerprint stored in the <rep-dir>/ports/<port-name>.hash file. This can be done by copying the content of the <port-dir>/<port-name>.hash file. The next time, you invoke the prepare_port tool, do not specify the 'CHECK_HASH=no' argument. So the fingerprint check will validate that the <port-name>.hash file corresponds to your <port-name>.port file. From now on, the contrib/dummy-<port-name> directory will no longer be used because the <port-name>.hash file points to the port directory named after the real fingerprint.
Transition to the new mechanism -------------------------------
In the last couple of days, I have reworked more than 60 ports to use the new mechanism. The ports not covered so far are:
* libports; qt5 * base-codezero * base-fiasco * base-pistachio * base-foc * dde_rump * dde_linux * dde_ipxe * ports-foc * ports: gcc * ports: gdb * ports: seoul * ports: virtualbox
During the transition phase (the next release cycle), we will keep the original 'make prepare' mechanism as a front end. So the 'make prepare' instructions as found in many tutorials will still work. But under the hood, 'make prepare' will just invoke the new _tool/ports/prepare_port_ tool.
In the current version at the staging branch, the hash sum check is disabled for all ports installed via the old 'make prepare' front end. Nearly all hash files contain "dummy". This is because the prepare_port tool is still in flux. Each change of the tool would require new fingerprints for all packages, which is inconvenient while working on the tool. However, once the development of the tools settles, the dummy hash sums will be replaced by the real hash sums.
I hope that the new mechanism will make Genode more convenient to use. Hopefully, the errors caused by missing or outdated 'make prepare' steps will be a thing of the past. However, the current change is just another step. Once we have fully adopted the prepare_port tool, we can fairly easily add further utilities to work with ports, e.g., adding dependency information between ports, garbage collecting stale versions, updating all installed ports, etc. Also, the new way of how ports are organized (having a layout similar to Genode repositories) will greatly help us to implement proper package management for Genode. For the latter, I'd love to embrace the Nix package manager.
As immediate steps, we should rework the remaining ports (as mentioned above) to use the new port mechanism and update the documentation, in particular the porting guide [4]
[4] https://github.com/genodelabs/genode/blob/master/doc/porting_guide.txt
Cheers Norman