Management of ported 3rd-party software

[Stefan Kalkowski]

Hi,

just a short remark for port file authors:

When a port preparation includes checking out a git repository, you can
state the revision explicitly by the "REV" variable, e.g. in
repos/libports/ports/lwip.port:

  REV(lwip) := fe63f36656bd66b4051bdfab93e351a584337d7c

I've already seen that people state a branch name instead of a revision
hash tag. That is non advisable, as branch names are not unique, and the
source of the branch may change without the new port mechanism noticing
that. To prevent users from experiencing inconsistencies between Genode
and 3rd party code in the future, I'd like to advise you to take always
hash tags instead of branch names.

Regards
Stefan

On 05/19/2014 12:15 PM, Norman Feske wrote:
> Hello,
> 
> now that Genode's new directory structure is in place, let's take the
> next step towards a bullet-proof solution for integrating 3rd-party code
> with Genode (see [1] for the corresponding issue). The new solution,
> which is very much inspired by the fabulous Nix package manager [2]
> comes in the form of new tools to be found at 'tool/ports/' on the
> staging branch [3].
> 
> [1] https://github.com/genodelabs/genode/issues/1082
> [2] http://nixos.org/nix/
> [3] https://github.com/genodelabs/genode/commits/staging
> 
> Hereby, I'd like to briefly explain how the new solution works from the
> viewpoint of a Genode user, describe the steps needed to add a new port
> to Genode, and outline how we will evolutionary move from the old 'make
> prepare' mechanism to the new concept.
> 
> Note that even though the port mechanism described herein looks a bit
> like "package management", it covers a different problem. The problem
> covered here is the integration of existing 3rd-party source code with
> the Genode source tree. Packaging, on the other hand, would provide a
> means to distribute self-contained portions of the Genode source tree
> including their respective 3rd-party counterparts as separate packages.
> Package management is not addressed yet.
> 
> 
> The use of the new mechanism
> ----------------------------
> 
> Genode's source-code repositories used to come with Makefiles in their
> respective base directories. Those makefile provided the rules 'prepare'
> and 'clean'. The 'prepare' rule allowed for the automated downloading
> and installation of 3rd-party code, whereas the 'clean' rule reverted
> the installation. Source-code archives were downloaded at
> <rep-dir>/download/ whereas the extracted 3rd-party code usually resided
> at <rep-dir>/contrib/. In the case of ported libraries, the 'make
> prepare' step also used to create a bunch of symlinks within
> <rep-dir/include/ that pointed to the respective header files within
> <rep-dir>/contrib/.
> 
> The old 'make prepare' approach was implemented for each individual
> repository. In contrast, the new solution unifies the procedure across
> all the repositories located at <genode-dir>/repos/. To install a port
> provided by any repository, just invoke the tool
> <genode-dir>/tool/port/prepare_port with the name of the port as
> argument. The tool will scan all repositories for the specified port and
> install the port to <genode-dir>/contrib/. Each version of an installed
> port resides in a dedicated subdirectory within the contrib/ directory.
> The port-specific directory is called port directory. It is named
> <port-fingerprint>-<port-name>. The <fingerprint> uniquely identifies
> the version of the port (it is a SHA1 hash of the ingredients of the
> port). If two versions of the same port are installed, each of them will
> have a different fingerprint. So they and up in different directories.
> 
> Within the Genode source tree, a port is represented by two files, a
> <port-name>.port and a <port-name>.hash file. Both files reside at the
> <rep-dir>/ports/ subdirectory of the corresponding repository. The
> <port-name>.port file is the port description, which declares the
> ingredients of the port, e.g., archives to download, patches to apply.
> The <port-name>.hash file contains the fingerprint of the corresponding
> port description, thereby uniquely identifying a version of the port.
> 
> To see, which ports are available, look out for <rep-dir>/port/*.port files:
> 
>   find <genode-dir> -mindepth 4 -maxdepth 4 -name "*.port"
> 
> So how does Genode's build system find the source codes for the right
> port directory to use? If the build system encounters a target that
> incorporates ported source code, it looks up the respective
> <port-name>.hash file in the repositories as specified in the build
> configuration. The fingerprint found in the hash file is used to
> construct the path to the port directory under contrib/.
> 
> 
> Adding a port
> -------------
> 
> The basic steps of adding a new port to Genode are as follows. Let us
> assume the 3rd-party source codes comes in the form of a tar.gz archive.
> 
> 1) Create '<rep-dir>/ports/<port-name>.port' file:
> 
>   LICENSE := unknown
>   VERSION := <version>
>   DOWNLOADS := <archive-name>.archive
>   URL(<archive-name>) := http://the-url-of-the.archive.tar.gz
> 
> The 'DOWNLOADS' declaration contains a list of items to download. Each
> item is suffixed with the type of the download. Supported types are
> 'file' (a plain file), 'archive' (an archive of the types tar.gz,
> tar.xz, tgz, tar.bz2, or zip), 'git' (a Git repository), or 'svn' (a
> Subversion repository). For each item, there have to be a few additional
> declarations, in particular the URL where to to download it from.
> 
> 2) Create 'ports/<port-name>.hash' file with the content 'dummy'.
> 
> This one will be used during the porting work and replaced once the port
> if finished. Because the hash file contains the string "dummy", the port
> directory will be located at <genode-dir>/contrib/dummy-<port-name>/.
> 
> 3) Declaring the hash sum of a downloaded archive
> 
> Try to execute the new port file to download the archive.
> 
>   <genode-dir>/tool/ports/prepare_port <port-name>
> 
> This step will create the port directory and download the archive to the
> base of this directory. The prepare_port tool tries to validate the
> correct version of the downloaded file using an SHA1 hash sum. Because
> the port description lack the known-good SHA1 sum, the check will fail:
> 
>   Error: Hash sum check for <port-name> failed
> 
> Calculate the SHA1 hash sum of the archive:
> 
>   sha1sum <port-dir>/<archive-name>.tar.gz
> 
> Declare the hash sum of the archive in the <port-name>.port file:
> 
>   SHA(<archive-name>) := <hash-value>
> 
> When executing the prepare_port step again, the integrity check for the
> downloaded archive should succeed. However, we get the following error
> message:
> 
>   Error: <rep-dir>/ports/<port-name>.port is out of date, expected
> <fingerprint>
> 
> We get this message because we had specified the "dummy" hash value in
> the <port-name>.hash file. The prepare_port tool computes a fingerprint
> of the actual version of the port and compares this fingerprint with the
> hash value specified in <port-name>.hash. The computed fingerprint can
> be found at <port-dir>/<port-name>.hash. In the final step of the port,
> we will replace the dummy fingerprint with the actual fingerprint of the
> port. But before finalizing the porting work, it is practical to keep
> using the dummy hash and suppress the fingerprint check. This can be
> done by adding 'CHECK_HASH=no' as argument to the prepare_port tool:
> 
>   <genode-dir>/tool/ports/prepare-port <port-name> CHECK_HASH=no
> 
> 4) Extracting the source code
> 
> When executing the prepare-step now, the tool will present you with the
> following message:
> 
>   Error: Missing definition of DIR(<port-name>) in <port-file>
> 
> We need to declare where to extract the downloaded archive. E.g.,
> 
>   DIR(<port-name>) := src/lib/<port-name>
> 
> Each port directory is principally organized like a Genode source-code
> repository. So it is good practice to place the extracted code at the
> location where it would reside if hosted within a repository. Most
> source packages distributed as tar archives contain a directory named
> after the package and the version number in the top-level directory of
> the archive. By default, the prepare tool will strip this directory when
> extracting the archive. So the actual content will be installed at the
> directory specified via the 'DIR(<archive-name>)' declaration. You can
> override the default extraction argument by specifying a custom
> 'TAR_OPT(<archive-name>)' declaration.
> 
> 5) Declaration of the license
> 
> Now that you have downloaded and extracted the 3rd-party source code
> within the port directory, revisit the code for its license. Update the
> LICENSE declaration in the <port-name>.port file accordingly.
> 
> 6) Assembling the include directory exported by the port
> 
> Define include files to be presented to the Genode build system, e.g.:
> 
>   DIRS := include
>   DIR_CONTENT(include) src/lib/<port-name>/include/*.h
> 
> This declaration tells the prepare_port tool to copy the files found in
> the include/ directory of the extracted archive to the
> <port-dir>/include/ directory. Using this mechanism, arbitrary directory
> structures can be constructed out of the downloaded content. Validate
> that the DIRS declarations work as expected by executing the
> prepare_port step again and revisiting the content of the port directory.
> 
> 7) Using the ported code from Genode compilation targets
> 
> Now the preparation step is complete. The final piece of the puzzle is
> telling the Genode build system to use the port. Within any library
> description file, target.mk file, or import-*.mk file, you can use the
> function 'select_from_ports' to query a port directory using the port
> name as argument. E.g., assuming <port-name> refers to a library, the
> corresponding import-<port-name>.mk file may contain the following
> declaration:
> 
>   INC_DIR += $(call select_from_ports,<port-name>)/include
> 
> This declaration will add the <port-name>/include directory of the port
> to the include-search path of every target that uses the library.
> Similarly, the library description file may use the 'select_from_ports'
> function to define the vpath of the 3rd-party source codes.
> 
> Under the hood, the 'select_from_ports' function looks up the
> fingerprint of the specified port by reading the corresponding
> <port-name>.hash file. It then uses this hash value to construct the
> directory path within the 'contrib/' directory that belongs to the
> matching version of the port. If there is no hash file that matches the
> port name, or if the port directory does not exist, the build system
> will present you with an error message.
> 
> Finally, after having tested that both the preparation-step and the
> build of the ported source code works as expected, it is time to
> finalize the fingerprint stored in the <rep-dir>/ports/<port-name>.hash
> file. This can be done by copying the content of the
> <port-dir>/<port-name>.hash file. The next time, you invoke the
> prepare_port tool, do not specify the 'CHECK_HASH=no' argument. So the
> fingerprint check will validate that the <port-name>.hash file
> corresponds to your <port-name>.port file. From now on, the
> contrib/dummy-<port-name> directory will no longer be used because the
> <port-name>.hash file points to the port directory named after the real
> fingerprint.
> 
> 
> Transition to the new mechanism
> -------------------------------
> 
> In the last couple of days, I have reworked more than 60 ports to use
> the new mechanism. The ports not covered so far are:
> 
> * libports; qt5
> * base-codezero
> * base-fiasco
> * base-pistachio
> * base-foc
> * dde_rump
> * dde_linux
> * dde_ipxe
> * ports-foc
> * ports: gcc
> * ports: gdb
> * ports: seoul
> * ports: virtualbox
> 
> During the transition phase (the next release cycle), we will keep the
> original 'make prepare' mechanism as a front end. So the 'make prepare'
> instructions as found in many tutorials will still work. But under the
> hood, 'make prepare' will just invoke the new _tool/ports/prepare_port_
> tool.
> 
> In the current version at the staging branch, the hash sum check is
> disabled for all ports installed via the old 'make prepare' front end.
> Nearly all hash files contain "dummy". This is because the prepare_port
> tool is still in flux. Each change of the tool would require new
> fingerprints for all packages, which is inconvenient while working on
> the tool. However, once the development of the tools settles, the dummy
> hash sums will be replaced by the real hash sums.
> 
> 
> I hope that the new mechanism will make Genode more convenient to use.
> Hopefully, the errors caused by missing or outdated 'make prepare' steps
> will be a thing of the past. However, the current change is just another
> step. Once we have fully adopted the prepare_port tool, we can fairly
> easily add further utilities to work with ports, e.g., adding dependency
> information between ports, garbage collecting stale versions, updating
> all installed ports, etc. Also, the new way of how ports are organized
> (having a layout similar to Genode repositories) will greatly help us to
> implement proper package management for Genode. For the latter, I'd love
> to embrace the Nix package manager.
> 
> As immediate steps, we should rework the remaining ports (as mentioned
> above) to use the new port mechanism and update the documentation, in
> particular the porting guide [4]
> 
> [4] https://github.com/genodelabs/genode/blob/master/doc/porting_guide.txt
> 
> Cheers
> Norman
> 

-- 
Stefan Kalkowski
Genode Labs

http://www.genode-labs.com/ · http://genode.org/