Management of ported 3rd-party software

[Sartakov A. Vasily]

Hello, Norman. 

Maybe I had to say this early, but: 
We, and Maybe someone else, have own repository with packages. We can/would not merge it with main branch of Genode. We have used directory in root Genode source tree, like dde_linux, before. So, how third party repos should look like in new directory layout? 

> 
> Hello,
> 
> now that Genode's new directory structure is in place, let's take the
> next step towards a bullet-proof solution for integrating 3rd-party code
> with Genode (see [1] for the corresponding issue). The new solution,
> which is very much inspired by the fabulous Nix package manager [2]
> comes in the form of new tools to be found at 'tool/ports/' on the
> staging branch [3].
> 
> [1] https://github.com/genodelabs/genode/issues/1082
> [2] http://nixos.org/nix/
> [3] https://github.com/genodelabs/genode/commits/staging
> 
> Hereby, I'd like to briefly explain how the new solution works from the
> viewpoint of a Genode user, describe the steps needed to add a new port
> to Genode, and outline how we will evolutionary move from the old 'make
> prepare' mechanism to the new concept.
> 
> Note that even though the port mechanism described herein looks a bit
> like "package management", it covers a different problem. The problem
> covered here is the integration of existing 3rd-party source code with
> the Genode source tree. Packaging, on the other hand, would provide a
> means to distribute self-contained portions of the Genode source tree
> including their respective 3rd-party counterparts as separate packages.
> Package management is not addressed yet.
> 
> 
> The use of the new mechanism
> ----------------------------
> 
> Genode's source-code repositories used to come with Makefiles in their
> respective base directories. Those makefile provided the rules 'prepare'
> and 'clean'. The 'prepare' rule allowed for the automated downloading
> and installation of 3rd-party code, whereas the 'clean' rule reverted
> the installation. Source-code archives were downloaded at
> <rep-dir>/download/ whereas the extracted 3rd-party code usually resided
> at <rep-dir>/contrib/. In the case of ported libraries, the 'make
> prepare' step also used to create a bunch of symlinks within
> <rep-dir/include/ that pointed to the respective header files within
> <rep-dir>/contrib/.
> 
> The old 'make prepare' approach was implemented for each individual
> repository. In contrast, the new solution unifies the procedure across
> all the repositories located at <genode-dir>/repos/. To install a port
> provided by any repository, just invoke the tool
> <genode-dir>/tool/port/prepare_port with the name of the port as
> argument. The tool will scan all repositories for the specified port and
> install the port to <genode-dir>/contrib/. Each version of an installed
> port resides in a dedicated subdirectory within the contrib/ directory.
> The port-specific directory is called port directory. It is named
> <port-fingerprint>-<port-name>. The <fingerprint> uniquely identifies
> the version of the port (it is a SHA1 hash of the ingredients of the
> port). If two versions of the same port are installed, each of them will
> have a different fingerprint. So they and up in different directories.
> 
> Within the Genode source tree, a port is represented by two files, a
> <port-name>.port and a <port-name>.hash file. Both files reside at the
> <rep-dir>/ports/ subdirectory of the corresponding repository. The
> <port-name>.port file is the port description, which declares the
> ingredients of the port, e.g., archives to download, patches to apply.
> The <port-name>.hash file contains the fingerprint of the corresponding
> port description, thereby uniquely identifying a version of the port.
> 
> To see, which ports are available, look out for <rep-dir>/port/*.port files:
> 
>  find <genode-dir> -mindepth 4 -maxdepth 4 -name "*.port"
> 
> So how does Genode's build system find the source codes for the right
> port directory to use? If the build system encounters a target that
> incorporates ported source code, it looks up the respective
> <port-name>.hash file in the repositories as specified in the build
> configuration. The fingerprint found in the hash file is used to
> construct the path to the port directory under contrib/.
> 
> 
> Adding a port
> -------------
> 
> The basic steps of adding a new port to Genode are as follows. Let us
> assume the 3rd-party source codes comes in the form of a tar.gz archive.
> 
> 1) Create '<rep-dir>/ports/<port-name>.port' file:
> 
>  LICENSE := unknown
>  VERSION := <version>
>  DOWNLOADS := <archive-name>.archive
>  URL(<archive-name>) := http://the-url-of-the.archive.tar.gz
> 
> The 'DOWNLOADS' declaration contains a list of items to download. Each
> item is suffixed with the type of the download. Supported types are
> 'file' (a plain file), 'archive' (an archive of the types tar.gz,
> tar.xz, tgz, tar.bz2, or zip), 'git' (a Git repository), or 'svn' (a
> Subversion repository). For each item, there have to be a few additional
> declarations, in particular the URL where to to download it from.
> 
> 2) Create 'ports/<port-name>.hash' file with the content 'dummy'.
> 
> This one will be used during the porting work and replaced once the port
> if finished. Because the hash file contains the string "dummy", the port
> directory will be located at <genode-dir>/contrib/dummy-<port-name>/.
> 
> 3) Declaring the hash sum of a downloaded archive
> 
> Try to execute the new port file to download the archive.
> 
>  <genode-dir>/tool/ports/prepare_port <port-name>
> 
> This step will create the port directory and download the archive to the
> base of this directory. The prepare_port tool tries to validate the
> correct version of the downloaded file using an SHA1 hash sum. Because
> the port description lack the known-good SHA1 sum, the check will fail:
> 
>  Error: Hash sum check for <port-name> failed
> 
> Calculate the SHA1 hash sum of the archive:
> 
>  sha1sum <port-dir>/<archive-name>.tar.gz
> 
> Declare the hash sum of the archive in the <port-name>.port file:
> 
>  SHA(<archive-name>) := <hash-value>
> 
> When executing the prepare_port step again, the integrity check for the
> downloaded archive should succeed. However, we get the following error
> message:
> 
>  Error: <rep-dir>/ports/<port-name>.port is out of date, expected
> <fingerprint>
> 
> We get this message because we had specified the "dummy" hash value in
> the <port-name>.hash file. The prepare_port tool computes a fingerprint
> of the actual version of the port and compares this fingerprint with the
> hash value specified in <port-name>.hash. The computed fingerprint can
> be found at <port-dir>/<port-name>.hash. In the final step of the port,
> we will replace the dummy fingerprint with the actual fingerprint of the
> port. But before finalizing the porting work, it is practical to keep
> using the dummy hash and suppress the fingerprint check. This can be
> done by adding 'CHECK_HASH=no' as argument to the prepare_port tool:
> 
>  <genode-dir>/tool/ports/prepare-port <port-name> CHECK_HASH=no
> 
> 4) Extracting the source code
> 
> When executing the prepare-step now, the tool will present you with the
> following message:
> 
>  Error: Missing definition of DIR(<port-name>) in <port-file>
> 
> We need to declare where to extract the downloaded archive. E.g.,
> 
>  DIR(<port-name>) := src/lib/<port-name>
> 
> Each port directory is principally organized like a Genode source-code
> repository. So it is good practice to place the extracted code at the
> location where it would reside if hosted within a repository. Most
> source packages distributed as tar archives contain a directory named
> after the package and the version number in the top-level directory of
> the archive. By default, the prepare tool will strip this directory when
> extracting the archive. So the actual content will be installed at the
> directory specified via the 'DIR(<archive-name>)' declaration. You can
> override the default extraction argument by specifying a custom
> 'TAR_OPT(<archive-name>)' declaration.
> 
> 5) Declaration of the license
> 
> Now that you have downloaded and extracted the 3rd-party source code
> within the port directory, revisit the code for its license. Update the
> LICENSE declaration in the <port-name>.port file accordingly.
> 
> 6) Assembling the include directory exported by the port
> 
> Define include files to be presented to the Genode build system, e.g.:
> 
>  DIRS := include
>  DIR_CONTENT(include) src/lib/<port-name>/include/*.h
> 
> This declaration tells the prepare_port tool to copy the files found in
> the include/ directory of the extracted archive to the
> <port-dir>/include/ directory. Using this mechanism, arbitrary directory
> structures can be constructed out of the downloaded content. Validate
> that the DIRS declarations work as expected by executing the
> prepare_port step again and revisiting the content of the port directory.
> 
> 7) Using the ported code from Genode compilation targets
> 
> Now the preparation step is complete. The final piece of the puzzle is
> telling the Genode build system to use the port. Within any library
> description file, target.mk file, or import-*.mk file, you can use the
> function 'select_from_ports' to query a port directory using the port
> name as argument. E.g., assuming <port-name> refers to a library, the
> corresponding import-<port-name>.mk file may contain the following
> declaration:
> 
>  INC_DIR += $(call select_from_ports,<port-name>)/include
> 
> This declaration will add the <port-name>/include directory of the port
> to the include-search path of every target that uses the library.
> Similarly, the library description file may use the 'select_from_ports'
> function to define the vpath of the 3rd-party source codes.
> 
> Under the hood, the 'select_from_ports' function looks up the
> fingerprint of the specified port by reading the corresponding
> <port-name>.hash file. It then uses this hash value to construct the
> directory path within the 'contrib/' directory that belongs to the
> matching version of the port. If there is no hash file that matches the
> port name, or if the port directory does not exist, the build system
> will present you with an error message.
> 
> Finally, after having tested that both the preparation-step and the
> build of the ported source code works as expected, it is time to
> finalize the fingerprint stored in the <rep-dir>/ports/<port-name>.hash
> file. This can be done by copying the content of the
> <port-dir>/<port-name>.hash file. The next time, you invoke the
> prepare_port tool, do not specify the 'CHECK_HASH=no' argument. So the
> fingerprint check will validate that the <port-name>.hash file
> corresponds to your <port-name>.port file. From now on, the
> contrib/dummy-<port-name> directory will no longer be used because the
> <port-name>.hash file points to the port directory named after the real
> fingerprint.
> 
> 
> Transition to the new mechanism
> -------------------------------
> 
> In the last couple of days, I have reworked more than 60 ports to use
> the new mechanism. The ports not covered so far are:
> 
> * libports; qt5
> * base-codezero
> * base-fiasco
> * base-pistachio
> * base-foc
> * dde_rump
> * dde_linux
> * dde_ipxe
> * ports-foc
> * ports: gcc
> * ports: gdb
> * ports: seoul
> * ports: virtualbox
> 
> During the transition phase (the next release cycle), we will keep the
> original 'make prepare' mechanism as a front end. So the 'make prepare'
> instructions as found in many tutorials will still work. But under the
> hood, 'make prepare' will just invoke the new _tool/ports/prepare_port_
> tool.
> 
> In the current version at the staging branch, the hash sum check is
> disabled for all ports installed via the old 'make prepare' front end.
> Nearly all hash files contain "dummy". This is because the prepare_port
> tool is still in flux. Each change of the tool would require new
> fingerprints for all packages, which is inconvenient while working on
> the tool. However, once the development of the tools settles, the dummy
> hash sums will be replaced by the real hash sums.
> 
> 
> I hope that the new mechanism will make Genode more convenient to use.
> Hopefully, the errors caused by missing or outdated 'make prepare' steps
> will be a thing of the past. However, the current change is just another
> step. Once we have fully adopted the prepare_port tool, we can fairly
> easily add further utilities to work with ports, e.g., adding dependency
> information between ports, garbage collecting stale versions, updating
> all installed ports, etc. Also, the new way of how ports are organized
> (having a layout similar to Genode repositories) will greatly help us to
> implement proper package management for Genode. For the latter, I'd love
> to embrace the Nix package manager.
> 
> As immediate steps, we should rework the remaining ports (as mentioned
> above) to use the new port mechanism and update the documentation, in
> particular the porting guide [4]
> 
> [4] https://github.com/genodelabs/genode/blob/master/doc/porting_guide.txt
> 
> Cheers
> Norman
> 
> -- 
> Dr.-Ing. Norman Feske
> Genode Labs
> 
> http://www.genode-labs.com · http://genode.org
> 
> Genode Labs GmbH · Amtsgericht Dresden · HRB 28424 · Sitz Dresden
> Geschäftsführer: Dr.-Ing. Norman Feske, Christian Helmuth
> 
> ------------------------------------------------------------------------------
> "Accelerate Dev Cycles with Automated Cross-Browser Testing - For FREE
> Instantly run your Selenium tests across 300+ browser/OS combos.
> Get unparalleled scalability from the best Selenium testing platform available
> Simple to use. Nothing to install. Get started now for free."
> http://p.sf.net/sfu/SauceLabs
> _______________________________________________
> Genode-main mailing list
> Genode-main at lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/genode-main