Hello,
now that Genode's new directory structure is in place, let's take the next step towards a bullet-proof solution for integrating 3rd-party code with Genode (see [1] for the corresponding issue). The new solution, which is very much inspired by the fabulous Nix package manager [2] comes in the form of new tools to be found at 'tool/ports/' on the staging branch [3].
[1] https://github.com/genodelabs/genode/issues/1082 [2] http://nixos.org/nix/ [3] https://github.com/genodelabs/genode/commits/staging
Hereby, I'd like to briefly explain how the new solution works from the viewpoint of a Genode user, describe the steps needed to add a new port to Genode, and outline how we will evolutionary move from the old 'make prepare' mechanism to the new concept.
Note that even though the port mechanism described herein looks a bit like "package management", it covers a different problem. The problem covered here is the integration of existing 3rd-party source code with the Genode source tree. Packaging, on the other hand, would provide a means to distribute self-contained portions of the Genode source tree including their respective 3rd-party counterparts as separate packages. Package management is not addressed yet.
The use of the new mechanism ----------------------------
Genode's source-code repositories used to come with Makefiles in their respective base directories. Those makefile provided the rules 'prepare' and 'clean'. The 'prepare' rule allowed for the automated downloading and installation of 3rd-party code, whereas the 'clean' rule reverted the installation. Source-code archives were downloaded at <rep-dir>/download/ whereas the extracted 3rd-party code usually resided at <rep-dir>/contrib/. In the case of ported libraries, the 'make prepare' step also used to create a bunch of symlinks within <rep-dir/include/ that pointed to the respective header files within <rep-dir>/contrib/.
The old 'make prepare' approach was implemented for each individual repository. In contrast, the new solution unifies the procedure across all the repositories located at <genode-dir>/repos/. To install a port provided by any repository, just invoke the tool <genode-dir>/tool/port/prepare_port with the name of the port as argument. The tool will scan all repositories for the specified port and install the port to <genode-dir>/contrib/. Each version of an installed port resides in a dedicated subdirectory within the contrib/ directory. The port-specific directory is called port directory. It is named <port-fingerprint>-<port-name>. The <fingerprint> uniquely identifies the version of the port (it is a SHA1 hash of the ingredients of the port). If two versions of the same port are installed, each of them will have a different fingerprint. So they and up in different directories.
Within the Genode source tree, a port is represented by two files, a <port-name>.port and a <port-name>.hash file. Both files reside at the <rep-dir>/ports/ subdirectory of the corresponding repository. The <port-name>.port file is the port description, which declares the ingredients of the port, e.g., archives to download, patches to apply. The <port-name>.hash file contains the fingerprint of the corresponding port description, thereby uniquely identifying a version of the port.
To see, which ports are available, look out for <rep-dir>/port/*.port files:
find <genode-dir> -mindepth 4 -maxdepth 4 -name "*.port"
So how does Genode's build system find the source codes for the right port directory to use? If the build system encounters a target that incorporates ported source code, it looks up the respective <port-name>.hash file in the repositories as specified in the build configuration. The fingerprint found in the hash file is used to construct the path to the port directory under contrib/.
Adding a port -------------
The basic steps of adding a new port to Genode are as follows. Let us assume the 3rd-party source codes comes in the form of a tar.gz archive.
1) Create '<rep-dir>/ports/<port-name>.port' file:
LICENSE := unknown VERSION := <version> DOWNLOADS := <archive-name>.archive URL(<archive-name>) := http://the-url-of-the.archive.tar.gz
The 'DOWNLOADS' declaration contains a list of items to download. Each item is suffixed with the type of the download. Supported types are 'file' (a plain file), 'archive' (an archive of the types tar.gz, tar.xz, tgz, tar.bz2, or zip), 'git' (a Git repository), or 'svn' (a Subversion repository). For each item, there have to be a few additional declarations, in particular the URL where to to download it from.
2) Create 'ports/<port-name>.hash' file with the content 'dummy'.
This one will be used during the porting work and replaced once the port if finished. Because the hash file contains the string "dummy", the port directory will be located at <genode-dir>/contrib/dummy-<port-name>/.
3) Declaring the hash sum of a downloaded archive
Try to execute the new port file to download the archive.
<genode-dir>/tool/ports/prepare_port <port-name>
This step will create the port directory and download the archive to the base of this directory. The prepare_port tool tries to validate the correct version of the downloaded file using an SHA1 hash sum. Because the port description lack the known-good SHA1 sum, the check will fail:
Error: Hash sum check for <port-name> failed
Calculate the SHA1 hash sum of the archive:
sha1sum <port-dir>/<archive-name>.tar.gz
Declare the hash sum of the archive in the <port-name>.port file:
SHA(<archive-name>) := <hash-value>
When executing the prepare_port step again, the integrity check for the downloaded archive should succeed. However, we get the following error message:
Error: <rep-dir>/ports/<port-name>.port is out of date, expected <fingerprint>
We get this message because we had specified the "dummy" hash value in the <port-name>.hash file. The prepare_port tool computes a fingerprint of the actual version of the port and compares this fingerprint with the hash value specified in <port-name>.hash. The computed fingerprint can be found at <port-dir>/<port-name>.hash. In the final step of the port, we will replace the dummy fingerprint with the actual fingerprint of the port. But before finalizing the porting work, it is practical to keep using the dummy hash and suppress the fingerprint check. This can be done by adding 'CHECK_HASH=no' as argument to the prepare_port tool:
<genode-dir>/tool/ports/prepare-port <port-name> CHECK_HASH=no
4) Extracting the source code
When executing the prepare-step now, the tool will present you with the following message:
Error: Missing definition of DIR(<port-name>) in <port-file>
We need to declare where to extract the downloaded archive. E.g.,
DIR(<port-name>) := src/lib/<port-name>
Each port directory is principally organized like a Genode source-code repository. So it is good practice to place the extracted code at the location where it would reside if hosted within a repository. Most source packages distributed as tar archives contain a directory named after the package and the version number in the top-level directory of the archive. By default, the prepare tool will strip this directory when extracting the archive. So the actual content will be installed at the directory specified via the 'DIR(<archive-name>)' declaration. You can override the default extraction argument by specifying a custom 'TAR_OPT(<archive-name>)' declaration.
5) Declaration of the license
Now that you have downloaded and extracted the 3rd-party source code within the port directory, revisit the code for its license. Update the LICENSE declaration in the <port-name>.port file accordingly.
6) Assembling the include directory exported by the port
Define include files to be presented to the Genode build system, e.g.:
DIRS := include DIR_CONTENT(include) src/lib/<port-name>/include/*.h
This declaration tells the prepare_port tool to copy the files found in the include/ directory of the extracted archive to the <port-dir>/include/ directory. Using this mechanism, arbitrary directory structures can be constructed out of the downloaded content. Validate that the DIRS declarations work as expected by executing the prepare_port step again and revisiting the content of the port directory.
7) Using the ported code from Genode compilation targets
Now the preparation step is complete. The final piece of the puzzle is telling the Genode build system to use the port. Within any library description file, target.mk file, or import-*.mk file, you can use the function 'select_from_ports' to query a port directory using the port name as argument. E.g., assuming <port-name> refers to a library, the corresponding import-<port-name>.mk file may contain the following declaration:
INC_DIR += $(call select_from_ports,<port-name>)/include
This declaration will add the <port-name>/include directory of the port to the include-search path of every target that uses the library. Similarly, the library description file may use the 'select_from_ports' function to define the vpath of the 3rd-party source codes.
Under the hood, the 'select_from_ports' function looks up the fingerprint of the specified port by reading the corresponding <port-name>.hash file. It then uses this hash value to construct the directory path within the 'contrib/' directory that belongs to the matching version of the port. If there is no hash file that matches the port name, or if the port directory does not exist, the build system will present you with an error message.
Finally, after having tested that both the preparation-step and the build of the ported source code works as expected, it is time to finalize the fingerprint stored in the <rep-dir>/ports/<port-name>.hash file. This can be done by copying the content of the <port-dir>/<port-name>.hash file. The next time, you invoke the prepare_port tool, do not specify the 'CHECK_HASH=no' argument. So the fingerprint check will validate that the <port-name>.hash file corresponds to your <port-name>.port file. From now on, the contrib/dummy-<port-name> directory will no longer be used because the <port-name>.hash file points to the port directory named after the real fingerprint.
Transition to the new mechanism -------------------------------
In the last couple of days, I have reworked more than 60 ports to use the new mechanism. The ports not covered so far are:
* libports; qt5 * base-codezero * base-fiasco * base-pistachio * base-foc * dde_rump * dde_linux * dde_ipxe * ports-foc * ports: gcc * ports: gdb * ports: seoul * ports: virtualbox
During the transition phase (the next release cycle), we will keep the original 'make prepare' mechanism as a front end. So the 'make prepare' instructions as found in many tutorials will still work. But under the hood, 'make prepare' will just invoke the new _tool/ports/prepare_port_ tool.
In the current version at the staging branch, the hash sum check is disabled for all ports installed via the old 'make prepare' front end. Nearly all hash files contain "dummy". This is because the prepare_port tool is still in flux. Each change of the tool would require new fingerprints for all packages, which is inconvenient while working on the tool. However, once the development of the tools settles, the dummy hash sums will be replaced by the real hash sums.
I hope that the new mechanism will make Genode more convenient to use. Hopefully, the errors caused by missing or outdated 'make prepare' steps will be a thing of the past. However, the current change is just another step. Once we have fully adopted the prepare_port tool, we can fairly easily add further utilities to work with ports, e.g., adding dependency information between ports, garbage collecting stale versions, updating all installed ports, etc. Also, the new way of how ports are organized (having a layout similar to Genode repositories) will greatly help us to implement proper package management for Genode. For the latter, I'd love to embrace the Nix package manager.
As immediate steps, we should rework the remaining ports (as mentioned above) to use the new port mechanism and update the documentation, in particular the porting guide [4]
[4] https://github.com/genodelabs/genode/blob/master/doc/porting_guide.txt
Cheers Norman
Hello, Norman.
Maybe I had to say this early, but: We, and Maybe someone else, have own repository with packages. We can/would not merge it with main branch of Genode. We have used directory in root Genode source tree, like dde_linux, before. So, how third party repos should look like in new directory layout?
Hello,
now that Genode's new directory structure is in place, let's take the next step towards a bullet-proof solution for integrating 3rd-party code with Genode (see [1] for the corresponding issue). The new solution, which is very much inspired by the fabulous Nix package manager [2] comes in the form of new tools to be found at 'tool/ports/' on the staging branch [3].
[1] https://github.com/genodelabs/genode/issues/1082 [2] http://nixos.org/nix/ [3] https://github.com/genodelabs/genode/commits/staging
Hereby, I'd like to briefly explain how the new solution works from the viewpoint of a Genode user, describe the steps needed to add a new port to Genode, and outline how we will evolutionary move from the old 'make prepare' mechanism to the new concept.
Note that even though the port mechanism described herein looks a bit like "package management", it covers a different problem. The problem covered here is the integration of existing 3rd-party source code with the Genode source tree. Packaging, on the other hand, would provide a means to distribute self-contained portions of the Genode source tree including their respective 3rd-party counterparts as separate packages. Package management is not addressed yet.
The use of the new mechanism
Genode's source-code repositories used to come with Makefiles in their respective base directories. Those makefile provided the rules 'prepare' and 'clean'. The 'prepare' rule allowed for the automated downloading and installation of 3rd-party code, whereas the 'clean' rule reverted the installation. Source-code archives were downloaded at <rep-dir>/download/ whereas the extracted 3rd-party code usually resided at <rep-dir>/contrib/. In the case of ported libraries, the 'make prepare' step also used to create a bunch of symlinks within <rep-dir/include/ that pointed to the respective header files within <rep-dir>/contrib/.
The old 'make prepare' approach was implemented for each individual repository. In contrast, the new solution unifies the procedure across all the repositories located at <genode-dir>/repos/. To install a port provided by any repository, just invoke the tool <genode-dir>/tool/port/prepare_port with the name of the port as argument. The tool will scan all repositories for the specified port and install the port to <genode-dir>/contrib/. Each version of an installed port resides in a dedicated subdirectory within the contrib/ directory. The port-specific directory is called port directory. It is named <port-fingerprint>-<port-name>. The <fingerprint> uniquely identifies the version of the port (it is a SHA1 hash of the ingredients of the port). If two versions of the same port are installed, each of them will have a different fingerprint. So they and up in different directories.
Within the Genode source tree, a port is represented by two files, a <port-name>.port and a <port-name>.hash file. Both files reside at the <rep-dir>/ports/ subdirectory of the corresponding repository. The <port-name>.port file is the port description, which declares the ingredients of the port, e.g., archives to download, patches to apply. The <port-name>.hash file contains the fingerprint of the corresponding port description, thereby uniquely identifying a version of the port.
To see, which ports are available, look out for <rep-dir>/port/*.port files:
find <genode-dir> -mindepth 4 -maxdepth 4 -name "*.port"
So how does Genode's build system find the source codes for the right port directory to use? If the build system encounters a target that incorporates ported source code, it looks up the respective <port-name>.hash file in the repositories as specified in the build configuration. The fingerprint found in the hash file is used to construct the path to the port directory under contrib/.
Adding a port
The basic steps of adding a new port to Genode are as follows. Let us assume the 3rd-party source codes comes in the form of a tar.gz archive.
- Create '<rep-dir>/ports/<port-name>.port' file:
LICENSE := unknown VERSION := <version> DOWNLOADS := <archive-name>.archive URL(<archive-name>) := http://the-url-of-the.archive.tar.gz
The 'DOWNLOADS' declaration contains a list of items to download. Each item is suffixed with the type of the download. Supported types are 'file' (a plain file), 'archive' (an archive of the types tar.gz, tar.xz, tgz, tar.bz2, or zip), 'git' (a Git repository), or 'svn' (a Subversion repository). For each item, there have to be a few additional declarations, in particular the URL where to to download it from.
- Create 'ports/<port-name>.hash' file with the content 'dummy'.
This one will be used during the porting work and replaced once the port if finished. Because the hash file contains the string "dummy", the port directory will be located at <genode-dir>/contrib/dummy-<port-name>/.
- Declaring the hash sum of a downloaded archive
Try to execute the new port file to download the archive.
<genode-dir>/tool/ports/prepare_port <port-name>
This step will create the port directory and download the archive to the base of this directory. The prepare_port tool tries to validate the correct version of the downloaded file using an SHA1 hash sum. Because the port description lack the known-good SHA1 sum, the check will fail:
Error: Hash sum check for <port-name> failed
Calculate the SHA1 hash sum of the archive:
sha1sum <port-dir>/<archive-name>.tar.gz
Declare the hash sum of the archive in the <port-name>.port file:
SHA(<archive-name>) := <hash-value>
When executing the prepare_port step again, the integrity check for the downloaded archive should succeed. However, we get the following error message:
Error: <rep-dir>/ports/<port-name>.port is out of date, expected
<fingerprint>
We get this message because we had specified the "dummy" hash value in the <port-name>.hash file. The prepare_port tool computes a fingerprint of the actual version of the port and compares this fingerprint with the hash value specified in <port-name>.hash. The computed fingerprint can be found at <port-dir>/<port-name>.hash. In the final step of the port, we will replace the dummy fingerprint with the actual fingerprint of the port. But before finalizing the porting work, it is practical to keep using the dummy hash and suppress the fingerprint check. This can be done by adding 'CHECK_HASH=no' as argument to the prepare_port tool:
<genode-dir>/tool/ports/prepare-port <port-name> CHECK_HASH=no
- Extracting the source code
When executing the prepare-step now, the tool will present you with the following message:
Error: Missing definition of DIR(<port-name>) in <port-file>
We need to declare where to extract the downloaded archive. E.g.,
DIR(<port-name>) := src/lib/<port-name>
Each port directory is principally organized like a Genode source-code repository. So it is good practice to place the extracted code at the location where it would reside if hosted within a repository. Most source packages distributed as tar archives contain a directory named after the package and the version number in the top-level directory of the archive. By default, the prepare tool will strip this directory when extracting the archive. So the actual content will be installed at the directory specified via the 'DIR(<archive-name>)' declaration. You can override the default extraction argument by specifying a custom 'TAR_OPT(<archive-name>)' declaration.
- Declaration of the license
Now that you have downloaded and extracted the 3rd-party source code within the port directory, revisit the code for its license. Update the LICENSE declaration in the <port-name>.port file accordingly.
- Assembling the include directory exported by the port
Define include files to be presented to the Genode build system, e.g.:
DIRS := include DIR_CONTENT(include) src/lib/<port-name>/include/*.h
This declaration tells the prepare_port tool to copy the files found in the include/ directory of the extracted archive to the <port-dir>/include/ directory. Using this mechanism, arbitrary directory structures can be constructed out of the downloaded content. Validate that the DIRS declarations work as expected by executing the prepare_port step again and revisiting the content of the port directory.
- Using the ported code from Genode compilation targets
Now the preparation step is complete. The final piece of the puzzle is telling the Genode build system to use the port. Within any library description file, target.mk file, or import-*.mk file, you can use the function 'select_from_ports' to query a port directory using the port name as argument. E.g., assuming <port-name> refers to a library, the corresponding import-<port-name>.mk file may contain the following declaration:
INC_DIR += $(call select_from_ports,<port-name>)/include
This declaration will add the <port-name>/include directory of the port to the include-search path of every target that uses the library. Similarly, the library description file may use the 'select_from_ports' function to define the vpath of the 3rd-party source codes.
Under the hood, the 'select_from_ports' function looks up the fingerprint of the specified port by reading the corresponding <port-name>.hash file. It then uses this hash value to construct the directory path within the 'contrib/' directory that belongs to the matching version of the port. If there is no hash file that matches the port name, or if the port directory does not exist, the build system will present you with an error message.
Finally, after having tested that both the preparation-step and the build of the ported source code works as expected, it is time to finalize the fingerprint stored in the <rep-dir>/ports/<port-name>.hash file. This can be done by copying the content of the <port-dir>/<port-name>.hash file. The next time, you invoke the prepare_port tool, do not specify the 'CHECK_HASH=no' argument. So the fingerprint check will validate that the <port-name>.hash file corresponds to your <port-name>.port file. From now on, the contrib/dummy-<port-name> directory will no longer be used because the <port-name>.hash file points to the port directory named after the real fingerprint.
Transition to the new mechanism
In the last couple of days, I have reworked more than 60 ports to use the new mechanism. The ports not covered so far are:
- libports; qt5
- base-codezero
- base-fiasco
- base-pistachio
- base-foc
- dde_rump
- dde_linux
- dde_ipxe
- ports-foc
- ports: gcc
- ports: gdb
- ports: seoul
- ports: virtualbox
During the transition phase (the next release cycle), we will keep the original 'make prepare' mechanism as a front end. So the 'make prepare' instructions as found in many tutorials will still work. But under the hood, 'make prepare' will just invoke the new _tool/ports/prepare_port_ tool.
In the current version at the staging branch, the hash sum check is disabled for all ports installed via the old 'make prepare' front end. Nearly all hash files contain "dummy". This is because the prepare_port tool is still in flux. Each change of the tool would require new fingerprints for all packages, which is inconvenient while working on the tool. However, once the development of the tools settles, the dummy hash sums will be replaced by the real hash sums.
I hope that the new mechanism will make Genode more convenient to use. Hopefully, the errors caused by missing or outdated 'make prepare' steps will be a thing of the past. However, the current change is just another step. Once we have fully adopted the prepare_port tool, we can fairly easily add further utilities to work with ports, e.g., adding dependency information between ports, garbage collecting stale versions, updating all installed ports, etc. Also, the new way of how ports are organized (having a layout similar to Genode repositories) will greatly help us to implement proper package management for Genode. For the latter, I'd love to embrace the Nix package manager.
As immediate steps, we should rework the remaining ports (as mentioned above) to use the new port mechanism and update the documentation, in particular the porting guide [4]
[4] https://github.com/genodelabs/genode/blob/master/doc/porting_guide.txt
Cheers Norman
-- Dr.-Ing. Norman Feske Genode Labs
http://www.genode-labs.com · http://genode.org
Genode Labs GmbH · Amtsgericht Dresden · HRB 28424 · Sitz Dresden Geschäftsführer: Dr.-Ing. Norman Feske, Christian Helmuth
"Accelerate Dev Cycles with Automated Cross-Browser Testing - For FREE Instantly run your Selenium tests across 300+ browser/OS combos. Get unparalleled scalability from the best Selenium testing platform available Simple to use. Nothing to install. Get started now for free." http://p.sf.net/sfu/SauceLabs _______________________________________________ Genode-main mailing list Genode-main@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/genode-main
Hi Vasily,
Maybe I had to say this early, but: We, and Maybe someone else, have own repository with packages. We can/would not merge it with main branch of Genode. We have used directory in root Genode source tree, like dde_linux, before. So, how third party repos should look like in new directory layout?
there are two possible solutions.
First, you could place your own Git repository at '<genode-dir>/repos/'. E.g., 'repos/ksyslabs.git'. (it might be useful to add repos/*.git to the .gitignore file of Genode) Your repository will be automatically scanned for ports.
Alternatively, you may place your repository at any arbitrary place and just create a symlink at <genode-dir>/src/repos/ksyslabs pointing to /your/place/of/choice. To support the symlink variant, I have just added a small fixup commit to the staging branch.
Does that solve your problem?
Norman
Hello.
Maybe I had to say this early, but: We, and Maybe someone else, have own repository with packages. We can/would not merge it with main branch of Genode. We have used directory in root Genode source tree, like dde_linux, before. So, how third party repos should look like in new directory layout?
there are two possible solutions.
First, you could place your own Git repository at '<genode-dir>/repos/'. E.g., 'repos/ksyslabs.git'. (it might be useful to add repos/*.git to the .gitignore file of Genode) Your repository will be automatically scanned for ports.
Alternatively, you may place your repository at any arbitrary place and just create a symlink at <genode-dir>/src/repos/ksyslabs pointing to /your/place/of/choice. To support the symlink variant, I have just added a small fixup commit to the staging branch.
Does that solve your problem?
I think yes. Thank you. I will tell which way we will choose later.
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
Hello,
I just added a few more fixup commits to the staging branch:
https://github.com/genodelabs/genode/commits/staging
* Apparently, there are some versions of sha1sum that expect exactly two space characters between the hashsum and the filename when performing the check (using -c as argument). I had to adjust the prepare_port tool to generate this format.
* I was asked by Martin Stein why I picked <finger-print>-<port-name> as the name of the port directory. Putting the port name at the beginning of the name would be more convenient when listing the content of the contrib/ dir. There was actually no particular reason. So I changed the order. Thanks for the good suggestion!
Cheers Norman
On 05/19/2014 01:48 PM, Sartakov A. Vasily wrote:
Hello.
Maybe I had to say this early, but: We, and Maybe someone else, have own repository with packages. We can/would not merge it with main branch of Genode. We have used directory in root Genode source tree, like dde_linux, before. So, how third party repos should look like in new directory layout?
there are two possible solutions.
First, you could place your own Git repository at '<genode-dir>/repos/'. E.g., 'repos/ksyslabs.git'. (it might be useful to add repos/*.git to the .gitignore file of Genode) Your repository will be automatically scanned for ports.
Alternatively, you may place your repository at any arbitrary place and just create a symlink at <genode-dir>/src/repos/ksyslabs pointing to /your/place/of/choice. To support the symlink variant, I have just added a small fixup commit to the staging branch.
Does that solve your problem?
I think yes. Thank you. I will tell which way we will choose later.
- -- Dr.-Ing. Norman Feske Genode Labs
http://www.genode-labs.com · http://genode.org
Genode Labs GmbH · Amtsgericht Dresden · HRB 28424 · Sitz Dresden Geschäftsführer: Dr.-Ing. Norman Feske, Christian Helmuth
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
Hello,
as quick follow-up to my last email, I am happy to report that the transition to the new ports mechanism is almost complete. All tests are running fine and the master branch has been updated just now.
Thanks to everyone involved in doing the migration work, which went as smooth as I had wished for.
The only remaining ports not yet adapted to the new mechanism are qt5, gcc, gdb, and qt4. The former three will be adapted soon. The latter is scheduled for removal. So it won't be migrated to the new mechanism.
Cheers Norman
- -- Dr.-Ing. Norman Feske Genode Labs
http://www.genode-labs.com · http://genode.org
Genode Labs GmbH · Amtsgericht Dresden · HRB 28424 · Sitz Dresden Geschäftsführer: Dr.-Ing. Norman Feske, Christian Helmuth
Hi,
just a short remark for port file authors:
When a port preparation includes checking out a git repository, you can state the revision explicitly by the "REV" variable, e.g. in repos/libports/ports/lwip.port:
REV(lwip) := fe63f36656bd66b4051bdfab93e351a584337d7c
I've already seen that people state a branch name instead of a revision hash tag. That is non advisable, as branch names are not unique, and the source of the branch may change without the new port mechanism noticing that. To prevent users from experiencing inconsistencies between Genode and 3rd party code in the future, I'd like to advise you to take always hash tags instead of branch names.
Regards Stefan
On 05/19/2014 12:15 PM, Norman Feske wrote:
Hello,
now that Genode's new directory structure is in place, let's take the next step towards a bullet-proof solution for integrating 3rd-party code with Genode (see [1] for the corresponding issue). The new solution, which is very much inspired by the fabulous Nix package manager [2] comes in the form of new tools to be found at 'tool/ports/' on the staging branch [3].
[1] https://github.com/genodelabs/genode/issues/1082 [2] http://nixos.org/nix/ [3] https://github.com/genodelabs/genode/commits/staging
Hereby, I'd like to briefly explain how the new solution works from the viewpoint of a Genode user, describe the steps needed to add a new port to Genode, and outline how we will evolutionary move from the old 'make prepare' mechanism to the new concept.
Note that even though the port mechanism described herein looks a bit like "package management", it covers a different problem. The problem covered here is the integration of existing 3rd-party source code with the Genode source tree. Packaging, on the other hand, would provide a means to distribute self-contained portions of the Genode source tree including their respective 3rd-party counterparts as separate packages. Package management is not addressed yet.
The use of the new mechanism
Genode's source-code repositories used to come with Makefiles in their respective base directories. Those makefile provided the rules 'prepare' and 'clean'. The 'prepare' rule allowed for the automated downloading and installation of 3rd-party code, whereas the 'clean' rule reverted the installation. Source-code archives were downloaded at <rep-dir>/download/ whereas the extracted 3rd-party code usually resided at <rep-dir>/contrib/. In the case of ported libraries, the 'make prepare' step also used to create a bunch of symlinks within <rep-dir/include/ that pointed to the respective header files within <rep-dir>/contrib/.
The old 'make prepare' approach was implemented for each individual repository. In contrast, the new solution unifies the procedure across all the repositories located at <genode-dir>/repos/. To install a port provided by any repository, just invoke the tool <genode-dir>/tool/port/prepare_port with the name of the port as argument. The tool will scan all repositories for the specified port and install the port to <genode-dir>/contrib/. Each version of an installed port resides in a dedicated subdirectory within the contrib/ directory. The port-specific directory is called port directory. It is named <port-fingerprint>-<port-name>. The <fingerprint> uniquely identifies the version of the port (it is a SHA1 hash of the ingredients of the port). If two versions of the same port are installed, each of them will have a different fingerprint. So they and up in different directories.
Within the Genode source tree, a port is represented by two files, a <port-name>.port and a <port-name>.hash file. Both files reside at the <rep-dir>/ports/ subdirectory of the corresponding repository. The <port-name>.port file is the port description, which declares the ingredients of the port, e.g., archives to download, patches to apply. The <port-name>.hash file contains the fingerprint of the corresponding port description, thereby uniquely identifying a version of the port.
To see, which ports are available, look out for <rep-dir>/port/*.port files:
find <genode-dir> -mindepth 4 -maxdepth 4 -name "*.port"
So how does Genode's build system find the source codes for the right port directory to use? If the build system encounters a target that incorporates ported source code, it looks up the respective <port-name>.hash file in the repositories as specified in the build configuration. The fingerprint found in the hash file is used to construct the path to the port directory under contrib/.
Adding a port
The basic steps of adding a new port to Genode are as follows. Let us assume the 3rd-party source codes comes in the form of a tar.gz archive.
- Create '<rep-dir>/ports/<port-name>.port' file:
LICENSE := unknown VERSION := <version> DOWNLOADS := <archive-name>.archive URL(<archive-name>) := http://the-url-of-the.archive.tar.gz
The 'DOWNLOADS' declaration contains a list of items to download. Each item is suffixed with the type of the download. Supported types are 'file' (a plain file), 'archive' (an archive of the types tar.gz, tar.xz, tgz, tar.bz2, or zip), 'git' (a Git repository), or 'svn' (a Subversion repository). For each item, there have to be a few additional declarations, in particular the URL where to to download it from.
- Create 'ports/<port-name>.hash' file with the content 'dummy'.
This one will be used during the porting work and replaced once the port if finished. Because the hash file contains the string "dummy", the port directory will be located at <genode-dir>/contrib/dummy-<port-name>/.
- Declaring the hash sum of a downloaded archive
Try to execute the new port file to download the archive.
<genode-dir>/tool/ports/prepare_port <port-name>
This step will create the port directory and download the archive to the base of this directory. The prepare_port tool tries to validate the correct version of the downloaded file using an SHA1 hash sum. Because the port description lack the known-good SHA1 sum, the check will fail:
Error: Hash sum check for <port-name> failed
Calculate the SHA1 hash sum of the archive:
sha1sum <port-dir>/<archive-name>.tar.gz
Declare the hash sum of the archive in the <port-name>.port file:
SHA(<archive-name>) := <hash-value>
When executing the prepare_port step again, the integrity check for the downloaded archive should succeed. However, we get the following error message:
Error: <rep-dir>/ports/<port-name>.port is out of date, expected
<fingerprint>
We get this message because we had specified the "dummy" hash value in the <port-name>.hash file. The prepare_port tool computes a fingerprint of the actual version of the port and compares this fingerprint with the hash value specified in <port-name>.hash. The computed fingerprint can be found at <port-dir>/<port-name>.hash. In the final step of the port, we will replace the dummy fingerprint with the actual fingerprint of the port. But before finalizing the porting work, it is practical to keep using the dummy hash and suppress the fingerprint check. This can be done by adding 'CHECK_HASH=no' as argument to the prepare_port tool:
<genode-dir>/tool/ports/prepare-port <port-name> CHECK_HASH=no
- Extracting the source code
When executing the prepare-step now, the tool will present you with the following message:
Error: Missing definition of DIR(<port-name>) in <port-file>
We need to declare where to extract the downloaded archive. E.g.,
DIR(<port-name>) := src/lib/<port-name>
Each port directory is principally organized like a Genode source-code repository. So it is good practice to place the extracted code at the location where it would reside if hosted within a repository. Most source packages distributed as tar archives contain a directory named after the package and the version number in the top-level directory of the archive. By default, the prepare tool will strip this directory when extracting the archive. So the actual content will be installed at the directory specified via the 'DIR(<archive-name>)' declaration. You can override the default extraction argument by specifying a custom 'TAR_OPT(<archive-name>)' declaration.
- Declaration of the license
Now that you have downloaded and extracted the 3rd-party source code within the port directory, revisit the code for its license. Update the LICENSE declaration in the <port-name>.port file accordingly.
- Assembling the include directory exported by the port
Define include files to be presented to the Genode build system, e.g.:
DIRS := include DIR_CONTENT(include) src/lib/<port-name>/include/*.h
This declaration tells the prepare_port tool to copy the files found in the include/ directory of the extracted archive to the <port-dir>/include/ directory. Using this mechanism, arbitrary directory structures can be constructed out of the downloaded content. Validate that the DIRS declarations work as expected by executing the prepare_port step again and revisiting the content of the port directory.
- Using the ported code from Genode compilation targets
Now the preparation step is complete. The final piece of the puzzle is telling the Genode build system to use the port. Within any library description file, target.mk file, or import-*.mk file, you can use the function 'select_from_ports' to query a port directory using the port name as argument. E.g., assuming <port-name> refers to a library, the corresponding import-<port-name>.mk file may contain the following declaration:
INC_DIR += $(call select_from_ports,<port-name>)/include
This declaration will add the <port-name>/include directory of the port to the include-search path of every target that uses the library. Similarly, the library description file may use the 'select_from_ports' function to define the vpath of the 3rd-party source codes.
Under the hood, the 'select_from_ports' function looks up the fingerprint of the specified port by reading the corresponding <port-name>.hash file. It then uses this hash value to construct the directory path within the 'contrib/' directory that belongs to the matching version of the port. If there is no hash file that matches the port name, or if the port directory does not exist, the build system will present you with an error message.
Finally, after having tested that both the preparation-step and the build of the ported source code works as expected, it is time to finalize the fingerprint stored in the <rep-dir>/ports/<port-name>.hash file. This can be done by copying the content of the <port-dir>/<port-name>.hash file. The next time, you invoke the prepare_port tool, do not specify the 'CHECK_HASH=no' argument. So the fingerprint check will validate that the <port-name>.hash file corresponds to your <port-name>.port file. From now on, the contrib/dummy-<port-name> directory will no longer be used because the <port-name>.hash file points to the port directory named after the real fingerprint.
Transition to the new mechanism
In the last couple of days, I have reworked more than 60 ports to use the new mechanism. The ports not covered so far are:
- libports; qt5
- base-codezero
- base-fiasco
- base-pistachio
- base-foc
- dde_rump
- dde_linux
- dde_ipxe
- ports-foc
- ports: gcc
- ports: gdb
- ports: seoul
- ports: virtualbox
During the transition phase (the next release cycle), we will keep the original 'make prepare' mechanism as a front end. So the 'make prepare' instructions as found in many tutorials will still work. But under the hood, 'make prepare' will just invoke the new _tool/ports/prepare_port_ tool.
In the current version at the staging branch, the hash sum check is disabled for all ports installed via the old 'make prepare' front end. Nearly all hash files contain "dummy". This is because the prepare_port tool is still in flux. Each change of the tool would require new fingerprints for all packages, which is inconvenient while working on the tool. However, once the development of the tools settles, the dummy hash sums will be replaced by the real hash sums.
I hope that the new mechanism will make Genode more convenient to use. Hopefully, the errors caused by missing or outdated 'make prepare' steps will be a thing of the past. However, the current change is just another step. Once we have fully adopted the prepare_port tool, we can fairly easily add further utilities to work with ports, e.g., adding dependency information between ports, garbage collecting stale versions, updating all installed ports, etc. Also, the new way of how ports are organized (having a layout similar to Genode repositories) will greatly help us to implement proper package management for Genode. For the latter, I'd love to embrace the Nix package manager.
As immediate steps, we should rework the remaining ports (as mentioned above) to use the new port mechanism and update the documentation, in particular the porting guide [4]
[4] https://github.com/genodelabs/genode/blob/master/doc/porting_guide.txt
Cheers Norman
Hi,
On Wed, 21 May 2014 13:21:21 +0200 Stefan Kalkowski <stefan.kalkowski@...1...> wrote:
Hi,
just a short remark for port file authors:
When a port preparation includes checking out a git repository, you can state the revision explicitly by the "REV" variable, e.g. in repos/libports/ports/lwip.port:
REV(lwip) := fe63f36656bd66b4051bdfab93e351a584337d7c
I've already seen that people state a branch name instead of a revision hash tag. That is non advisable, as branch names are not unique, and the source of the branch may change without the new port mechanism noticing that. To prevent users from experiencing inconsistencies between Genode and 3rd party code in the future, I'd like to advise you to take always hash tags instead of branch names.
Do you want to re-invent "git submodule"?
--
- ptr
Hi Petr,
On Thu, May 22, 2014 at 09:10:35AM +0400, Petr Ovtchenkov wrote:
Do you want to re-invent "git submodule"?
That is not our intention.Instead, we're striving to develop unified and clear mechanism to incorporate third-party software of any kind not just Git repositories into Genode. "Porting" a software to Genode includes several steps besides fetching the source code. Please refer to Norman's original email on an explanation.
Regards
Hi Petr,
On 05/22/14 07:10, Petr Ovtchenkov wrote:
[…] Do you want to re-invent "git submodule"?
tl;dr No.
The new port mechanism merely unifies the already existing solution for integrating 3rd-party software with Genode. That being said, the port mechanism and git submodules both address different issues. The former obtains 3rd-party software from varying sources that in some cases happen to be git repositories and prepares the source code for utilization within Genode or rather its build system. Git submodules on the other extend the actual repository be embedding forgein repositories in it, e.g. make them available in the actual repository.
This would work for obtaining 3rd-party software available in git repositories only, obviously. By using this approach we would have to treat these 3rd-party software sources in a different way from all other 3rd-party sources and we would im embedded them in the Genode repository. Instead of creating diverging ways to obtain 3rd-party source code by using a special feature of one kind of origin of 3rd-party source code we treat them all equally and clearly dinstinguish between their origin.
If you really want to compare the new port mechanism in a loose way, it is more or less similar to the one used by one or the other BSD port framework.
Cheers Josen
Gave it a try now by myself and I'm really happy about the new way of preparing ports. Just a short question in the margin: Is it wanted (besides the eye-sugar like output of prepare_port :-) ) that git clone still routes its own output to stdout?
-
Christian Helmuth -
Josef Söntgen -
Martin Stein -
Norman Feske -
Petr Ovtchenkov -
Sartakov A. Vasily -
Stefan Kalkowski