Hello Genodians
As part of a project, I will port WasmEdge [1] to Genode.
As WasmEdge uses cmake as build system my idea is to use GOA to port it. There is already a port of WasmEdge to the SEL4 kernel [2].
WasmEdge uses several libraries of which some also use cmake as build system.
To start small I first want to port the `spdlog` library. Before working on the big dependency in form of the `lld` library from the `llvm` project.
If I issue `goa build` successfully runs `cmake` but building fails afterward. There are a lot of `[-Werror=effc++]`.
I tired different ways to set `CC_CXX_WARN_STRICT=` in the files `cmake_args` and `make_args`. Unfortunately none of them helped.
Can anybody give me some hints how I could achive this?
My project is located at [3].
Regards, Pirmin
[1] https://github.com/WasmEdge/WasmEdge [2] https://github.com/second-state/wasmedge-seL4.git [3] https://github.com/trimpim/wasmedge-genode
Hi Pirmin,
as a word of caution, we haven't used Goa for the porting of libraries yet. In particular, the linking arguments for building shared objects are currently not handled by Goa. Goa is also not equipped with a way to publish a library's ABI (symbol file + headers). We eventually want to go in this direction, but you are the first. ;-)
If I issue `goa build` successfully runs `cmake` but building fails afterward. There are a lot of `[-Werror=effc++]`.
I tired different ways to set `CC_CXX_WARN_STRICT=` in the files `cmake_args` and `make_args`. Unfortunately none of them helped.
The strict default can be overridden in the two following ways.
1. Telling goa to be less strict at the command line:
$ goa build --no-warn-strict
2. Placing a .goarc file in your project directory with the following line:
set warn_strict 0
BTW, while investigating your question, I noticed a slight inconsistency in the documentation of Goa's configuration ('goa help config') that I fixed just now. So thanks for bringing up the topic.
Cheers Norman
Hi Norman
On 11/11/22 09:05, Norman Feske wrote:
Hi Pirmin,
as a word of caution, we haven't used Goa for the porting of libraries yet. In particular, the linking arguments for building shared objects are currently not handled by Goa. Goa is also not equipped with a way to publish a library's ABI (symbol file + headers). We eventually want to go in this direction, but you are the first. ;-)
From what I had seen before, I already had guessed, that that might be the case. Thanks for the information, that is good to know.
If I issue `goa build` successfully runs `cmake` but building fails afterward. There are a lot of `[-Werror=effc++]`.
I tired different ways to set `CC_CXX_WARN_STRICT=` in the files `cmake_args` and `make_args`. Unfortunately none of them helped.
The strict default can be overridden in the two following ways.
- Telling goa to be less strict at the command line:
$ goa build --no-warn-strict
- Placing a .goarc file in your project directory with the
following line:
set warn_strict 0
This works like a charm.
BTW, while investigating your question, I noticed a slight inconsistency in the documentation of Goa's configuration ('goa help config') that I fixed just now. So thanks for bringing up the topic.
happy to help :-)
Regards Pirmin
Hello Genodians
On 11/11/22 12:37, Pirmin Duss wrote:
Hi Norman
On 11/11/22 09:05, Norman Feske wrote:
Hi Pirmin,
as a word of caution, we haven't used Goa for the porting of libraries yet. In particular, the linking arguments for building shared objects are currently not handled by Goa. Goa is also not equipped with a way to publish a library's ABI (symbol file + headers). We eventually want to go in this direction, but you are the first. ;-)
From what I had seen before, I already had guessed, that that might be the case. Thanks for the information, that is good to know.
I have now a first version of Goa that supports creation of libraries.
There are still some rough edges and the code needs some cleaning up and squashing of commits.
The current state can be found at [1].
Regards Pirmin
Hi Pirmin,
I have now a first version of Goa that supports creation of libraries.
[...]
thank you for sharing your work, which looks very promising.
The following thoughts crossed my mind when looking through the commits of your branch.
- It is very good to see how you followed Goa's internal patterns, even down to the documentation. :)
Down the road, it would be good to avoid the code repetitions in 'extract_api_artifacts'. From cursory look, there seem to be many similarities with the existing 'extract_artifacts_from_build_dir' function. So it may be worth factoring out the reusable parts of the existing code into small utility functions as an intermediate commit.
- The ABI (in the form of the symbols file) should better not be implicitely created from the linked library. A manual curation step should remain part of the work flow. The (manual) invocation of the abi_symbols utility (e.g., disguised as a goa command) can possibly streamline this step (see below) but it should not transparent.
The main reason is that the symbol sizes are sometimes critical. A symbol file created via the abi_symbols tool from a 32-bit .lib.so file won't always work on 64-bit architectures. A secondary reason is that an ABI is sometimes only a subset of the symbols found in an .lib.so file. Removing symbol noise is good for keeping the ABI stable over time. The subset can be best carved out by manual inspection.
Hence, the symbols files should be handled as part of the Goa project, not generated automatically. To do so, we'd need a convention, e.g., hosting symbol files always in project-local files called 'symbols/<libname>' in the project directory. The name of the <libname> would correspond to the base name of the .lib.so file. With such a convention in place, Goa could aid the update/extraction of symbols via a command like:
goa extract-abi-symbols
This command would trigger the library build (to get hold of the .lib.so files), call the abi_symbols for the libraries, and thereby update the content of the symbols/ directory. This new version could then undergo human inspection before being commited into the project.
Note that this command would only be needed by the maintainer of the library (or library port), not the users of the Goa project who merely want to reproduce the build for using the library.
- I was surprised to find Genode's linker script [2] genode_rel.ld absent from your commit series. Do the resulting shared libraries actually work with Genode's dynamic linker? I wouldn't have expected that.
- Goa should not depend on the Genode source tree. The tools required by Goa should be hosted locally under Goa's share/goa/ directory. I consistently did that for the (parts of the) depot, ports, and gosh tools and would like to continue this path. In a distant future, I imagine that regular Goa users won't need to look at Genode's source tree at all, similar to how application developers on Linux don't need to look at the Linux kernel's source tree.
Now, looking again at your latest branch, I saw that you already applied this pattern. Very nice!
[2] https://github.com/genodelabs/genode/blob/master/repos/base/src/ld/genode_re...
Cheers Norman
Hello Norman
On 24.11.22 11:52, Norman Feske wrote:
Hi Pirmin,
I have now a first version of Goa that supports creation of libraries.
[...]
thank you for sharing your work, which looks very promising.
Thanks for the kind words.
The following thoughts crossed my mind when looking through the commits of your branch.
- It is very good to see how you followed Goa's internal patterns,
even down to the documentation. :)
Down the road, it would be good to avoid the code repetitions in 'extract_api_artifacts'. From cursory look, there seem to be many similarities with the existing 'extract_artifacts_from_build_dir' function. So it may be worth factoring out the reusable parts of the existing code into small utility functions as an intermediate commit.
I agree with you. I had this on my radar already, but wanted to know the differences between the different uses of such a utility function first. I will create such a commit.
- The ABI (in the form of the symbols file) should better not be
implicitely created from the linked library. A manual curation step should remain part of the work flow. The (manual) invocation of the abi_symbols utility (e.g., disguised as a goa command) can possibly streamline this step (see below) but it should not transparent.
The main reason is that the symbol sizes are sometimes critical. A symbol file created via the abi_symbols tool from a 32-bit .lib.so file won't always work on 64-bit architectures. A secondary reason is that an ABI is sometimes only a subset of the symbols found in an .lib.so file. Removing symbol noise is good for keeping the ABI stable over time. The subset can be best carved out by manual inspection.
Hence, the symbols files should be handled as part of the Goa project, not generated automatically. To do so, we'd need a convention, e.g., hosting symbol files always in project-local files called 'symbols/<libname>' in the project directory. The name of the <libname> would correspond to the base name of the .lib.so file. With such a convention in place, Goa could aid the update/extraction of symbols via a command like:
goa extract-abi-symbols
I see your point on this, and will change the process accordingly.
Should I also support something like 'symbols/<arch>/<libname>' for libraries that need different symbol files per platform?
- I was surprised to find Genode's linker script [2] genode_rel.ld
absent from your commit series. Do the resulting shared libraries actually work with Genode's dynamic linker? I wouldn't have expected that.
When I fist tried to build a library for the `arm_v8a` platform, I noticed this and other errors that were in the library part.
With my latest commit (pushed just after receiving your mail), I fixed that. I can now run a the test program of the `spdlog` library on x86 using the library created with Goa [1].
This is currently only tested with cmake libraries, as I do not have any qmake of autoconf to test this with. Should I add a warning / error if somebody tries to use the library feature in the untested cases?
I added an other commit that uses the depot_user for the archives when building the Linux run environment.
When using the depot directory of my Genode source tree, I noticed one minor "annoyance" now that the `pubkey` and `download` files have moved to a different location, Goa doesn't find them and complains. If you wont mind, I will add an option to specify the location(s) of the depot users.
[1] https://github.com/trimpim/wasmedge-genode/tree/spdlog_library
Cheers Pirmin
Hi Pirmin,
Should I also support something like 'symbols/<arch>/<libname>' for libraries that need different symbol files per platform?
Since we harmonized the binary interfaces between 32-bit and 64-bit (see [1]), the symbol files look almost identical. The only difference is the size of global data symbols that sometimes are larger on 64-bit. Fortunately, those symbols are rare, and the larger sizes work well for both 32-bit and 64-bit.
A distinction between architectures would conserve a bit of memory for 32-bit platforms while incurring a substantial burden in terms of bureaucracy and maintenance. So we deliberately dropped this distinction for Genode.
[1] https://genode.org/documentation/release-notes/17.02#Genode_Application_Bina...
To mitigate the risk of symbol files generated from a 32-bit .lib.so to be used on a 64-bit architecture, the Genode build system invokes the check_abi tool [2] after linking each shared library. Besides safety-checking the symbol sizes, the tool also performs a few other useful sanity checks. I think that Goa should follow these footsteps by invoking this tool, safeguarding the maintenance of ABIs.
[2] https://github.com/genodelabs/genode/blob/master/tool/check_abi
With my latest commit (pushed just after receiving your mail), I fixed that. I can now run a the test program of the `spdlog` library on x86 using the library created with Goa [1].
Thank you for clarifying, and congratulation for making it work!
I find it is interesting that the generated shared libraries work, which are - in your latest version - still linked without the genode_rel.ld script.
I'm not quite sure that the unconditional linkage of ld.lib.so is the right way to go, though. Why should a plain POSIX application or library depend on Genode's ABI after all?
The important part should be the linking of the 'ldso_so_support' library (as provided by the api/so depot archive) to .lib.so files and presumably the use of genode_rel.ld - essentially mirroring what Genode's build system does.
This is currently only tested with cmake libraries, as I do not have any qmake of autoconf to test this with. Should I add a warning / error if somebody tries to use the library feature in the untested cases?
If such a warning can be added without much trouble, I'm for it. Mentioning this limitation in the documentation would be perfectly fine also.
I added an other commit that uses the depot_user for the archives when building the Linux run environment.
When using the depot directory of my Genode source tree, I noticed one minor "annoyance" now that the `pubkey` and `download` files have moved to a different location, Goa doesn't find them and complains. If you wont mind, I will add an option to specify the location(s) of the depot users.
Does that mean that adding depot keys via 'goa add-depot-user' does not work?
goa add-depot-user <name> --depot-url <url> --pubkey-file <file>
When configuring a depot location via the 'depot_dir' configuration variable, this step should add the keys to the referenced depot.
I think what you are observing is that the "key ring" is initially empty because genode/depot/ no longer hosts any keys by default. But ultimately this is how it should be, doesn't it?
Regarding your commit "cmake support sub directory for cmake", have you considered solving this issue by placing a CMakeLists.txt file with the following line in your project's src/ directory?
add_subdirectory(relative/path/to/subdirectory)
By keeping this tweak as patch inside the project's patches/ directory (to be applied on 'goa import'), you should be able to refer to any sub directory (or even multiple of them) via Goa's existing cmake support.
Cheers Norman
Hello Norman
To mitigate the risk of symbol files generated from a 32-bit .lib.so to be used on a 64-bit architecture, the Genode build system invokes the check_abi tool [2] after linking each shared library. Besides safety-checking the symbol sizes, the tool also performs a few other useful sanity checks. I think that Goa should follow these footsteps by invoking this tool, safeguarding the maintenance of ABIs.
[2] https://github.com/genodelabs/genode/blob/master/tool/check_abi
Thanks for the directional explanation, I will also integrate the `check_abi` tool.
Thank you for clarifying, and congratulation for making it work!
I find it is interesting that the generated shared libraries work, which are - in your latest version - still linked without the genode_rel.ld script.
I'm not quite sure that the unconditional linkage of ld.lib.so is the right way to go, though. Why should a plain POSIX application or library depend on Genode's ABI after all?
The important part should be the linking of the 'ldso_so_support' library (as provided by the api/so depot archive) to .lib.so files and presumably the use of genode_rel.ld - essentially mirroring what Genode's build system does.
Thanks for explaining these internals. Using the Genode build system I never had to think about these things.
I will adapt my changes to take everything mentioned above in to account.
If such a warning can be added without much trouble, I'm for it. Mentioning this limitation in the documentation would be perfectly fine also.
Looking at the code, I think, the only place where the warning can easily be added, is when extracting the library symbols. Everywhere else it isn't clear if a library, a binary or both are built.
Adding the information to the documentation is definitively really important. I'll do that.
When using the depot directory of my Genode source tree, I noticed one minor "annoyance" now that the `pubkey` and `download` files have moved to a different location, Goa doesn't find them and complains. If you wont mind, I will add an option to specify the location(s) of the depot users.
Does that mean that adding depot keys via 'goa add-depot-user' does not work?
goa add-depot-user <name> --depot-url <url> --pubkey-file <file>
When configuring a depot location via the 'depot_dir' configuration variable, this step should add the keys to the referenced depot.
I think what you are observing is that the "key ring" is initially empty because genode/depot/ no longer hosts any keys by default. But ultimately this is how it should be, doesn't it?
As I used the shared depot, I thought that I could skip over this step. But I agree with your explanation, and the general work-flow.
What do you think about adding the possibility to copy the files from a sculpt directory the user already uses for "normal" Genode work?
goa add-depot-user <name> --sculpt-path <path/to/sculpt>
When used like that, there would probably no need to reset the depot directory. That way one could save some time recompiling artifacts already existing.
Regarding your commit "cmake support sub directory for cmake", have you considered solving this issue by placing a CMakeLists.txt file with the following line in your project's src/ directory?
add_subdirectory(relative/path/to/subdirectory)
By keeping this tweak as patch inside the project's patches/ directory (to be applied on 'goa import'), you should be able to refer to any sub directory (or even multiple of them) via Goa's existing cmake support.
As I do not really know CMake, I didn't think of that. With a few more lines added this works. Therefore this patch will no longer be included.
Regards, Pirmin
Hello Norman
I'm still working on the support to build libraries with Goa.
Currently I'm debugging a page fault that happens, when I load `libunwind.lib.so`. This library was built with Goa from the sources of llvm 11.1.0.
Yesterday I recognized an other problem, that I will have to figure out a solution for. One of the libraries that is used to build wasmedge, doesn't have a nice include directory that can be copied. The header files are located in the source directory. I will have to adapt my solution for extracting the header files.
I also was able to build my first library which uses autoconf/libtool to configure and build. This required (a lot) of patching in the input files thou and clearly isn't ready for prime time in the current state.
The current state can be found at https://github.com/trimpim/goa/tree/library_support. This still requires heavy cleanup, which I will do next year, after I have completed some of my other tasks.
Merry Christmas and a happy new year to everybody Pirmin
Hi Pirmin,
I'm still working on the support to build libraries with Goa. [...] The current state can be found at https://github.com/trimpim/goa/tree/library_support. This still requires heavy cleanup, which I will do next year, after I have completed some of my other tasks.
thank you for the welcome update. I'm happy to see that you took my advice about the check_abi tool to heart. By casually browsing the commits, I got the impression that you had to modify it. We should keep in mind that those changes should eventually be backported to the Genode repository. I always try to keep the files shared between Genode and Goa (like the ports/mk/ content) identical between both source trees.
Yesterday I recognized an other problem, that I will have to figure out a solution for. One of the libraries that is used to build wasmedge, doesn't have a nice include directory that can be copied. The header files are located in the source directory. I will have to adapt my solution for extracting the header files.
That's an interesting point I wasn't fully aware of. In the general case, such headers may become available not before building the 3rd-party code (if they are generated by a script executed as part of the build process). So the extraction of "API artifacts" follows similar patterns as the extraction of the binaries as build artifacts.
But in contrast to build artifacts that originate from the build directory, the content of an API archive is assembled from the source directory (headers imported as plain source code), the build directory (generated headers), and the symbols directory. I'm puzzled how this assemblage could be expressed best.
I also was able to build my first library which uses autoconf/libtool to configure and build. This required (a lot) of patching in the input files thou and clearly isn't ready for prime time in the current state.
I'm in awe! Building Genode-compatible shared objects with autoconf looks like an impossible maze to me.
If the Genode tool chain had built-in heuristics regarding linker scripts and startup code following closer the lines of regular Linux tool chains, this may become easier. E.g., when calling gcc on Linux, the compiler knows where to find the libc, the libc headers, and the startup code. In contrast, with Genode's tool chain, we give explicit arguments.
Admittedly, I'm divided. On the one hand, I'd wish to remove pain points that stand in the way of porting of existing software. On the other hand, I think of builtin magic as muddy. I wonder, could that conflict in principle be solved by providing a separate tool chain - containing the libc API, etc. - for the use with Goa? Not a suggestion, merely sharing my vague thoughts about it.
Cheers Norman
Hello Norman
On 03.01.23 14:58, Norman Feske wrote:
Hi Pirmin,
I'm still working on the support to build libraries with Goa. [...]
I have cleaned up the commit series.
50dc357 is a bit big for my liking, but I failed to split it up in to smaller, reasonable commits.
The current state can be found at https://github.com/trimpim/goa/tree/library_support. This still requires heavy cleanup, which I will do next year, after I have completed some of my other tasks.
thank you for the welcome update. I'm happy to see that you took my advice about the check_abi tool to heart. By casually browsing the commits, I got the impression that you had to modify it. We should keep in mind that those changes should eventually be backported to the Genode repository. I always try to keep the files shared between Genode and Goa (like the ports/mk/ content) identical between both source trees.
I agree, and will crate an issue on the main Repository that "ports back" the changes.
One question I have, is regarding the output format. The original writes to stdout. For the Goa integration I changed it to write to a file, as I did fail to parse the output of `abi_symbols_from_library`. I guess, the "native" implementation should still write to stdout in the end? Therefore I think I will change `abi_symbols_from_library` to take a file descriptor as the second parameter. Do you think this is a reasonable approach? Os should it work in a completely different way?
[...]
That's an interesting point I wasn't fully aware of. In the general case, such headers may become available not before building the 3rd-party code (if they are generated by a script executed as part of the build process). So the extraction of "API artifacts" follows similar patterns as the extraction of the binaries as build artifacts.
I have now changed the extraction of headers to be taken from the build directory.
I had to patch some CMakeLists.txt files in my project, but in the long run I think, this makes the whole process much cleaner. This way the case of automatically generated headers (autoconf/automake) is also handled properly.
I also was able to build my first library which uses autoconf/libtool to configure and build. This required (a lot) of patching in the input files thou and clearly isn't ready for prime time in the current state.
I'm in awe! Building Genode-compatible shared objects with autoconf looks like an impossible maze to me.
I fully agree with you, it wasn't at all a pleasing experience, especially compared to projects using CMake.
[...]
Admittedly, I'm divided. On the one hand, I'd wish to remove pain points that stand in the way of porting of existing software. On the other hand, I think of builtin magic as muddy. I wonder, could that conflict in principle be solved by providing a separate tool chain - containing the libc API, etc. - for the use with Goa? Not a suggestion, merely sharing my vague thoughts about it.
This sounds like a cool idea. But will probably be a major effort to undertake.
With all the changes, I'm now able to build WasmEdge which uses the libcxx, libcxxabi and libunwind from the LLVM project.
Cheers Pirmin
-
Duss Pirmin -
Norman Feske -
Pirmin Duss