Dear Genodians,
I'm back at learning how to do packaging.
By following the developer-resources/porting_libraries guide, I ported the Hammer parser combinator library into a Genode port [1]. It builds and runs a simple test. (commit 665e71d).
Now I want to make it a depot package. I copied the structure from libports/expat and adapted it. I have a recipes/{api,src}/content.mk (commit XXXXXX)
I created the lib/symbols/hammer file by
! tool/abi_symbols bin/hammer.lib.so
Then I run the build:
! ~/genode/tool/depot/create guido/bin/x86_64/hammer UPDATE_VERSIONS=1 FORCE=1 KEEP_BUILD_DIR=1
It starts good: ! created guido/api/base/2018-11-23 ! created guido/api/hammer/2019-01-08 ! created guido/api/libc/2018-11-26 ! created guido/src/hammer/2019-01-08-a
However, I get this error during the compile of hammer. ! MERGE hammer.lib.so ! Error: hammer ABI contains Genode-internal symbol '__bss_start', ! Error: hammer ABI contains Genode-internal symbol '__eh_frame_start__' all from the symbol_blacklist defined in check_abi.
It ends with this error: recipe for target 'hammer.lib.checked' failed
So where did I go wrong?
(Maybe this is an item for the list of missing documentation).
Hello Guido,
On Wed, Jan 09, 2019 at 11:55:50PM +0100, Guido Witmond wrote:
I created the lib/symbols/hammer file by
! tool/abi_symbols bin/hammer.lib.so
Then I run the build:
! ~/genode/tool/depot/create guido/bin/x86_64/hammer UPDATE_VERSIONS=1 FORCE=1 KEEP_BUILD_DIR=1
It starts good: ! created guido/api/base/2018-11-23 ! created guido/api/hammer/2019-01-08 ! created guido/api/libc/2018-11-26 ! created guido/src/hammer/2019-01-08-a
However, I get this error during the compile of hammer. ! MERGE hammer.lib.so ! Error: hammer ABI contains Genode-internal symbol '__bss_start', ! Error: hammer ABI contains Genode-internal symbol '__eh_frame_start__' all from the symbol_blacklist defined in check_abi.
It ends with this error: recipe for target 'hammer.lib.checked' failed
I suggest you remove symbols from the list that should not be exported by the library. The same is true for expat
/tool/abi_symbols build/x86_64/var/libcache/expat/expat.lib.so > /tmp/expat.symbols diff -u repos/libports/lib/symbols/expat /tmp/expat.symbols
--- repos/libports/lib/symbols/expat 2018-12-05 16:27:41.427653704 +0100 +++ /tmp/expat.symbols 2019-01-10 09:26:40.289152230 +0100 @@ -79,3 +79,16 @@ XmlSizeOfUnknownEncoding T XmlUtf16Encode T XmlUtf8Encode T +__bss_start R 0 +__eh_frame_start__ R 0 +__exidx_end D 0 +__exidx_start D 0 +__l4sys_invoke_indirect D 0 +_ctors_end D 0 +_ctors_start D 0 +_edata R 0 +_end R 0 +_init T +_parent_cap D 0 +_parent_cap_local_name D 0 +_parent_cap_thread_id D 0
The abi_symbols tool is just a handy utility to generate a list of potential ABI symbols, which must be manually amended/checked.
Greets
Hello,
The abi_symbols tool is just a handy utility to generate a list of potential ABI symbols, which must be manually amended/checked.
I wonder, should we consider adding a black list of known Genode symbols to the abi_symbols tool? This would make the tool a bit more convenient. On the other hand, it may foster the (wrong) expectation that the output of the tool *is* an ABI as is. In practice, the manual curing step is important. E.g., symbols of library-internal interfaces should be removed. This step needs a human brain.
Given Guido's question, we should definitely enhance the documentation about using the abi_symbols tool. Since Guido referred to the porting guide, would that document be a suitable place?
Cheers Norman
Hello Norman,
I'm unsure if you addressed me by your questions but...
On Thu, Jan 10, 2019 at 11:04:49AM +0100, Norman Feske wrote:
I wonder, should we consider adding a black list of known Genode symbols to the abi_symbols tool? This would make the tool a bit more convenient. On the other hand, it may foster the (wrong) expectation that the output of the tool *is* an ABI as is. In practice, the manual curing step is important. E.g., symbols of library-internal interfaces should be removed. This step needs a human brain.
When writing my response I also tended to propose to add those symbols to the tool but came to the same conclusion: The ABI ssymbols list is the equivalent of an ordinary linker version script which explicitly defines the exported symbols of dynamic libraries in traditional systems. Therefore, the human-brain intervention during the ABI creation should be at least emphasized if not enforced.
Given Guido's question, we should definitely enhance the documentation about using the abi_symbols tool. Since Guido referred to the porting guide, would that document be a suitable place?
You're right, adding extended documentation of the tool usage to the guide supports the goal mentioned above, but currently I'm somewhat occupied by other labors to improve the documentation myself. Maybe Guido could contribute some words based on his practical experiences?
Regards
On 01/10/19 11:21, Christian Helmuth wrote:
The abi_symbols tool is just a handy utility to generate a list of potential ABI symbols, which must be manually amended/checked.
The release notes for 17.02 mentioned a manual process but did not offer any insight in what needs to be done.
On Thu, Jan 10, 2019 at 11:04:49AM +0100, Norman Feske wrote:
I wonder, should we consider adding a black list of known Genode symbols to the abi_symbols tool? This would make the tool a bit more convenient.
Since both tools go together, I think it would help a lot to filter out the 'forbidden' symbols in abi_symbols. I see two benefits: 1: it saves mental effort and menial labour that could be automated; 2: it prevents questions like mine :-)
On the other hand, it may foster the (wrong) expectation that the output of the tool *is* an ABI as is.
Now, that's good to write in the documentation.
In practice, the manual curing step is important. E.g., symbols of library-internal interfaces should be removed. This step needs a human brain.
This consideration was left out too. And the 'e.g.' suggests there are other considerations too. Now I'm curious to those.
Therefore, the human-brain intervention during the ABI creation should be at least emphasized if not enforced.
Please don't enforce, I have another consideration:
As application programmer, I look up the API in the documentation, examples, header files and source files (in that order). If I would call internal interfaces, I risk breakage when the library gets updated.
Give that, the ABI file does not seem to provide much intellectual value to me, I see it as a way to speed up compilation.
What is the effect on security when I leave internal symbols in the ABI? Would a process - that is linked to my library - be severely easier to subvert by an attacker who exploits a buffer overflow?
If not, I propose to filter the blacklist in abi_symbols (and check in check_abi) and call that the 'official' ABI. It may not be the cleanest approach but it saves mental effort (== time and money) in porting libraries. Please make it easy for programmers, not harder.
If it has not been build, I'll add the blacklist filter to abi_symbols and submit a merge request for it.
Given Guido's question, we should definitely enhance the documentation about using the abi_symbols tool. Since Guido referred to the porting guide, would that document be a suitable place?
I think a document to complement the porting guide would make sense.
You're right, adding extended documentation of the tool usage to the guide supports the goal mentioned above, but currently I'm somewhat occupied by other labors to improve the documentation myself. Maybe Guido could contribute some words based on his practical experiences?
If I get my depot package to work I'll blog about it, especially if the documentation is not there yet. Now that you have Genodians, it saves me the hassle to setup something myself. :-)
PS. About genodians.org. To make it easy, I'd put it behind a Caddy webserver proxy (running on linux). Caddy provides automatic Let's Encrypt certificates.
Cheers, Guido.
Hello Guido,
your posting does not bode well with me.
On the one hand, you acknowledge that your are unaware of the ramifications of the symbols definition.
On the other hand, you chose to ignore the sentiments brought forward by Christan Helmuth and me.
Your pull request [1] following-up your posting addresses your inconvenience while putting the burden of added technical debt on us by duplicating the list. So we find ourselves not only ignored but also confronted with the expectation that we should be happy to carry forward the added technical debt. That cannot have been your intention.
[1] https://github.com/genodelabs/genode/pull/3112
Given your response, I suspect that our concerns were not expressed clearly enough. So let me try again, using an analogy: There is a country road that enters a village. At the side of the road, just in front of the school, there is a warning sign. The sign is meant to raise attention, to prompt drivers to decelerate, to get conscious about the situation, and - if everything is clear - to slowly accelerate again.
You drive this road for the first time. You see the sign, you get conscious of the situation, but at this time, no children are crossing the road. So your conclusion is to remove the warning sign to make the traffic run smoother though the village. I hope you agree that this wouldn't be a positive contribution to society. ;-)
Granted, the existing "warning sign" is certainly far from ideal. Thanks to your original posting, we recognize that it is (1) confusing and (2) that there is currently no good documentation to interpret the error message. So we should definitely improve.
In [2] you find my take on point (1) - making the warning sign more clear. Point (2) will need to wait until the next time I'll undergo the process of porting a library. I'll enhance the porting guide then.
[2] https://github.com/nfeske/genode/commit/d7fd53cc8e933cdc8b967d7159965b540c18...
What is the effect on security when I leave internal symbols in the ABI? Would a process - that is linked to my library - be severely easier to subvert by an attacker who exploits a buffer overflow?
This is not a security problem but a potential robustness problem. ABI inconsistencies can result in runtime errors. Someone who offers a library should pay attention to the ABI in order to catch errors of library users at linking time, not at runtime.
Generally speaking, an ABI that exposes unofficial symbols is similarly bad as dead code. It smells.
But it's is not solely an aesthetic issue. Imagine porting a library where you use a wildcard to select the source files to be incorporated into the library. If the source directory happens to also contain the source code of a test program, this program will accidentally end up in the library. A review of the ABI symbols catches these kind of problems on the spot. In contrast, if such a problem goes unnoticed, it can produce the most weird kind of runtime errors. I assure you that the debugging of symbol aliasing problems can be very painful.
In short, treating the ABI symbols as a black box is a mistake. I think your desire for convenience is misplaced here.
Reviewing a library ABI is simple enough whenever library symbols are nicely prefixed with the library name. A human reviewer can easily spot such patterns. In contrast, symbols starting with '_' are usually not part of the official library interface.
Norman
Hello Norman, Christian,
On 01/15/19 13:11, Norman Feske wrote:
Hello Guido,
your posting does not bode well with me.
My sincere apologies if my previous email came across as too harsh. I never had the intention to be rude. I think it came from lack of knowledge on my side to correctly understand your view.
Your latest response provided lots of background so I think I know where our misunderstanding came from.
... using an analogy: There is a country road that enters a village. At the side of the road, just in front of the school, there is a warning sign. The sign is meant to raise attention, to prompt drivers to decelerate, to get conscious about the situation, and - if everything is clear - to slowly accelerate again.
You drive this road for the first time. You see the sign, you get conscious of the situation, but at this time, no children are crossing the road. So your conclusion is to remove the warning sign to make the traffic run smoother though the village. I hope you agree that this wouldn't be a positive contribution to society. ;-)
I like the analogy, it fits well:
It was my first time on that road. But what you designed as a warning sign, felt like a road block to me: I could not build the package to the point that I could test it works. I was completely stuck, without any road signs (documentation, release notes, emails, blogs) on how to navigate around it. So I tried to remove the road block, not to kill children, just to get moving :-)
From the earlier responses, I understood that it is possible to leave internal symbols in the ABI-file but it seemed you gravitated to make the warning/road-block enforced. Being stuck behind it, I wanted to prevent that from happening. Hence my request for convenience. But I failed in articulating my view of the road-block.
When I filtered out the blacklist in abi_symbols, I could compile my depot package. After some evenings of fruitless struggle, it felt like a victory to me. That's when I made the PR. It solved my problem, so I assumed it could be of value to you.
[...]
Someone who offers a library should pay attention to the ABI in order to catch errors of library users at linking time, not at runtime.
Agree. But I wasn't yet at the stage of offering it to anyone. I still needed to compile it.
It took me some iterations to get the recipe files correct. Having to review at every cycle - and losing my review work when changing the recipes - did not seem attractive to me. So I wanted to postpone the ABI-cleanup until my test code could compile against the package.
So I end with a plea to make that warning easily bypassed during package development. Not just for me but also for others who start packaging.
Regards, Guido.
Hi Guido,
thank you for clarifying your posting. Don't worry. It's all fine.
It took me some iterations to get the recipe files correct. Having to review at every cycle - and losing my review work when changing the recipes - did not seem attractive to me. So I wanted to postpone the ABI-cleanup until my test code could compile against the package.
In my experience, the creation of the ABI-symbols file is a step that comes not before the actual porting work is finished. A target can use/link a library without an ABI symbols file when built from the build directory (not from a depot source archive). So during the repetitive porting process, the library can be repeatedly linked and tested from within the build directory using a regular target. The packaging does not stand in the way here.
The creation of the symbols file it is typically a one-off task with subsequent manual curation - at packaging time, not at library-porting time. As far as I remember, I have rarely called the abi_symbols tool more than once while porting/packaging a library.
It is clear by now, that our typical workflow needs to be better documented.
So I end with a plea to make that warning easily bypassed during package development. Not just for me but also for others who start packaging.
Would the following command possibly work for you?
abi_symbols <your-library>.lib.so | grep -v '#' > <your-symbols-file>
Cheers Norman
-
Christian Helmuth -
Guido Witmond -
Norman Feske