as a part of efforts related to golang port I make separate patch for 21.02 genode related to implementation of setcontext/getcontext/makecontext/swapcontext family in libc
as a base I borrow some code from glibc, and remove related files (mostly _ucontext.h/ucontext.h) from genode
related 2 patches are in https://github.com/tor-m6/genode.git in 21.02 repo main one contains implementation to be applied to libc port aux patch contains only simple test for these functions, you can run it as
I need also to add C support for alloc_secondary_stack/free_secondary_stack in dummies.cc to be able to use them during allocation of stack for makecontext/etc (it should work even in C files - current code do requires C++, I even can’t test patch without them). May be this is wrong place for the wrappers…
current problems with the patch: is support only x86_64 (while can be easily extended to arm64) on nova (while should work on other OSes), and it do not support signals save/restore (while seems that not many of them supported by genode currently). Also some removal of code from libc is not that clear and correct (implement it in patch).
Question for Norman (and may be other maintainers): is it suitable for main genode code, or I should keep patch only in my fork? Is it worth to make a pull request?
Sincerely, Alexander
Hello Alexander,
thanks for keeping the golang port effort going.
as a part of efforts related to golang port I make separate patch for 21.02 genode related to implementation of setcontext/getcontext/makecontext/swapcontext family in libc
as a base I borrow some code from glibc, and remove related files (mostly _ucontext.h/ucontext.h) from genode
May I ask what is the reason for choosing and incorporating the glibc implementation rather than using/enabling the one in the FreeBSD libc?
related 2 patches are in https://github.com/tor-m6/genode.git in 21.02 repo main one contains implementation to be applied to libc port aux patch contains only simple test for these functions, you can run it as
I need also to add C support for alloc_secondary_stack/free_secondary_stack in dummies.cc to be able to use them during allocation of stack for makecontext/etc (it should work even in C files - current code do requires C++, I even can’t test patch without them). May be this is wrong place for the wrappers…
current problems with the patch: is support only x86_64 (while can be easily extended to arm64) on nova (while should work on other OSes), and it do not support signals save/restore (while seems that not many of them supported by genode currently). Also some removal of code from libc is not that clear and correct (implement it in patch).
Question for Norman (and may be other maintainers): is it suitable for main genode code, or I should keep patch only in my fork? Is it worth to make a pull request?
Strictly speaking containg such a large patch makes the commit unsuitable for the Genode main repo as the burden of maintaing it falls on our shoulders at the end and the reason for including it in the first place is not immediately clear.
For better or worse, we still lack the resources (read man-power) as well as a deeper understanding for the issues at hand when it comes to the requirements of the golang-runtime (which somewhat is a consequence of the lack of resources) that such an undertaking arguably deserves.
Regards Josef
Hello Josef, Some relatively long answers below
as a base I borrow some code from glibc, and remove related files (mostly _ucontext.h/ucontext.h) from genode
May I ask what is the reason for choosing and incorporating the glibc implementation rather than using/enabling the one in the FreeBSD libc?
1. current implementation of context switch in FreeBSD done completely in kernel via sys call. E.g. this is full code of getcontext function (stored in genode, not used):
#include <machine/asm.h> __FBSDID("$FreeBSD: releng/12.0/lib/libc/i386/sys/getcontext.S 258451 2013-11-21 22:31:18Z andreast $");
#include <SYS.h>
/* * This has to be magic to handle the multiple returns. * Otherwise, the setcontext() syscall will return here and we'll * pop off the return address and go to the *setcontext* call. */ WEAK_REFERENCE(__sys_getcontext, _getcontext) WEAK_REFERENCE(__sys_getcontext, getcontext) ENTRY(__sys_getcontext) movl (%esp),%ecx /* save getcontext return address */ mov $SYS_getcontext,%eax KERNCALL jb HIDENAME(cerror) addl $4,%esp /* remove stale (setcontext) return address */ jmp *%ecx /* restore return address */ END(__sys_getcontext)
.section .note.GNU-stack,"",%progbits
2. context family support do require complex and misleading structure, e.g. take a look for setcontext code: #include <sys/cdefs.h> __FBSDID("$FreeBSD: releng/12.0/lib/libc/sys/setcontext.c 326576 2017-12-05 20:19:13Z emaste $");
#include <sys/types.h> #include <ucontext.h> #include "libc_private.h"
__weak_reference(__sys_setcontext, __setcontext); __sym_compat(setcontext, __impl_setcontext, FBSD_1.0); __weak_reference(setcontext, __impl_setcontext); __sym_default(setcontext, setcontext, FBSD_1.2);
int setcontext(const ucontext_t *uc) {
return (((int (*)(const ucontext_t *)) __libc_interposing[INTERPOS_setcontext])(uc)); }
I try to understand how all these __libc_interposing works - and fail, this is not currently supported in genode as I understand
3. ucontext.h and related is isolated in current libc and could be easily replaced (because it used only in *context family of functions and signals which has only rudimentary support
4. glibc implementation is straightforward and well supported, relatively small and suitable for big set of platforms, without complex kernel/__libc_interposing analogous and could be borrowed (copied) with minimal efforts, I can confirm this.
5. glibc implementation of these functions do contain potentially useful security feature of shadow stack support (under SHSTK_ENABLED define - not enabled in my patch)
So, I consider option to use existing freeBSD version and replacement of code of functions to ones from glibc - and think that this make everything too complex, related infrastructure not clear and not easy to support. May be I am wrong.
for me simplest for implementation and, especially, for future support - is chosen approach (with part of glibc port). It is simple and straightforward, easy to extend to arm/etc.
Another option I consider is a usage of native context-switch functions for genode low level Os (OS - dependent). Anyway, it will significantly lower performance, and not portable option, so, I do not choose it.
Strictly speaking containg such a large patch makes the commit unsuitable for the Genode main repo as the burden of maintaing it falls on our shoulders at the end and the reason for including it in the first place is not immediately clear.
significant part of patch is a removal of old files like ucontext.h and plain addition of new files, take a look for statistics, no complex updates of files involved: fgrep @@ ./repos/libports/src/lib/libc/patches/zmcontext.patch @@ -0,0 +1,1 @@ @@ -0,0 +1,1 @@ @@ -0,0 +1,1 @@ @@ -0,0 +1,7 @@ @@ -0,0 +1,7 @@ @@ -0,0 +1,1092 @@ @@ -0,0 +1,3 @@ @@ -0,0 +1,3 @@ @@ -0,0 +1,279 @@ @@ -0,0 +1,590 @@ @@ -0,0 +1,3 @@ @@ -0,0 +1,18 @@ @@ -0,0 +1,124 @@ @@ -0,0 +1,137 @@ @@ -1,109 +1,155 @@ @@ -0,0 +1,201 @@ @@ -0,0 +1,256 @@ @@ -0,0 +1,28 @@ @@ -1,48 +0,0 @@ @@ -0,0 +1,97 @@ @@ -0,0 +1,113 @@ @@ -0,0 +1,129 @@ @@ -1,167 +0,0 @@ @@ -1,54 +0,0 @@
By the way, I can make patch smaller if you give me a direction how to remove some files during install or patch process without patch itself, may be via .mk file (e.g. libc-gen.mk)
For better or worse, we still lack the resources (read man-power) as well as a deeper understanding for the issues at hand when it comes to the requirements of the golang-runtime (which somewhat is a consequence of the lack of resources) that such an undertaking arguably deserves.
I need *context() implementation, and it can’t be moved to world repository - its a part of libc.
Also to use functions I need C interface for allocate_seconday_stack() which should be somewhere inside (I choose libc dummies.cchttp://dummies.cc file) Question: may be better other place? e.g. inline C interface function as a part of e.g. thread.h? Any recommendations are welcome!
I understand your point, so, another question is: how I can continue if patch can’t be stored inside genode core? it can’t be significantly smaller (4 functions/files to be add at least)...
Everything else for golang support I will have in world, this is the only part of core (may be together with extended anonymous mmap support).
Sincerely, Alexander
Hello Alexander,
as a base I borrow some code from glibc, and remove related files (mostly _ucontext.h/ucontext.h) from genode
May I ask what is the reason for choosing and incorporating the glibc implementation rather than using/enabling the one in the FreeBSD libc?
- current implementation of context switch in FreeBSD done completely
in kernel via sys call. E.g. this is full code of getcontext function (stored in genode, not used):
[…]
- context family support do require complex and misleading structure,
e.g. take a look for setcontext code:
[…]
- ucontext.h and related is isolated in current libc and could be
easily replaced (because it used only in *context family of functions and signals which has only rudimentary support
- glibc implementation is straightforward and well supported,
relatively small and suitable for big set of platforms, without complex kernel/__libc_interposing analogous and could be borrowed (copied) with minimal efforts, I can confirm this.
- glibc implementation of these functions do contain potentially
useful security feature of shadow stack support (under SHSTK_ENABLED define - not enabled in my patch)
So, I consider option to use existing freeBSD version and replacement of code of functions to ones from glibc - and think that this make everything too complex, related infrastructure not clear and not easy to support. May be I am wrong.
for me simplest for implementation and, especially, for future support
- is chosen approach (with part of glibc port). It is simple and
straightforward, easy to extend to arm/etc.
Another option I consider is a usage of native context-switch functions for genode low level Os (OS - dependent). Anyway, it will significantly lower performance, and not portable option, so, I do not choose it.
Strictly speaking containg such a large patch makes the commit unsuitable for the Genode main repo as the burden of maintaing it falls on our shoulders at the end and the reason for including it in the first place is not immediately clear.
significant part of patch is a removal of old files like ucontext.h and plain addition of new files, take a look for statistics, no complex updates of files involved:
[…]
By the way, I can make patch smaller if you give me a direction how to remove some files during install or patch process without patch itself, may be via .mk file (e.g. libc-gen.mk)
Thanks for the thorough answer, which puts things into perspective. We briefly discussed the matter offline and somehow implementing the API with a mechanism that is already there (e.g. the setjmp/longjmp based task model that we use in various drivers and the libc) feels more natural. That being said, we do not know if there are any requirements in the *context API that would prevent us from going down that road.
Do you know of any, besides the potentially worse performance?
For better or worse, we still lack the resources (read man-power) as well as a deeper understanding for the issues at hand when it comes to the requirements of the golang-runtime (which somewhat is a consequence of the lack of resources) that such an undertaking arguably deserves.
I need *context() implementation, and it can’t be moved to world repository - its a part of libc.
Also to use functions I need C interface for allocate_seconday_stack() which should be somewhere inside (I choose libc dummies.cchttp://dummies.cc file) Question: may be better other place? e.g. inline C interface function as a part of e.g. thread.h? Any recommendations are welcome!
I understand your point, so, another question is: how I can continue if patch can’t be stored inside genode core? it can’t be significantly smaller (4 functions/files to be add at least)...
Everything else for golang support I will have in world, this is the only part of core (may be together with extended anonymous mmap support).
Judging by the freestanding nature of the '*context()' implementation should it not be possible to get away with putting it into world as well? For example as part of a supplementary static run-time library that shadows the libc by resolving everything locally, including the stack functions.
Regards Josef
Hello Josef,
Thanks for the thorough answer, which puts things into perspective. We briefly discussed the matter offline and somehow implementing the API with a mechanism that is already there (e.g. the setjmp/longjmp based task model that we use in various drivers and the libc) feels more natural. That being said, we do not know if there are any requirements in the *context API that would prevent us from going down that road.
Do you know of any, besides the potentially worse performance?
So, you want to keep FreeBSD structure and related functions by replacement of the code inside old infrastructure to direct switch ones? See some doubts below (except some potential problems related to «native» FreeBSD infrastructure, as mentioned in previous letter).
Note: Under "worse performance" I mean that current FreeBSD borrowed code assume context switch to kernel and back for every get/set context. In theory, we can call native context-switching functions from underlaying kernel (while it will be unique for every kernel, no common code). I don’t like this approach as non-portable.
And I still have these questions to be answered to move further:
Also to use functions I need C interface for allocate_seconday_stack() which should be somewhere inside (I choose libc dummies.cchttp://dummies.cchttp://dummies.cc file) Question: may be better other place? e.g. inline C interface function as a part of e.g. thread.h? Any recommendations are welcome!
I understand your point, so, another question is: how I can continue if patch can’t be stored inside genode core? it can’t be significantly smaller (4 functions/files to be add at least)...
Everything else for golang support I will have in world, this is the only part of core (may be together with extended anonymous mmap support).
Judging by the freestanding nature of the '*context()' implementation should it not be possible to get away with putting it into world as well? For example as part of a supplementary static run-time library that shadows the libc by resolving everything locally, including the stack functions.
the problem that this family together with headers and ucontext structure is a part of general libc. how I can mask its usage, e.g. in configure scripts? I potentially need to install / modify part of files installed by libc, Eg signal.h from libc (include/libc/signal.h) do include #include <machine/ucontext.h> #include <sys/_ucontext.h> and it is not clear for me how to mask existing files if I have 3-d party application (not mine which can include own ucontext.h in repos/world ), e.g. in repos/ports? It I replace header search path - it will be complex mix of native FreeBSD-esh headers from Genode, own application code and my world replacement code for *context...
Another problem that setcontex/getcontext should be called on the same nested level, you can’t put easily wrapper above them (stack should be restored correctly).
Due to specific of Genode we have another potential problem - we can’t use standard malloc() for stack allocation because stack should be from specific memory area to allow context switch, so, external packages compatibility will be limited anyhow.
By the way, what is a reason for this? it is relatively fast and easy to make a separation between «main» and «other» threads using, e.g. registry...
And, last but not least - setcontex/getcontext should store and restore signal context (including signal handlers) which is not that clear for me in Genode. Currently is is limited, and where it should go later? how we can store/restore state of signals for current thread?
So, returning back. I can try to put everything in *context with uniq name (replace original files and function names) into world repo, and force libgo to call these function. Anyway, any other software porting will require manual efforts mainly because of old/rest of headers from libc.
And I still need to patch part of libc related to anonymous memory mapping. Updated implementation for libc 21.02:
https://github.com/tor-m6/genode/commit/c64bfeeba6e8cca4376636bdd2e9e8bbf3aa...
One more attempt to implement [set,get,make,swap]context inside libc scope, much smaller - only asm files moved from previous attempt (overall around 400+- lines): git apply --stat ../mc.patch repos/libports/lib/import/import-libc-context.mk | 1 repos/libports/lib/mk/libc.mk | 2 repos/libports/lib/mk/spec/x86_64/libc-context.mk | 9 + repos/libports/lib/symbols/libc | 2 repos/libports/ports/libc.hash | 2 repos/libports/recipes/api/libc_setjmp/content.mk | 2 repos/libports/src/lib/libc/dummies.cc | 28 +- repos/libports/src/lib/libc/patches/mcontext.patch | 346 ++++++++++++++++++++ 8 files changed, 387 insertions(+), 5 deletions(-)
https://github.com/tor-m6/genode/commit/b5ed6a49b0b6c09b20161b1ea726dbb685ae...
Also add test for them, see separate patch. Not yet choose where to put, potentially should be near makecontext() declaration inside libc include/libc/ucontext.h. extern "C" void *alloc_secondary_stack(char const *name, size_t stack_size);
Josef, please, take a look for this patch and for anon_mmap!
My understanding of limitations: - x86_64 only (easy to extend to others like arm64) - ignore set/restore of signals (need kernel support?) - replace/add old code from libc references to interpose table/etc; not 100% sure that all related aux functions like __getcontextx will work as FreeBSD intended… - require alloc_secondary_stack for correct operations in makecontext (the only way to work) - free_secondary_stack not yet tested actively - could be some problems with alignment/etc related to secondary stack allocation - do not restore temporary/etc registers (for amd64 preserve only rbp/rbx/r12/r13/r14/r15/rsi/rdi/rdx/rcx/r8/r9/rip/rsp general registers and fp_state and MXCSR) - not sure that fpstate stored and restored correctly, not tested yet
And I still need to patch part of libc related to anonymous memory mapping. Updated implementation for libc 21.02:
https://github.com/tor-m6/genode/commit/c64bfeeba6e8cca4376636bdd2e9e8bbf3aa...
Hi Alexander,
On 3/18/21 10:33 AM, Alexander Tormasov via users wrote:
One more attempt to implement [set,get,make,swap]context inside libc scope, much smaller
- only asm files moved from previous attempt (overall around 400+- lines):
git apply --stat ../mc.patch repos/libports/lib/import/import-libc-context.mk | 1 repos/libports/lib/mk/libc.mk | 2 repos/libports/lib/mk/spec/x86_64/libc-context.mk | 9 + repos/libports/lib/symbols/libc | 2 repos/libports/ports/libc.hash | 2 repos/libports/recipes/api/libc_setjmp/content.mk | 2 repos/libports/src/lib/libc/dummies.cc | 28 +- repos/libports/src/lib/libc/patches/mcontext.patch | 346 ++++++++++++++++++++ 8 files changed, 387 insertions(+), 5 deletions(-)
https://github.com/tor-m6/genode/commit/b5ed6a49b0b6c09b20161b1ea726dbb685ae...
Also add test for them, see separate patch. Not yet choose where to put, potentially should be near makecontext() declaration inside libc include/libc/ucontext.h. extern "C" void *alloc_secondary_stack(char const *name, size_t stack_size);
Josef, please, take a look for this patch and for anon_mmap!
Me an Josef took a look at your anon_mmap patch. First of all we used your Genode 21.02 [1] branch and Genode world master [2]. In order to compile your "go_app" run script the patches in [3] had to be applied by us.
The call to 'alloc_secondary_stack' in 'proc.c' of libgo should use the name as its first argument, the size is the second argument. See patch [4], otherwise an arbitrary stack size will be allocated (whatever is in the rsi register).
After this 'alloc_secondary_stack' caused a page fault, because 'stack->top' will give the stack beginning aligned to 16 byte + 8 byte for x86. In case the stack size is a multiple of page size, this causes a page fault in 'memset' because the page at the end of the stack is crossed. Adding 16 byte to the allocation size quick fixes the problem. In the next iteration we received a page fault at a high address in 'src/lib/gcc/libgo/runtime/proc.c:610' in 'makeGContext' caused by the value of 'uc'. We have not debugged this any further for now.
Regarding some of your questions:
I would suggest that you move 'alloc_secondary_stack' and 'free_secondary_stack' into it's own library lets call it 'libgo_support' into world and link it into libgo. In the same library I would implement the mmap_anon version. For this to work you can change the name of the 'mmap' call within libgo in file [6] there is the definition of 'sysMmap'
!//extern mmap !func sysMmap(addr unsafe.Pointer, n uintptr, prot, flags, fd int32, off uintptr) unsafe.Pointer
If one changes the "extern" directive a call to a different function is generated. For example,
! //extern mmap_anon
will lead to 'mmap_anon' being called instead of 'mmap'. The 'mmap_anon' function could handle the ANON|FIXED cases and wrap the other functionality by calling libc's mmap. This way your map anon code could reside next to libc without the risk of breaking anything generic.
I hope this helps a little,
Sebastian
[1] https://github.com/tor-m6/genode/tree/21.02 [2] https://github.com/tor-m6/genode-world [3] world.patch [4] src.patch [5] alloc_secondary_stack.patch [6] src/lib/gcc/libgo/go/runtime/mem_gccgo.go
Hi Sebastian, thanks for answer
Josef, please, take a look for this patch and for anon_mmap!
Me an Josef took a look at your anon_mmap patch. First of all we used your Genode 21.02 [1] branch and Genode world master [2]. In order to compile your "go_app" run script the patches in [3] had to be applied by us.
The call to 'alloc_secondary_stack' in 'proc.c' of libgo should use the name as its first argument, the size is the second argument. See patch [4], otherwise an arbitrary stack size will be allocated (whatever is in the rsi register).
yes, I fix this in the local copy and forget to update a patch for lingo port, because it is not tracked by git… my fault, will submit later fix with it and other problems. it is still not working in my go_app in the same reason as below, I assume alloc_secondary_stack to store stack structure and requested (rounded) size safely which seems not true (may be it worth to fix in genode code in general in base/src/lib/base/thread.cc Thread::_alloc_stack(size_t stack_size, char const *name, bool main_thread) - it seems a real mistake (I ask for my stack of stack_size but return something which I can’t use from first to last byte?)
After this 'alloc_secondary_stack' caused a page fault, because 'stack->top' will give the stack beginning aligned to 16 byte + 8 byte for x86. In case the stack size is a multiple of page size, this causes a page fault in 'memset' because the page at the end of the stack is crossed. Adding 16 byte to the allocation size quick fixes the problem. In the next iteration we received a page fault at a high address in 'src/lib/gcc/libgo/runtime/proc.c:610' in 'makeGContext' caused by the value of 'uc'. We have not debugged this any further for now.
so, will debug this, may be this is because of new setcontext implementation, and then submit a patch.
Regarding some of your questions:
I would suggest that you move 'alloc_secondary_stack' and 'free_secondary_stack' into it's own library lets call it 'libgo_support' into world and link it into libgo. In the same library I would implement the mmap_anon version. For this to work you can change the name of the 'mmap' call within libgo in file [6] there is the definition of 'sysMmap'
!//extern mmap !func sysMmap(addr unsafe.Pointer, n uintptr, prot, flags, fd int32, off uintptr) unsafe.Pointer
If one changes the "extern" directive a call to a different function is generated. For example,
! //extern mmap_anon
will lead to 'mmap_anon' being called instead of 'mmap'. The 'mmap_anon' function could handle the ANON|FIXED cases and wrap the other functionality by calling libc's mmap. This way your map anon code could reside next to libc without the risk of breaking anything generic.
in general I will move both functions to the separate lib as suggested, anon_mmap.c is a bit more complicated in logics…
ok, will do.
Forget to mention that current state of golang/libgo 21.02 support is not intended for evaluation
After this 'alloc_secondary_stack' caused a page fault, because 'stack->top' will give the stack beginning aligned to 16 byte + 8 byte for x86. In case the stack size is a multiple of page size, this causes a page fault in 'memset' because the page at the end of the stack is crossed. Adding 16 byte to the allocation size quick fixes the problem. In the next iteration we received a page fault at a high address in 'src/lib/gcc/libgo/runtime/proc.c:610' in 'makeGContext' caused by the value of 'uc'. We have not debugged this any further for now.
I start porting of it from old version and it is in transit state now. As you can see, I do not yet finish debug of new version of libgo, so, no reasons in this moment to check it.
for *context functions I add separate test here, please use it
https://github.com/tor-m6/genode/commit/73649e4a5d98cf83c8bed56ca17f300dd8a0...
I think that good idea to finalise context/anon parts of required genode core functionality (move them to separate support lib) and only after I will return to libgo fixes.
Hello Alexander,
So, you want to keep FreeBSD structure and related functions by replacement of the code inside old infrastructure to direct switch ones? See some doubts below (except some potential problems related to «native» FreeBSD infrastructure, as mentioned in previous letter).
Note: Under "worse performance" I mean that current FreeBSD borrowed code assume context switch to kernel and back for every get/set context. In theory, we can call native context-switching functions from underlaying kernel (while it will be unique for every kernel, no common code). I don’t like this approach as non-portable.
That is true - normally you do not have access to the kernel directly on Genode (I am not even sure if all kernels supported by Genode provide such functionality) and that will not change, especially for entirely libc-based components.
the problem that this family together with headers and ucontext structure is a part of general libc. how I can mask its usage, e.g. in configure scripts?
That is indeed a problem and one of the reasons why I already mentioned that using configure for building *libraries* for Genode is not the way to go. For the time being the manual effort of writing library .mk files to bring a library to Genode is the preferred way.
In doing so the library is better integrated into Genode's build-system (amongst others, this also eases cross compilation), you gain more insight into the target you are currently porting as you are required to check the compilation units as well as the used compiler flags. Generally speaking you are more in control of the compilation process and do not rely on the original “black-box”.
In the case of GCC's libgo the overhead of this manual approach should not be so high because I would imagine it would be updated together with the tool-chain, which is around every 2 years (I do not how intertwined the library is with the compiler).
That being said, with better tooling in place it is quite possible to reuse the existing build tools in the future.
Due to specific of Genode we have another potential problem - we can’t use standard malloc() for stack allocation because stack should be from specific memory area to allow context switch, so, external packages compatibility will be limited anyhow.
I would argue that if go-packages operate on a level where they need to know where or how the stack is allocated you properly want a Genode-specific backend anyway. Or is that an implementation detail that leaks through the standard library?
By the way, what is a reason for this? it is relatively fast and easy to make a separation between «main» and «other» threads using, e.g. registry...
And, last but not least - setcontex/getcontext should store and restore signal context (including signal handlers) which is not that clear for me in Genode. Currently is is limited, and where it should go later? how we can store/restore state of signals for current thread?
That is a good question where I do not have an answer. In general the signal handling in our libc is minimal and we rely on user-code to call an preemtion point (“system-call”, I/O) to deliver signals, i.e., there is no preemption through signals.
So, returning back. I can try to put everything in *context with uniq name (replace original files and function names) into world repo, and force libgo to call these function. Anyway, any other software porting will require manual efforts mainly because of old/rest of headers from libc.
Also to use functions I need C interface for allocate_seconday_stack() which should be somewhere inside (I choose libc dummies.cchttp://dummies.cchttp://dummies.cc file) Question: may be better other place? e.g. inline C interface function as a part of e.g. thread.h? Any recommendations are welcome!
I understand your point, so, another question is: how I can continue if patch can’t be stored inside genode core? it can’t be significantly smaller (4 functions/files to be add at least)...
Everything else for golang support I will have in world, this is the only part of core (may be together with extended anonymous mmap support).
Please see my branch [1] where I have changed your code in a way that the whole mcontext implementation is now part of a static library “mcontext-support” that is linked against your mcontext test component. Hopefully I did not break anything in the process; the test seems to be fine. It also contains the 'allocate_seconday_stack' implementation.
[1] https://github.com/cnuke/genode/commits/mcontext-2021-03-23
The context API is still stubbed in the libc and you will see the dummy calls at run-time but you can override them by providing your own implementation. Naturally the implementation should use the ucontext_t data structure from the libc to prevent any memory layout issues. If that is possible to uphold you could as well bring the glibc implementation back.
(I removed the available makecontext implementation because it seems to be merely a left-over that is not required. Stubbing it as well feels more in line. I moved the implementation into the support library which is why it depends on libc to get access to the header files because I omitted making the compilation unit standalone.)
So in case of go, the libgo library should be linked against such an support library, that - as it uses the Genode API - needs to be linked against the Genode base library. If that is not possible, presumeably because adding additional libraries to libgo's build-system is cumbersum, linking your go component against the library should do the trick.
It goes without saying that this approach only works if you are fine with linking against the base library but I think for the time being that is workable solution. It removes extending the libc from your critical path and gives us time to decide how to deal with it properly.
Please let me know if you share the assessment or if I am still missing something obvious.
Regards Josef
Hello Josef,
the problem that this family together with headers and ucontext structure is a part of general libc. how I can mask its usage, e.g. in configure scripts?
That is indeed a problem and one of the reasons why I already mentioned that using configure for building *libraries* for Genode is not the way to go. For the time being the manual effort of writing library .mk files to bring a library to Genode is the preferred way.
In doing so the library is better integrated into Genode's build-system (amongst others, this also eases cross compilation), you gain more insight into the target you are currently porting as you are required to check the compilation units as well as the used compiler flags. Generally speaking you are more in control of the compilation process and do not rely on the original “black-box”.
In the case of GCC's libgo the overhead of this manual approach should not be so high because I would imagine it would be updated together with the tool-chain, which is around every 2 years (I do not how intertwined the library is with the compiler).
The problem is that current confgure from gcc do generate a list of files on the fly, depending of the subsystem. It is not easy to extract them even from listing: in .go files you can use pseudo comment like // +build aix linux darwin dragonfly freebsd openbsd netbsd solaris where the fact of compilation is defined on the correspondence of subsystem, file name (e.g. file_bsd.go included into any psd-related systems, memmove_linux_amd64_test.go will be included only for linux) and arch (cipherhw_amd64.go).
Moreover, some files are generated «on the fly» during configure run and then preprocessed couple times to generate some other files .c or .go files, and they should be placed to inside build directory (not inside source tree) for compilation.
in theory, I can copy all files and add them into all arch/etc, but this is error-prone process (especially repeating after 2 years when you make it last time ;-). IMHO easier to run original configure, as I do now - I see all compilation options in the listing and know what’s happens. What do you think?
Also we have generated file «macros.h» from your patch with offsets of current ucontext.h structures which should be manually regenerated from attached to mcontext.c test after any changes in these structures (not sure how to force error in case of structures update).
That being said, with better tooling in place it is quite possible to reuse the existing build tools in the future.
see above, this is definitely not that straight forward
Due to specific of Genode we have another potential problem - we can’t use standard malloc() for stack allocation because stack should be from specific memory area to allow context switch, so, external packages compatibility will be limited anyhow.
I would argue that if go-packages operate on a level where they need to know where or how the stack is allocated you properly want a Genode-specific backend anyway. Or is that an implementation detail that leaks through the standard library?
yes, definitely so - the only place where stack for new threads is generated is hidden inside library in proc.c (it done for every goroutine which could be plenty for application)
And, last but not least - setcontex/getcontext should store and restore signal context (including signal handlers) which is not that clear for me in Genode. Currently is is limited, and where it should go later? how we can store/restore state of signals for current thread?
That is a good question where I do not have an answer. In general the signal handling in our libc is minimal and we rely on user-code to call an preemtion point (“system-call”, I/O) to deliver signals, i.e., there is no preemption through signals.
hm, anyway, may be it worth to call a setprocmask()/__libc_sigprocmask() from libc, just to be sure that if it will be supported then we can run it (and call preemption, not sure that it will work)
Everything else for golang support I will have in world, this is the only part of core (may be together with extended anonymous mmap support).
Please see my branch [1] where I have changed your code in a way that the whole mcontext implementation is now part of a static library “mcontext-support” that is linked against your mcontext test component. Hopefully I did not break anything in the process; the test seems to be fine. It also contains the 'allocate_seconday_stack' implementation.
[1] https://github.com/cnuke/genode/commits/mcontext-2021-03-23
good, thank, will use it as a part of implementation.
(I removed the available makecontext implementation because it seems to be merely a left-over that is not required. Stubbing it as well feels more in line. I moved the implementation into the support library which is why it depends on libc to get access to the header files because I omitted making the compilation unit standalone.)
Anyway, I use makecontext.c from original implementation, so, I think that I need to return it back (it fit to original ucontext.h from genode). While it requires to setup specific «size" field inside it to be suitable for makecontext. This is not supported inside proc.c and I need to patch it (it was a reason for error in the proc.c code you try to debug in last letter).
So in case of go, the libgo library should be linked against such an support library, that - as it uses the Genode API - needs to be linked against the Genode base library. If that is not possible, presumeably because adding additional libraries to libgo's build-system is cumbersum, linking your go component against the library should do the trick.
libgo.a is not a shared library, so, it just can be linked to application directly together with mcontext.a library, should not be a problem
It goes without saying that this approach only works if you are fine with linking against the base library but I think for the time being
probably fine - except that I still have a problem in moving anon_mmap support from libc to separate library. Problem that I need kernel heap reference from Libc::Kernel instance which is somehow private. Seems that I can’t just call Kernel heap allocator (need for metadata store inside anon_mmap), and I should patch file_operations.cchttp://file_operations.cc to expose reference to _heap via some function.
that is workable solution. It removes extending the libc from your critical path and gives us time to decide how to deal with it properly.
ok, I make some patch for moving parts into world (see my git forks for genode and world), now I will change it to proposed scheme, and will test again golang test. last time I have a problem - version without debugger works ok, with debugger fail in try to obtain some more caps (4) in extend_cpu_session for unclear reason (I have 2800 caps for gdb and 1300 caps for test-go). Do debugger quota for caps include caps for internal process to be debug, or they are separate?
And I still have a problem when try to run code with 2 CPU in SMP mode, some nested mutex call appears (I know that this is potentially because of internal call from unwind stack, seems that jmp_slot code invoked for strlen() function inside, compilation/linking error I think).
Please let me know if you share the assessment or if I am still missing something obvious.
potentially it works. my plan: I need to fix a problem with anon_mmap initialisation (need kernel heap reference). Need to fix a problem with 2+CPU. Later , thanks to S. Platonov help, I want to add a network support. For network I need samples of run script configuration to provide tcp capabilities for executable.
Regards Josef
-- Josef Söntgen Genode Labs
On 3/24/21 9:22 AM, Alexander Tormasov via users wrote:
So in case of go, the libgo library should be linked against such an support library, that - as it uses the Genode API - needs to be linked against the Genode base library. If that is not possible, presumeably because adding additional libraries to libgo's build-system is cumbersum, linking your go component against the library should do the trick.
libgo.a is not a shared library, so, it just can be linked to application directly together with mcontext.a library, should not be a problem
It goes without saying that this approach only works if you are fine with linking against the base library but I think for the time being
probably fine - except that I still have a problem in moving anon_mmap support from libc to separate library. Problem that I need kernel heap reference from Libc::Kernel instance which is somehow private. Seems that I can’t just call Kernel heap allocator (need for metadata store inside anon_mmap), and I should patch file_operations.cc http://file_operations.cc to expose reference to _heap via some function.
I think you could use the 'Libc::Allocator' (libports/include/libc/allocator.h) or a simple 'malloc' in 'map_anon' instead of the heap.
Sebastian
Hello Josef, I finalise move of mcontext and libgo_support libraries, as well as update libgo a bit to reflect changes, and fix go test application.
result for 21.02 available here: https://github.com/tor-m6/genode/commits/21.02 https://github.com/tor-m6/genode-world/commits/master
please, take a look here https://github.com/tor-m6/genode-world/blob/master/src/test/go_app/README.go...
I used for my compilation command like make -C build/x86_64/ KERNEL=nova CFLAGS="-g -O0" VERBOSE= run/go_app before you need to make additional libs like written in README. I am not sure for 100% that I fix all problems, may be something still not works, please, report.
I need to understand what to do with unclosed questions to move further. May be you can add something below. I need to prioritise this list as well, this order is somehow random.
1. fix SMP support (now it work only with 1 CPU, 2+ CPU give «deadlock ahead» err) 2. fix debugging latest version do not normally start debug process with go_app_gdb.run, somewhere in the start of the go program in gogo() function log say [init -> gdb_monitor] upgrading quota donation for PD session (0 bytes, 4 caps) [init] child "gdb_monitor" requests resources: ram_quota=0, cap_quota=4 and hangs afterward, changes in number of caps in run file do not helps. I don’t find a way to print current cap quota yet... 3. create makefile for libgo and related libraries as static from the listing of compilation, as you suggest. Still not for 100% sure that this is worth do do - it will require periodic update, will have different set of files for different platforms, and not clear what to do with generated files (like .h for headers and .go for syscall/etc stubs) - whether it also should be generated once (like I do for mcontext now) or it worth to run via support scripts generated by configure. Also it is not clear how to add these generated scripts for port - should they appears as a patch file for main sources in contrib/* tree or it should be inside more local places like libgo_support in repos/world (then it will be complex mix of internal files and headers only for compilation of them and the rest - not so suitable for other usage of libgo_support). 4. port to seL4. Now I can compile, some problems with kernel.s file - but it works with total recompilation. I see a problem that for nova and sel4 we could have different object files in common tree (e.g. because of different offsets of internal os-specific headers), not sure how to handle it, e.g. in libgo stubs for C OS interfaces. 5. port to arm64 for nova/etc. Need to create a mcontext staff (hopefully that it relatively easy) 6. fix signals processing in mcontext now I completely omit call to SIG_MASK set/reset in the code, not sure which applications will have a problems 7. check Go support for wider set of features like floating point, os interfaces, etc, including cleanup of stubs in libgo related to unused OS features from other subsystems (see README for list of broken calls) 8. add golang support for TCP network need not only patch back some .go code, but also a set of sample .run files and test examples for configuration 9. port feature which allow to generate make file from «go build» process in this moment go build and related compilation infrastructure not supported, I use gccgo 10. start port of docker libraries to Genode in the same way as it was done by me for another microkernel before (kind of complex task requiring some additional workforce support)
Probably something else?
Sincerely, Alexander
Hallo Alexander,
I finalise move of mcontext and libgo_support libraries, as well as update libgo a bit to reflect changes, and fix go test application.
thanks, I will take a look today and get back to you.
Regards Josef
Hello Alexander,
I am going to focus on your most recent E-Mail as the list at the end also covers most of the question of the previous one.
I finalise move of mcontext and libgo_support libraries, as well as update libgo a bit to reflect changes, and fix go test application.
[…]
Thanks, I used your branches and could successfully run the test component. I then took both your branches and refactored the code somewhat [1], [2]. In the end adding support for building .go files with the Genode build-system is all that is left in the Genode repository.
[1] https://github.com/cnuke/genode/commits/golang-2021-03-29 [2] https://github.com/cnuke/genode-world/commits/tor-m6-master-2021-03-26--gola...
The world repository now contains all the support code that is needed in the 'libgo_support' library at the moment. The library is only made available for x86_64 to make it clear it is the only support architecture. Please take a look at my commits - I am not entirely sure if the local anon mmap registry is “safe” to use but at least the test component seems to be fine.
I need to understand what to do with unclosed questions to move further. May be you can add something below. I need to prioritise this list as well, this order is somehow random.
- fix SMP support
(now it work only with 1 CPU, 2+ CPU give «deadlock ahead» err)
You should see the address of the mutex in question that leads to the deadlock situation.
- fix debugging
latest version do not normally start debug process with go_app_gdb.run, somewhere in the start of the go program in gogo() function log say [init -> gdb_monitor] upgrading quota donation for PD session (0 bytes, 4 caps) [init] child "gdb_monitor" requests resources: ram_quota=0, cap_quota=4 and hangs afterward, changes in number of caps in run file do not helps. I don’t find a way to print current cap quota yet...
I am not well versed in the ways of the gdbserver, maybe Christian Prochaska wants to pitch in?
- create makefile for libgo and related libraries as static from the
listing of compilation, as you suggest. Still not for 100% sure that this is worth do do - it will require periodic update, will have different set of files for different platforms, and not clear what to do with generated files (like .h for headers and .go for syscall/etc stubs) - whether it also should be generated once (like I do for mcontext now) or it worth to run via support scripts generated by configure. Also it is not clear how to add these generated scripts for port - should they appears as a patch file for main sources in contrib/* tree or it should be inside more local places like libgo_support in repos/world (then it will be complex mix of internal files and headers only for compilation of them and the rest - not so suitable for other usage of libgo_support).
By now we we keep such generated files in a different repository that is referenced in the .port file and those files should be only generated by the developer. The generated files are then stored in the contrib directory of the port. Of course they could be generated on the fly while preparing the port - we do that for a number of ports - but that requires everyone to have the needed host tools installed and depending on the version of the tools leads to slightly different files which in return lead to different hashes in the archive recipes. The scripts or the mechanism could still be part of the .port file or could be stored in the repository for the generated files depending on how extensive it is.
The library .mk files are in part responsible to assemble the contrib sources properly for each platform or architecture.
- port to seL4.
Now I can compile, some problems with kernel.s file - but it works with total recompilation. I see a problem that for nova and sel4 we could have different object files in common tree (e.g. because of different offsets of internal os-specific headers), not sure how to handle it, e.g. in libgo stubs for C OS interfaces.
I am not sure which OS specific headers that would involve but normally we try to make the components as kernel-agnostic as possible. Of course there are differences, like the virtual address-space layout differs between kernels, but such things are enclosed and abstracted away by ld.lib.so. Now it could be different for a language runtime in which case I am curious which functionality that is?
The remaining points primarly deal with missing features where I have not much to comment - IIRC Gapfruit has already worked on TCP support.
- port to arm64 for nova/etc.
Need to create a mcontext staff (hopefully that it relatively easy)
- fix signals processing in mcontext
now I completely omit call to SIG_MASK set/reset in the code, not sure which applications will have a problems
- check Go support for wider set of features
like floating point, os interfaces, etc, including cleanup of stubs in libgo related to unused OS features from other subsystems (see README for list of broken calls)
- add golang support for TCP network
need not only patch back some .go code, but also a set of sample .run files and test examples for configuration
- port feature which allow to generate make file from «go build» process
in this moment go build and related compilation infrastructure not supported, I use gccgo
- start port of docker libraries to Genode in the same way as it was done
by me for another microkernel before (kind of complex task requiring some additional workforce support)
Regards Josef
Hello Josef, I try to fix a problem with SMP support with golang relate to nested mutex call. As I mention couple month ago here https://lists.genode.org/pipermail/users/2020-October/007269.html
I still have a problem with nested jmp_slot() related to call to strlen() during stack unwind call and related nested mutex leading to deadlock. Seems that strlen() was instantiated as a call to shared libc from libgcc compiled and linked to main test-go. This is a part of compiled
What I found is the following
#0 strlen (str=str@entry=0x127d029 "zR") at /var/services/homes/admin/gen/21.02/contrib/libc-3ec28a350593879749a015a584ed24ffd7d3edce/src/lib/libc/lib/libc/string/strlen.c:100 #1 0x00000000011a8ca5 in get_cie_encoding (cie=0x127d020) at /genodelabs-genode-2491eee/contrib/gcc-3cade18cf9defeefa714aa91de3b157fbad4aa18/src/noux-pkg/gcc/libgcc/unwind-dw2-fde.c:300 #2 0x00000000011a93a3 in get_fde_encoding (f=0x12a8e8c) at /genodelabs-genode-2491eee/contrib/gcc-3cade18cf9defeefa714aa91de3b157fbad4aa18/src/noux-pkg/gcc/libgcc/unwind-dw2-fde.h:157 #3 _Unwind_IteratePhdrCallback (info=<optimized out>, size=<optimized out>, ptr=0x405fe190) at /genodelabs-genode-2491eee/contrib/gcc-3cade18cf9defeefa714aa91de3b157fbad4aa18/src/noux-pkg/gcc/libgcc/unwind-dw2-fde-dip.c:418 #4 0x00000000000eba9e in dl_iterate_phdr (callback=callback@entry=0x11a8f60 <_Unwind_IteratePhdrCallback>, data=data@entry=0x405fe190) at /var/services/homes/admin/gen/21.02/repos/base/src/lib/ldso/exception.cc:53 #5 0x00000000011aa1cf in _Unwind_Find_FDE (pc=0x11a8797 <_Unwind_Backtrace+55>, bases=bases@entry=0x405fe4c8) at /genodelabs-genode-2491eee/contrib/gcc-3cade18cf9defeefa714aa91de3b157fbad4aa18/src/noux-pkg/gcc/libgcc/unwind-dw2-fde-dip.c:469 #6 0x00000000011a6783 in uw_frame_state_for (context=context@entry=0x405fe420, fs=fs@entry=0x405fe270) at /genodelabs-genode-2491eee/contrib/gcc-3cade18cf9defeefa714aa91de3b157fbad4aa18/src/noux-pkg/gcc/libgcc/unwind-dw2.c:1257 #7 0x00000000011a77a0 in uw_init_context_1 (context=context@entry=0x405fe420, outer_cfa=outer_cfa@entry=0x405fe6d0, outer_ra=0x119ad62 <backtrace_full+180>) at /genodelabs-genode-2491eee/contrib/gcc-3cade18cf9defeefa714aa91de3b157fbad4aa18/src/noux-pkg/gcc/libgcc/unwind-dw2.c:1586 #8 0x00000000011a8798 in _Unwind_Backtrace (trace=0x119abbc <unwind>, trace_argument=0x405fe700) at /genodelabs-genode-2491eee/contrib/gcc-3cade18cf9defeefa714aa91de3b157fbad4aa18/src/noux-pkg/gcc/libgcc/unwind.inc:295 #9 0x000000000119ad62 in backtrace_full (state=0x29000, skip=0x0, callback=0x119804e <callback>, error_callback=0x1198503 <error_callback>, data=0x405fe770) at /var/services/homes/admin/gen/21.02/contrib/libbacktrace-cabcd83bc903137f607eda3a65eb09feaa5f507e/src/lib/gcc/libbacktrace/backtrace.c:127 #9 0x000000000119ad62 in backtrace_full (state=0x29000, skip=0x0, callback=0x119804e <callback>, error_callback=0x1198503 <error_callback>, data=0x405fe770) at /var/services/homes/admin/gen/21.02/contrib/libbacktrace-cabcd83bc903137f607eda3a65eb09feaa5f507e/src/lib/gcc/libbacktrace/backtrace.c:127 #10 0x00000000011985d1 in runtime_callers (skip=0x2, locbuf=0xc42000e8d8, m=0x20, keep_thunks=0x0) at /var/services/homes/admin/gen/21.02/contrib/libgo-281260d9bdc27fefb62c00310025c54e1e629a2e/src/lib/gcc/libgo/runtime/go-callers.c:207
this is compiled inside toolchain, disassembly from here: /usr/local/genode/tool/19.05/lib/gcc/x86_64-pc-elf/8.3.0/64/libgcc_eh.a
0000000000000460 <get_cie_encoding>: get_cie_encoding(): /genodelabs-genode-2491eee/contrib/gcc-3cade18cf9defeefa714aa91de3b157fbad4aa18/src/noux-pkg/gcc/libgcc/unwind-dw2-fde.c:293 460: 55 push %rbp 461: 48 89 fd mov %rdi,%rbp 464: 53 push %rbx /genodelabs-genode-2491eee/contrib/gcc-3cade18cf9defeefa714aa91de3b157fbad4aa18/src/noux-pkg/gcc/libgcc/unwind-dw2-fde.c:299 465: 48 8d 5f 09 lea 0x9(%rdi),%rbx /genodelabs-genode-2491eee/contrib/gcc-3cade18cf9defeefa714aa91de3b157fbad4aa18/src/noux-pkg/gcc/libgcc/unwind-dw2-fde.c:300 469: 48 89 df mov %rbx,%rdi /genodelabs-genode-2491eee/contrib/gcc-3cade18cf9defeefa714aa91de3b157fbad4aa18/src/noux-pkg/gcc/libgcc/unwind-dw2-fde.c:293 46c: 48 83 ec 18 sub $0x18,%rsp /genodelabs-genode-2491eee/contrib/gcc-3cade18cf9defeefa714aa91de3b157fbad4aa18/src/noux-pkg/gcc/libgcc/unwind-dw2-fde.c:300 470: e8 00 00 00 00 callq 475 <get_cie_encoding+0x15>
call above inside test-go linked to strlen() to inside libc, as you see from stack above and from disassemble test-go, where it call shared libc instance: 00000000011a8c90 <get_cie_encoding>: get_cie_encoding(): /genodelabs-genode-2491eee/contrib/gcc-3cade18cf9defeefa714aa91de3b157fbad4aa18/src/noux-pkg/gcc/libgcc/unwind-dw2-fde.c:293 11a8c90: 55 push %rbp 11a8c91: 48 89 fd mov %rdi,%rbp 11a8c94: 53 push %rbx /genodelabs-genode-2491eee/contrib/gcc-3cade18cf9defeefa714aa91de3b157fbad4aa18/src/noux-pkg/gcc/libgcc/unwind-dw2-fde.c:299 11a8c95: 48 8d 5f 09 lea 0x9(%rdi),%rbx /genodelabs-genode-2491eee/contrib/gcc-3cade18cf9defeefa714aa91de3b157fbad4aa18/src/noux-pkg/gcc/libgcc/unwind-dw2-fde.c:300 11a8c99: 48 89 df mov %rbx,%rdi /genodelabs-genode-2491eee/contrib/gcc-3cade18cf9defeefa714aa91de3b157fbad4aa18/src/noux-pkg/gcc/libgcc/unwind-dw2-fde.c:293 11a8c9c: 48 83 ec 18 sub $0x18,%rsp /genodelabs-genode-2491eee/contrib/gcc-3cade18cf9defeefa714aa91de3b157fbad4aa18/src/noux-pkg/gcc/libgcc/unwind-dw2-fde.c:300 11a8ca0: e8 db 6b 00 00 callq 11af880 strlen@plt
inside callstack you can see a call to dl_iterate_phdr which contain in the very beginning extern "C" int dl_iterate_phdr(int (*callback) (Phdr_info *info, size_t size, void *data), void *data) { int err = 0; Phdr_info info;
Mutex::Guard guard(mutex());
so, we take same mutex twice here and in attempt to search for strlen() from @plt via jmp_slot… This happens only with SMP, if I use UP configuration it is not happens.
Frankly, I am stuck a bit… this lib is a part of general toolchain, not clear why this stack unwind staff call shared libc… may be it is necessary to recompile toolchain and library and force strlen() to be expanded to inlined one?
Sincerely, Alexander
Hello Alexander,
I still have a problem with nested jmp_slot() related to call to strlen() during stack unwind call and related nested mutex leading to deadlock. […] call above inside test-go linked to strlen() to inside libc, as you see from stack above and from disassemble test-go, where it call shared libc […] /genodelabs-genode-2491eee/contrib/gcc-3cade18cf9defeefa714aa91de3b157fbad4aa18/src/noux-pkg/gcc/libgcc/unwind-dw2-fde.c:300 11a8ca0: e8 db 6b 00 00 callq 11af880 strlen@plt
so, we take same mutex twice here and in attempt to search for strlen() from @plt via jmp_slot… This happens only with SMP, if I use UP configuration it is not happens. Frankly, I am stuck a bit… this lib is a part of general toolchain, not clear why this stack unwind staff call shared libc… may be it is necessary to recompile toolchain and library and force strlen() to be expanded to inlined one?
Please give the following two commits [1], [2] a try. The first exports the required Unwind functions needed by the libgo runtime and the second removes the direct usage of the toolchain libraries.
With both commits applied the 'go_app' test component still works and hopefully it addresses your SMP issue as well.
[1] https://github.com/cnuke/genode/commit/e19b19e0367e544654bc917e09c2cb991b71b... [2] https://github.com/cnuke/genode-world/commit/30ef976ebdd58becb1785993cd18434...
Your problem boils down to using the toolchain libraries directly, which is not supported and forced the compiler to produce the PLT relocations (as strlen() is unresolved at this point). On Genode you are supposed to go through the base library, i.e. ld.lib.so, that already takes care of setting things up the proper way. As the Unwind symbols required by the libgo runtime were not exported the linking step of the 'go_app' probably failed.
Regards Josef
Please give the following two commits [1], [2] a try. The first exports the required Unwind functions needed by the libgo runtime and the second removes the direct usage of the toolchain libraries.
With both commits applied the 'go_app' test component still works and hopefully it addresses your SMP issue as well.
[1] https://github.com/cnuke/genode/commit/e19b19e0367e544654bc917e09c2cb991b71b... [2] https://github.com/cnuke/genode-world/commit/30ef976ebdd58becb1785993cd18434...
Your problem boils down to using the toolchain libraries directly, which is not supported and forced the compiler to produce the PLT relocations (as strlen() is unresolved at this point). On Genode you are supposed to go through the base library, i.e. ld.lib.so, that already takes care of setting things up the proper way. As the Unwind symbols required by the libgo runtime were not exported the linking step of the 'go_app' probably failed.
hm, I apply patch- it works ok for UP. But, for SMP (2 CPU - just replace -smp 1 option in .run file) it return the following:
[init] child "test-go" [init] RAM quota: 20232K [init] cap quota: 216 [init] ELF binary: test-go [init] priority: 0 [init -> timer] 0x1000000 .. 0x10ffffff: linker area [init -> timer] 0x40000000 .. 0x4fffffff: stack area [init -> timer] 0x30000 .. 0x161fff: ld.lib.so [init -> test-go] 0x1000000 .. 0x10ffffff: linker area [init] child "timer" announces service "Timer" [init -> test-go] 0x40000000 .. 0x4fffffff: stack area [init -> test-go] 0x30000 .. 0x161fff: ld.lib.so [init -> test-go] 0x10e0c000 .. 0x10ffffff: libc.lib.so [init -> test-go] 0x10d70000 .. 0x10e0bfff: vfs.lib.so [init -> test-go] 0x10d36000 .. 0x10d6ffff: libm.lib.so [init -> test-go] 0x1328000 .. 0x14dffff: stdcxx.lib.so [init -> test-go] Error: LD: jump slot relocation failed for symbol: '_Unwind_Backtrace' [init -> test-go] Error: Uncaught exception of type 'Linker::Not_found' [init -> test-go] Warning: abort called - thread: ep
interesting that I do not find a code for _Unwind_Backtrace inside libc.lib.so
Seems that I have a bit different code path for UP and SMP cases. In particular, I found that code related to the read of myself executable file invoked only in case of SMP (I still not solve the problem of open for myself, I receive some recommendation from Uwe, but seems that I need to have some more code samples to access to the file).
so, this time problem could be the same, need to dig deeper: source for this «Not_found» problem below - seems that again it try to do something with stack parsing:
#2 Genode::Placeable::Placeable (args#2=..., args#6=..., args#5=..., args#4=..., args#3=..., args#1=..., args#0=..., this=0x11b0740 <type..func.8.9.8>) at /var/services/homes/admin/gen/21.02/repos/base/include/util/construct_at.h:56 #3 Genode::construct_at<Genode::Local_connectionGenode::Rom_connection, Genode::Child::Env_connectionGenode::Rom_connection::Env_service&, Genode::Id_spaceGenode::Parent::Client&, Genode::Id_spaceGenode::Parent::Client::Id const&, Genode::String<256> const&, Genode::Affinity&, Genode::Session_label const&, Genode::Session::Diag const&> (at=0x11b0740 <type..func.8.9.8>) at /var/services/homes/admin/gen/21.02/repos/base/include/util/construct_at.h:82 #4 Genode::Reconstructible<Genode::Local_connectionGenode::Rom_connection >::_do_construct<Genode::Child::Env_connectionGenode::Rom_connection::Env_service&, Genode::Id_spaceGenode::Parent::Client&, Genode::Id_spaceGenode::Parent::Client::Id const&, Genode::String<256ul> const&, Genode::Affinity&, Genode::Session_label const&, Genode::Session::Diag const&> (this=0x11b0740 <type..func.8.9.8>) at /var/services/homes/admin/gen/21.02/repos/base/include/util/reconstructible.h:55 #5 Genode::Reconstructible<Genode::Local_connectionGenode::Rom_connection >::construct<Genode::Child::Env_connectionGenode::Rom_connection::Env_service&, Genode::Id_spaceGenode::Parent::Client&, Genode::Id_spaceGenode::Parent::Client::Id const&, Genode::String<256ul> const&, Genode::Affinity, Genode::Session_label const&, Genode::Session::Diag const&> (this=0x11b0740 <type..func.8.9.8>) at /var/services/homes/admin/gen/21.02/repos/base/include/util/reconstructible.h:110 #6 Genode::Child::Env_connectionGenode::Rom_connection::initiate (this=0xc42000c038) at /var/services/homes/admin/gen/21.02/repos/base/include/base/child.h:582 #7 0x00000000011985d1 in runtime_callers (skip=0x2, locbuf=0xc42000e8d8, m=0x20, keep_thunks=0x0) at /var/services/homes/admin/gen/21.02/contrib/libgo-281260d9bdc27fefb62c00310025c54e1e629a2e/src/lib/gcc/libgo/runtime/go-callers.c:207 #8 0x00000000010fdee8 in runtime.callers (skip=0x1, locbuf=...) at /var/services/homes/admin/gen/21.02/contrib/libgo-281260d9bdc27fefb62c00310025c54e1e629a2e/src/lib/gcc/libgo/go/runtime/traceback_gccgo.go:56 #9 0x00000000010d8f3d in runtime.mcommoninit (mp=0xc42000e800) at /var/services/homes/admin/gen/21.02/contrib/libgo-281260d9bdc27fefb62c00310025c54e1e629a2e/src/lib/gcc/libgo/go/runtime/proc.go:546 #10 0x00000000010db685 in runtime.allocm (_p_=0xc420008000, fn=0x1219ed8 <runtime.mspinning..f>, allocatestack=0x0) at /var/services/homes/admin/gen/21.02/contrib/libgo-281260d9bdc27fefb62c00310025c54e1e629a2e/src/lib/gcc/libgo/go/runtime/proc.go:1466
Hallo Alexander,
hm, I apply patch- it works ok for UP. But, for SMP (2 CPU - just replace -smp 1 option in .run file) it return the following:
[init] child "test-go" [init] RAM quota: 20232K [init] cap quota: 216 [init] ELF binary: test-go [init] priority: 0 [init -> timer] 0x1000000 .. 0x10ffffff: linker area [init -> timer] 0x40000000 .. 0x4fffffff: stack area [init -> timer] 0x30000 .. 0x161fff: ld.lib.so [init -> test-go] 0x1000000 .. 0x10ffffff: linker area [init] child "timer" announces service "Timer" [init -> test-go] 0x40000000 .. 0x4fffffff: stack area [init -> test-go] 0x30000 .. 0x161fff: ld.lib.so [init -> test-go] 0x10e0c000 .. 0x10ffffff: libc.lib.so [init -> test-go] 0x10d70000 .. 0x10e0bfff: vfs.lib.so [init -> test-go] 0x10d36000 .. 0x10d6ffff: libm.lib.so [init -> test-go] 0x1328000 .. 0x14dffff: stdcxx.lib.so [init -> test-go] Error: LD: jump slot relocation failed for symbol: '_Unwind_Backtrace' [init -> test-go] Error: Uncaught exception of type 'Linker::Not_found' [init -> test-go] Warning: abort called - thread: ep
My bad - the first commit from yesterday is incomplete. Merely making the symbols available for linking is not enough. Since the implementation is global hidden we have to make it also accessible. Please also apply the following commit [1].
[1] https://github.com/cnuke/genode/commit/3eb8312daee2ae24f7eea43adf7697dbfd0c9...
interesting that I do not find a code for _Unwind_Backtrace inside libc.lib.so
That is correct because the code is not there. We make the symbols available so that the linking stage for components that solely link against the libc will not fail. At run-time the implementation is provided by the dynamic linker (ld.lib.so). If you look around in the base repository's library .mk files and check the resulting unstripped shared-objects (e.g. objdump -x) the picture should become clear.
(All in all, long-term it is probably better to provide the unwind functionality in way that accommodates the needs of the Genode base system as well as as the needs of different languague runtimes rather (ab-)using the one that is primarily tailored to Genode's base system.)
Regards Josef
Hello Josef,
My bad - the first commit from yesterday is incomplete. Merely making the symbols available for linking is not enough. Since the implementation is global hidden we have to make it also accessible. Please also apply the following commit [1].
[1] https://github.com/cnuke/genode/commit/3eb8312daee2ae24f7eea43adf7697dbfd0c9...
good, now it work as expected. Now I need to cleanup the codepath related to self-open of executable file for SMP version to run:
[init] child "timer" announces service "Timer" [init -> test-go] 0x40000000 .. 0x4fffffff: stack area [init -> test-go] 0x30000 .. 0x161fff: ld.lib.so [init -> test-go] 0x10e0c000 .. 0x10ffffff: libc.lib.so [init -> test-go] 0x10d70000 .. 0x10e0bfff: vfs.lib.so [init -> test-go] 0x10d36000 .. 0x10d6ffff: libm.lib.so [init -> test-go] 0x1328000 .. 0x14dffff: stdcxx.lib.so [init -> test-go] fd(2): fatal error: libbacktrace could not find executable to open [init -> test-go] fd(2): [init -> test-go] fd(2): goroutine 1 [running, locked to thread]: [init -> test-go] fd(2): fatal error: libbacktrace could not find executable to open [init -> test-go] fd(2): panic during panic [init -> test-go] fd(2): [init -> test-go] fd(2): goroutine 1 [running, locked to thread]: [init -> test-go] fd(2): fatal error: libbacktrace could not find executable to open [init -> test-go] fd(2): stack trace unavailable [init] child "test-go" exited with exit value 4
I plan (as Uwe suggested) to dup2() rom session for executable file and provide it to the stack unwind code - while it is not so clear for me what is a way to «cast» rom session object to the posix (libc) file handle, need to find an example in the genode code base.
it you have some suggestions - will be happy to follow
Sincerely, Alexander
-
Alexander Tormasov -
Josef Söntgen -
Sebastian Sumpf