During port of libbacktrace code from gcc compiler I found that I need to be able to read a.out or similar file in runtime. This library try to read original executable file and parse it for functions addresses/etc, as a part of garbage collection for golang (it parse call stack in some specific way using gcc-specific calls).
I have to port to genode the following code from src/lib/gcc/libbacktrace/fileline.c:
for (pass = 0; pass < 5; ++pass) { int does_not_exist;
switch (pass) { case 0: filename = state->filename; break; case 1: filename = getexecname (); break; case 2: filename = "/proc/self/exe"; break; case 3: filename = "/proc/curproc/file"; break; case 4: snprintf (buf, sizeof (buf), "/proc/%ld/object/a.out", (long) getpid ()); filename = buf; break; default: abort (); }
best idea is to implement library call getexecname(), e.g. as a part of libc.
Anyway, this is complex question, because we need to be able to open it and read a content later. How to configure this in run file?
PS for better understanding why I need this - call stack where it is called, from gdb:
#0 fileline_initialize (state=0x29000, error_callback=0x1198543 <error_callback>, data=0x405fe770) at /var/services/homes/admin/gen/21.02/contrib/libbacktrace-cabcd83bc903137f607eda3a65eb09feaa5f507e/src/lib/gcc/libbacktrace/fileline.c:102 #1 0x000000000119a9a5 in backtrace_pcinfo (state=0x29000, pc=0x1198610, callback=0x119808e <callback>, error_callback=0x1198543 <error_callback>, data=0x405fe770) at /var/services/homes/admin/gen/21.02/contrib/libbacktrace-cabcd83bc903137f607eda3a65eb09feaa5f507e/src/lib/gcc/libbacktrace/fileline.c:177 #2 0x000000000119accc in unwind (context=0x405fe420, vdata=0x405fe700) at /var/services/homes/admin/gen/21.02/contrib/libbacktrace-cabcd83bc903137f607eda3a65eb09feaa5f507e/src/lib/gcc/libbacktrace/backtrace.c:91 #3 0x00000000011a87e9 in _Unwind_Backtrace (trace=0x119abfc <unwind>, trace_argument=0x405fe700) at /genodelabs-genode-2491eee/contrib/gcc-3cade18cf9defeefa714aa91de3b157fbad4aa18/src/noux-pkg/gcc/libgcc/unwind.inc:307 #4 0x000000000119ada2 in backtrace_full (state=0x29000, skip=0x0, callback=0x119808e <callback>, error_callback=0x1198543 <error_callback>, data=0x405fe770) at /var/services/homes/admin/gen/21.02/contrib/libbacktrace-cabcd83bc903137f607eda3a65eb09feaa5f507e/src/lib/gcc/libbacktrace/backtrace.c:127 #5 0x0000000001198611 in runtime_callers (skip=0x2, locbuf=0xc42000e8d8, m=0x20, keep_thunks=0x0) at /var/services/homes/admin/gen/21.02/contrib/libgo-281260d9bdc27fefb62c00310025c54e1e629a2e/src/lib/gcc/libgo/runtime/go-callers.c:207 #6 0x00000000010fdf28 in runtime.callers (skip=0x1, locbuf=...) at /var/services/homes/admin/gen/21.02/contrib/libgo-281260d9bdc27fefb62c00310025c54e1e629a2e/src/lib/gcc/libgo/go/runtime/traceback_gccgo.go:56 #7 0x00000000010d8f7d in runtime.mcommoninit (mp=0xc42000e800) at /var/services/homes/admin/gen/21.02/contrib/libgo-281260d9bdc27fefb62c00310025c54e1e629a2e/src/lib/gcc/libgo/go/runtime/proc.go:546 #8 0x00000000010db6c5 in runtime.allocm (_p_=0xc420008000, fn=0x121e298 <runtime.mspinning..f>, allocatestack=0x0) at /var/services/homes/admin/gen/21.02/contrib/libgo-281260d9bdc27fefb62c00310025c54e1e629a2e/src/lib/gcc/libgo/go/runtime/proc.go:1466 #9 0x00000000010dbdb2 in runtime.newm (fn=0x121e298 <runtime.mspinning..f>, _p_=0xc420008000) at /var/services/homes/admin/gen/21.02/contrib/libgo-281260d9bdc27fefb62c00310025c54e1e629a2e/src/lib/gcc/libgo/go/runtime/proc.go:1769 #10 0x00000000010dc360 in runtime.startm (_p_=0xc420008000, spinning=0x1) at /var/services/homes/admin/gen/21.02/contrib/libgo-281260d9bdc27fefb62c00310025c54e1e629a2e/src/lib/gcc/libgo/go/runtime/proc.go:1925 #11 0x00000000010dc702 in runtime.wakep () at /var/services/homes/admin/gen/21.02/contrib/libgo-281260d9bdc27fefb62c00310025c54e1e629a2e/src/lib/gcc/libgo/go/runtime/proc.go:2006 #12 0x00000000010dfabf in __go_go (fn=0x10d72f2, arg=0x0) at /var/services/homes/admin/gen/21.02/contrib/libgo-281260d9bdc27fefb62c00310025c54e1e629a2e/src/lib/gcc/libgo/go/runtime/proc.go:3144
Gesendet: Sonntag, 04. April 2021 um 15:18 Uhr Von: "Alexander Tormasov via users" users@lists.genode.org An: "Genode users mailing list" users@lists.genode.org Cc: "Alexander Tormasov" a.tormasov@innopolis.ru Betreff: how to open elf file image from inside genode? libbacktrace question
During port of libbacktrace code from gcc compiler I found that I need to be able to read a.out or similar file in runtime. This library try to read original executable file and parse it for functions addresses/etc, as a part of garbage collection for golang (it parse call stack in some specific way using gcc-specific calls).
I have to port to genode the following code from src/lib/gcc/libbacktrace/fileline.c:
for (pass = 0; pass < 5; ++pass) { int does_not_exist;
switch (pass){ case 0: filename = state->filename; break; case 1: filename = getexecname (); break; case 2: filename = "/proc/self/exe"; break; case 3: filename = "/proc/curproc/file"; break; case 4: snprintf (buf, sizeof (buf), "/proc/%ld/object/a.out", (long) getpid ()); filename = buf; break; default: abort (); }
best idea is to implement library call getexecname(), e.g. as a part of libc.
Your parent needs a ROMFile capability to run your program. Ask it for that capability to read your own file. You can call that RPC getexecname or whatever strikes your fancy. Of course you would have to extend the official interface to the parent else your program could be started from a component that doesn't implement that function. Or you can write a wrapper that guarantees to implement that function and start your program exclusively with that wrapper. You can call your wrapper "go run" or whatever strikes your fancy.
Anyway, this is complex question, because we need to be able to open it and read a content later. How to configure this in run file?
PS for better understanding why I need this - call stack where it is called, from gdb:
#0 fileline_initialize (state=0x29000, error_callback=0x1198543 <error_callback>, data=0x405fe770) at /var/services/homes/admin/gen/21.02/contrib/libbacktrace-cabcd83bc903137f607eda3a65eb09feaa5f507e/src/lib/gcc/libbacktrace/fileline.c:102 #1 0x000000000119a9a5 in backtrace_pcinfo (state=0x29000, pc=0x1198610, callback=0x119808e <callback>, error_callback=0x1198543 <error_callback>, data=0x405fe770) at /var/services/homes/admin/gen/21.02/contrib/libbacktrace-cabcd83bc903137f607eda3a65eb09feaa5f507e/src/lib/gcc/libbacktrace/fileline.c:177 #2 0x000000000119accc in unwind (context=0x405fe420, vdata=0x405fe700) at /var/services/homes/admin/gen/21.02/contrib/libbacktrace-cabcd83bc903137f607eda3a65eb09feaa5f507e/src/lib/gcc/libbacktrace/backtrace.c:91 #3 0x00000000011a87e9 in _Unwind_Backtrace (trace=0x119abfc <unwind>, trace_argument=0x405fe700) at /genodelabs-genode-2491eee/contrib/gcc-3cade18cf9defeefa714aa91de3b157fbad4aa18/src/noux-pkg/gcc/libgcc/unwind.inc:307 #4 0x000000000119ada2 in backtrace_full (state=0x29000, skip=0x0, callback=0x119808e <callback>, error_callback=0x1198543 <error_callback>, data=0x405fe770) at /var/services/homes/admin/gen/21.02/contrib/libbacktrace-cabcd83bc903137f607eda3a65eb09feaa5f507e/src/lib/gcc/libbacktrace/backtrace.c:127 #5 0x0000000001198611 in runtime_callers (skip=0x2, locbuf=0xc42000e8d8, m=0x20, keep_thunks=0x0) at /var/services/homes/admin/gen/21.02/contrib/libgo-281260d9bdc27fefb62c00310025c54e1e629a2e/src/lib/gcc/libgo/runtime/go-callers.c:207 #6 0x00000000010fdf28 in runtime.callers (skip=0x1, locbuf=...) at /var/services/homes/admin/gen/21.02/contrib/libgo-281260d9bdc27fefb62c00310025c54e1e629a2e/src/lib/gcc/libgo/go/runtime/traceback_gccgo.go:56 #7 0x00000000010d8f7d in runtime.mcommoninit (mp=0xc42000e800) at /var/services/homes/admin/gen/21.02/contrib/libgo-281260d9bdc27fefb62c00310025c54e1e629a2e/src/lib/gcc/libgo/go/runtime/proc.go:546 #8 0x00000000010db6c5 in runtime.allocm (_p_=0xc420008000, fn=0x121e298 <runtime.mspinning..f>, allocatestack=0x0) at /var/services/homes/admin/gen/21.02/contrib/libgo-281260d9bdc27fefb62c00310025c54e1e629a2e/src/lib/gcc/libgo/go/runtime/proc.go:1466 #9 0x00000000010dbdb2 in runtime.newm (fn=0x121e298 <runtime.mspinning..f>, _p_=0xc420008000) at /var/services/homes/admin/gen/21.02/contrib/libgo-281260d9bdc27fefb62c00310025c54e1e629a2e/src/lib/gcc/libgo/go/runtime/proc.go:1769 #10 0x00000000010dc360 in runtime.startm (_p_=0xc420008000, spinning=0x1) at /var/services/homes/admin/gen/21.02/contrib/libgo-281260d9bdc27fefb62c00310025c54e1e629a2e/src/lib/gcc/libgo/go/runtime/proc.go:1925 #11 0x00000000010dc702 in runtime.wakep () at /var/services/homes/admin/gen/21.02/contrib/libgo-281260d9bdc27fefb62c00310025c54e1e629a2e/src/lib/gcc/libgo/go/runtime/proc.go:2006 #12 0x00000000010dfabf in __go_go (fn=0x10d72f2, arg=0x0) at /var/services/homes/admin/gen/21.02/contrib/libgo-281260d9bdc27fefb62c00310025c54e1e629a2e/src/lib/gcc/libgo/go/runtime/proc.go:3144
Genode users mailing list users@lists.genode.org https://lists.genode.org/listinfo/users
Thanks for suggestion, Uwe
Your parent needs a ROMFile capability to run your program. Ask it for that capability to read your own file. You can call that RPC getexecname or whatever strikes your fancy.
the question is what path for open() call I should use for ROMfile?
Sincerely, Alexander
Gesendet: Montag, 05. April 2021 um 00:59 Uhr Von: "Alexander Tormasov via users" users@lists.genode.org An: "Genode users mailing list" users@lists.genode.org Cc: "Alexander Tormasov" a.tormasov@innopolis.ru Betreff: Re: how to open elf file image from inside genode? libbacktrace question
Thanks for suggestion, Uwe
Your parent needs a ROMFile capability to run your program. Ask it for that capability to read your own file. You can call that RPC getexecname
Oops, that is possibly the source of a misunderstanding. I should have proposed getexechandle.
or whatever strikes your fancy.
the question is what path for open() call I should use for ROMfile?
You don't open() the session, that does the parent, because it needs to do it anyway. You only ask for the already opened file handle. I don't have the exact startup sequence In memory, but it could be that the handle is already communicated as part of the startup sequence to build the ds session. Then you only need to extend the startup sequence with a copy of that session.
Sincerely, Alexander
Genode users mailing list users@lists.genode.org https://lists.genode.org/listinfo/users
Your parent needs a ROMFile capability to run your program. Ask it for that capability to read your own file. You can call that RPC getexecname or whatever strikes your fancy.
the question is what path for open() call I should use for ROMfile?
Not sure if that's relevant to the discussion as I'm jumping in mid-stream, but in my Genode run scenarios I use the following, in order to access data files (not exe files) :
<start name="my_application"> <config> <vfs> <rom name="background.jpeg"/> ..etc
And then I can use Genode's libc POSIX functions (open() or fopen() etc) with the above "files", as they are located at the very top root of the VFS, i.e. "/background.jpeg" (just prepended with a slash)
Cedric
Returning to my question, could I use posix open(«/test-go») to open my binary from which I run executable, if I have the following code in run file:
<start name="test-go" caps="250"> <resource name="RAM" quantum="20M"/> <config verbose="yes" ld_verbose="yes" > <vfs> <dir name="dev"> <log/> </dir> </vfs> <libc stdout="/dev/log" stderr="/dev/log"/> </config> </start>
or I need to modify it and add another «double» option, something like
<start name="test-go" caps="250"> <resource name="RAM" quantum="20M"/> <config verbose="yes" ld_verbose="yes" > <vfs> <dir name="dev"> <log/> </dir> <rom name="test-go"/> </vfs> <libc stdout="/dev/log" stderr="/dev/log"/> </config> </start>
5 апр. 2021 г., в 14:36, ttcoder@netcourrier.com написал(а):
Your parent needs a ROMFile capability to run your program. Ask it for that capability to read your own file. You can call that RPC getexecname or whatever strikes your fancy.
the question is what path for open() call I should use for ROMfile?
Not sure if that's relevant to the discussion as I'm jumping in mid-stream, but in my Genode run scenarios I use the following, in order to access data files (not exe files) :
<start name="my_application"> <config> <vfs> <rom name="background.jpeg"/> ..etc
Sincerely, Alexander
Gesendet: Montag, 05. April 2021 um 19:48 Uhr Von: "Alexander Tormasov via users" users@lists.genode.org An: "Genode users mailing list" users@lists.genode.org Cc: "Alexander Tormasov" a.tormasov@innopolis.ru Betreff: Re: how to open elf file image from inside genode? libbacktrace question
Returning to my question, could I use posix open(«/test-go») to open my binary from which I run executable, if I have the following code in run file:
<start name="test-go" caps="250"> <resource name="RAM" quantum="20M"/> <config verbose="yes" ld_verbose="yes" > <vfs> <dir name="dev"> <log/> </dir> </vfs> <libc stdout="/dev/log" stderr="/dev/log"/> </config> </start>or I need to modify it and add another «double» option, something like
You need this configuration. But why do you want to use a way that is dependent upon config, when you need it for a language detail. That is why make something compiled in depend on something configured. And why has it to be POSIX open ()? Alternatively you can store that capability on startup in a static var in open () where it is returned from upon encountering a special parameter combination for instance an empty string as filename.
<start name="test-go" caps="250"> <resource name="RAM" quantum="20M"/> <config verbose="yes" ld_verbose="yes" > <vfs> <dir name="dev"> <log/> </dir> <rom name="test-go"/> </vfs> <libc stdout="/dev/log" stderr="/dev/log"/> </config> </start>5 апр. 2021 г., в 14:36, ttcoder@netcourrier.com написал(а):
Your parent needs a ROMFile capability to run your program. Ask it for that capability to read your own file. You can call that RPC getexecname or whatever strikes your fancy.
the question is what path for open() call I should use for ROMfile?
Not sure if that's relevant to the discussion as I'm jumping in mid-stream, but in my Genode run scenarios I use the following, in order to access data files (not exe files) :
<start name="my_application"> <config> <vfs> <rom name="background.jpeg"/> ..etc
Sincerely, Alexander
Genode users mailing list users@lists.genode.org https://lists.genode.org/listinfo/users
or I need to modify it and add another «double» option, something like
You need this configuration. But why do you want to use a way that is dependent upon config, when you need it for a language detail. That is why make something compiled in depend on something configured. And why has it to be POSIX open ()? Alternatively you can store that capability on startup in a static var in open () where it is returned from upon encountering a special parameter combination for instance an empty string as filename.
of course, I would prefer to be able to use posix open (or even just obtain a dup() of handle for my own executable file) to use it for read()/write() operations later. This is not mine code (libbacktrace is a part of original gcc), and I just want to be able to run it maximally smoothly inside genode. In this moment it try to call posix open() for some names like /proc/self/exe or similar, and later returned handle used deeply inside code. I will be happy to find ANY smooth way to have a handle for myself…
and, I am not so sure that loaded into ram current executable is suitable for such operations like read of elf header… probably I need to open it explicitly, but to do so I need to find it inside genode file system mounted name space (/vfs?)
Gesendet: Montag, 05. April 2021 um 22:05 Uhr Von: "Alexander Tormasov via users" users@lists.genode.org An: "Genode users mailing list" users@lists.genode.org Cc: "Alexander Tormasov" a.tormasov@innopolis.ru Betreff: Re: how to open elf file image from inside genode? libbacktrace question
or I need to modify it and add another «double» option, something like
You need this configuration. But why do you want to use a way that is dependent upon config, when you need it for a language detail. That is why make something compiled in depend on something configured. And why has it to be POSIX open ()? Alternatively you can store that capability on startup in a static var in open () where it is returned from upon encountering a special parameter combination for instance an empty string as filename.
of course, I would prefer to be able to use posix open (or even just obtain a dup() of handle for my own executable file) to use it for read()/write() operations later. This is not mine code (libbacktrace is a part of original gcc), and I just want to be able to run it maximally smoothly inside genode. In this moment it try to call posix open() for some names like /proc/self/exe or similar, and later returned handle used deeply inside code. I will be happy to find ANY smooth way to have a handle for myself…
and, I am not so sure that loaded into ram current executable is suitable for such operations like read of elf header… probably I need to open it explicitly, but to do so I need to find it inside genode file system mounted name space (/vfs?)
And "probably " is the watchword here. Genode is an improbable system. For instance all systems treat the file system as given, in genode it's optional. Therefore open () can fail in a very roundabout manner. Your excerpt of the function was too short. You didn't even complete the for loop. I would bet there is a static var in the function that caches the handle. And only if that is empty the for loop is entered. And if it is so then I would give that function a parameter of type handle (whatever is stored in the statistic) with default empty and an additional statement at the beginning of the function that in the case that this parameter is not empty stores that parameter in the statistic var for later retrieval and for skipping the for loop. And at startup you retrieve the open handle (which is another name for capability) from the parent (or, if the parent provides it voluntarily, copy it) and call that function with that handle once. If that cannot work because the function is constructed otherwise I would ask you to post the whole function.
Genode users mailing list users@lists.genode.org https://lists.genode.org/listinfo/users
Gesendet: Montag, 05. April 2021 um 22:05 Uhr Von: "Alexander Tormasov via users" users@lists.genode.org An: "Genode users mailing list" users@lists.genode.org Cc: "Alexander Tormasov" a.tormasov@innopolis.ru Betreff: Re: how to open elf file image from inside genode? libbacktrace question
First let me tell sorry for the spellchecker ran amok in my last post. It smuggled the ist into two instances of static to create statistic. Please ignore the contraband.
Now that I was able to look into the library source I can give you a recipe. 1) Define a number that is used as descriptor next 2) On startup dup2() the capability (possibly after casting to a POSIX Descriptor) to the defined number 3) In fileline.c introduce a last option of using the empty filename "" 4) In posix.c in function backtrace_open () check for empty filename 5) If so, blindly dup() the defined number. If successful return the created (By dup()) handle. If error you can either declare failure or try first to really open () the (empty) filename.
or I need to modify it and add another «double» option, something like
You need this configuration. But why do you want to use a way that is dependent upon config, when you need it for a language detail. That is why make something compiled in depend on something configured. And why has it to be POSIX open ()? Alternatively you can store that capability on startup in a static var in open () where it is returned from upon encountering a special parameter combination for instance an empty string as filename.
of course, I would prefer to be able to use posix open (or even just obtain a dup() of handle for my own executable file) to use it for read()/write() operations later. This is not mine code (libbacktrace is a part of original gcc), and I just want to be able to run it maximally smoothly inside genode. In this moment it try to call posix open() for some names like /proc/self/exe or similar, and later returned handle used deeply inside code. I will be happy to find ANY smooth way to have a handle for myself…
and, I am not so sure that loaded into ram current executable is suitable for such operations like read of elf header… probably I need to open it explicitly, but to do so I need to find it inside genode file system mounted name space (/vfs?)
Genode users mailing list users@lists.genode.org https://lists.genode.org/listinfo/users
Hallo Alexander,
Returning to my question, could I use posix open(«/test-go») to open my binary from which I run executable, if I have the following code in run file: […]
Yes, using 'open()' is the way to go. As libbacktrace hardcodes a few options you could piggy-back on one of them. I picked the '/proc/self/exe' option:
! <start name="test-go" caps="250"> ! <resource name="RAM" quantum="12M"/> ! <config verbose="yes" ld_verbose="yes"> ! <vfs> ! <dir name="dev"> <log/> </dir> ! ! <dir name="proc"> ! <dir name="self"> ! <rom name="exe" label="test-go"/> ! </dir> ! </dir> ! </vfs> ! <libc stdout="/dev/log" stderr="/dev/log"/> ! </config> ! </start>
With this VFS configuration in place libbacktrace opens the binary successfully. However, you will be greeted with the by now infamous deadlock ahead error message when using '-smp 2':
! [init -> test-go] Error: deadlock ahead, mutex=0x164040, return ip=0xf3383
The mutex in question is the 'Linker::mutex()' object. Again this is due to the fact that as side-effect of 'dl_iterate_phdr' a jump-slot relocation for 'mknod' takes place (*). Why that is I have not checked.
Now we are not entirely sure if taking the mutex in 'dl_iterate_phdr' is strictly necessary - that is something we have to investigate. Removing the mutex allows the 'go_app' to run through. So you are free to do the same. But to be clear, we do not know if this leads to other regressions. We are going to look into that.
(*) You can instrument that by applying the patch I attached that makes the linker verbose and shows you the object and index for the relocation:
! […] ! [init -> test-go] LD: SLOT libc.lib.so 0x20c ! [init -> test-go] LD: SLOT binary 0xe ! [init -> test-go] Error: deadlock ahead, mutex=0x164040, return ip=0xf3383
With this information you can check to which symbol the index belongs to:
$ readelf --dyn-syms test-go|grep '<14:' 14: 0000000000000000 0 FUNC GLOBAL DEFAULT UND mknod
Regards Josef
Gesendet: Donnerstag, 08. April 2021 um 15:23 Uhr Von: "Josef Söntgen" josef.soentgen@genode-labs.com An: "Alexander Tormasov via users" users@lists.genode.org Betreff: Re: how to open elf file image from inside genode? libbacktrace question
Hallo Alexander,
Returning to my question, could I use posix open(«/test-go») to open my binary from which I run executable, if I have the following code in run file: […]
Yes, using 'open()' is the way to go. As libbacktrace hardcodes a few
Is that really a good idea? Depending on vfs for something as bas(e)ic as stack tracing seems to me as overly broad. Because the needed capability is already a (private) member of the environment as _binary- >cap(). I only need to find a way to make it accessible by POSIX later. Or make it directly accessible by the stack tracing part.
options you could piggy-back on one of them. I picked the '/proc/self/exe' option:
! <start name="test-go" caps="250"> ! <resource name="RAM" quantum="12M"/> ! <config verbose="yes" ld_verbose="yes"> ! <vfs> ! <dir name="dev"> <log/> </dir> ! ! <dir name="proc"> ! <dir name="self"> ! <rom name="exe" label="test-go"/> ! </dir> ! </dir> ! </vfs> ! <libc stdout="/dev/log" stderr="/dev/log"/> ! </config> ! </start>
With this VFS configuration in place libbacktrace opens the binary successfully. However, you will be greeted with the by now infamous deadlock ahead error message when using '-smp 2':
! [init -> test-go] Error: deadlock ahead, mutex=0x164040, return ip=0xf3383
The mutex in question is the 'Linker::mutex()' object. Again this is due to the fact that as side-effect of 'dl_iterate_phdr' a jump-slot relocation for 'mknod' takes place (*). Why that is I have not checked.
Now we are not entirely sure if taking the mutex in 'dl_iterate_phdr' is strictly necessary - that is something we have to investigate. Removing the mutex allows the 'go_app' to run through. So you are free to do the same. But to be clear, we do not know if this leads to other regressions. We are going to look into that.
(*) You can instrument that by applying the patch I attached that makes the linker verbose and shows you the object and index for the relocation:
! […] ! [init -> test-go] LD: SLOT libc.lib.so 0x20c ! [init -> test-go] LD: SLOT binary 0xe ! [init -> test-go] Error: deadlock ahead, mutex=0x164040, return ip=0xf3383 With this information you can check to which symbol the index belongs to: $ readelf --dyn-syms test-go|grep '\<14:' 14: 0000000000000000 0 FUNC GLOBAL DEFAULT UND mknodRegards Josef
-- Josef Söntgen Genode Labs
http://www.genode-labs.com/ · http://genode.org/ _______________________________________________ Genode users mailing list users@lists.genode.org https://lists.genode.org/listinfo/users
I tried this patch for mutex and receive the following:
36M kernel memory
Genode 20.11-203-g9c8a05c12 <local changes> 530 MiB RAM and 63253 caps assigned to init [init] 0x1000000 .. 0x10ffffff: linker area [init] 0x40000000 .. 0x4fffffff: stack area [init] 0x30000 .. 0x161fff: ld.lib.so [init] LD: obj: binary 0x46 [init] LD: obj: binary 0x2e [init] LD: obj: binary 0x3d [init] LD: obj: binary 0x22 [init] LD: obj: binary 0x6 [init] LD: obj: binary 0x5 [init] LD: obj: binary 0x32 [init] LD: obj: binary 0xa [init] LD: obj: binary 0x44 [init] LD: obj: binary 0x15 [init] LD: obj: binary 0x25 [init] LD: obj: binary 0x16 [init] LD: obj: binary 0x3a [init] LD: obj: binary 0x3f [init] LD: obj: binary 0x34 [init] LD: obj: binary 0x8 [init] LD: obj: binary 0x42 [init] LD: obj: binary 0x2f [init] LD: obj: binary 0x3c [init] LD: obj: binary 0x2d [init] LD: obj: binary 0x38 [init] LD: obj: binary 0x19 [init] LD: obj: binary 0x1f [init] LD: obj: binary 0x30 [init] LD: obj: binary 0x1b [init] LD: obj: binary 0x27 page fault, pd='init' thread='ep' cpu=0 ip=0xb1d73 address=0x401eeffc stack pointer=0x401eeff0 qualifiers=0x6 irUWp reason=1
which seems to appear in thread->myself():
b1d5a: 48 8d 1d df ff ff ff lea -0x21(%rip),%rbx # b1d40 Genode::Thread::myself() b1d61: 4c 01 db add %r11,%rbx /var/services/homes/admin/gen/21.02/repos/base/src/lib/base/thread_myself.cc:32 if (sp < Genode::stack_area_virtual_base() || b1d64: 48 01 dd add %rbx,%rbp b1d67: 49 89 df mov %rbx,%r15 /var/services/homes/admin/gen/21.02/repos/base/src/lib/base/thread_myself.cc:23 { b1d6a: 48 83 ec 18 sub $0x18,%rsp /var/services/homes/admin/gen/21.02/repos/base/src/lib/base/thread_myself.cc:31 addr_t sp = (addr_t)(&dummy); b1d6e: 4c 8d 64 24 0c lea 0xc(%rsp),%r12 /var/services/homes/admin/gen/21.02/repos/base/src/lib/base/thread_myself.cc:24 int dummy = 0; /* used for determining the stack pointer */ b1d73: c7 44 24 0c 00 00 00 movl $0x0,0xc(%rsp) b1d7a: 00 /var/services/homes/admin/gen/21.02/repos/base/src/lib/base/thread_myself.cc:32 if (sp < Genode::stack_area_virtual_base() || b1d7b: ff d5 callq *%rbp b1d7d: 49 39 c4 cmp %rax,%r12 b1d80: 72 46 jb b1dc8 Genode::Thread::myself()+0x88 /var/services/homes/admin/gen/21.02/repos/base/src/lib/base/thread_myself.cc:33 sp >= Genode::stack_area_virtual_base() + Genode::stack_area_virtual_size()) b1d82: ff d5 callq *%rbp b1d84: 48 05 00 00 00 10 add $0x10000000,%rax /var/services/homes/admin/gen/21.02/repos/base/src/lib/base/thread_myself.cc:32 if (sp < Genode::stack_area_virtual_base() ||
Now we are not entirely sure if taking the mutex in 'dl_iterate_phdr' is strictly necessary - that is something we have to investigate. Removing the mutex allows the 'go_app' to run through. So you are free to do the same. But to be clear, we do not know if this leads to other regressions. We are going to look into that.
(*) You can instrument that by applying the patch I attached that makes the linker verbose and shows you the object and index for the relocation:
! […] ! [init -> test-go] LD: SLOT libc.lib.so 0x20c ! [init -> test-go] LD: SLOT binary 0xe ! [init -> test-go] Error: deadlock ahead, mutex=0x164040, return ip=0xf3383
With this information you can check to which symbol the index belongs to:
$ readelf --dyn-syms test-go|grep '<14:' 14: 0000000000000000 0 FUNC GLOBAL DEFAULT UND mknod
Hello Alexander,
On Thu, Apr 08, 2021 at 18:18:01 CEST, Alexander Tormasov via users wrote:
page fault, pd='init' thread='ep' cpu=0 ip=0xb1d73 address=0x401eeffc stack pointer=0x401eeff0 qualifiers=0x6 irUWp reason=1
The fault address is in the stack area and looks suspicious: 0x401eeff0 is 4100 bytes below 64K of stack. Could it be your stack size is 64K which does not suffice your use case?
Greets
Hello Christian,
On Thu, Apr 08, 2021 at 18:18:01 CEST, Alexander Tormasov via users wrote:
page fault, pd='init' thread='ep' cpu=0 ip=0xb1d73 address=0x401eeffc stack pointer=0x401eeff0 qualifiers=0x6 irUWp reason=1
The fault address is in the stack area and looks suspicious: 0x401eeff0 is 4100 bytes below 64K of stack. Could it be your stack size is 64K which does not suffice your use case?
it is really hard to say something about, but in the past 64k was definitely enough. I suspect nested "panic in panic" during processing of c++ exceptions…
while it is hard to say something because of strange debugger work (request 4 more caps from init and hang), I try to find any methods to understand/debug situation…
Alexander
Alexander,
On Fri, Apr 09, 2021 at 13:07:37 CEST, Alexander Tormasov via users wrote:
On Thu, Apr 08, 2021 at 18:18:01 CEST, Alexander Tormasov via users wrote:
page fault, pd='init' thread='ep' cpu=0 ip=0xb1d73 address=0x401eeffc stack pointer=0x401eeff0 qualifiers=0x6 irUWp reason=1
The fault address is in the stack area and looks suspicious: 0x401eeff0 is 4100 bytes below 64K of stack. Could it be your stack size is 64K which does not suffice your use case?
it is really hard to say something about, but in the past 64k was definitely enough. I suspect nested "panic in panic" during processing of c++ exceptions…
while it is hard to say something because of strange debugger work (request 4 more caps from init and hang), I try to find any methods to understand/debug situation…
Sorry but I completely ignored the fact that the failing component is the root init instance. So, it might help to just remove ld_verbose="yes" from its config while debugging with Josef's instrumentation. The change seems to trigger unexpected stack usage in init which you don't seem to debug currently.
Regards
Hello Josef, I apply patch and remove some ld_verbose options from run file (to allow it to work), and got another interesting backtrace during unwind attempt, see below. In short, it try to open ld.lib.so (from inside genode), open() fail, and it try to obtain libc errno (via jmp_slot/etc from shared library). So, this is inside dl_iterate_phdr under mutex, as usual, and double mutex call related to search of errno. I suspect other potential code path where we can hit «seach for address of some functions from shared libraries»...
Finally I will remove, as suggested , mutex guard from dl_iterate_phdr completely - while not so sure that this will not lead to error. Also I add the following to the run file to allow read of ld.lib.so from inside (not sure that this is correct - may be need something else? like direct reference to rom session?) <dir name=«ld.lib.so»> </dir>
Seems that after that operation my SMP small test with even 4 CPU works somehow (need more extensive tests to be run for golang and goroutunes).
Question: how to verify that removed mutex call does not break something important?
#0 __error () at /var/services/homes/admin/gen/21.02/repos/libports/src/lib/libc/errno.cc:20 #1 0x0000000010f80a16 in Libc::Vfs_plugin::stat (this=<optimized out>, path=<optimized out>, buf=<optimized out>) at /var/services/homes/admin/gen/21.02/repos/libports/src/lib/libc/vfs_plugin.cc:809 #2 0x0000000010f2f07e in Libc::resolve_symlinks (path=<optimized out>, resolved_path=...) at /var/services/homes/admin/gen/21.02/repos/os/include/os/path.h:239 #3 0x0000000010f30cbb in __sys_open (pathname=0x161380 Linker::Ld::linker()::_linker+32 "ld.lib.so", flags=0x100000) at /var/services/homes/admin/gen/21.02/repos/os/include/os/path.h:239 #4 0x000000000119aa56 in backtrace_open (filename=0x161380 Linker::Ld::linker()::_linker+32 "ld.lib.so", error_callback=0x1198503 <error_callback>, data=0x405fe770, does_not_exist=0x405fe1c4) at /var/services/homes/admin/gen/21.02/contrib/libbacktrace-cabcd83bc903137f607eda3a65eb09feaa5f507e/src/lib/gcc/libbacktrace/posix.c:67 #5 0x000000000119fd00 in phdr_callback (info=0x405fe200, size=0x20, pdata=0x405fe290) at /var/services/homes/admin/gen/21.02/contrib/libbacktrace-cabcd83bc903137f607eda3a65eb09feaa5f507e/src/lib/gcc/libbacktrace/elf.c:3264 #6 0x00000000000ebe6e in dl_iterate_phdr (callback=0x119fc27 <phdr_callback>, data=0x405fe290) at /var/services/homes/admin/gen/21.02/repos/base/src/lib/ldso/exception.cc:53 #7 0x000000000119fe7a in backtrace_initialize (state=0x29000, filename=0x126d0c6 "/proc/self/exe", descriptor=0x3, error_callback=0x1198503 <error_callback>, data=0x405fe770, fileline_fn=0x405fe370) at /var/services/homes/admin/gen/21.02/contrib/libbacktrace-cabcd83bc903137f607eda3a65eb09feaa5f507e/src/lib/gcc/libbacktrace/elf.c:3313 #8 0x000000000119a8c1 in fileline_initialize (state=0x29000, error_callback=0x1198503 <error_callback>, data=0x405fe770) at /var/services/homes/admin/gen/21.02/contrib/libbacktrace-cabcd83bc903137f607eda3a65eb09feaa5f507e/src/lib/gcc/libbacktrace/fileline.c:143 #9 0x000000000119a965 in backtrace_pcinfo (state=0x29000, pc=0x11985d0, callback=0x119804e <callback>, error_callback=0x1198503 <error_callback>, data=0x405fe770) at /var/services/homes/admin/gen/21.02/contrib/libbacktrace-cabcd83bc903137f607eda3a65eb09feaa5f507e/src/lib/gcc/libbacktrace/fileline.c:177 #10 0x000000000119ac8c in unwind (context=0x405fe420, vdata=0x405fe700) at /var/services/homes/admin/gen/21.02/contrib/libbacktrace-cabcd83bc903137f607eda3a65eb09feaa5f507e/src/lib/gcc/libbacktrace/backtrace.c:91 #11 0x00000000000d7179 in _Unwind_Backtrace (trace=0x119abbc <unwind>, trace_argument=0x405fe700) at /genodelabs-genode-2491eee/contrib/gcc-3cade18cf9defeefa714aa91de3b157fbad4aa18/src/noux-pkg/gcc/libgcc/unwind.inc:307 #12 0x000000000119ad62 in backtrace_full (state=0x29000, skip=0x0, callback=0x119804e <callback>, error_callback=0x1198503 <error_callback>, data=0x405fe770) at /var/services/homes/admin/gen/21.02/contrib/libbacktrace-cabcd83bc903137f607eda3a65eb09feaa5f507e/src/lib/gcc/libbacktrace/backtrace.c:127 #13 0x00000000011985d1 in runtime_callers (skip=0x2, locbuf=0xc4200148d8, m=0x20, keep_thunks=0x0) at /var/services/homes/admin/gen/21.02/contrib/libgo-281260d9bdc27fefb62c00310025c54e1e629a2e/src/lib/gcc/libgo/runtime/go-callers.c:207
Hello Alexander,
[…] So, this is inside dl_iterate_phdr under mutex, as usual, and double mutex call related to search of errno. I suspect other potential code path where we can hit «seach for address of some functions from shared libraries»...
Finally I will remove, as suggested , mutex guard from dl_iterate_phdr completely - while not so sure that this will not lead to error.
Please replace your mutex-remove commit with the following commit [1], which replaces the linker mutex with the shared-object mutex. This behaviour is in line with other dynamic-linkers.
[1] https://github.com/genodelabs/genode/commit/c45d964bc23f4f
(I moved the quotation down a bit to get concise block of text.)
In short, it try to open ld.lib.so (from inside genode), open() fail, and it try to obtain libc errno (via jmp_slot/etc from shared library). So, this is inside dl_iterate_phdr under mutex, as usual, and double mutex call related to search of errno. I suspect other potential code path where we can hit «seach for address of some functions from shared libraries»... Also I add the following to the run file to allow read of ld.lib.so from inside (not sure that this is correct - may be need something else? like direct reference to rom session?) <dir name=«ld.lib.so»> </dir>
In case libbacktrace wants to access other shared-objects directly, besides the binary itself, I would deal with it in the same way, e.g:
! […] ! <vfs> ! <dir name="dev"> <log/> </dir> ! ! <rom name="ld.lib.so"/> ! <rom name="xyz.lib.so"/> ! ! <dir name="proc"> ! <dir name="self"> ! <rom name="exe" label="test-go"/> ! </dir> ! </dir> ! </vfs> ! […]
This config operates under the assumption that libbacktrace wants to access the shared-objects via 'open("/ld.lib.so")' - I have not checked it that's true.
Seems that after that operation my SMP small test with even 4 CPU works somehow (need more extensive tests to be run for golang and goroutunes).
Question: how to verify that removed mutex call does not break something important?
Please just use the commit I referenced above. As far as our automated tests go, it did not introduce any regressions (of course that is no guarantee that all potential corner-cases are covered).
Regards Josef
-
Alexander Tormasov -
Christian Helmuth -
Josef Söntgen -
ttcoder＠netcourrier.com -
Uwe