So, I checked out your 20.11 branch and tried to follow your instructions. Unfortunately, I could not compile libgo out-of-the-box because libtool was complaining about “unrecognized options” and I stopped at this point.
sorry, forget to remove it - it does not matter, everything configured correctly even with them. To avoid misunderstanding I fix this and coding style in additional patch for the same 20.11 branch, please, take a look
This brings me to my first question
- Is there a technical reason for executing the auto-tools build system
rather than porting the libraries, i.e., create the .mk files for Genode's build system?
configuration of the system is VERY complex, result of 10+ years legacy code development as a part of gcc, and IMHO better to get rid of manual handling of it. In particular, list of files to be compiled for golang runtime sources generated «on the fly» and complied using libtool script with complex 2-3 times in a row generation process (from .m4 to Makefile and inside for every go package in relation to choosed subsystem)
Also, I assume that using «native» build system from gcc could simplify future port of newer version of golang runtime and related gcc libraries/parts
The instructions in the README lack the motivation behind that decision.
my porting experience is the main motivator - closer to original build system, easier to support it later.
and furthermore
- Is it strictly necessary to solely produce static libraries or could
we get away with a dynamic shared-object, libgo.lib.so, that incorporates everything and the target application is linked against or is it possible to have multiple shared-objects. After all, there is already a libffi port
- ffi - that is built as .lib.so.
I do not check this carefully, but standalone libffi and libffi from inside gcc distribution (which I port smoothly) -IMHO different beasts. At least, gcc build system assume part of it (e.g. generated file ffi.h) to be placed inside specific place of gcc build environment; and configuration options could be different (I do not check this in details). May be I am wrong, and we can use standard libffi (which is already ported)
about shared vs .a libraries … frankly, I do not remember exactly a reason, probably the same - default build system do generate .a file which is good for golang application. Do not investigated yet, may be it will be easy.
While stepping through the libgo commit I noticed the patches:
- You choose to add a new run-time target, “inno”, that is based on NetBSD
rather than using the existing FreeBSD target. For all intents and purposes, Genode appears to be FreeBSD for ported software due to using its libc¹. Is there a reason why that is not possible for Golang's run-time?
Golang runtime deeply integrated with OS services and sys calls, use a lot of them. Genode implementation of FreeBSD ones has significant implementation gaps. NetBSD is not that advanced, has smaller set of features and, therefore, size of gaps is also smaller. So, from plain porting perspective in the very beginning, volume of work for NetBSD version port seems significantly smaller than ones based on FreeBSD (which Genode port does not have full feature sets available in original OS). This was for me a reason. Again, may be on this stage we can add some features from freebsd version as well.
IIRC, on some platforms, e.g. Solaris(?), the normal Golang run-time solely uses the libc to interface with the OS instead of wrapping system-calls directly. Is it the same with gccgo's run-time?
There are some semantical differences between libc, sys calls, POSIX and even glibc. I can’t say that there are single simple answer - I see some random files related to subsystems whch partially belongs to any BSD-based implementation, some just directly call sys calls via generated golang or even c wrappers. This is a result of 11 years golang ports evolution and support of 9 very different subsystems. Runtime developers try to mimic/mask differences in features from main code, not always correctly IMHO.
With regard to the first questions, if we remove auto-tools from the equation we could shadow specific run-time parts where the upstream implementation does not work on Genode.
This is not that straightforward and easy task. For example, Golang use concept of packages, and packages do have initializers (like constructors) which called before main.main(). So, even if you do not use some of them, they can potentially call low-level functions (e.g. system calls) and, if they are not implemented, or we put dummy stubs - logic can fail and we will not able to start the program.
This refer to more common question about porting strategy. What we can do is 1. Modify or not modify golang runtime. 2. Modify or not modify Genode OS services (especially on per-OS basis) 3. Do create universal dummy/proxy functions to mimic problems from runtime. I think that 2 is most complicated. I use partially approach 1 from above (by introducing additional subsystem with minimization of changes in runtime), and file the rest by 3. Unfortunately, due to historical reasons, I have all 3 approaches combined - some dirty hacks on golang go runtime to avoid call of unimplemented functions, Provide some dummy implementation of small set of system calls, and small modification of Genode related to ANON mmap and set/getcontext support.
¹) Granted, our libc port is not feature-complete and might lack expected functionality, e.g. kqueue/sysctl or as you already noticed mmap handling, but we could take this as an opportunity to extended our libc port.
In that case it is relatively easy to return support of some features what I manually cut from go code (they are relatively isolated), eg take them from freeBSD golang runtime.
- I noticed that you use internal headers for implementing the stack
handling. Is in this context the available Genode::Thread API not enough?
My somewhat naive point of view would be to use our POSIX layer wherever possible and only diverge where necessary - that only boils down to go-routine and stack handling or does it not?
No, because to implement context switching I need to create a support for set/getcontext family of functions. They are from one point of view, similar to set jump/longjump, but they also has internal call to set/restore signals mask using sys call API which is absent in Genode. And, moreover, it has hidden problem because Genode threads should keep their stack in some pre-defined area which is tracked in myself() function. goroutines behaves like user-space fibers, they should be switched manually using manually crafted context. Makecontex() should obtain stack from mentioned area - this is the main reason why I use internal function to create my own stack. Posix threads do not have such problem, so, I can’t use just them.
Finally, the most important question:
- From what I can tell the port itself still somewhat feels like a
proof-of-concept that as you mentioned in the README is the side result of a research project. It would require further clean-up before we could add it to Genode.
(This includes the current short-comings like missing context support for go-routines, reliance on -fno-omit-framepointer, etc. The support code that is currently part of the component should be moved to a central location like you already suggested.)
Would you be willing to do that clean-up work or is it from your point of view enough to just keep the port as an example in case somebody else wants to do that? Your reaction so far, creating the github issue, suggest that you do have some motivation.
I am limited in time which I can spend around this project. Anyway, I think that work done can be good basis for future development and in general I want to bring it to the level which will be suitable for people interested in utilization and continuation. For better understanding, I had a project to port docker container support to some microkernel os, which started (as a experimental testbed) from Genode. Initially I do not assume to deal with go at all, but later found that it probably will be the easiest step (taking into account the availability of gccgo in Genode). So, I make initial dirty port of runtime as proof-of-concept and try to compile docker itself. I found a lot of interesting things about direct golang connection to os services/syscalls on different platforms, and have finished minimal port of it using makefiles (generated ones), not «go build». So, golang runtime port became «side project» of it.
In general I assume that I can do some work (at least this year) toward bringing the project to more usable state, together with potential testbed and relation of improvement of my knowledge about Genode in context of another potential project related to port of part of OS (related to persistency model).
Even docker port can be considered later (having in mind my experience of its port attempt to Genode)
I ask because in the past we got a welcome one-time contribution for Rust support on Genode. However, since we do not use Rust, we merely dragged it along and eventually removed it². As support for Golang has no immediate priority for us, we hesitate to spend our limited time on this at the moment.
That being said, I wrote our CI-tools web front end in Golang and running it on Genode certainly would be nice and could be a reason to nonetheless set some time aside to get that up and running :-). Judging by the reaction on your issue, we could very well make a community effort out of supporting Golang.
²) To not make the same mistake I imagine adding support to our tooling for creating Golang based components is necessary. Maybe it is worthwhile to extend 'goa'[sic] in that regard so that a Golang-developer can use 'go' to initially develop his or her application and later one use 'goa' to create a binary for Genode, while 'goa' does the heavy-lifting behind the scenes. I think you briefly touched the “go build → Makefile” topic in one of your mails. After all, rather than writing specific Genode components in Golang I guess people want to deploy their existing applications first and would be fine with interacting with Genode via the POSIX/VFS layer.
In general I think that it could be a good idea to list a set of sub task to make golang port and related infrastructure more feasible
In particular, I think that it is necessary to create in genode mainstream the following 1. setcontext family of calls (I am not happy with current one) 2. Anon mmap extended support (better than current one, potentially mine can be intermediate version) 3. Go as language support in genode build infrastructure 4. Some of aux libraries could be compiled as a part of tools (libatomic/libbacktrace, may be even libffi)
Everything else (including libgo and test apps) could be started as a part of “world” repo
Probably we need to specify a list of things to cleanup/rebuild current port
What do you think?