I made an attempt to port libunistring (needed for guile) trying to follow porting guide. I didn't go far but feel that what I've done did not go in the right direction and would like to ask for opinions about my feelings.
Short story
I would like to be able to do:
./configure --host=genode-x86 && make
Longer story
List of compiled sources in libunistring depends on results of running configure and there is more than 500 source files. Even if I would prepare list of files to compile probably it would be different in next revision so it would have to be at least updated during upgrade. I feel it would be waste of resources.
From different perspective as on of the goals of Genode for this year is
to have more software packages ported I think the procedure to port should be as straightforward as possible. Probably most of the ported software will be based on libc (I think that packages that will not be linked against libc mostly will constitute base of genode not userspace programs) and to be able to quickly port libc based libraries and applications build system should not replaced but reused.
My feeling is that current Genode build system is great for Genode specific components but for porting general purpose requires too much maintenance.
I don't know detailes underneath current makefiles based build system but I imagine steps to compile using configure could look somehow like this:
1. Prepare spec with dependencies containing libc - just like it is right now
2. make deps - build dependencies and install headers somewhere - this is probably done right now in current build system
3. Run configure in this prepared environment. Running it without prepared environment fails when testing if genode-x86-gcc can create binaries - so quite quickly :-)
4. make - real porting work - hopefully not much
5. make install - to some prepared place?
6. from installed files take headers in case of libraries and binaries - this somehow works right now
Firstly I'd like to hear what you think about the idea in general.
Eventually could you provide some technical informations where are the most important parts of build system where I should look to be able to make such changes?
I don't know exactly how to best port a program or library to Genode, so I don't really have the best answer, but I suggest that you look at the makefiles in libports. You can also look at the ones in ports, but the noux-pkg makefiles won't be particularly helpful, except when porting utilities to noux. The makefiles don't all list the source files individually. On Feb 15, 2016 3:00 PM, "Tomasz Gajewski" <tomga@...300...> wrote:
I made an attempt to port libunistring (needed for guile) trying to follow porting guide. I didn't go far but feel that what I've done did not go in the right direction and would like to ask for opinions about my feelings.
Short story
I would like to be able to do:
./configure --host=genode-x86 && make
Longer story
List of compiled sources in libunistring depends on results of running configure and there is more than 500 source files. Even if I would prepare list of files to compile probably it would be different in next revision so it would have to be at least updated during upgrade. I feel it would be waste of resources.
From different perspective as on of the goals of Genode for this year is
to have more software packages ported I think the procedure to port should be as straightforward as possible. Probably most of the ported software will be based on libc (I think that packages that will not be linked against libc mostly will constitute base of genode not userspace programs) and to be able to quickly port libc based libraries and applications build system should not replaced but reused.
My feeling is that current Genode build system is great for Genode specific components but for porting general purpose requires too much maintenance.
I don't know detailes underneath current makefiles based build system but I imagine steps to compile using configure could look somehow like this:
Prepare spec with dependencies containing libc - just like it is right now
make deps - build dependencies and install headers somewhere - this is probably done right now in current build system
Run configure in this prepared environment. Running it without prepared environment fails when testing if genode-x86-gcc can create binaries - so quite quickly :-)
make - real porting work - hopefully not much
make install - to some prepared place?
from installed files take headers in case of libraries and binaries - this somehow works right now
Firstly I'd like to hear what you think about the idea in general.
Eventually could you provide some technical informations where are the most important parts of build system where I should look to be able to make such changes?
-- Tomasz Gajewski
Site24x7 APM Insight: Get Deep Visibility into Application Performance APM + Mobile APM + RUM: Monitor 3 App instances at just $35/Month Monitor end-to-end web transactions and take corrective actions now Troubleshoot faster and improve end-user experience. Signup Now! http://pubads.g.doubleclick.net/gampad/clk?id=272487151&iu=/4140 _______________________________________________ genode-main mailing list genode-main@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/genode-main
Hi Tomasz,
Am 15.02.2016 um 22:59 schrieb Tomasz Gajewski:
I would like to be able to do:
./configure --host=genode-x86 && make
As far as I know, we currently do not issue 'configure' through the ports mechanism. Instead the customized config.* files of the ports are provided. An example for this is libav:
libports/src/lib/libav/config.h libports/src/lib/libav/config.mak
Then the Genode sided directory is added to the include paths, so the config header applies when building the port and the config makefile is included and applied in the Genode config:
libports/lib/mk/av.inc: ... INC_DIR += $(REP_DIR)/src/lib/libav ... include $(REP_DIR)/src/lib/libav/config.mak SRC_C += $(OBJS:.o=.c) SRC_C += $(OBJS-yes:.o=.c) ...
Cheers, Martin
Hello Tomasz,
* Tomasz Gajewski <tomga@...300...> [2016-02-15 22:59:23 +0100]:
I would like to be able to do:
./configure --host=genode-x86 && make
At the first glance that would be convient indeed but on the second not so much — mostly because it involves having a build environment ready, which means having autotools, pkg-config and the like available. That only works well if your host system is quite POSIXish and if the distributed autotools files of the program/library in question are aware of the host target.
Longer story
List of compiled sources in libunistring depends on results of running configure and there is more than 500 source files. Even if I would prepare list of files to compile probably it would be different in next revision so it would have to be at least updated during upgrade. I feel it would be waste of resources.
From different perspective as on of the goals of Genode for this year is to have more software packages ported I think the procedure to port should be as straightforward as possible. Probably most of the ported software will be based on libc (I think that packages that will not be linked against libc mostly will constitute base of genode not userspace programs) and to be able to quickly port libc based libraries and applications build system should not replaced but reused.
My feeling is that current Genode build system is great for Genode specific components but for porting general purpose requires too much maintenance.
You have to maintain the build system either way. Since all of the projects using autotools are not aware of Genode you have to patch the various scripts, makefiles or m4 macros files to recognize it or rather you have to execute autoreconf — if the project even ships its .ac files — or replace the distributed files directly. Futhermore, nowadays pkg-config is used to query certain dependencies, so you have to provide the pkg-config database and to manage the .pc files as well.
I don't know detailes underneath current makefiles based build system but I imagine steps to compile using configure could look somehow like this:
Prepare spec with dependencies containing libc - just like it is right now
make deps - build dependencies and install headers somewhere - this is probably done right now in current build system
Run configure in this prepared environment. Running it without prepared environment fails when testing if genode-x86-gcc can create binaries - so quite quickly :-)
make - real porting work - hopefully not much
make install - to some prepared place?
from installed files take headers in case of libraries and binaries - this somehow works right now
I am not sure if that really saves you much time in the long run. Providing a mechanism that integrates well with the Genode build system is one thing. Making this mechanism useable by a majority of GNU build tool based build systems so that you can conviently just execute 'configure', is something completely different.
Firstly I'd like to hear what you think about the idea in general.
To be honest I personally do not think that its such a good idea but that is mostly based on my past experience in dealing with autotools. Been there done that while using the OpenBSD ports system and when porting programs to Haiku — everytime I had to use autotools on a system that was unknown or slightly different than the status quo, e.g. GNU/Linux, it was uncomfortable to say the least(*). And I do not expect it be any different in this case.
I would rather spend my time doing the actual porting work than messing around with short comings of the original build system or breaking things for one particular program while fixing the build tools another one.
The other thing that annoys me about autotools is the fact, that you would have to run it _every_ time you want compile the source, although the target system has not changed at all. Sure, we could cache the result for later use, but… In short, I would rather avoid the checking and bookkeeping that comes free when using it altogether.
(*) One can argue that this is not the fault of the GNU build system itself but rather of the way it certainly is used in the wild.
Eventually could you provide some technical informations where are the most important parts of build system where I should look to be able to make such changes?
There is _repos/ports/mk/noux.mk_ which basically is a wrapper around the autotools suite for building noux-pkgs of programs that use it from within the Genode build system. This wrapper creates an environment that contains the important compiler options and flags to build the program for running in Noux. If you start from there and make your way along to _repos/base/mk_ you will hit the most important parts.
Regards Josef
Josef Söntgen <josef.soentgen@...1...> writes:
You have to maintain the build system either way. Since all of the projects using autotools are not aware of Genode you have to patch the various scripts, makefiles or m4 macros files to recognize it or rather you have to execute autoreconf — if the project even ships its .ac files — or replace the distributed files directly. Futhermore, nowadays pkg-config is used to query certain dependencies, so you have to provide the pkg-config database and to manage the .pc files as well.
In theory autoconf tests should test for features not for systems although reality is probably different.
I am not sure if that really saves you much time in the long run. Providing a mechanism that integrates well with the Genode build system is one thing. Making this mechanism useable by a majority of GNU build tool based build systems so that you can conviently just execute 'configure', is something completely different.
I understand your points but I can't imagine another way than incrementally adding support for different build systems/engines. Maybe I'm too much influenced by existing linux package systems though, where they use "native" packages' build systems.
To be honest I personally do not think that its such a good idea but that is mostly based on my past experience in dealing with autotools. Been there done that while using the OpenBSD ports system and when porting programs to Haiku — everytime I had to use autotools on a system that was unknown or slightly different than the status quo, e.g. GNU/Linux, it was uncomfortable to say the least(*). And I do not expect it be any different in this case.
I know that plenty of packages will not build on such system without modifications but what is the alterntaive? I believe that even maintaining lists of source files for some greater number of packges is infeasible. And maybe I'm naive but fixes for autoconf tests could be merged into source packages but Genode specific patches definitely not.
There is _repos/ports/mk/noux.mk_ which basically is a wrapper around the autotools suite for building noux-pkgs of programs that use it from within the Genode build system. This wrapper creates an environment that contains the important compiler options and flags to build the program for running in Noux. If you start from there and make your way along to _repos/base/mk_ you will hit the most important parts.
That was really the information that I needed. Thank you.
After fighting with my lack of knowledge about advanced make usage I've managed to prepare a version of noux.mk for a library (without noux dependency) and build libunistring - without any source changes. I haven't tested it yet though.
What I think I understand about current Genode build system it consists of two layers:
* a package manager like layer that maintains dependencies between programs/libraries
* build rules for building concrete programs/libraries
Both are implemented using make. I believe (although I haven't verified it) that work of Emery to port nix to Genode is an attempt to replace the former layer with nix.
I think that using guix and guile could be a better approach due to using general purpose language (although not liked too much :-) ) and Genode specific requirements, which could be harder to achieve in a language dedicated specifically to build packages. But I have to check it myself first. If I'm able to have a proof of concept and know more about Emery's work I'll try to make some conclusions share them with you here.
libunistring that I compiled is a dependency for guile and compiling it is first step in my attempt to check if guix can become package manager for Genode.
My take on this is that in the future it will be possible to construct Noux environments such that autotools can be run in Noux to produce a native Noux build (and maybe cross-compiling), but the practical thing to do will be to try to and fool existing autotools tests, and only inside Noux.
I do think that Nix can trick autotools into building on Noux, but first Noux needs to be faster and I want to find out how far merging file systems can go before hitting the point of diminishing returns.
On Wed, Feb 24, 2016 at 09:49:01PM +0100, Tomasz Gajewski wrote:
What I think I understand about current Genode build system it consists of two layers:
a package manager like layer that maintains dependencies between programs/libraries
build rules for building concrete programs/libraries
Both are implemented using make. I believe (although I haven't verified it) that work of Emery to port nix to Genode is an attempt to replace the former layer with nix.
Yes, I want to create an alternate layer for dependencies and locating the required makefiles. When the package output specification is ready then both build systems should produce cross-compatible packages.
I think that using guix and guile could be a better approach due to using general purpose language (although not liked too much :-) ) and Genode specific requirements, which could be harder to achieve in a language dedicated specifically to build packages. But I have to check it myself first. If I'm able to have a proof of concept and know more about Emery's work I'll try to make some conclusions share them with you here.
You might be right, sometimes I think that Nix is a bit heavyweight for simply generating text files like it often does, but I don't know much about Guile.
As far as porting Guix, it might look like a good idea to try and make it work on top of Noux, but my experience with Nix was that to fix every required unix-ism will take more time and in the end you will loose some important potential features. It may be possible to reuse my Nix libraries for Guix, but I made some changes that break compabality with upstream Nix for the sake of security and content addressing, which you might not agree with.
The build controller component I have actually contains no 'Nix' code or libraries, so if Guix can produce the some low-level build recipe files, that would deduplicate a lot of work.
Cheers, Emery
On Thu, Feb 25, 2016 at 12:45:40AM +0100, emery@...261... wrote:
My take on this is that in the future it will be possible to construct Noux environments such that autotools can be run in Noux to produce a native Noux build (and maybe cross-compiling), but the practical thing to do will be to try to and fool existing autotools tests, and only inside Noux.
I think this is a good assumption
I do think that Nix can trick autotools into building on Noux, but first Noux needs to be faster and I want to find out how far merging file systems can go before hitting the point of diminishing returns.
On Wed, Feb 24, 2016 at 09:49:01PM +0100, Tomasz Gajewski wrote:
What I think I understand about current Genode build system it consists of two layers:
a package manager like layer that maintains dependencies between programs/libraries
build rules for building concrete programs/libraries
Both are implemented using make. I believe (although I haven't verified it) that work of Emery to port nix to Genode is an attempt to replace the former layer with nix.
Yes, I want to create an alternate layer for dependencies and locating the required makefiles. When the package output specification is ready then both build systems should produce cross-compatible packages.
You might be right, sometimes I think that Nix is a bit heavyweight for simply generating text files like it often does, but I don't know much about Guile.
Guile's a bit more heavyweight I think. However, Guix has the ability to do a lot of really cool stuff when it comes to build environments- I'm not sure if Nix has the same support. One interesting thing I've been toying with recently is the 'build a VM', which has some Guile code to take a derivation and run it in a VM. I imagine we could have something like that in Noux, then take the output and use that in Genode. This might be already how you do it though.
As far as porting Guix, it might look like a good idea to try and make it work on top of Noux, but my experience with Nix was that to fix every required unix-ism will take more time and in the end you will loose some important potential features. It may be possible to reuse my Nix libraries for Guix, but I made some changes that break compabality with upstream Nix for the sake of security and content addressing, which you might not agree with.
Guix is based on POSIX-isms which Noux could certainly help out with, currently there's work to bring HURD support in which is testing Linux-specific assumptions. It'd be very interesting to survey the assumptions Guix makes for it's host-side code.
The build controller component I have actually contains no 'Nix' code or libraries, so if Guix can produce the some low-level build recipe files, that would deduplicate a lot of work.
Assuming you mean the build controller just runs Bash code, then I imagine it'd be a problem given instead of using shells to build, Guix uses a well-defined subset of build-side modules that can run derivations.
Cheers, Emery
Now that I look, a Guix port could use much of the same underside of Nix, the derivation recipe format is the same: http://git.savannah.gnu.org/cgit/guix.git/tree/README#n128
The problem is that I interpret the format differently than on unix, which is not so bad, because Genode derivations are always marked as such. I use the environment variables section to rewrite requests for ROMs and File_system. It was a better solution than assuming that derivations must be built with Noux, and its less work to write a few variations of a Noux wrapper at the higher level than it is to make Noux work everytime at the C++ level.
I'm still not that interested in Guix, but, if someone could port the Guile/Guix evaluator, I would help hook it to my store and builder components. What would be really helpful is if someone could come up with a good system of distributing binary packages that would be compatible between the two.
Emery
On Thu, Feb 25, 2016 at 10:08:57AM +0100, emery@...261... wrote:
I'm still not that interested in Guix, but, if someone could port the Guile/Guix evaluator, I would help hook it to my store and builder components. What would be really helpful is if someone could come up with a good system of distributing binary packages that would be compatible between the two.
I don't think you'll get binary compatibility between Nix and Guix.
Emery
Jookia.
On Thu, Feb 25, 2016 at 08:15:08PM +1100, Jookia wrote:
On Thu, Feb 25, 2016 at 10:08:57AM +0100, emery@...261... wrote:
I'm still not that interested in Guix, but, if someone could port the Guile/Guix evaluator, I would help hook it to my store and builder components. What would be really helpful is if someone could come up with a good system of distributing binary packages that would be compatible between the two.
I don't think you'll get binary compatibility between Nix and Guix.
Sorry, I just meant compatible infrastructure that can serve both variants over a network.
Emery
Hello Tomasz,
After fighting with my lack of knowledge about advanced make usage I've managed to prepare a version of noux.mk for a library (without noux dependency) and build libunistring - without any source changes. I haven't tested it yet though.
What I think I understand about current Genode build system it consists of two layers:
a package manager like layer that maintains dependencies between programs/libraries
build rules for building concrete programs/libraries
Both are implemented using make. I believe (although I haven't verified it) that work of Emery to port nix to Genode is an attempt to replace the former layer with nix.
it is not as simple as that. Actually, it is far too easy to mix things up. For a structured discussion, I propose to keep different topics cleanly separated:
1. Integration of 3rd-party code with Genode's genuine code (Genode's ports mechanism) 2. Building Genode components (the Genode build system) 3. Introducing the notion of "source code packages" (does not exist yet). A source-code package may be structurally similar to what a Genode source repository is today, but it would contain 3rd-party source code also. 4. Introducing the notion of "binary packages" (does not exist yet). A binary package may be similar to what a build directory is today, but specific for a certain source-code package and its dependencies. 5. Building binary packages out of source packages 6. Deployment of binary packages on top of Genode system 7. Distribution of source and binary packages 8. Updating packages
Apparently, Emery's Nix line of work is primarily concerned about the last three points. The discussion about the GNU build system vs. the Genode build system is just a detail of point 5. Of course, it is possible to wrap the GNU build system as I did with the (arguably ugly) noux.mk wrapper. But for low-level libraries in particular, it is sensible to realize their build procedure directly with the Genode build system to benefit from Genode's inner- and inter-library dependency tracking and to accommodate work flows where no "packages" are used, like the run-script-based work flow we use today.
Until May, I'll try to develop a tangible notion of what we understand as a "source package". Once this is in place, we can move on with the points 6 to 8. Does this make sense to you? ;-)
Btw, it might be insightful to revisit the previous discussion on the subject at our issue tracker:
https://github.com/genodelabs/genode/issues/1082
I think that using guix and guile could be a better approach due to using general purpose language (although not liked too much :-) ) and Genode specific requirements, which could be harder to achieve in a language dedicated specifically to build packages. But I have to check it myself first. If I'm able to have a proof of concept and know more about Emery's work I'll try to make some conclusions share them with you here.
libunistring that I compiled is a dependency for guile and compiling it is first step in my attempt to check if guix can become package manager for Genode.
Interesting. Personally, I do not think that there needs to be one single way of package management for Genode. What I like about Emery's work is that he managed to keep even the existence of the Nix mechanism hidden from the "user". The more we experiment, the better our understanding of the problem will get. So your initiative is very much appreciated.
Cheers Norman
-
unknown＠example.com -
Jookia -
Josef Söntgen -
Martin Stein -
Nobody III -
Norman Feske -
Tomasz Gajewski