Hello Genodians
Our continuous integration pipeline is split into several stages which correspond with the Genode depot tools. The artifacts created by the depot tool are passed to the subsequent stages:
- "Extract"-stage: extract all API/source archives.
- "Build"-stage: build binary archives from the source archives extracted during the previous stage. For each architecture (arm_v7a, arm_v8a, x86_64, etc.) a dedicated build job is triggered. These jobs run in parallel and possibly on different hardware.
- "Run"-stage: execute the run scripts. Everything is "imported from depot" so that nothing needs to be compiled during this stage.
- "Publish"-stage: publish API/source/binary/package archives
For the "Build"-stage, we introduced meta packages like "pkg/gapfruit_ci-x86_64", "pkg/gapfruit_ci-arm_v7a", etc. that include every archive we want to build for that architecture.
These meta-packages are also used by the "Extract"-stage, so that for example `tool/depot/extract rite/pkg/gapfruit_ci-x86_64 rite/pkg/gapfruit_ci-arm_v7a` can be called.
Now that the number of archives became too large and `depot/extract` complains: `make: execvp: bash: Argument list too long`.
Apparently, the aggregated argument list passed to `make` contains every archive including all dependencies. Duplicates (packages and dependencies that are build for x86_64 as well as arm_v7a, etc.) are not removed.
Is this currently a limitation of the `tool/depot/extract`? Before I try my luck to tackle it myself, I'd appreciate every hint how to do so.
Or am I using `depot/extract` wrong? Do you see an alternative regarding the use case described above?
Thanks Roman
PS, I disabled the dependency check in `tool/depot/mk/dependencies.inc` completely. See the discussion [1,2] on the Genode mailing list.
[1] https://lists.genode.org/pipermail/users/2018-May/006079.html [2] https://lists.genode.org/pipermail/users/2018-June/006085.html
On 07.06.21 17:00, Roman Iten wrote:
Now that the number of archives became too large and `depot/extract` complains: `make: execvp: bash: Argument list too long`.
Apparently, the aggregated argument list passed to `make` contains every archive including all dependencies. Duplicates (packages and dependencies that are build for x86_64 as well as arm_v7a, etc.) are not removed.
After some more debugging, I understand now what the actual problem is:
The arguments for the `make`-command assembled by the rule `checked_ports_exists` can get quite long for many archives even though it doesn't contain any duplicates. As a workaround for the CI pipeline, I can just disable this check. Do you think it's worth creating a proper fix that works without passing all input as command line parameter?
Cheers, Roman
Hello Roman,
On Mon, Jun 07, 2021 at 21:48:31 CEST, Roman Iten wrote:
After some more debugging, I understand now what the actual problem is:
The arguments for the `make`-command assembled by the rule `checked_ports_exists` can get quite long for many archives even though it doesn't contain any duplicates. As a workaround for the CI pipeline, I can just disable this check. Do you think it's worth creating a proper fix that works without passing all input as command line parameter?
I did not encounter this issue before despite I use meta pkgs sometimes. Does this happen only with PREPARE_PORTS=1?
In my opinion a proper fix that prevents exceeding command line arguments is much appreciated and could serve as valuable blueprint in other places too. It sounds like the issue needs a mechanism similar to the xargs tool.
Regards
Gesendet: Dienstag, 08. Juni 2021 um 09:29 Uhr Von: "Christian Helmuth" christian.helmuth@genode-labs.com An: users@lists.genode.org Betreff: Re: depot/extract: make: execvp: bash: Argument list too long
Hello Roman,
On Mon, Jun 07, 2021 at 21:48:31 CEST, Roman Iten wrote:
After some more debugging, I understand now what the actual problem is:
The arguments for the `make`-command assembled by the rule `checked_ports_exists` can get quite long for many archives even though it doesn't contain any duplicates. As a workaround for the CI pipeline, I can just disable this check. Do you think it's worth creating a proper fix that works without passing all input as command line parameter?
I did not encounter this issue before despite I use meta pkgs sometimes. Does this happen only with PREPARE_PORTS=1?
In my opinion a proper fix that prevents exceeding command line arguments is much appreciated and could serve as valuable blueprint in other places too. It sounds like the issue needs a mechanism similar to the xargs tool.
How about using exactly that, `xargs`? It should be easy to do some $(for ...) in make to put the arguments in some temporary file, and xargs -1 can extract them line by line. Or we could develop some patch for xargs that allows the default behavior (split by blank until maximum command length) to prefer(!) to terminate the command line at the last newline over the last blank. Another good trick would be to prepare the temporary file to be a library in the format supported by ar. Make can update the members indepently and `grep` can extract all commands as needed or `xargs ar p` can extract exactly the parts that are defined beforehand to not overflow. That definition is then probably a manual task.
Regards
Christian Helmuth Genode Labs
https://www.genode-labs.com/ · https://genode.org/ https://twitter.com/GenodeLabs · /ˈdʒiː.nəʊd/
Genode Labs GmbH · Amtsgericht Dresden · HRB 28424 · Sitz Dresden Geschäftsführer: Dr.-Ing. Norman Feske, Christian Helmuth
Genode users mailing list users@lists.genode.org https://lists.genode.org/listinfo/users
This will probably be of little to no help (or maybe as inspiration ?) since we're talking gmake here, but I'll mention in passing that in other build tools there is sometimes support for a "piecemeal" modifier for compilation units : this modifier instructs the build tool to repeatedly invoke the compilation/build unit with a subset of parameters small enough to fit the OS buffer size (e.g. 2048 bytes) : https://duckduckgo.com/?q=jam+redux+%22piecemeal%22&t=ffab&ia=web
Cedric
De : Uwe geno.de@public-files.de Sujet : Aw: Re: depot/extract: make: execvp: bash: Argument list too long
In my opinion a proper fix that prevents exceeding command line arguments is much appreciated and could serve as valuable blueprint in other places too. It sounds like the issue needs a mechanism similar to the xargs tool.
How about using exactly that, `xargs`? It should be easy to do some $(for ...) in make to put the arguments in some temporary file, and xargs -1 can extract them line by line. Or we could develop some patch for xargs that allows the
Hello Christian
On Mon, Jun 07, 2021 at 21:48:31 CEST, Roman Iten wrote:
The arguments for the `make`-command assembled by the rule `checked_ports_exists` can get quite long for many archives even though it doesn't contain any duplicates.
More precisely, the culprit is the generation of `gen_missing_ports.mk`, see [1].
On 08.06.21 09:29, Christian Helmuth wrote:
I did not encounter this issue before despite I use meta pkgs sometimes. Does this happen only with PREPARE_PORTS=1?
No, it happens in both cases. As far as I understand, the check for missing ports doesn't depend on this variable - which is good because otherwise, I guess there will be hard to decipher error messages if a required port isn't prepared.
In my opinion a proper fix that prevents exceeding command line arguments is much appreciated and could serve as valuable blueprint in other places too. It sounds like the issue needs a mechanism similar to the xargs tool.
I'm not very experienced with make, and writing makefiles using make is way out of my "comfort zone" ;) Although conceptually it suffers from the same flaw, this workaround [2] works for our use case for now.
Thanks for everyone's advice, Roman
[1] https://github.com/genodelabs/genode/blob/c0150f97e524bdb90ad455ac0837c400e8...
[2] https://github.com/rite/genode/commit/1753574d3d0484520688872b84f4e241b2965c...
-
Christian Helmuth -
Roman Iten -
ttcoder＠netcourrier.com -
Uwe