As long as "they" do not want to reserve (512 << 20) KiB there must be some miscalculation.
Unfortunately, they do want, as explicitly written in the comment -they even try to claim 544gb mem space initially... not real alloc, just reserve .
Anyway, system should not hang even if I request 512gb of ram, it should return with error... This is standard approach - we try to alloc until we can, receiving ENOMEM or something similar? Why it is not working?
The current mmap(MAP_ANONYMOUS) implementation seems to strive for satisfaction of the application by not probing the allocation but enforcing it. This ends up in the request for more RAM resources as you traced.
Yes, but why this call (wait for response from parent) just hangs forever instead of return and say that “I can’t”?