Early memory allocation

[Norman Feske]

Hi Sven,

> This time, I have a question related to the allocation of memory during
> system startup. At the moment, I'm just working with qemu/
> l4ka-pistachio, and I've noticed that the initial memory allcation
> ("Trying to allocate 4K pages from sigma0") requests over 100MB, and
> this allocation takes about 3 seconds on my machine (without kqemu
> acceleration).
> 
> In core/platform.cc I saw, that you ask sigma0 for the entire amount of
> memory it has concerning this page size and that you also memset the
> returned memory to zero.
> 
> When playing around with that, I saw that omitting the zeroing does not
> seem to have any impact on the running system (I only have a dde driver
> running...).
> 
> I also saw that I may - for a start - limit the memory allocation to a
> much smaller size. This involved a much better startup performance.

at the startup, we use sigma0 as a tool to determine the physical memory
regions to initialize core's pyhsical-memory allocator with. This is
useful because Sigma0 knows which pages are RAM and which pages are
non-RAM (e.g., memory-mapped I/O). Furthermore, on Pistachio, each
physical page that is going to be mapped by core to another process must
be locally mapped within core anyway. So we initialize this mapping
right at the startup. This way, there is no further interaction with
sigma0 needed during the actual runtime.

You are right, the delay is annoying. It is caused by two things.
First, we do not request 4M pages, only 4K pages, from sigma0 because
we experienced problems with further mapping 4K pages out of a 4M page
on Pistachio and will need to investigate that. So there is far to
much interaction going on between core and sigma0. Second, as you
noticed, there is the zeroing of the pages, which, indeed, is not
needed. Before handing out RAM dataspaces to the outside of core, we
zero them out anyway. So there is no benefit to be gained from this
early initialization.

So far we have mostly worked with only 64M of memory (on Qemu). On
hardware with much memory, we experienced a further problem with
the 4K allocations as the kernel tends to run out of kernel memory
then. As a quick work-around for this case, we restricted the maximum
amount of memory via the following kickstart argument:

  kernel /pistachio/kickstart maxmem=64M

> So I wonder why the zeroing is needed, and I'd like to know if your
> memory management allowed for subsequent allocations from sigma0, so
> that I could save some startup time and allocate the remaining memory later.
> 
> (Is this question - from your point of view - worth the investigation at
> all? Or is the delay only related to my slow qemu having no impact on an
> average hardware?)

You definitly spotted an area that we would like to see improved. The
zeroing of memory can be discarded right away. However, the 4M-page
issue and the kernel-memory issue are still unresolved problems.

If you are venturesome, you may try out using 4M mappings by
removing the line

  page_size_mask &= 1 << 12;

from the 'Platform::_setup_mem_alloc' function. ;-) You will notice that
the startup delay gets much reduced. But you will certainly also notice
corruptions in Pistachio's mapping database.

Thanks for your investigations and for pointing out the superfluous
zeroing of memory!

Best regards

Norman