Hi,
I wrote a simple run script 'manytimer', based on 'timer': https://github.com/rite/genode/tree/manytimer When the number of components exceeds 19, I get a "Quota exceeded!" warning. It seems that it doesn't matter if I use timer or other components. It also doesn't matter how much RAM quota I configure per timer.
- Is this behaviour intended?
- Whose quota exceeds (init, core, ...)?
- Can I resolve the warning by increasing its quota?
- In a scenario with 19 or more components (within one init), is it still possible to "assign all remaining resources to the last child by simply specifying an overly large quantum" as described in "Genode 16.05 Foundations" (see Chapter 6.2.2 "Resource quota saturation"). Or would there be no more slack memory available for init and core respectively?
Thanks, Roman
Hi Roman,
On 04.10.2016 20:04, Roman Iten wrote:
Hi, I wrote a simple run script 'manytimer', based on 'timer': https://github.com/rite/genode/tree/manytimerWhen the number of components exceeds 19, I get a "Quota exceeded!" warning. It seems that it doesn't matter if I use timer or other components. It also doesn't matter how much RAM quota I configure per timer. - Is this behaviour intended? - Whose quota exceeds (init, core, ...)? - Can I resolve the warning by increasing its quota? - In a scenario with 19 or more components (within one init), is it still possible to "assign all remaining resources to the last child by simply specifying an overly large quantum" as described in "Genode 16.05 Foundations" (see Chapter 6.2.2 "Resource quota saturation"). Or would there be no more slack memory available for init and core respectively?
I encountered the same problem in the Turmvilla scenario. In the presence of the "overly large quantum", init transfers all remaining quota from itself to the corresponding child and preserves just a tiny bit of quota for itself. This preserved quota is needed to accommodate the few metadata allocations that cannot be easily allocated from a specific child (i.e., child-thread meta data). If the number of children becomes too large, this preservation does not suffice. But you can increase the value as done by the following commit:
https://github.com/nfeske/genode/commit/619ce3ff2c81df8e24b10a675ccc78b83ee3...
Cheers Norman
Hi Norman
Thanks for the insights. I still don't understand why init's(?) quota exceeds. I get the warning no matter of the presence of the "overly large quantum". Is this behavior intended?
- How big is the initial quota of init?
- Can I change that quota - or only the preserved slack memory?
- If not - is there another way to prevent the "Quota exceeded!" warning?
- Does the size of the metadata allocation for a child depends on whether I'm using a 32 or 64 bit system?
- Is a scenario with 19 or more components within one init considered 'large'?
Thanks, Roman
On Mit, Okt 5, 2016 at 9:05 , Norman Feske <norman.feske@...1...> wrote:
Hi Roman,
On 04.10.2016 20:04, Roman Iten wrote:
Hi, I wrote a simple run script 'manytimer', based on 'timer': https://github.com/rite/genode/tree/manytimer. When the number of components exceeds 19, I get a "Quota exceeded!" warning. It seems that it doesn't matter if I use timer or other components. It also doesn't matter how much RAM quota I configure per timer. - Is this behaviour intended? - Whose quota exceeds (init, core, ...)? - Can I resolve the warning by increasing its quota? - In a scenario with 19 or more components (within one init), is it still possible to "assign all remaining resources to the last child by simply specifying an overly large quantum" as described in "Genode 16.05 Foundations" (see Chapter 6.2.2 "Resource quota saturation"). Or would there be no more slack memory available for init and core respectively?
I encountered the same problem in the Turmvilla scenario. In the presence of the "overly large quantum", init transfers all remaining quota from itself to the corresponding child and preserves just a tiny bit of quota for itself. This preserved quota is needed to accommodate the few metadata allocations that cannot be easily allocated from a specific child (i.e., child-thread meta data). If the number of children becomes too large, this preservation does not suffice. But you can increase the value as done by the following commit:
https://github.com/nfeske/genode/commit/619ce3ff2c81df8e24b10a675ccc78b83ee3...
Cheers Norman
-- Dr.-Ing. Norman Feske Genode Labs
http://www.genode-labs.com · http://genode.org
Genode Labs GmbH · Amtsgericht Dresden · HRB 28424 · Sitz Dresden Geschäftsführer: Dr.-Ing. Norman Feske, Christian Helmuth
Check out the vibrant tech community on one of the world's most engaging tech sites, SlashDot.org! http://sdm.link/slashdot _______________________________________________ genode-main mailing list genode-main@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/genode-main
Hi Roman,
- How big is the initial quota of init?
init receives the quota for all the available physical memory from core. The amount depends on the physical memory of the machine. It is printed by core when init is started. Look for a message like this:
24 MiB RAM assigned to init
- Can I change that quota - or only the preserved slack memory?
If you are using Qemu, you may adjust the memory of the virtual machine by putting the following line (for 256 MiB) in your run script:
append qemu_args " -m 256 "
In your run script, the value is 64 MiB, which leaves merely 24 MiB (on 64 bit) for init.
- If not - is there another way to prevent the "Quota exceeded!" warning?
I looked a bit closer at your scenario. Here is what happens:
For each child started by init, init needs to create several capabilities (e.g., the parent capability presented to the child, or the capability for the local ROM session for providing the child's binary as "binary" ROM module). The allocation of those capabilities is performed via the 'Nova_native_pd::alloc_rpc_cap' RPC function. This function naturally consumes session quota of the corresponding PD session (init's PD session). At one point, the initial session quota (that was passed to core when init's PD session was created) is depleted. In this case, core prints the diagnostic message and returns an error to the client (init). Init responds to this error by upgrading the session quota of its PD session using the preserved slack memory. The session upgrading is handled at 'base-nova/src/lib/base/rpc_cap_alloc.cc'.
In your case, the message you see is merely a diagnostic message. The condition is handled properly. In cases where proper error handing of the 'Out_of_metadata' condition is missing, the message used to be quite valuable to spot the problem. So we decided to kept it.
Of course, you may opt to suppress the message by adjusting the code in 'base/include/base/allocator_guard.h'.
- Does the size of the metadata allocation for a child depends on
whether I'm using a 32 or 64 bit system?
Yes. I.e., your scenario produces the message only on 64 bit, not on 32 bit.
- Is a scenario with 19 or more components within one init considered
'large'?
The static part of the current scenarios (like the ones tested by our autopilot) typically consists of fewer components. Personally, I only hit the limit with the Turmvilla scenario.
That said, the limit turned out not to be a problem in your case. The message is a false-positive warning. The default limit becomes a problem not before the PD-session upgrade fails. I can trigger the problem with your run script when configuring Qemu with 64 MiB of memory and starting 76 children. Sorry that my previous email pointed you to the wrong direction.
Cheers Norman
Hi Norman
- How big is the initial quota of init?
init receives the quota for all the available physical memory from core. The amount depends on the physical memory of the machine. It is printed by core when init is started.
Thats what I thought. But I couldn't imagine how init's quota could possibly exceed on a x86-64 machine with several GiB physical memory :)
For each child started by init, init needs to create several capabilities (e.g., the parent capability presented to the child, or the capability for the local ROM session for providing the child's binary as "binary" ROM module). The allocation of those capabilities is performed via the 'Nova_native_pd::alloc_rpc_cap' RPC function. This function naturally consumes session quota of the corresponding PD session (init's PD session). At one point, the initial session quota (that was passed to core when init's PD session was created) is depleted. In this case, core prints the diagnostic message and returns an error to the client (init). Init responds to this error by upgrading the session quota of its PD session using the preserved slack memory. The session upgrading is handled at 'base-nova/src/lib/base/rpc_cap_alloc.cc'.
In your case, the message you see is merely a diagnostic message. The condition is handled properly. In cases where proper error handing of the 'Out_of_metadata' condition is missing, the message used to be quite valuable to spot the problem. So we decided to kept it.
How can I distinguish if the condition is handled properly or not? Are there any preceding or following log messages in either case?
- Does the size of the metadata allocation for a child depends on
whether I'm using a 32 or 64 bit system?
Yes. I.e., your scenario produces the message only on 64 bit, not on 32 bit.
Is it worth thinking about calculating the slack memory size based on Genode::addr_t? Or make the value even configurable?
That said, the limit turned out not to be a problem in your case. The message is a false-positive warning. The default limit becomes a problem not before the PD-session upgrade fails. I can trigger the problem with your run script when configuring Qemu with 64 MiB of memory and starting 76 children. Sorry that my previous email pointed you to the wrong direction.
It didn't. I'm trying to improve my understanding of memory configuration and allocation in Genode. So every hint helps ;)
Thanks, Roman
I've had problems with init failing when a component was configured to use all remaining available RAM. I've had to alter some of the standard run scripts to make them work. I don't remember which ones right now, but I think arora was one of them. Maybe init should have an XML option to set its reserved RAM quota, or it could instead automatically calculate the required quota based on the number of components it starts. Also, it would be nice to have both reserved and maximum quotas for components in init's configuration.
On Wed, Oct 5, 2016 at 8:48 AM, Roman Iten <roman.iten@...453...> wrote:
Hi Norman
- How big is the initial quota of init?
init receives the quota for all the available physical memory from core. The amount depends on the physical memory of the machine. It is printed by core when init is started.
Thats what I thought. But I couldn't imagine how init's quota could possibly exceed on a x86-64 machine with several GiB physical memory :)
For each child started by init, init needs to create several capabilities (e.g., the parent capability presented to the child, or the capability for the local ROM session for providing the child's binary as "binary" ROM module). The allocation of those capabilities is performed via the 'Nova_native_pd::alloc_rpc_cap' RPC function. This function naturally consumes session quota of the corresponding PD session (init's PD session). At one point, the initial session quota (that was passed to core when init's PD session was created) is depleted. In this case, core prints the diagnostic message and returns an error to the client (init). Init responds to this error by upgrading the session quota of its PD session using the preserved slack memory. The session upgrading is handled at 'base-nova/src/lib/base/rpc_cap_alloc.cc'.
In your case, the message you see is merely a diagnostic message. The condition is handled properly. In cases where proper error handing of the 'Out_of_metadata' condition is missing, the message used to be quite valuable to spot the problem. So we decided to kept it.
How can I distinguish if the condition is handled properly or not? Are there any preceding or following log messages in either case?
- Does the size of the metadata allocation for a child depends on whether
I'm using a 32 or 64 bit system?
Yes. I.e., your scenario produces the message only on 64 bit, not on 32 bit.
Is it worth thinking about calculating the slack memory size based on Genode::addr_t? Or make the value even configurable?
That said, the limit turned out not to be a problem in your case. The message is a false-positive warning. The default limit becomes a problem not before the PD-session upgrade fails. I can trigger the problem with your run script when configuring Qemu with 64 MiB of memory and starting 76 children. Sorry that my previous email pointed you to the wrong direction.
It didn't. I'm trying to improve my understanding of memory configuration and allocation in Genode. So every hint helps ;)
Thanks, Roman
Check out the vibrant tech community on one of the world's most engaging tech sites, SlashDot.org! http://sdm.link/slashdot _______________________________________________ genode-main mailing list genode-main@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/genode-main
Hi Ben,
thanks for chiming in!
On 05.10.2016 22:22, Nobody III wrote:
Maybe init should have an XML option to set its reserved RAM quota, or it could instead automatically calculate the required quota based on the number of components it starts.
The automatic calculation would work if init was solely used in a static way. But init can actually respond to configuration changes at runtime. Hence, the number of children may vary over the lifetime of init, which makes it impossible to calculate the value up-front.
To address the problem in a fundamental way, init should allocate all child-specific capabilities from the PD session of the respective child. However, this is not a simple change. E.g., it would require us to pass the PD session as argument to 'Entrypoint::manage' and to keep the association of each RPC object with the PD session where its capability was allocated from. Since this is a rare problem (actually only for init at this point), I am not convinced to make the API more complicated for everyone. However, I have no strong opinion yet.
In the shorter term, it would be sensible to make the preservation configurable via init's configuration instead of hard-coding the value. I plan to rework init soon and will keep this suggestion in mind.
Also, it would be nice to have both reserved and maximum quotas for components in init's configuration.
I agree. I will consider this while reworking init.
Cheers Norman
Hi
Thank you very much for this very good explanation.
On 07.10.2016 10:47, Norman Feske wrote:
In the shorter term, it would be sensible to make the preservation configurable via init's configuration instead of hard-coding the value. I plan to rework init soon and will keep this suggestion in mind.
I agree. Making the preservation configurable makes the memory reservations more transparent, i.e. everything that requires memory is visible in init's configuration - well, at least if init is used in a static way. Or are there still other memory 'pools' one might not be aware of?
Also, it would be nice to have both reserved and maximum quotas for components in init's configuration.
I agree. I will consider this while reworking init.
Great, I like the idea of having both, reserved/soft and a maximum/hard quota.
One last question: how do I calculate the required memory preservation for init on nova_x86_64, based on the number of children?
Thanks, Roman
PS: sorry for screwing up the quoting in my previous mail. Tried another mail client - I'm back to thunderbird now :)
Hi Roman,
I agree. Making the preservation configurable makes the memory reservations more transparent, i.e. everything that requires memory is visible in init's configuration - well, at least if init is used in a static way. Or are there still other memory 'pools' one might not be aware of?
on NOVA, the most important one is the kernel's memory pool, which has a fixed size that is defined in the kernel's linker script. The linker script is located in nova/src/hypervisor.ld (look for '_mempool_f').
Another limited resource is core's capability space, in particular the meta data required to manage the lifetime of capabilities. The details differ from kernel to kernel. On most base platforms, those information are kept in statically allocated arrays, which are dimensioned to accommodate the current scenarios. Core is in a special position because it has to keep track of all capabilities in the system (capabilities are allocated via core's PD service). Since the capability space of core is limited, we should apply Genode's resource-trading concept to capabilities too. In fact, I plan to implement this idea sometime next year. Until then, we have to life with the situation that capability allocations are not properly accounted (which is a potential denial-of-service issue).
One last question: how do I calculate the required memory preservation for init on nova_x86_64, based on the number of children?
I cannot give you a precise formula. My previously reported experiment where I started over 70 children with the default preservation of 128 KiB suggests that 2 KiB per child should suffice. Make it 16 KiB per child and you should be really fine. ;-)
Cheers Norman
Hi Roman,
How can I distinguish if the condition is handled properly or not? Are there any preceding or following log messages in either case?
I remember the following situations where the messages were somehow useful to diagnose a problem:
- If the scenario got stuck with the this being the last message. This happened in the phase when we introduced the dynamic resource handling originally (around version 13.11). In the meanwhile, we have equipped all places where clients may encounter the out-of- metadata condition with corresponding handlers (i.e., in the form of the 'base/src/include/base/internal/expanding_*_client.h' classes). Therefore, I haven't observed this effect for a long time now.
- If a component aborts right after a quota-exceeded message, this is a strong hint that an exception handler is missing. An example is the issue recently reported by Denis (and fixed with [1]) where core missed to reflect an 'Allocator::Out_of_memory' condition as an 'Rm_session::Out_of_metadata' exception. That said, in this case, the message does not add too much information because the abort message already prints the exception type, which should suffice to diagnose the problem.
- If an unusual amount of quota-exceeded messages appears, this hints at a resource leak or at least an opportunity for optimization. The initial session quotas and the quota upgrades are dimensioned such that the quota updates are needed only sporadically. If the log is flooded by quota-exceeded messages, a superficial number of quota upgrades is performed, which should better be investigated.
The more I think about it, the better I like the idea to remove the message. In the log, it gives a wrong impression of presumed problems where none exists because the quota upgrading is a normal part of life of Genode components. The diagnostic value added by the messages is indeed quite low.
[1] https://github.com/genodelabs/genode/commit/fac69eff48b3d70a9d37ebc5d3ae6746...
- Does the size of the metadata allocation for a child depends on whether I'm using a 32 or 64 bit system?Yes. I.e., your scenario produces the message only on 64 bit, not on 32 bit.
Is it worth thinking about calculating the slack memory size based on Genode::addr_t? Or make the value even configurable?
Making the value configurable seems to be the most sensible idea. I will respond to Ben's (Nobody III) posting with an explanation why.
Cheers Norman
-
Nobody III -
Norman Feske -
Roman Iten