I sent this last year, but nobody responded, and I still need an answer.
The RAM preservation feature in init is very nice; I used to have issues with init running out of memory. However, the RAM reservation applies after first reserving the RAM for all of the child processes based on their quotas. That may be fine in most cases, but it doesn't work properly when you give a child process a huge RAM quota to ensure that it gets all of the available RAM. In that situation, the RAM preservation happens after all of the available RAM has been reserved for the child processes, leaving no RAM available for init to preserve for itself. Something needs to change here.
Changing the ordering is the simplest solution, but there may be a better solution. When we reserve all the remaining RAM for a particular process, we supply a huge RAM quota that is much larger than the process actually needs. It seems that the solution would be to split the quota numbers into a minimum (amount *reserved*) and a maximum (amount *available*). It would work very nicely for standard desktop workloads, especially when web browsers are involved. For example, I'd like to ensure that chromium has access to up to 2G of RAM, if it is available, but I don't want to reserve 2G of RAM specifically for chromium.
Would this solution work well? Or is there something that I'm missing here?
Hi Ben,
sorry that your previous posting remained unanswered.
The RAM preservation feature in init is very nice; I used to have issues with init running out of memory. However, the RAM reservation applies after first reserving the RAM for all of the child processes based on their quotas. That may be fine in most cases, but it doesn't work properly when you give a child process a huge RAM quota to ensure that it gets all of the available RAM. In that situation, the RAM preservation happens after all of the available RAM has been reserved for the child processes, leaving no RAM available for init to preserve for itself. Something needs to change here.
Thanks for raising this issue. Just to make sure it does not go unresolved, would you mind to open an issue at GitHub and possibly give instructions how to reproduce it?
Changing the ordering is the simplest solution, but there may be a better solution. When we reserve all the remaining RAM for a particular process, we supply a huge RAM quota that is much larger than the process actually needs. It seems that the solution would be to split the quota numbers into a minimum (amount *reserved*) and a maximum (amount *available*). It would work very nicely for standard desktop workloads, especially when web browsers are involved. For example, I'd like to ensure that chromium has access to up to 2G of RAM, if it is available, but I don't want to reserve 2G of RAM specifically for chromium.
Would this solution work well? Or is there something that I'm missing here?
I am not convinced of this approach. Even though I can relate to the described scenario, to me, the two statements sound like a contradiction:
1. Ensure that chromium has access to up to 2G of RAM 2. Don't want to reserve 2G of RAM specifically for chromium
The proper way to deal with the situation you described would be to dynamically adjust the quota values for chromium (a "desktop manager" could do that automatically). Init supports the dynamic change of RAM quotas in the following ways:
* If the quota of a start node is increased, init transfers RAM quota from init to the component.
* If the quota is decreased, init tries to transfer RAM quota from the component to init. If this is not possible (because the component's PD has not enough unused RAM quota), init sends a 'yield' request to the component. A nice-behaving component could respond to such a request by freeing up the requested amount of RAM.
With this way, we can leave out the policy (like a softer notion of quotas) outside of init.
Norman
Hmm. I still strongly prefer my original idea. In particular, I don't want to run out of RAM quota with hundreds of MB of physical RAM unused. Can you help me understand your reasoning here?
What is the status of how init handles quotas? Specifically, can a component go above its quota if init has more RAM available?
What happens when a component runs out of RAM quota? How much RAM quota is required to start a process?
How are you imagining such a "desktop manager" would work? In particular, how would it know when a process needs more RAM (or CAPs)?
Is there a way to know how much RAM (and CAP) quota init has left?
On Feb 16, 2018 2:01 AM, "Norman Feske" <norman.feske@...1...> wrote:
Hi Ben,
sorry that your previous posting remained unanswered.
The RAM preservation feature in init is very nice; I used to have issues with init running out of memory. However, the RAM reservation applies after first reserving the RAM for all of the child processes based on their quotas. That may be fine in most cases, but it doesn't work properly when you give a child process a huge RAM quota to ensure that it gets all of the available RAM. In that situation, the RAM preservation happens after all of the available RAM has been reserved for the child processes, leaving no RAM available for init to preserve for itself. Something needs to change here.
Thanks for raising this issue. Just to make sure it does not go unresolved, would you mind to open an issue at GitHub and possibly give instructions how to reproduce it?
Changing the ordering is the simplest solution, but there may be a better solution. When we reserve all the remaining RAM for a particular process, we supply a huge RAM quota that is much larger than the process actually needs. It seems that the solution would be to split the quota numbers into a minimum (amount *reserved*) and a maximum (amount *available*). It would work very nicely for standard desktop workloads, especially when web browsers are involved. For example, I'd like to ensure that chromium has access to up to 2G of RAM, if it is available, but I don't want to reserve 2G of RAM specifically for
chromium.
Would this solution work well? Or is there something that I'm missing
here?
I am not convinced of this approach. Even though I can relate to the described scenario, to me, the two statements sound like a contradiction:
1. Ensure that chromium has access to up to 2G of RAM 2. Don't want to reserve 2G of RAM specifically for chromium
The proper way to deal with the situation you described would be to dynamically adjust the quota values for chromium (a "desktop manager" could do that automatically). Init supports the dynamic change of RAM quotas in the following ways:
* If the quota of a start node is increased, init transfers RAM quota from init to the component.
* If the quota is decreased, init tries to transfer RAM quota from the component to init. If this is not possible (because the component's PD has not enough unused RAM quota), init sends a 'yield' request to the component. A nice-behaving component could respond to such a request by freeing up the requested amount of RAM.
With this way, we can leave out the policy (like a softer notion of quotas) outside of init.
Norman
-- Dr.-Ing. Norman Feske Genode Labs
https://www.genode-labs.com · https://genode.org
Genode Labs GmbH · Amtsgericht Dresden · HRB 28424 · Sitz Dresden Geschäftsführer: Dr.-Ing. Norman Feske, Christian Helmuth
------------------------------------------------------------ ------------------ Check out the vibrant tech community on one of the world's most engaging tech sites, Slashdot.org! http://sdm.link/slashdot _______________________________________________ genode-main mailing list genode-main@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/genode-main
Hi Ben,
On 16.02.2018 11:18, Nobody III wrote:
Hmm. I still strongly prefer my original idea. In particular, I don't want to run out of RAM quota with hundreds of MB of physical RAM unused. Can you help me understand your reasoning here?
Instead of adding heuristics (of handing out unused memory) to init, the policy can be implemented outside of init by the one who controls init's configuration.
The idea is that init is accompanied by another "manager" component that generates init's configuration and consumes init's status reports. There is a feedback loop between init and the manager component. This puts the manager component in the natural position to balance the quotas of init's children, and control the lifetimes of the children. However your quota policy may look like, you may replace/customize the manager component but leave init unchanged.
In Sculpt EA, the user plays the role of the "manager" component. But this role can be played by a program, too.
What is the status of how init handles quotas? Specifically, can a component go above its quota if init has more RAM available?
No. But init does reflect this situation in its state report. So a manager component can respond to such a condition by increasing the quota in init's config.
What happens when a component runs out of RAM quota?
If the component takes no special precautions, it would block until its RAM quota gets upgraded.
How much RAM quota is required to start a process?
This depends on the component, e.g., the binary size, BSS size, shared libraries.
Is there a way to know how much RAM (and CAP) quota init has left?
In init's reports, you can see how much RAM quota each child uses.
Cheers Norman
Thanks. How about CAP quotas? What happens when a component runs out of CAPs? And init doesn't seem to indicate the number or CAPs each child is using. Shouldn't CAPs be added to the report?
Also, how can a process determine how much RAM quota is required to start a process? Is there a good heuristic at least, perhaps in noux?
On Fri, Feb 16, 2018 at 5:17 AM, Norman Feske <norman.feske@...1...> wrote:
Hi Ben,
On 16.02.2018 11:18, Nobody III wrote:
Hmm. I still strongly prefer my original idea. In particular, I don't want to run out of RAM quota with hundreds of MB of physical RAM unused. Can you help me understand your reasoning here?
Instead of adding heuristics (of handing out unused memory) to init, the policy can be implemented outside of init by the one who controls init's configuration.
The idea is that init is accompanied by another "manager" component that generates init's configuration and consumes init's status reports. There is a feedback loop between init and the manager component. This puts the manager component in the natural position to balance the quotas of init's children, and control the lifetimes of the children. However your quota policy may look like, you may replace/customize the manager component but leave init unchanged.
In Sculpt EA, the user plays the role of the "manager" component. But this role can be played by a program, too.
What is the status of how init handles quotas? Specifically, can a component go above its quota if init has more RAM available?
No. But init does reflect this situation in its state report. So a manager component can respond to such a condition by increasing the quota in init's config.
What happens when a component runs out of RAM quota?
If the component takes no special precautions, it would block until its RAM quota gets upgraded.
How much RAM quota is required to start a process?
This depends on the component, e.g., the binary size, BSS size, shared libraries.
Is there a way to know how much RAM (and CAP) quota init has left?
In init's reports, you can see how much RAM quota each child uses.
Cheers Norman
-- Dr.-Ing. Norman Feske Genode Labs
https://www.genode-labs.com · https://genode.org
Genode Labs GmbH · Amtsgericht Dresden · HRB 28424 · Sitz Dresden Geschäftsführer: Dr.-Ing. Norman Feske, Christian Helmuth
Check out the vibrant tech community on one of the world's most engaging tech sites, Slashdot.org! http://sdm.link/slashdot _______________________________________________ genode-main mailing list genode-main@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/genode-main
Hi Ben,
On 11.03.2018 04:06, Nobody III wrote:
Thanks. How about CAP quotas? What happens when a component runs out of CAPs? And init doesn't seem to indicate the number or CAPs each child is using. Shouldn't CAPs be added to the report?
you can add this information to init's state report via the 'child_caps="yes"' attribute. Currently, unlike RAM, init does not support the dynamic adjustment of cap quota. This is not an inherent limitation but just not implemented.
Also, how can a process determine how much RAM quota is required to start a process? Is there a good heuristic at least, perhaps in noux?
This information should be covered by the 'runtime' file of a pkg archive. For an example, have a look at the pkg/noux-system as used by Sculpt:
https://github.com/genodelabs/genode/blob/master/repos/ports/recipes/pkg/nou...
When thinking of building blocks, we should start thinking about depot pkgs, not ELF executables. E.g., one ELF executable may be referred to by several pkgs (with different configurations). The RAM and caps demands may very well vary depending on the configuration. By introducing a pkg for each flavor, we can nicely capture this. E.g., the pkg/nic_router-nat runtime employs the NIC router in one particular flavour.
https://github.com/genodelabs/genode/blob/master/repos/os/recipes/pkg/nic_ro...
The same NIC router could be used as an ingredient of many other building blocks.
Cheers Norman
Thanks. If nobody else gets to it first, I'll probably implement dynamic cap quota adjustment in init. As for determining starting quotas, I'm wondering mostly for porting QProcess, so I won't be working with packages.
On Tue, Mar 13, 2018 at 3:55 AM, Norman Feske <norman.feske@...1...> wrote:
Hi Ben,
On 11.03.2018 04:06, Nobody III wrote:
Thanks. How about CAP quotas? What happens when a component runs out of CAPs? And init doesn't seem to indicate the number or CAPs each child is using. Shouldn't CAPs be added to the report?
you can add this information to init's state report via the 'child_caps="yes"' attribute. Currently, unlike RAM, init does not support the dynamic adjustment of cap quota. This is not an inherent limitation but just not implemented.
Also, how can a process determine how much RAM quota is required to start a process? Is there a good heuristic at least, perhaps in noux?
This information should be covered by the 'runtime' file of a pkg archive. For an example, have a look at the pkg/noux-system as used by Sculpt:
https://github.com/genodelabs/genode/blob/master/repos/ ports/recipes/pkg/noux-system/runtime
When thinking of building blocks, we should start thinking about depot pkgs, not ELF executables. E.g., one ELF executable may be referred to by several pkgs (with different configurations). The RAM and caps demands may very well vary depending on the configuration. By introducing a pkg for each flavor, we can nicely capture this. E.g., the pkg/nic_router-nat runtime employs the NIC router in one particular flavour.
https://github.com/genodelabs/genode/blob/master/repos/os/ recipes/pkg/nic_router-nat/runtime
The same NIC router could be used as an ingredient of many other building blocks.
Cheers Norman
-- Dr.-Ing. Norman Feske Genode Labs
https://www.genode-labs.com · https://genode.org
Genode Labs GmbH · Amtsgericht Dresden · HRB 28424 · Sitz Dresden Geschäftsführer: Dr.-Ing. Norman Feske, Christian Helmuth
Check out the vibrant tech community on one of the world's most engaging tech sites, Slashdot.org! http://sdm.link/slashdot _______________________________________________ genode-main mailing list genode-main@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/genode-main
-
Nobody III -
Norman Feske