Hello,
after an exhaustive evaluation of different scenarios for testing the coherent execution of components on distinct cores, the following question arises:
Is it basically possible to separate a component running on core 1 completely (e.g. pd, ram, rom, log) from the remaining system running on core 0?
Our results so far demonstrate that, in any case, a component running on core 1 utilizes at least the rom and log services provided by init which is running on core 0. Also the usage of a second level init component (which is suggested by the genode book) didn't change the resulting behavior.
Of course, genode is not meant to be a separation kernel by design, but would it still be possible to map/assign resources/partitions to corresponding cores? Or do all components running on different cores still have to contact the genode CORE component which is running on core 0?
Best regards Daniel
Hi Daniel,
Our results so far demonstrate that, in any case, a component running on core 1 utilizes at least the rom and log services provided by init which is running on core 0. Also the usage of a second level init component (which is suggested by the genode book) didn't change the resulting behavior.
this observation is correct. All session interfaces provided by core are served by a single thread running on the boot CPU. Also the init instance spawned by core is started on the same CPU. There are two reasons for this way of operation.
First, however complex your Genode system is, the ultimate allocation and arbitration of physical resources must happen - and be synchronized across CPUs - at a central point. Whether the synchronization happens via multiple threads (running on a different CPU each) contending for a lock or by serializing RPC calls to a single thread, cross-CPU synchronization is inevitable. We picked the latter path because it is simpler.
Second, core has no prior knowledge about the demands of the user land and does not even try to be clever (e.g., it does not balance threads among CPUs automatically). The first point where such policy enters the picture is the configuration of init. Hence, a secondary init can be moved to a different CPU by the means of the configuration of the primary init, but the primary init instance is bound to the boot CPU.
Of course, genode is not meant to be a separation kernel by design, but would it still be possible to map/assign resources/partitions to corresponding cores? Or do all components running on different cores still have to contact the genode CORE component which is running on core 0?
Should multi-core scalability become a concern (our today's workloads clearly don't suffer from the current design), I see three principle ways for improvement.
1. Combining the notions of a low-count SMP Genode system and a "distributed" Genode system (in multikernel fashion). In this picture, there would be multiple loosely-coupled Genode systems running on the same machine, using only a few coarsely shared resources. Each of these Genode systems would control multiple CPUs that are close to each other according to the physical CPU topology. So the current way of inter-CPU synchronization can work well within each of these systems.
2. Core could create an entrypoint for each physical CPU. For certain core services (PD, IRQ), it could take the client-provided session- affinity information into account to create the respective session object on the same CPU as the client. So RPC calls could be served independently. As noted above, the entrypoints must still be synchronized with each other. For IRQ sessions, this could generally be beneficial since each IRQ session is fairly free-standing. On the other hand, kernels like base-hw and NOVA don't actually need any interplay between the user-level device driver and core for servicing interrupts.
3. Should we observe a high contention on a specific core service, let's say the RAM service, one could implement "cached" versions of this service as a separate component. This could be a RAM server that initially allocates dataspaces from core and hands them out to its clients. The physical RAM could be partitioned among several of such cache services - each running on a different CPU. This could work also well for the ROM service.
That said, there are currently no concrete plans to pursue any of those ideas.
Cheers Norman
On Genode, how do we ensure that CPU cores are being used efficiently? What mechanisms are in place for thread balancing?
On Thu, Mar 8, 2018 at 2:14 AM, Norman Feske <norman.feske@...1...> wrote:
Hi Daniel,
Our results so far demonstrate that, in any case, a component running on core 1 utilizes at least the rom and log services provided by init which is running on core 0. Also the usage of a second level init component (which is suggested by the genode book) didn't change the resulting behavior.
this observation is correct. All session interfaces provided by core are served by a single thread running on the boot CPU. Also the init instance spawned by core is started on the same CPU. There are two reasons for this way of operation.
First, however complex your Genode system is, the ultimate allocation and arbitration of physical resources must happen - and be synchronized across CPUs - at a central point. Whether the synchronization happens via multiple threads (running on a different CPU each) contending for a lock or by serializing RPC calls to a single thread, cross-CPU synchronization is inevitable. We picked the latter path because it is simpler.
Second, core has no prior knowledge about the demands of the user land and does not even try to be clever (e.g., it does not balance threads among CPUs automatically). The first point where such policy enters the picture is the configuration of init. Hence, a secondary init can be moved to a different CPU by the means of the configuration of the primary init, but the primary init instance is bound to the boot CPU.
Of course, genode is not meant to be a separation kernel by design, but would it still be possible to map/assign resources/partitions to corresponding cores? Or do all components running on different cores still have to contact the genode CORE component which is running on core
0?
Should multi-core scalability become a concern (our today's workloads clearly don't suffer from the current design), I see three principle ways for improvement.
Combining the notions of a low-count SMP Genode system and a "distributed" Genode system (in multikernel fashion). In this picture, there would be multiple loosely-coupled Genode systems running on the same machine, using only a few coarsely shared resources. Each of these Genode systems would control multiple CPUs that are close to each other according to the physical CPU topology. So the current way of inter-CPU synchronization can work well within each of these systems.
Core could create an entrypoint for each physical CPU. For certain core services (PD, IRQ), it could take the client-provided session- affinity information into account to create the respective session object on the same CPU as the client. So RPC calls could be served independently. As noted above, the entrypoints must still be synchronized with each other. For IRQ sessions, this could generally be beneficial since each IRQ session is fairly free-standing. On the other hand, kernels like base-hw and NOVA don't actually need any interplay between the user-level device driver and core for servicing interrupts.
Should we observe a high contention on a specific core service, let's say the RAM service, one could implement "cached" versions of this service as a separate component. This could be a RAM server that initially allocates dataspaces from core and hands them out to its clients. The physical RAM could be partitioned among several of such cache services - each running on a different CPU. This could work also well for the ROM service.
That said, there are currently no concrete plans to pursue any of those ideas.
Cheers Norman
-- Dr.-Ing. Norman Feske Genode Labs
https://www.genode-labs.com · https://genode.org
Genode Labs GmbH · Amtsgericht Dresden · HRB 28424 · Sitz Dresden Geschäftsführer: Dr.-Ing. Norman Feske, Christian Helmuth
Check out the vibrant tech community on one of the world's most engaging tech sites, Slashdot.org! http://sdm.link/slashdot _______________________________________________ genode-main mailing list genode-main@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/genode-main
Hello Ben,
On 08.03.2018 20:23, Nobody III wrote:
On Genode, how do we ensure that CPU cores are being used efficiently?
it is not ensured.
What mechanisms are in place for thread balancing?
None. (well, except on base-linux where Linux performs the balancing)
The general idea is that the balancing is not a concern for the base system but should be solved at a higher level. E.g., when assigning multiple CPUs to a VM, the guest OS will do the balancing. In a similar way, noux could in principle spawn processes on different CPUs while monitoring CPU usage (e.g., via core's trace service).
Cheers Norman
It seems like we ought to have a balancer component or at least basic balancing built into init. A lack of balancing is a huge performance issue on modern SMP CPUs. How could we go about this? Could a server do the balancing through the CPU session interface?
On Tue, Mar 13, 2018 at 3:40 AM, Norman Feske <norman.feske@...1...> wrote:
Hello Ben,
On 08.03.2018 20:23, Nobody III wrote:
On Genode, how do we ensure that CPU cores are being used efficiently?
it is not ensured.
What mechanisms are in place for thread balancing?
None. (well, except on base-linux where Linux performs the balancing)
The general idea is that the balancing is not a concern for the base system but should be solved at a higher level. E.g., when assigning multiple CPUs to a VM, the guest OS will do the balancing. In a similar way, noux could in principle spawn processes on different CPUs while monitoring CPU usage (e.g., via core's trace service).
Cheers Norman
-- Dr.-Ing. Norman Feske Genode Labs
https://www.genode-labs.com · https://genode.org
Genode Labs GmbH · Amtsgericht Dresden · HRB 28424 · Sitz Dresden Geschäftsführer: Dr.-Ing. Norman Feske, Christian Helmuth
Check out the vibrant tech community on one of the world's most engaging tech sites, Slashdot.org! http://sdm.link/slashdot _______________________________________________ genode-main mailing list genode-main@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/genode-main
CPU balancing is pretty easy to manage at a higher level. For example, the Nim runtime balances additional Genode threads automatically, because the common high-level case is to create a thread pool for each CPU and push jobs to it. In other cases I assume that a thread will be blocked or block an external component, and running components on multiple CPUs only comes with the performance cost of communicating across multiple CPUs.
https://github.com/nim-lang/Nim/blob/55a5dcf8a47a47028707c46491072c438eb2f6c...
On Tue, 13 Mar 2018 05:19:46 -0600 Nobody III <hungryninja101@...9...> wrote:
It seems like we ought to have a balancer component or at least basic balancing built into init. A lack of balancing is a huge performance issue on modern SMP CPUs. How could we go about this? Could a server do the balancing through the CPU session interface?
Right now, doesn't Genode just put everything on CPU 0 unless told otherwise? It seems best for an available CPU to run the next thread that has that CPU in its affinity space. How can that policy (or something similarly efficient) be implemented?
On Wed, Mar 14, 2018 at 4:12 AM, Emery Hemingway <ehmry@...548...> wrote:
CPU balancing is pretty easy to manage at a higher level. For example, the Nim runtime balances additional Genode threads automatically, because the common high-level case is to create a thread pool for each CPU and push jobs to it. In other cases I assume that a thread will be blocked or block an external component, and running components on multiple CPUs only comes with the performance cost of communicating across multiple CPUs.
https://github.com/nim-lang/Nim/blob/55a5dcf8a47a47028707c46491072c 438eb2f6ce/lib/system/threads.nim#L570
On Tue, 13 Mar 2018 05:19:46 -0600 Nobody III <hungryninja101@...9...> wrote:
It seems like we ought to have a balancer component or at least basic balancing built into init. A lack of balancing is a huge performance issue on modern SMP CPUs. How could we go about this? Could a server do the balancing through the CPU session interface?
Check out the vibrant tech community on one of the world's most engaging tech sites, Slashdot.org! http://sdm.link/slashdot _______________________________________________ genode-main mailing list genode-main@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/genode-main
-
Daniel Krefft -
Emery Hemingway -
Nobody III -
Norman Feske