Hi, I am using Genode with the seL4 backend for my research and have questions about some of the core's services.
Address Space: --------------------
1. Referring to section 3.4.4 from the Genode foundation's book. I am not sure I follow how using a different RM for the stack segment helps us provide buffers in between stacks of different threads in a PD.
2. A PD is defined as Address Space and CSpace. I wondered if it is possible to have multiple address spaces inside a PD. The RM for each of these different Address spaces would still be separate capabilities in the singular CSpace for that PD. This way, a thread can potentially jump across different address spaces like [1, 2]. Or multiple threads can have slightly different address spaces.
3. Where am I going with this? I am trying to implement a new type of thread. In this thread(let's call it thread-stack), each thread's stack will be completely isolated from the other threads. That is, threads will not have access to each other's stacks. They will still share the code-section, library, and heap sections. This is a toy idea, so I can get my hand dirty with the Genode basics. I think everything I need to implement is already there in Genode but wanted to get your opinion.
4. Are they any helper functions to clone an entire address space? Replicate the page table hierarchy with the same underlying page frames.
Scheduling: ---------------
Since scheduling is done inside the kernel in seL4, I think we might be limited to the options exposed by the kernel, but I still wanted to run it by you guys to see if I am missing anything.
1. Using the notion of Affinity, I can restrict a thread to run on a particular CPU, but is there a way to restrict other threads from running on a given CPU. I am exploring isolating two threads by ensuring that they run on separate cores. 2. Is there a way to restrict the time slices given to a particular thread. I am thinking of something like cgroups. I see that seL4 has the MCS kernel, which shares some of the motivations. But I think Genode does not support the MCS sel4 kernel.
Debugging Capability Spaces: ---------------------------------------
1. Is the CSpace for a PD the same as CSpace in seL4, or is there a different notion of Cspace in Genode? 2. Is there an easy way to print the entire CSpace for debugging purposes?
Thanks a lot.
Sid sid-agrawal.ca
[1] Light-Weight Contexts: An OS Abstraction for Safety and Performance | USENIX [2] SpaceJMP: Programming with Multiple Virtual Address Space | ACM SIGPLAN
Hello Sid,
thanks for your interest in Genode and welcome to the mailing list!
- Referring to section 3.4.4 from the Genode foundation's book. I am not
sure I follow how using a different RM for the stack segment helps us provide buffers in between stacks of different threads in a PD.
When attaching a dataspace to a region map without telling a specific address, core picks a suitable free address range using a best-fit allocation. When attaching dataspaces to a PD's regular region map, the region may be placed adjacent to an existing region, which is normally fine. However, in the case of stacks, we want do enforce two things:
- A certain alignment (1 MiB) of the stack's region so that the corresponding thread can infer its TLS pointer from its stack address using a mere bit-logical operation (alleviating the need for a special register holding a TLS pointer).
- Leaving guard pages around the stacks.
Both constraints can be satisfied by managing the region allocation within a dedicated stack-region map manually.
- A PD is defined as Address Space and CSpace. I wondered if it is
possible to have multiple address spaces inside a PD. The RM for each of these different Address spaces would still be separate capabilities in the singular CSpace for that PD. This way, a thread can potentially jump across different address spaces like [1, 2]. Or multiple threads can have slightly different address spaces.
Early versions of Genode indeed decoupled address spaces from PDs. However, as we never found reasonable use cases of this presumed flexibility, we abandoned this artificial separation of mechanisms. As a result, the implementation became less complex and thereby more robust.
[1] https://genode.org/documentation/release-notes/16.05#Consolidation_of_core_s...
BTW, it is still possible to share a portion of the virtual address space between multiple PDs by using core's RM service. E.g., the cached_fs_rom component uses this mechanism to make files available as read-only dataspaces shared by multiple clients.
Regarding the idea of a thread "jumping across address spaces", that is really what's happening when issuing an RPC. Genode's RPC mechanism mimics plain synchronous function calls.
- Where am I going with this? I am trying to implement a new type of
thread. In this thread(let's call it thread-stack), each thread's stack will be completely isolated from the other threads. That is, threads will not have access to each other's stacks. They will still share the code-section, library, and heap sections. This is a toy idea, so I can get my hand dirty with the Genode basics. I think everything I need to implement is already there in Genode but wanted to get your opinion.
Genode's mechanisms allow you to build that scenario. Just a rough sketch:
A custom parent component (runtime) would create a new PD for each "thread". But instead of loading the real program into the PD, it would load a simple bootstrapping program. Once executed, this bootstrapping program would request ROM and RAM dataspaces containing the text and data segments of the designated program from the parent. The parent is free to hand out the same dataspaces for each child.
What I just described is actually very similar to the regular function of our dynamic linker.
- Are they any helper functions to clone an entire address space?
Replicate the page table hierarchy with the same underlying page frames.
Not an entire address space. But a region map can be used as a dataspace. On kernels other than Linux, this - so called managed dataspace" - can be shared between PDs. It can also be used to implement on-demand page-fault handling. Think of automatically growing threads or page swapping. You can find an example at [2] However, in practice the mechanism is not used much.
[2] https://github.com/genodelabs/genode/blob/master/repos/base/src/test/rm_nest...
- Using the notion of Affinity, I can restrict a thread to run on a
particular CPU, but is there a way to restrict other threads from running on a given CPU. I am exploring isolating two threads by ensuring that they run on separate cores.
It's best to think in terms of PDs when partitioning resources. At the granularity of PDs, you can express the assignment of two PDs to a mutually exclusive set of CPU cores using init's configuration: defining an affinity space of 2x1, assigning the "left" part to one PD, and the "right" part to the other. Note however, that the boot CPU is effectively shared with other components and the execution of core's services. So to minimize the chance for cross talk, one would need to use a CPU with more than two cores (one for the boot CPU and one for each resource partition).
BTW, you can readily experiment with the mechanism using the Genode-based Sculpt OS as described at [3].
[3] http://genodians.org/nfeske/2021-03-24-sculpt-os
- Is there a way to restrict the time slices given to a particular thread.
I am thinking of something like cgroups. I see that seL4 has the MCS kernel, which shares some of the motivations. But I think Genode does not support the MCS sel4 kernel.
We haven't used seL4's MCS kernel, yet.
That said, our custom base-hw microkernel supports a similar scheduling feature since version 14.11 [4].
[4] https://genode.org/documentation/release-notes/14.11#Trading_CPU_time_betwee...
- Is the CSpace for a PD the same as CSpace in seL4, or is there a
different notion of Cspace in Genode?
It is the same notion. It is only used in the seL4-specific part of Genode (in base-sel4/).
- Is there an easy way to print the entire CSpace for debugging purposes?
Unfortunately not. However, while working with the code, you may chose to make any class you like printable (a valid argument to 'log', 'warning', 'error') by implementing a const 'print' method in the class. You can find simple examples at [5].
[5] https://github.com/genodelabs/genode/blob/master/repos/os/include/util/geome...
Have fun with the further exploration!
Norman
Hi Norman, Thanks for the response. I have replied inline below whilst quoting the relevant parts. Let me know if this list has some other preferred way of organizing longer email exchanges.
Thanks, Sid
On Mon, Jan 10, 2022 at 6:14 AM Norman Feske norman.feske@genode-labs.com wrote:
Both constraints can be satisfied by managing the region allocation within a dedicated stack-region map manually.
Thanks. This makes sense.
- A PD is defined as Address Space and CSpace. I wondered if it is
possible to have multiple address spaces inside a PD. The RM for each of these different Address spaces would still be separate capabilities in
the
singular CSpace for that PD. This way, a thread can potentially jump
across
different address spaces like [1, 2]. Or multiple threads can have
slightly
different address spaces.
Early versions of Genode indeed decoupled address spaces from PDs. However, as we never found reasonable use cases of this presumed flexibility, we abandoned this artificial separation of mechanisms. As a result, the implementation became less complex and thereby more robust.
[1]
https://genode.org/documentation/release-notes/16.05#Consolidation_of_core_s...
BTW, it is still possible to share a portion of the virtual address space between multiple PDs by using core's RM service. E.g., the cached_fs_rom component uses this mechanism to make files available as read-only dataspaces shared by multiple clients.
Thanks, I will look at fs_rom.
I want to share a portion of the virtual address space between multiple threads in the same PD (and not different PD), this was both the threads have access to the same Cspace.
Regarding the idea of a thread "jumping across address spaces", that is really what's happening when issuing an RPC. Genode's RPC mechanism mimics plain synchronous function calls.
In the case of issuing an RPC, the Cspace is also changing, right? Since we making the RPC call from 1 PD to another.
- Where am I going with this? I am trying to implement a new type of
thread. In this thread(let's call it thread-stack), each thread's stack will be completely isolated from the other threads. That is, threads will not have access to each other's stacks. They will still share the code-section, library, and heap sections. This is a toy idea, so I can
get
my hand dirty with the Genode basics. I think everything I need to implement is already there in Genode but wanted to get your opinion.
Genode's mechanisms allow you to build that scenario. Just a rough sketch:
A custom parent component (runtime) would create a new PD for each "thread". But instead of loading the real program into the PD, it would load a simple bootstrapping program. Once executed, this bootstrapping program would request ROM and RAM dataspaces containing the text and data segments of the designated program from the parent. The parent is free to hand out the same dataspaces for each child.
What I just described is actually very similar to the regular function of our dynamic linker.
Thanks. I will take a look at that.
- Are they any helper functions to clone an entire address space?
Replicate the page table hierarchy with the same underlying page frames.
Not an entire address space. But a region map can be used as a dataspace. On kernels other than Linux, this - so called managed dataspace" - can be shared between PDs. It can also be used to implement on-demand page-fault handling. Think of automatically growing threads or page swapping. You can find an example at [2] However, in practice the mechanism is not used much.
[2]
https://github.com/genodelabs/genode/blob/master/repos/base/src/test/rm_nest...
thanks. I will take a look at that.
- Using the notion of Affinity, I can restrict a thread to run on a
particular CPU, but is there a way to restrict other threads from running on a given CPU. I am exploring isolating two threads by ensuring that
they
run on separate cores.
It's best to think in terms of PDs when partitioning resources. At the granularity of PDs, you can express the assignment of two PDs to a mutually exclusive set of CPU cores using init's configuration: defining an affinity space of 2x1, assigning the "left" part to one PD, and the "right" part to the other. Note however, that the boot CPU is effectively shared with other components and the execution of core's services. So to minimize the chance for cross talk, one would need to use a CPU with more than two cores (one for the boot CPU and one for each resource partition).
BTW, you can readily experiment with the mechanism using the Genode-based Sculpt OS as described at [3].
[3] http://genodians.org/nfeske/2021-03-24-sculpt-os
- Is there a way to restrict the time slices given to a particular
thread.
I am thinking of something like cgroups. I see that seL4 has the MCS kernel, which shares some of the motivations. But I think Genode does not support the MCS sel4 kernel.
We haven't used seL4's MCS kernel, yet.
That said, our custom base-hw microkernel supports a similar scheduling feature since version 14.11 [4].
If a component has the capability for every thread in the system it can implement any arbitrary scheduling policy using suspend/resume on the seL4 TCB. I think, my initial statement about not being able to do scheduling in userspace in seL4 might have been premature.
[4]
https://genode.org/documentation/release-notes/14.11#Trading_CPU_time_betwee...
- Is the CSpace for a PD the same as CSpace in seL4, or is there a
different notion of Cspace in Genode?
It is the same notion. It is only used in the seL4-specific part of Genode (in base-sel4/).
- Is there an easy way to print the entire CSpace for debugging
purposes?
Unfortunately not. However, while working with the code, you may chose to make any class you like printable (a valid argument to 'log', 'warning', 'error') by implementing a const 'print' method in the class. You can find simple examples at [5].
[5]
https://github.com/genodelabs/genode/blob/master/repos/os/include/util/geome...
Have fun with the further exploration!
Norman
-- Dr.-Ing. Norman Feske Genode Labs
https://www.genode-labs.com · https://genode.org
Genode Labs GmbH · Amtsgericht Dresden · HRB 28424 · Sitz Dresden Geschäftsführer: Dr.-Ing. Norman Feske, Christian Helmuth
Genode users mailing list users@lists.genode.org https://lists.genode.org/listinfo/users
-
Norman Feske -
Sid Agrawal