Hi, there.
I'm a college student majoring Information Security in HUST, and I'm interested in genode's GSoC project on microkernelizaing the Linux kernel. I've been quite interested in microkernel OSs and virtualization, with limited experiences working on xen and the linux kernel. and have been in the MINIX community for a short time, so I was attracted by this challenge at the first sight.
However, I'm still new to the genode project. So I'll appreciate it if I could get from you some suggestions on where to get started. :)
Best Regards,
Zhongze Liu.
Hello Sky Liu,
thank you for your interest in Genode!
I'm a college student majoring Information Security in HUST, and I'm interested in genode's GSoC project on microkernelizaing the Linux kernel. I've been quite interested in microkernel OSs and virtualization, with limited experiences working on xen and the linux kernel. and have been in the MINIX community for a short time, so I was attracted by this challenge at the first sight.
However, I'm still new to the genode project. So I'll appreciate it if I could get from you some suggestions on where to get started. :)
The best way to start exploring Genode is the "Genode Foundations" book, which you can download here:
http://genode.org/documentation/genode-foundations-16-05.pdf
I recommend you to at follow the getting-started section, and skim over the Chapter 3 (Architecture) and Section 4.7 (Component compositions) to get a tangible feeling for Genode.
To practically tackle the "microkernelization of Linux", there are actually two approaches. (1) The first approach is to enable Genode to access devices of the Linux system you are working on. This is convenient, but it bears the risk that a device driver (running in a Genode component) may interfere with the Linux kernel, crashing the system. The other approach (2) is booting a custom-made Linux system + Genode's core as init process in Qemu. The latter approach would be equivalent to how we work with the various microkernels.
Depending on your interests, you may quickly dive in into the actual Genode code (1) or work on a custom run environment for Linux-based Genode system (2) first.
Regarding the actual topic, the overall challenge is allowing Genode's device drivers access to the hardware that is normally accessed by the Linux kernel only. From Linux' point of view, Genode appears as a user-level device driver. So one piece of the puzzle is to gain a good understanding of Linux' user-level device-driver support. From Genode's perspective, device hardware is accessed through the IO_MEM, IO_PORT, and IRQ services of Genode's core component. So in principle, the topic comes down to implementing these services for the 'base-linux' version of core by using Linux' interfaces for user-level device drivers.
There are several stages:
1. By implementing the IO_PORT service, Genode components can interact with simple port-I/O-devices such as the PIT timer. The implementation should by straight-forward: When running core at I/O privilege level (IOPL) 3, core can execute the regular 'inb', 'outb', etc. instructions. A simple test component could request a Genode IO_PORT session for, let's say, the PIT, program the PIT as a running counter, and repeatedly read the current counter value.
2. Non-trivial devices need access to memory-mapped I/O registers. Core's IO_MEM service makes such registers available to its clients as a dataspace (the concept is explained in the book). On Linux, dataspaces are represented as memory-mapped files, passed from core to the driver component by passing a file descriptor, and attached to the driver's address space via 'mmap'. Consequently, the problem comes down to core obtaining a file descriptor for a given physical address region. Here the challenge is to find a suitable Linux kernel mechanism that hands out portions of physical memory as a file descriptor.
With the principle support for memory-mapped I/O and port I/O in place, it should be possible to run the VESA framebuffer driver.
3. Most drivers use device interrupts. To use those drivers, core's IRQ service needs to be implemented. Again, this calls for an investigation of Linux' user-level device driver support.
4. Direct memory access. Many drivers use DMA to let the device write directly into memory. For this to work, the driver must supply the targeted memory address to the device. On systems without IOMMU, this is the physical bus address. In order to use DMA on Genode/Linux, DMA buffers need to be allocated as contiguous physical memory and their physical addresses must become known to the user-level driver component. In systems with IOMMU, the device-physical addresses a virtualized. So there is more freedom. But the details ultimately depends on the Linux handling of IOMMUs.
5. In Genode, PCI devices are managed by the so-called platform driver. When a regular device driver needs access to a certain device, it does not use core's IO_MEM, IO_PORT, and IRQ services directly but it requests a platform session. A platform session is like a virtual bus where one or multiple devices are present, depending on the platform driver's policy. The platform session makes the device resources of those devices available to the client (the device driver). Under the hood, the platform driver opens IO_MEM/IO_PORT/IRQ sessions at core.
In order to use Genode's existing arsenal of device drivers on Linux, we need either a platform-driver version specifically adapted for Linux (when re-using the Linux PCI driver), or investigate a way to use our regular platform driver (including the PCI driver) directly (this would be pretty cool!).
I hope that the level of detail does not frighten you. We certainly don't expect you to complete all these stages. E.g., if completing stage 1 to 3, there would already be demonstratable results. Stage 4 certainly requires an investigation into the ways about the interplay of the IOMMU handling of the kernel with user-level device drivers. Stage 5 also has a atrong design aspect to it.
It goes without saying that we won't leave you on your own. :-)
Cheers Norman
Hi Norman,
Thanks for the detailed suggestions with so much patience. I'm a practitioner, so I really appreciate details, and I'll check out the stuff that you've mentioned and try to get started ASAP.
BTW, do you have any suggestions on small related tasks that I can take in prior to my formal application for the GSoC project? I think committing small patches would be a great starting point to getting familiar with the Genode codebase.
Best Regards,
Zhongze Liu.
2017-03-01 19:37 GMT+08:00 Norman Feske <norman.feske@...1...>:
Hello Sky Liu,
thank you for your interest in Genode!
I'm a college student majoring Information Security in HUST, and I'm interested in genode's GSoC project on microkernelizaing the Linux kernel. I've been quite interested in microkernel OSs and virtualization, with limited experiences working on xen and the linux kernel. and have been in the MINIX community for a short time, so I was attracted by this challenge at the first sight.
However, I'm still new to the genode project. So I'll appreciate it if I could get from you some suggestions on where to get started. :)
The best way to start exploring Genode is the "Genode Foundations" book, which you can download here:
http://genode.org/documentation/genode-foundations-16-05.pdf
I recommend you to at follow the getting-started section, and skim over the Chapter 3 (Architecture) and Section 4.7 (Component compositions) to get a tangible feeling for Genode.
To practically tackle the "microkernelization of Linux", there are actually two approaches. (1) The first approach is to enable Genode to access devices of the Linux system you are working on. This is convenient, but it bears the risk that a device driver (running in a Genode component) may interfere with the Linux kernel, crashing the system. The other approach (2) is booting a custom-made Linux system + Genode's core as init process in Qemu. The latter approach would be equivalent to how we work with the various microkernels.
Depending on your interests, you may quickly dive in into the actual Genode code (1) or work on a custom run environment for Linux-based Genode system (2) first.
Regarding the actual topic, the overall challenge is allowing Genode's device drivers access to the hardware that is normally accessed by the Linux kernel only. From Linux' point of view, Genode appears as a user-level device driver. So one piece of the puzzle is to gain a good understanding of Linux' user-level device-driver support. From Genode's perspective, device hardware is accessed through the IO_MEM, IO_PORT, and IRQ services of Genode's core component. So in principle, the topic comes down to implementing these services for the 'base-linux' version of core by using Linux' interfaces for user-level device drivers.
There are several stages:
By implementing the IO_PORT service, Genode components can interact with simple port-I/O-devices such as the PIT timer. The implementation should by straight-forward: When running core at I/O privilege level (IOPL) 3, core can execute the regular 'inb', 'outb', etc. instructions. A simple test component could request a Genode IO_PORT session for, let's say, the PIT, program the PIT as a running counter, and repeatedly read the current counter value.
Non-trivial devices need access to memory-mapped I/O registers. Core's IO_MEM service makes such registers available to its clients as a dataspace (the concept is explained in the book). On Linux, dataspaces are represented as memory-mapped files, passed from core to the driver component by passing a file descriptor, and attached to the driver's address space via 'mmap'. Consequently, the problem comes down to core obtaining a file descriptor for a given physical address region. Here the challenge is to find a suitable Linux kernel mechanism that hands out portions of physical memory as a file descriptor.
With the principle support for memory-mapped I/O and port I/O in place, it should be possible to run the VESA framebuffer driver.
Most drivers use device interrupts. To use those drivers, core's IRQ service needs to be implemented. Again, this calls for an investigation of Linux' user-level device driver support.
Direct memory access. Many drivers use DMA to let the device write directly into memory. For this to work, the driver must supply the targeted memory address to the device. On systems without IOMMU, this is the physical bus address. In order to use DMA on Genode/Linux, DMA buffers need to be allocated as contiguous physical memory and their physical addresses must become known to the user-level driver component. In systems with IOMMU, the device-physical addresses a virtualized. So there is more freedom. But the details ultimately depends on the Linux handling of IOMMUs.
In Genode, PCI devices are managed by the so-called platform driver. When a regular device driver needs access to a certain device, it does not use core's IO_MEM, IO_PORT, and IRQ services directly but it requests a platform session. A platform session is like a virtual bus where one or multiple devices are present, depending on the platform driver's policy. The platform session makes the device resources of those devices available to the client (the device driver). Under the hood, the platform driver opens IO_MEM/IO_PORT/IRQ sessions at core.
In order to use Genode's existing arsenal of device drivers on Linux, we need either a platform-driver version specifically adapted for Linux (when re-using the Linux PCI driver), or investigate a way to use our regular platform driver (including the PCI driver) directly (this would be pretty cool!).
I hope that the level of detail does not frighten you. We certainly don't expect you to complete all these stages. E.g., if completing stage 1 to 3, there would already be demonstratable results. Stage 4 certainly requires an investigation into the ways about the interplay of the IOMMU handling of the kernel with user-level device drivers. Stage 5 also has a atrong design aspect to it.
It goes without saying that we won't leave you on your own. :-)
Cheers Norman
-- Dr.-Ing. Norman Feske Genode Labs
http://www.genode-labs.com · http://genode.org
Genode Labs GmbH · Amtsgericht Dresden · HRB 28424 · Sitz Dresden Geschäftsführer: Dr.-Ing. Norman Feske, Christian Helmuth
Check out the vibrant tech community on one of the world's most engaging tech sites, SlashDot.org! http://sdm.link/slashdot _______________________________________________ genode-main mailing list genode-main@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/genode-main
Hi, Norman,
I think this explanation you gave would fit greatly in the Genode site documentation. It would help a lot if the documentation had some guidance on where to start, essential reading (like Tanenbaum's book on Operating Systems), etc. Personally it would be great because I work with systems development (Java, etc) but have a lot of interest on microkernels and such areas are far from each other. I already programmed in C and enjoy optimizing things. A high level path on what should be read by people with some programming background could attract more help.
I think microkernels have a fantastic potential and particularly this project. The inherent security, the concept of delegation of responsibility of sharing resources through other microkernels, the reduction on the trusted code base by an order of magnitude and the consequence of making it auditable is a revolution. Once it begin being used in some distribution with only the essential it will get momentum.
By the way, some time ago I saw some discussion here about how to make this effort more visible. I remember it ended because the focus of the project was about developing the framework. I understand and agree with that, but if/when the idea comes back, I suggest considering releasing a very basic server installation: essential GNU tools, TCP/IP stack, login through SSH, a compiler, etc. The problem would be the drivers. But to counter that, you could release the ISO at least for Raspberry Pi which is a fixed hardware and is very widespread. It could be used as a home server by hobbyists as a firewall (where security is paramount), as a file server, etc. Or, maybe by handling the work of a dedicated server that don't need to withstand great loads. To call more attention, it should run starting at Raspberry Pi 1 model B, which has 512MB. About this last requirement (to run on Raspberry Pi 1), I don't remember very well, but I think running Samba, Apache and a firewall (which is built in Linux TCP/IP stack) should not exhaust the memory, so I think it's possible.
Sorry for having digressed. ;)
Best regards,
José Roberto
2017-03-01 12:58 GMT-03:00 Sky Liu <blackskygg@...9...>:
Hi Norman,
Thanks for the detailed suggestions with so much patience. I'm a practitioner, so I really appreciate details, and I'll check out the stuff that you've mentioned and try to get started ASAP.
BTW, do you have any suggestions on small related tasks that I can take in prior to my formal application for the GSoC project? I think committing small patches would be a great starting point to getting familiar with the Genode codebase.
Best Regards,
Zhongze Liu.
2017-03-01 19:37 GMT+08:00 Norman Feske <norman.feske@...1...>:
Hello Sky Liu,
thank you for your interest in Genode!
I'm a college student majoring Information Security in HUST, and I'm interested in genode's GSoC project on microkernelizaing the Linux kernel. I've been quite interested in microkernel OSs and virtualization, with limited experiences working on xen and the linux kernel. and have been in the MINIX community for a short time, so I was attracted by this challenge at the first sight.
However, I'm still new to the genode project. So I'll appreciate it if I could get from you some suggestions on where to get started. :)
The best way to start exploring Genode is the "Genode Foundations" book, which you can download here:
http://genode.org/documentation/genode-foundations-16-05.pdf
I recommend you to at follow the getting-started section, and skim over the Chapter 3 (Architecture) and Section 4.7 (Component compositions) to get a tangible feeling for Genode.
To practically tackle the "microkernelization of Linux", there are actually two approaches. (1) The first approach is to enable Genode to access devices of the Linux system you are working on. This is convenient, but it bears the risk that a device driver (running in a Genode component) may interfere with the Linux kernel, crashing the system. The other approach (2) is booting a custom-made Linux system + Genode's core as init process in Qemu. The latter approach would be equivalent to how we work with the various microkernels.
Depending on your interests, you may quickly dive in into the actual Genode code (1) or work on a custom run environment for Linux-based Genode system (2) first.
Regarding the actual topic, the overall challenge is allowing Genode's device drivers access to the hardware that is normally accessed by the Linux kernel only. From Linux' point of view, Genode appears as a user-level device driver. So one piece of the puzzle is to gain a good understanding of Linux' user-level device-driver support. From Genode's perspective, device hardware is accessed through the IO_MEM, IO_PORT, and IRQ services of Genode's core component. So in principle, the topic comes down to implementing these services for the 'base-linux' version of core by using Linux' interfaces for user-level device drivers.
There are several stages:
By implementing the IO_PORT service, Genode components can interact with simple port-I/O-devices such as the PIT timer. The implementation should by straight-forward: When running core at I/O privilege level (IOPL) 3, core can execute the regular 'inb', 'outb', etc. instructions. A simple test component could request a Genode IO_PORT session for, let's say, the PIT, program the PIT as a running counter, and repeatedly read the current counter value.
Non-trivial devices need access to memory-mapped I/O registers. Core's IO_MEM service makes such registers available to its clients as a dataspace (the concept is explained in the book). On Linux, dataspaces are represented as memory-mapped files, passed from core to the driver component by passing a file descriptor, and attached to the driver's address space via 'mmap'. Consequently, the problem comes down to core obtaining a file descriptor for a given physical address region. Here the challenge is to find a suitable Linux kernel mechanism that hands out portions of physical memory as a file descriptor.
With the principle support for memory-mapped I/O and port I/O in place, it should be possible to run the VESA framebuffer driver.
Most drivers use device interrupts. To use those drivers, core's IRQ service needs to be implemented. Again, this calls for an investigation of Linux' user-level device driver support.
Direct memory access. Many drivers use DMA to let the device write directly into memory. For this to work, the driver must supply the targeted memory address to the device. On systems without IOMMU, this is the physical bus address. In order to use DMA on Genode/Linux, DMA buffers need to be allocated as contiguous physical memory and their physical addresses must become known to the user-level driver component. In systems with IOMMU, the device-physical addresses a virtualized. So there is more freedom. But the details ultimately depends on the Linux handling of IOMMUs.
In Genode, PCI devices are managed by the so-called platform driver. When a regular device driver needs access to a certain device, it does not use core's IO_MEM, IO_PORT, and IRQ services directly but it requests a platform session. A platform session is like a virtual bus where one or multiple devices are present, depending on the platform driver's policy. The platform session makes the device resources of those devices available to the client (the device driver). Under the hood, the platform driver opens IO_MEM/IO_PORT/IRQ sessions at core.
In order to use Genode's existing arsenal of device drivers on Linux, we need either a platform-driver version specifically adapted for Linux (when re-using the Linux PCI driver), or investigate a way to use our regular platform driver (including the PCI driver) directly (this would be pretty cool!).
I hope that the level of detail does not frighten you. We certainly don't expect you to complete all these stages. E.g., if completing stage 1 to 3, there would already be demonstratable results. Stage 4 certainly requires an investigation into the ways about the interplay of the IOMMU handling of the kernel with user-level device drivers. Stage 5 also has a atrong design aspect to it.
It goes without saying that we won't leave you on your own. :-)
Cheers Norman
-- Dr.-Ing. Norman Feske Genode Labs
http://www.genode-labs.com · http://genode.org
Genode Labs GmbH · Amtsgericht Dresden · HRB 28424 · Sitz Dresden Geschäftsführer: Dr.-Ing. Norman Feske, Christian Helmuth
Check out the vibrant tech community on one of the world's most engaging tech sites, SlashDot.org! http://sdm.link/slashdot _______________________________________________ genode-main mailing list genode-main@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/genode-main
Check out the vibrant tech community on one of the world's most engaging tech sites, SlashDot.org! http://sdm.link/slashdot _______________________________________________ genode-main mailing list genode-main@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/genode-main
Hi José,
thanks for your kind words.
I think this explanation you gave would fit greatly in the Genode site documentation. It would help a lot if the documentation had some guidance on where to start, essential reading (like Tanenbaum's book on Operating Systems), etc. Personally it would be great because I work with systems development (Java, etc) but have a lot of interest on microkernels and such areas are far from each other.
Actually, I merely recommended the "Genode Foundations" book, which is already prominently displayed at the front page of genode.org. ;-)
By the way, some time ago I saw some discussion here about how to make this effort more visible. I remember it ended because the focus of the project was about developing the framework. I understand and agree with that, but if/when the idea comes back, I suggest considering releasing a very basic server installation: essential GNU tools, TCP/IP stack, login through SSH, a compiler, etc. The problem would be the drivers. But to counter that, you could release the ISO at least for Raspberry Pi which is a fixed hardware and is very widespread. It could be used as a home server by hobbyists as a firewall (where security is paramount), as a file server, etc. Or, maybe by handling the work of a dedicated server that don't need to withstand great loads. To call more attention, it should run starting at Raspberry Pi 1 model B, which has 512MB. About this last requirement (to run on Raspberry Pi 1), I don't remember very well, but I think running Samba, Apache and a firewall (which is built in Linux TCP/IP stack) should not exhaust the memory, so I think it's possible.
I like the idea and don't find it unrealistic at all. But that said, someone has to step forward and actually do it. In the light of the many topics presented on our road map, it is unlikely that any of the regular developers will be able to pursue this direction in the near future.
In my opinion, the best we can do is to lower for the bar for a wider community to pursue such ideas. The package management as planned for the upcoming release will hopefully contribute to that.
Cheers Norman
Hi Norman,
2017-03-03 7:46 GMT-03:00 Norman Feske <norman.feske@...1...>:
Hi José,
thanks for your kind words.
I think this explanation you gave would fit greatly in the Genode site documentation. It would help a lot if the documentation had some guidance on where to start, essential reading (like Tanenbaum's book on Operating Systems), etc. Personally it would be great because I work with systems development (Java, etc) but have a lot of interest on microkernels and such areas are far from each other.
Actually, I merely recommended the "Genode Foundations" book, which is already prominently displayed at the front page of genode.org. ;-)
I already got that book, and I think it's great. I even already knew it's a second edition. I though I needed more background to read it due to my lack of a deeper knowledge on Operating Systems - actually I'm a Mechanical Engineer who migrated to systems analysis. You mean that reading "Genode Foundations" is enough to begin understanding? Maybe some more background in C and C++ and some reading on Tanenbaum's book, isn't it?
By the way, some time ago I saw some discussion here about how to make this effort more visible. I remember it ended because the focus of the project was about developing the framework. I understand and agree with that, but if/when the idea comes back, I suggest considering releasing a very basic server installation: essential GNU tools, TCP/IP stack, login through SSH, a compiler, etc. The problem would be the drivers. But to counter that, you could release the ISO at least for Raspberry Pi which is a fixed hardware and is very widespread. It could be used as a home server by hobbyists as a firewall (where security is paramount), as a file server, etc. Or, maybe by handling the work of a dedicated server that don't need to withstand great loads. To call more attention, it should run starting at Raspberry Pi 1 model B, which has 512MB. About this last requirement (to run on Raspberry Pi 1), I don't remember very well, but I think running Samba, Apache and a firewall (which is built in Linux TCP/IP stack) should not exhaust the memory, so I think it's possible.
I like the idea and don't find it unrealistic at all. But that said, someone has to step forward and actually do it. In the light of the many topics presented on our road map, it is unlikely that any of the regular developers will be able to pursue this direction in the near future.
I know. If had more background I would probably try to help on that. By now I can only watch this list and try to understand the talks. Still sounds like japanese for me. ;)
In my opinion, the best we can do is to lower for the bar for a wider community to pursue such ideas. The package management as planned for the upcoming release will hopefully contribute to that.
Yes, when I saw the intent on the package management I found it would be
the right direction towards an easier adoption.
Best regards,
José Roberto
Hello Zhongze Liu,
On Wed, Mar 01, 2017 at 11:58:41PM +0800, Sky Liu wrote:
BTW, do you have any suggestions on small related tasks that I can take in prior to my formal application for the GSoC project? I think committing small patches would be a great starting point to getting familiar with the Genode codebase.
While reading your question I recalled the missing feature in our x86 RTC driver, which is only able to read out the current time from the RTC. It would be really nice if the driver could also update the time on request.
In detail, the feature has two aspects. First, the RTC driver has to be extended by the code for setting the CMOS clock according to the specification. Further, it needs to provide means to update the clock on request. The method of choice is the XML config of the component which could be extended by an XML node for the clock update. To support this feature the component code has to request and parse the config ROM initially but also register for updates of the configuration during runtime. The addition of this feature would not only help to get in touch with low-level driver code but also learn about the interaction of the RTC component with the surrounding system.
If you're interested in this hacking project I'd be pleased to provide further guidance. The current driver can be found in the Genode sources at repos/os/src/drivers/rtc/spec/x86.
Greets
Hi Christian Helmuth,
Sorry for my late reply. I've been quite busy these days. Thanks for telling me this task, I think it would be of great help as an initial step. And yes, I would like to try this hack. I would first try to read the code and the RTC spec. And I would appreciate it if you could provide me with your precious guidance.
Cheers,
Zhongze Liu
2017-03-02 17:11 GMT+08:00 Christian Helmuth <christian.helmuth@...1...>:
Hello Zhongze Liu,
On Wed, Mar 01, 2017 at 11:58:41PM +0800, Sky Liu wrote:
BTW, do you have any suggestions on small related tasks that I can take in prior to my formal application for the GSoC project? I think committing small patches would be a great starting point to getting familiar with the Genode codebase.
While reading your question I recalled the missing feature in our x86 RTC driver, which is only able to read out the current time from the RTC. It would be really nice if the driver could also update the time on request.
In detail, the feature has two aspects. First, the RTC driver has to be extended by the code for setting the CMOS clock according to the specification. Further, it needs to provide means to update the clock on request. The method of choice is the XML config of the component which could be extended by an XML node for the clock update. To support this feature the component code has to request and parse the config ROM initially but also register for updates of the configuration during runtime. The addition of this feature would not only help to get in touch with low-level driver code but also learn about the interaction of the RTC component with the surrounding system.
If you're interested in this hacking project I'd be pleased to provide further guidance. The current driver can be found in the Genode sources at repos/os/src/drivers/rtc/spec/x86.
Greets
Christian Helmuth Genode Labs
https://www.genode-labs.com/ · https://genode.org/ https://twitter.com/GenodeLabs · /ˈdʒiː.nəʊd/
Genode Labs GmbH · Amtsgericht Dresden · HRB 28424 · Sitz Dresden Geschäftsführer: Dr.-Ing. Norman Feske, Christian Helmuth
Check out the vibrant tech community on one of the world's most engaging tech sites, SlashDot.org! http://sdm.link/slashdot _______________________________________________ genode-main mailing list genode-main@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/genode-main
-
Christian Helmuth -
José Roberto Teixeira -
Norman Feske -
Sky Liu -
Zhongze Liu