Hallo everyone,
hereby, I'd like to follow up on our tradition to discuss Genode's road map on the mailing list. Let's take the turn of the year to recapture the events of 2018 and make plans for the next twelve months. Please feel welcome to chime in! By mid of January, I'll finalize Genode's official road map and would like to take your input into account.
In the following, let me share my personal reflections and goals.
The Year of Sculpt
When I declared 2018 as Genode's Year of Sculpt, I had no concrete idea how Sculpt OS would shape up. I was convinced that we had - functionality-wise - all building blocks of a general-purpose OS in place. But it was rather unclear how to best put them together to attain a practical system. The unconquered design space seemed vast, which was both exciting but also - at times - a bit paralyzing.
I ultimately had to bang a few nails into the wall, taking decisions with uncertain outcome. The Year of Sculpt was - more than anything - a design-space exploration, not an up-front planned activity. In the process, I came up with many half-baked thoughts and had the fortune of having our team at Genode Labs scrutinizing my ideas while also taking me seriously. Whenever an idea was indefensible, we jointly came up with a better one. Whenever uncertain about a proposed solution, I went forward with implementing it and took our team as guinea pigs. Whenever an idea was blocked by a limitation of an existing component, I could count on the support of the others to lend a helping hand with tweaking the components as needed.
Unsurprisingly, many topics of the past year had a direct connection to Sculpt, e.g., the NIC router, the huge device-driver efforts, the GUI-stack improvements, and the work on the file-system and networks stacks.
For me personally, this ride was certainly the most rewarding period in Genode's history so far. Now, when looking at the result, I am overwhelmed about what we achieved together! Whenever I have the chance to showing off Sculpt running on my laptop, Sculpt doesn't fail to impress.
That said, during the course of the year, this positive sense of achievement eventually blended with the rather dull feeling that our hard work remains largely unnoticed outside the inner circle of Genode enthusiasts. The public at large remains quite indifferent (e.g., I was unable to capture the interest of FOSDEM to feature a talk about Sculpt OS on their main track). For a long time, I tricked myself into believing that once we overcome all technical road blocks, Genode will eventually become widely recognized automatically. There was always a technical challenge to take on. With Sculpt, we have reached a point where this excuse doesn't hold anymore. There is no technical piece missing.
This leaves me with the question of how to make Genode relevant at a larger scale? Since this is not a technical question, I admittedly struggle to find a good answer. When thinking of the overall leitmotif for 2019, I always come back to this question.
Plan for 2019
I see three directions to help Genode to become more widely recognized.
1) Widening the audience of Sculpt OS 2) Fostering the community spirit around our project 3) Marketing of Genode-based products
Let me outline my thoughts behind each point.
1) Widening the audience of Sculpt OS
Sculpt works well, but it is arguably still too hard to use for non-technical users and it lacks a lot of software that users take for granted today. Consequently, we should improve its ease of use. In particular, I'd love to explore the more consequent use of Sculpt's component graph as an intuitive user interface for both the composition and the configuration of components. The ultimate goal would be to eliminate the need for editing any text files. With respect to software availability, I have high hopes in the Genode SDK that we introduced in Genode 18.11. Let's follow this route further.
2) Fostering the community spirit around our project
Our project enjoys a healthy community of users and contributors. But the community is rather scattered and the various groups work pretty much in isolation. As a result, the work of one group is often invisible to the others, not to speak of any public visibility. Could we possibly establish an instrument that would help each participant to gain more visibility and thereby help the community at large to become more relevant? I'll come up with a concrete proposal for such an instrument soon.
One technical element may be a feature of Sculpt that would allow the user to browse the software depots provided by various community members and install/deploy packages by just a few clicks. This would vastly ease the discoverability of the available software and highlight the roles of the respective developers.
A non-technical element may be a dissemination of Genode-related news at a higher rate than the current release announcements.
3) Marketing of Genode-based products
Let's build Genode-based solutions to real-world problems and talk about them. The more diverse the solutions, the better. Be it server appliances, network equipment, controlling devices, or other forms of embedded systems, there are so many opportunities.
One important trait of such products is the trustworthiness of Genode. So we constantly need to reinforce the confidence in Genode's quality. We should continue and expand our quality-assurance and software-hardening activities. For example, I see a huge potential in cultivating the use of Ada/SPARK for certain security-critical components.
These are my high-level thoughts about the direction of the project. I deliberately left out a list of technical items from this initial posting. Let's fill in the technical substance together. :-)
Please don't hesitate to share your thoughts. You may consider the following questions:
* What are your ambitions for 2018?
* Which areas of Genode would you like to see improved? How would you possibly contribute to these improvements?
* Which technical topics do you find tempting?
* If you imagine a Genode-based system one year in the future, how would it look like?
* Do you have further ideas that would help making Genode relevant at a larger scale than today?
Cheers Norman
Hi Norman and the whole Genode team,
First, I recognize the feeling that you are enthusiastic about your work and nobody seems to care. And how devastating that can feel.
I know that feeling from my attempt at designing and promoting my authentication protocol that (I believe) could decimate most phishing and identity theft attacks. I got so disappointed at the lack of response that I disabled web statistics so I didn't have to see how little visitors my site got. Now I've more or less given up on it.
Yet every time I see a news article about phishing or stolen (and abused) passwords I'm reminded that I came up with something that could have (helped to) prevent it.
Second, it's hard to create a new platform from scratch. As a rule of thumb, for any new invention it takes roughly 15 years from invention to market-readyness. So far, Genode Labs succeeded where others gave up. Thanks for not giving up.
Third, it's very hard to bring innovation into the IT-world. The IT community is very conservative.
Sandboxing to contain viruses is something that HP-Labs did 15 years ago for Windows XP [1]. Regrettably HP didn't succeed in selling it widely. Just last week, Microsoft has added a sandbox to Win10 in an attempt to contain malware.
An example of the conservatism: I explained some ideas of Genode (micro-kernel, sandboxing, separation, pola) to a 30yo manager of a software development company that advertised itself for writing secure software. He found the ideas interesting but did not want to spend time to research it. Not even reading on the topic. He said it was "academic stuff that eventually shows up in Linux". I couldn't convince him that it might be a good idea to be ahead of the curve. He was risk averse, as are most IT-people. Genode needs to find people willing to take a risk and reap the reward.
Every time I read about crypto malware holding a user or company hostage for ransom I'm reminded that had they used Genode it could have (helped to) prevent it.
I believe there is a huge market out there. The difficulty is getting there. I hope my answers to this mail give some ideas how to proceed.
Now about your goals and questions:
- Widening the audience of Sculpt OS
Consequently, we should improve its ease of use.
For me a 1000 times this.
To me, the learning curve of Genode is steep. The learning curve of (changing) Sculpt is steep too. Although I know most of the concepts of the platform, I find it still hard to develop in. Lately I went deep into changing the configuration of Virtualbox in Sculpt. What should be a few hours took me more than a week. And the result was still hacky :-(
Although there are many examples, I'd like to see them listed from simple to complex, each explaining a concept of Genode, builing on top of the previous ones. It could grow into a Genode for Dummies.
Other wish: please describe how I can enable all kinds of debugging modes. For example, when I make a XML syntax error in an init.config, the app doesn't start but the log remains quiet.
Other example, at one point I added printf-logging to init-components to trace the routing decisions so I could check that my config was correct. It would be great if there was a config option for that.
- Fostering the community spirit around our project
I think you have a nice (but small) community already, with your team ready to answer any question quicky. Just keep doing that.
You've seen my recent write-up on adding a raw partition to virtualbox in Sculpt. I thought of making it a blog so others can learn from it too. (I'm not sure if that example makes a nice showcase of the ease of Sculpt :-).
- Marketing of Genode-based products
I like the idea of cross-promotion between Genode and some companies that use it. Put some showcases on your site. Perhaps with a small testimonial from each company why they chose Genode.
Given the impossibility of convincing (conservative) people why Genode is better than their current system, don't focus on technical details. Instead focus on concrete benefits for (end) users: - safe by design; - protects privacy; - little errors do not become catastrophes; - robust against malware; - no need for regular updates; - updates don't break existing functionality; - easy to use, also for non-computer users (see my dream system below).
- What are your ambitions for 2019?
I still have my wish for running Genode on my server, running a webserver for static and dynamic content. I ran a static web site on Genode a few years ago. It lasted a week until it crashed due to a resource leak, (memory, VFS-file descriptors). I could not debug it, so the box runs Linux again.
I'd like to run a (small) dynamic web site on it. I.e. start with a blog and comments section and private messaging. For parsing I use the a composable parser generator such as the Hammer library [2] from the Langsec [3] community. I think Langsec and Genode complement each other nicely.
I know some hackers (professional pentesters) interested in the idea of a site that's very hard to hack, even if there are errors in my implementation. I'm curious to their findings.
My goal is to get Genode on a server with some VMs for things not yet ported. And get some hackers to pentest it.
- Which areas of Genode would you like to see improved? How would you possibly contribute to these improvements?
Documentation. (I'll tell you my struggles, you improve the docs).
- If you imagine a Genode-based system one year in the future, how would it look like?
My long term dream is Genode on a Desktop.
It has a desktop UI interface, double clicking opens an application in a sandbox. The application has only access to its dependencies and resources like fonts, etc.
However, the application does not have access to the user's files, email, etc., not even network access. It the user wants to open a file, the sandbox detects the application opening a file-browser and an attempt to read /home/<user> and opens a Powerbox. The powerbox is a trusted part of the OS that lets the user select one or more files. Only these are the files that the application can read. (The powerbox is described in HP-Labs paper [1] and other capability security papers on the net).
The user of this system is a non-technical user, say a clerk at city hall dealing with building permits. Their need is to approve or reject building plans. As they get data from the outside, it must be considered hostile. With current Linux, Mac and Windows systems, this clerk needs to make a decision whether to open a certain email or not. After all, it could be crypto-malware. So without opening the email, the clerk must make a value judgement on its contents. That's a mission impossible. Especially for a clerk without IT programming skills.
Genode can help here. Every parser (email, zip-files, photos, fonts, audio, video, etc) run in separated sandboxes, all the user's resources (files, emails, photos, address books, etc) are protected by powerboxes, so if an email contains malware, it can't sneakily encrypt anything. In fact, if the malware misbehaves, it probably triggers a powerbox for a file-open dialog that the clerk did not request. The clerk forwards the mail to the IT-department for analysis.
- Do you have further ideas that would help making Genode relevant at a larger scale than today?
Since you ask about my Santa list :-)
Make it easy (both in documenting) and code support to port exisiting Linux or Windows software to Genode. It could be a preconfigured Noux instance that provides just enough to start the application. It needs /usr, a bit of /etc, a private var and a powerbox to /home/<user>. It needs a NIC environment with a configurable ingress and egress firewall.
Make it easy for an end-user to create separate of these sandboxes for a single application, so the user can create separate mail-readers for private mail, office mail, etc.
It should be easy to port and effortless to upgrade. Upgrades should be build automatically by a build server so a small number of people can manage a large set of programs. I'm thinking user application such as libreoffice, gimp, photo, video, audio, bookkeeping. Software that doesn't need linux kernel access, drivers or distribution packaging specific things.
I love to run a mail-stack consisting of server programs such as postfix, dovecot, spamassissin, DNS-servers, DNSSEC signers, etc on Genode.
Best wishes for the holidays,
Guido.
1: http://www.hpl.hp.com/techreports/2004/HPL-2004-221.html 2: https://github.com/UpstandingHackers/hammer 3: http://langsec.org/
Hi Guido,
thank you for the elaborate and insightful response.
I'm very sorry to hear the fate of your eccentric-authentication project as I still vividly remember your enthusiasm about it. Hopefully, it's not buried but only temporarily suspended.
Now about your goals and questions:
- Widening the audience of Sculpt OS
Consequently, we should improve its ease of use.
For me a 1000 times this.
To me, the learning curve of Genode is steep. The learning curve of (changing) Sculpt is steep too. Although I know most of the concepts of the platform, I find it still hard to develop in. Lately I went deep into changing the configuration of Virtualbox in Sculpt. What should be a few hours took me more than a week. And the result was still hacky :-(
Hats off for trying! You certainly went outside the designated (and documented) scope of the current version, which is impressive. Your feedback is valuable for planning the upcoming improvements of Sculpt.
Although there are many examples, I'd like to see them listed from simple to complex, each explaining a concept of Genode, builing on top of the previous ones. It could grow into a Genode for Dummies.
This raises an interesting question: Should we prioritize documenting the use of Sculpt or the use of Genode?
The former would look only at the building blocks (but not inside them). Such documentation can be in the form of light-hearted and practical use-case anecdotes, which are fun to read and immediately applicable by Sculpt users without any programming. It also takes a few complicated topics like the build system or the run scripts out of view.
The latter would come down to a review and description of existing run-script scenarios, which is probably less rewarding but more valuable for people who want to go deeper below the surface of Sculpt.
Intuitively, for capturing a larger audience, I'd first put the focus on use cases based on Sculpt.
Other wish: please describe how I can enable all kinds of debugging modes. For example, when I make a XML syntax error in an init.config, the app doesn't start but the log remains quiet.
Other example, at one point I added printf-logging to init-components to trace the routing decisions so I could check that my config was correct. It would be great if there was a config option for that.
That's an interesting information because it does not overlap much with my personal experience. Making init (optionally) more verbose would be no problem at all. I just need to know which kind of verbosity is actually helpful. So your experience can guide us here.
- Fostering the community spirit around our project
I think you have a nice (but small) community already, with your team ready to answer any question quicky. Just keep doing that.
You've seen my recent write-up on adding a raw partition to virtualbox in Sculpt. I thought of making it a blog so others can learn from it too. (I'm not sure if that example makes a nice showcase of the ease of Sculpt :-).
The blogging topic popped up also in the other responses. So let me prematurely leak some information about a concrete idea in this respect:
We plan to start a kind of federated microblogging platform around Genode called "genodians.org", which enables everyone interested in Genode to publish posts. The content will be hosted at individual git repositories and remains completely under control and ownership of each individual author. The site merely aggregates the content from a list of authors and takes care of the layout and style. This will give Genode developers and users an easy way to share insights and experiences. For discussion, each posting will be equipped with a reddit-submission link.
It goes without saying that the site will be hosted on Genode.
If you like, you can already have a peek here:
https://github.com/genodelabs/genodians.org
Would you find the participation in such a federated publishing platform attractive?
- Which areas of Genode would you like to see improved?
How would you possibly contribute to these improvements?
Documentation. (I'll tell you my struggles, you improve the docs).
:-)
- If you imagine a Genode-based system one year in the future,
how would it look like?
My long term dream is Genode on a Desktop.
It has a desktop UI interface, double clicking opens an application in a sandbox. The application has only access to its dependencies and resources like fonts, etc.
However, the application does not have access to the user's files, email, etc., not even network access. It the user wants to open a file, the sandbox detects the application opening a file-browser and an attempt to read /home/<user> and opens a Powerbox. The powerbox is a trusted part of the OS that lets the user select one or more files. Only these are the files that the application can read. (The powerbox is described in HP-Labs paper [1] and other capability security papers on the net).
The desktop is a recurring topic, also highlighted by Ben and Cedric.
I'm probably too opinionated about user interfaces to be very helpful here. So it might take multiple people with different ideas to find a solution that is attractive to actual end users.
As an example for my opinionated view, I remain unconvinced about the powerbox paradigm. Since the file selector (aka powerbox) is triggered by the application, I'm afraid that the user ends up with the mental model that the file selector is part of the application. From the user's subjective view, the application seems to see all the files. In my opinion, the concept is similarly flawed as the UAC mechanism in Windows OS where the software - not the user - decides when an authentication popup dialog is smashed into the user's face. I would much prefer a user-interaction model where the user - not the application - takes the active role. E.g., delegating capabilities via drag-and-drop or your example with spawning a view for a clicked-on file supports the mental model I'm after.
As another indicator for my unfitness for creating a desktop environment, whereas I very much appreciated desktops of old lore like Jinnee (on Atari), BeOS, and RiscOS, for some reason, I tend to avoid interactive desktop applications of today. I spend almost all my time on the command line.
Regarding technicalities like the choice of the toolkit, I wholeheartedly share Cedric's sentiment. Qt is established and powerful. But it's also huge and conservative. With the menu-view approach that I use in Sculpt's Leitzentrale, I enjoy exploring a very different idea. So for my own work, I'd love to follow this unconventional direction. But for building a desktop for Genode, my spleens might just stand in the way.
To sum it up, for pursuing a desktop for Genode in the foreseeable future (the upcoming year), someone other than me should better take the lead.
- Do you have further ideas that would help making Genode relevant
at a larger scale than today?
Since you ask about my Santa list :-)
Make it easy (both in documenting) and code support to port exisiting Linux or Windows software to Genode. It could be a preconfigured Noux instance that provides just enough to start the application. It needs /usr, a bit of /etc, a private var and a powerbox to /home/<user>. It needs a NIC environment with a configurable ingress and egress firewall.
Make it easy for an end-user to create separate of these sandboxes for a single application, so the user can create separate mail-readers for private mail, office mail, etc.
It should be easy to port and effortless to upgrade. Upgrades should be build automatically by a build server so a small number of people can manage a large set of programs. I'm thinking user application such as libreoffice, gimp, photo, video, audio, bookkeeping. Software that doesn't need linux kernel access, drivers or distribution packaging specific things.
I love to run a mail-stack consisting of server programs such as postfix, dovecot, spamassissin, DNS-servers, DNSSEC signers, etc on Genode.
This is a very sensible wish list. Most items come down to removing the friction of porting existing software to Genode. This supports the importance of our SDK. I also agree that the application management you mention is very important.
Regarding the creation of a central build infrastructure, I'm afraid that we won't be able to accommodate you in 2019. I'd rather prefer distributing the responsibility of packages among individual contributors.
Thank you again for the very thoughtful and thought-provoking posting!
Cheers Norman
As a newcomer I should probably step up to the plate, while the 'first impressions' are still fresh in my mind.
I'm still making baby steps; but based on past experience I should hit the ascending part of the learning S-curve soon and become productive. Still have tons of documentation to read before I'll feel half-literate about Genode (it seems to be made of just a handful of ground- breaking principles, but in practice they have such far-reaching consequences that it takes some time to get "fluent" with all ramifications). Hopefully some of the below makes sense anyway. This much is sure -- as we say in french, if Genode didn't exist, someone'd have to invent it *g*.
My whole C.S. perspective (and thus also this message) is haiku centric. So each question & concern will be addressed from that perspective. Hopefully this will bring value to the discussion in at least a few of the cases, if not all. Criticism welcome.
Not much I can add (other than a deep sigh *s*) as to why people don't adopt superior technology in higher numbers.. As a frequent 'early adopter' type I've seen the same thing over time. People stick with the familiar as being "good enough".. Been guilty of that myself too. To be fair though, people who download and run Scult/VC currently are greeted with the leitzencentral, not with a desktop (or what in Haiku I call "a userland"); so the Genode team would need to develop a 'userland' on top of leitzencentral.. In order to go beyond the technological demo, and instead produce a general-purpose OS demo. More on that below.
I'll now 'do my part' with Genode though: will open a micro-blog in a few days about my technical forays into Genode, my mistakes and misunderstandings and how I solve them.. So that others will save time. Including some additional thoughts on this email thread -- to help keep this message short(er than a novel). Once I have more exciting news items in store, I'll give my micro-blog wider circulation, on the haiku forums (they don't have a gigantic readorship, but they are very enthusiastic early-adopter types so likely worth the effort). Waiting until I announce "our flagship app runs on Genode" will carry more weight.
When doing (low key) advocacy in haiku forums, I'll leverage the cultural aspect: in the haiku crowd, they/we adopted Neil Stephenson's saying that we're using the "batmobile" of operating systems: I'll build on that by saying Genode is potentially the "UFO" of operating systems, out of this world :-)
Will give back to the Genode community probably in a similar way I did with Haiku over the years (tickets e.g.). If I had to help out with the "challenges" page later, I suppose it might be with the MAID-SAFE part (decentralized internet). But even if within the edge of my abilities, I would have to get a lot of ducks in a row first. Their litterature mentions developing in something called "electron" (?) whereas I'm most productive in C++ coding against my beloved "Interface Kit".
*******
Finding the way:
The unconquered design space seemed vast, which was both exciting but also - at times - a bit paralyzing.
This might relate to a similar topic I have in mind these days: if e.g. the genode team does not come up with an "official" desktop but leaves it to the community to do (several) implementations, we might end up with the fragmentation that is prevalent in the Linux world: there are many "distros", which I hear are more or less (sometimes "less") compatible with each other.
The way this decision making difficulty has always been addressed in the Be/Haiku world is something like, "choose reasonable defaults", "allow some (limited) easy customisations", and "keep power-user customisation accessible but under the hood".
On a related note, I've also noticed over the years that OS designers are often bad application designers, and vice-versa. I guess the point I'm trying to make is something like, "if you want a rich userland, make an SDK that is a joy to use and help app developers help you" :-)
Of course any SDK design decision has to be weighted with the absolute Genode priorities: security, stability, separation.
*****
Concrete ideas for "spreading the word" on packages:
The single biggest issue I have with the 'depot' system and packages in Genode, is the lack of an introductory paragraph, explaining what each package is about. Small potatoes, right? Power users probably know software names by heart in their sleep? But there are lots of intermediate-style users who don't. It does make a difference.
Do note -- said texts can be simply copy-pasted, as they are in the example below with haiku recipes. There's no word-smith work involved at all. Just a matter of extending the depot format with a new field, and pasting pre-existing texts into that field. Especially important, taking into account that the genode depots are destined to grow into the hundreds of packages (I'd hope) or more, over the years.
On a secondary note, packages should probably be organized by categories, with both a one-liner label and the paragraph-length description mentionned above.
For comparison purposes, here's a similar recipe, as done by Genode and by Haikuports: https://github.com/genodelabs/genode/blob/master/repos/ports/ports/arora.por... https://github.com/haikuports/haikuports/blob/master/www-client/otter-browse...
For how it looks on screen, see the screenshot below.
***
Regarding the need to "get the word out" re. the existence of packages, and the fact that people's efforts tend to remain below-the-radar:
There should be a central depot manager, with a very easy way to add third-party repos (depots).
E.g. in the example I am familiar with, https://www.haiku-os.org/docs/userguide/en/applications/haikudepot.html one just has to invoke Tools > Manage Depots, add a new depot URL, and you may immediately download (and run) whatever is in that depot.
Those URLs are not 'auto discovered' though, one still has to read up about them e.g. in a discussion forum: someone would announce "I've created my own depot with my own app builds, here it is, use at your own risk". Then it's up to you to copy-paste the depot's address in the depot manager. Either that or, using a terminal, I'd type "pkgman add http...some-repo-here", and pkgman downloads the metadata and from now on "pkgman search foo" will return results also from that newly added repo (depot).
***
The "spreading the word" problem goes beyond just packages of course. In the haiku forums I sometimes learn about interesting uses of the OS that get no publicity (i.e. people keep them mostly under wraps) beyond a post here and there on which I stumble by chance.
A decade back, I have memories of that kind of problem being addressed by an enthusiastic supporter setting up a full fledged blog, doing journalist-like work, interviewing people, welcoming guest articles and the like. Maybe something like this would be needed for Genode, by someone who wants to help but cannot code.
******
"Build it and they will come"
Now I have to tread carefully but... A sine qua none condition (for me at least) for helping to make the 'userland' happen, is a good -- strike that, a *great* -- GUI toolkit and app framework.
Qt seems to have become almost an industry standard.. But the documentation alone is dozens of MB big; at least that's how big it shows in the haiku depots here:
pkgman search qt5 ... qt5_devel A comprehensive C++ application development fra qt5_docs A comprehensive C++ application development fra qt5_examples A comprehensive C++ application development fra
Not to mention the runtime itself:
~/Desktop> ll /system/packages/qt5-5.11.2-4-x86_64.hpkg -rw-r--r-- 1 user root 45810197 oct. 20 17:56 /system/packages/qt5-5.11.2-4-x86_64.hpkg
For that, and for other reasons I guess it will not be used by Genode as its "native" API of course. Looking at the nitpicker code seems to confirm that in spades.
To my (biased) eyes, the Be/Haiku API (organized in "kits") compares favorably to what I have seen in my coding life (incl. Java: the JDK is so broad and powerful.. but also of a more difficult access). There are some imperfections (BMessenger in the app kit, some aspects of the InterfaceKit, the Media Kit) that should obviously be fixed by whoever uses that API as inspiration. But it could indeed provide some inspiration to Genode. Developer should be able to tap classes like BButton, BMenu, BListView, automatic font-sensitive layout and the like. Modernized, cleaned up of deprecated calls, wrapped in namespaces instead of polluting the top namespace of course. Reading the source code of demo/scout and of sculpt/ I see lots of familiar things (repaint() is what I call Invalidate(), Window and NitPicker_Connection seem to be what I call a BWindow ..etc) but there's a big gap to fill.
Anyway, work will start soon on 'porting' the kits to Genode.. Will do some micro-blogging about it, post screenshots (still haven't found how to do a screen grab in Sculpt BTW.. now realizing maybe this is un-implemented on purpose, as it poses a security risk?) and the like, in case it's of value.
==================
So in short, history hints at an interesting (but not fail-proof) recipe for adoption: make the OS "sexy" with a "killer app", such that a core of enthusiasts will be drawn to it and refuse to use anything else.. Then make the API "sexy", so that developers will flock to it and develop more apps.. Then build on all that energy to add features that make it attractive to a wider public too, not just early adopters.
Extending on the "genode in one year time" topic:
By year 2019's end I would be very happy if..
- I've gone "cross platform", my apps can be compiled against both Haiku and Genode with little if-def'ing, thanks to adding a compatibility layer 'bridging' the userland API I'm used to and which (to me) is close to perfection. - The systems we are selling are running TT-CC on Genode, customers are happy. - I've started using Genode for non-coding day to day tasks (reading the news with arora? relaxing with Midwinter on dosbox ?). - I've made some progress toward using Genode as a "self hosted" coding system. - I've convinced at least a few in the Haiku community to take a look at Genode, and that working together could be a win-win rather than a competition or a zero sum game. - Tapping that "reservoir" of early adopter types proves to be a good idea, bringing developer and user resources to built momentum (to both OSes). - I wrote some code and uploaded it to chiselapp.com (probably MIT-licensed) that can be useful to others for day to day use of Genode (lots of potential for writing apps, with the right SDK.. With the Be/Haiku API I can build a skeleton app in a day; maybe others would follow suit ..etc).
My 2 euro-cents,
Cedric @ TTS
*Desktop Environment*
I also see having a desktop environment as priority, and I have already made some progress in that direction. I have my plans along those lines, and if anyone wants to discuss details, I'd be happy to do so in another thread on this mailing list. However, I have run into some hurdles that I could use some help with. In particular, to enable easy IPC with libc-based applications, a named pipe (FIFO) VFS plugin would be extremely useful. I wrote one myself, only to discover that the VFS and File_system interfaces do not adequately support partial writes, as needed by such a plugin. With those interfaces reworked, I would happily modify and provide my VFS FIFO plugin code, as well as my Libc pipe plugin that uses it as the backend.
My other main technical hurdle for creating a desktop environment has been converting the init component into a library to allow for easy customization beyond what is possible through configuration files alone. The current method of writing a monitor component to read init's state and modify its configuration works fine for simple cases, but it has performance penalties that can range from mild to severe. The configuration parsing overhead grows with the number of child components, which isn't ideal, but the real show-stopper that I see is dynamic quota management. Each quota upgrade has to wait for a new report, and reports happen at regular intervals, usually every second. Thus multiple quota upgrades, e.g. from opening one or more new web browser windows/tabs can take several seconds. With an init library, such upgrades could be handled very quickly. I can handle this particular issue myself as soon as it becomes a high enough priority.
*Hardware Support*
Aside from a desktop environment, hardware support is another likely show-stopper for new users. I've been working on this myself, and hope to be able to make Genode/Sculpt usable on my new desktop computer in 2019. I accept that support for my hardware is mostly my issue, but improvements in debugging tools and documentation would help a lot with this.
*Debugging*
As for debugging tools, I suggest building both remote (serial and/or (W)LAN) and on-screen GDB component debugging scenarios. This would help with many other goals, and might also help make Genode more attractive to developers.
*Package Management*
I'd like to second and add to what ttcoder said about package management. It would be very helpful to provide a way for Sculpt users to discover and add depots. This would increase usability, and would encourage developers outside Genode Labs to write and port more software for Genode.
Currently, depots don't provide package or launcher lists. Ideally, I would like to see Sculpt have an option to fetch the latest launchers from the repositories selected by the user, allowing them to easily discover components. This would also fix the issue I've seen with Sculpt launchers expecting package versions only available locally on the build machine. Currently, when I modify a component used by Sculpt and update its recipe version, Sculpt expects my new version to be available from depot.genode.org. If a launcher list is fetched from the Genode Labs depot (possibly with a correct default launcher list provided in the Sculpt image), this problem will be solved.
*Performance*
All current Genode scenarios (AFAIK), including Sculpt, run almost everything on a single hardware thread. An SMP balancing component would dramatically increase performance (and likely responsiveness) on modern multi-core CPUs. If nobody else gets around to it first, I'll probably write such a component, but if anyone else here more familiar with Genode's scheduling mechanisms is interested enough, they can probably write a much better implementation than I would.
*Summary*
In short, I'd like to see the following things on the 2019 roadmap: 1. Rework VFS and File_system interfaces to support partial writes, as needed by FIFOs 2. Interactive local and remote GDB debugging scenarios for real hardware 3. Depot and Launcher discovery for Sculpt 4. SMP balancing component 5. Improved documentation
We don't all see eye-to-eye on how a desktop environment should operate, so I'm particularly interested in getting help with (1) so I can make more progress on writing my own, and with (3) so I can make it easily available so others can try it out and give me feedback.
On Mon, Dec 24, 2018 at 8:42 AM ttcoder@netcourrier.com wrote:
As a newcomer I should probably step up to the plate, while the 'first impressions' are still fresh in my mind.
I'm still making baby steps; but based on past experience I should hit the ascending part of the learning S-curve soon and become productive. Still have tons of documentation to read before I'll feel half-literate about Genode (it seems to be made of just a handful of ground- breaking principles, but in practice they have such far-reaching consequences that it takes some time to get "fluent" with all ramifications). Hopefully some of the below makes sense anyway. This much is sure -- as we say in french, if Genode didn't exist, someone'd have to invent it *g*.
My whole C.S. perspective (and thus also this message) is haiku centric. So each question & concern will be addressed from that perspective. Hopefully this will bring value to the discussion in at least a few of the cases, if not all. Criticism welcome.
Not much I can add (other than a deep sigh *s*) as to why people don't adopt superior technology in higher numbers.. As a frequent 'early adopter' type I've seen the same thing over time. People stick with the familiar as being "good enough".. Been guilty of that myself too. To be fair though, people who download and run Scult/VC currently are greeted with the leitzencentral, not with a desktop (or what in Haiku I call "a userland"); so the Genode team would need to develop a 'userland' on top of leitzencentral.. In order to go beyond the technological demo, and instead produce a general-purpose OS demo. More on that below.
I'll now 'do my part' with Genode though: will open a micro-blog in a few days about my technical forays into Genode, my mistakes and misunderstandings and how I solve them.. So that others will save time. Including some additional thoughts on this email thread -- to help keep this message short(er than a novel). Once I have more exciting news items in store, I'll give my micro-blog wider circulation, on the haiku forums (they don't have a gigantic readorship, but they are very enthusiastic early-adopter types so likely worth the effort). Waiting until I announce "our flagship app runs on Genode" will carry more weight.
When doing (low key) advocacy in haiku forums, I'll leverage the cultural aspect: in the haiku crowd, they/we adopted Neil Stephenson's saying that we're using the "batmobile" of operating systems: I'll build on that by saying Genode is potentially the "UFO" of operating systems, out of this world :-)
Will give back to the Genode community probably in a similar way I did with Haiku over the years (tickets e.g.). If I had to help out with the "challenges" page later, I suppose it might be with the MAID-SAFE part (decentralized internet). But even if within the edge of my abilities, I would have to get a lot of ducks in a row first. Their litterature mentions developing in something called "electron" (?) whereas I'm most productive in C++ coding against my beloved "Interface Kit".
Finding the way:
The unconquered design space seemed vast, which was both exciting but also - at times - a bit paralyzing.
This might relate to a similar topic I have in mind these days: if e.g. the genode team does not come up with an "official" desktop but leaves it to the community to do (several) implementations, we might end up with the fragmentation that is prevalent in the Linux world: there are many "distros", which I hear are more or less (sometimes "less") compatible with each other.
The way this decision making difficulty has always been addressed in the Be/Haiku world is something like, "choose reasonable defaults", "allow some (limited) easy customisations", and "keep power-user customisation accessible but under the hood".
On a related note, I've also noticed over the years that OS designers are often bad application designers, and vice-versa. I guess the point I'm trying to make is something like, "if you want a rich userland, make an SDK that is a joy to use and help app developers help you" :-)
Of course any SDK design decision has to be weighted with the absolute Genode priorities: security, stability, separation.
Concrete ideas for "spreading the word" on packages:
The single biggest issue I have with the 'depot' system and packages in Genode, is the lack of an introductory paragraph, explaining what each package is about. Small potatoes, right? Power users probably know software names by heart in their sleep? But there are lots of intermediate-style users who don't. It does make a difference.
Do note -- said texts can be simply copy-pasted, as they are in the example below with haiku recipes. There's no word-smith work involved at all. Just a matter of extending the depot format with a new field, and pasting pre-existing texts into that field. Especially important, taking into account that the genode depots are destined to grow into the hundreds of packages (I'd hope) or more, over the years.
On a secondary note, packages should probably be organized by categories, with both a one-liner label and the paragraph-length description mentionned above.
For comparison purposes, here's a similar recipe, as done by Genode and by Haikuports:
https://github.com/genodelabs/genode/blob/master/repos/ports/ports/arora.por...
https://github.com/haikuports/haikuports/blob/master/www-client/otter-browse...
For how it looks on screen, see the screenshot below.
Regarding the need to "get the word out" re. the existence of packages, and the fact that people's efforts tend to remain below-the-radar:
There should be a central depot manager, with a very easy way to add third-party repos (depots).
E.g. in the example I am familiar with,
https://www.haiku-os.org/docs/userguide/en/applications/haikudepot.html one just has to invoke Tools > Manage Depots, add a new depot URL, and you may immediately download (and run) whatever is in that depot.
Those URLs are not 'auto discovered' though, one still has to read up about them e.g. in a discussion forum: someone would announce "I've created my own depot with my own app builds, here it is, use at your own risk". Then it's up to you to copy-paste the depot's address in the depot manager. Either that or, using a terminal, I'd type "pkgman add http...some-repo-here", and pkgman downloads the metadata and from now on "pkgman search foo" will return results also from that newly added repo (depot).
The "spreading the word" problem goes beyond just packages of course. In the haiku forums I sometimes learn about interesting uses of the OS that get no publicity (i.e. people keep them mostly under wraps) beyond a post here and there on which I stumble by chance.
A decade back, I have memories of that kind of problem being addressed by an enthusiastic supporter setting up a full fledged blog, doing journalist-like work, interviewing people, welcoming guest articles and the like. Maybe something like this would be needed for Genode, by someone who wants to help but cannot code.
"Build it and they will come"
Now I have to tread carefully but... A sine qua none condition (for me at least) for helping to make the 'userland' happen, is a good -- strike that, a *great* -- GUI toolkit and app framework.
Qt seems to have become almost an industry standard.. But the documentation alone is dozens of MB big; at least that's how big it shows in the haiku depots here:
pkgman search qt5 ... qt5_devel A comprehensive C++ applicationdevelopment fra qt5_docs A comprehensive C++ application development fra qt5_examples A comprehensive C++ application development fra
Not to mention the runtime itself:
~/Desktop> ll /system/packages/qt5-5.11.2-4-x86_64.hpkg -rw-r--r-- 1 user root 45810197 oct. 20 17:56/system/packages/qt5-5.11.2-4-x86_64.hpkg
For that, and for other reasons I guess it will not be used by Genode as its "native" API of course. Looking at the nitpicker code seems to confirm that in spades.
To my (biased) eyes, the Be/Haiku API (organized in "kits") compares favorably to what I have seen in my coding life (incl. Java: the JDK is so broad and powerful.. but also of a more difficult access). There are some imperfections (BMessenger in the app kit, some aspects of the InterfaceKit, the Media Kit) that should obviously be fixed by whoever uses that API as inspiration. But it could indeed provide some inspiration to Genode. Developer should be able to tap classes like BButton, BMenu, BListView, automatic font-sensitive layout and the like. Modernized, cleaned up of deprecated calls, wrapped in namespaces instead of polluting the top namespace of course. Reading the source code of demo/scout and of sculpt/ I see lots of familiar things (repaint() is what I call Invalidate(), Window and NitPicker_Connection seem to be what I call a BWindow ..etc) but there's a big gap to fill.
Anyway, work will start soon on 'porting' the kits to Genode.. Will do some micro-blogging about it, post screenshots (still haven't found how to do a screen grab in Sculpt BTW.. now realizing maybe this is un-implemented on purpose, as it poses a security risk?) and the like, in case it's of value.
==================
So in short, history hints at an interesting (but not fail-proof) recipe for adoption: make the OS "sexy" with a "killer app", such that a core of enthusiasts will be drawn to it and refuse to use anything else.. Then make the API "sexy", so that developers will flock to it and develop more apps.. Then build on all that energy to add features that make it attractive to a wider public too, not just early adopters.
Extending on the "genode in one year time" topic:
By year 2019's end I would be very happy if..
- I've gone "cross platform", my apps can be compiled against both Haiku
and Genode with little if-def'ing, thanks to adding a compatibility layer 'bridging' the userland API I'm used to and which (to me) is close to perfection.
- The systems we are selling are running TT-CC on Genode, customers are
happy.
- I've started using Genode for non-coding day to day tasks (reading the
news with arora? relaxing with Midwinter on dosbox ?).
- I've made some progress toward using Genode as a "self hosted" coding
system.
- I've convinced at least a few in the Haiku community to take a look at
Genode, and that working together could be a win-win rather than a competition or a zero sum game.
- Tapping that "reservoir" of early adopter types proves to be a good
idea, bringing developer and user resources to built momentum (to both OSes).
- I wrote some code and uploaded it to chiselapp.com (probably
MIT-licensed) that can be useful to others for day to day use of Genode (lots of potential for writing apps, with the right SDK.. With the Be/Haiku API I can build a skeleton app in a day; maybe others would follow suit ..etc).
My 2 euro-cents,
Cedric @ TTS
Genode users mailing list users@lists.genode.org https://lists.genode.org/listinfo/users
Hi Ben,
I very much appreciate your plans and suggestions. Thanks for chiming in!
On 24.12.18 20:22, Nobody III wrote:
*Desktop Environment*
I also see having a desktop environment as priority, and I have already made some progress in that direction. I have my plans along those lines, and if anyone wants to discuss details, I'd be happy to do so in another thread on this mailing list. However, I have run into some hurdles that I could use some help with. In particular, to enable easy IPC with libc-based applications, a named pipe (FIFO) VFS plugin would be extremely useful. I wrote one myself, only to discover that the VFS and File_system interfaces do not adequately support partial writes, as needed by such a plugin. With those interfaces reworked, I would happily modify and provide my VFS FIFO plugin code, as well as my Libc pipe plugin that uses it as the backend.
These are no fundamental show stoppers and I'm very hopeful that we can remove these limitations soon.
My other main technical hurdle for creating a desktop environment has been converting the init component into a library to allow for easy customization beyond what is possible through configuration files alone. The current method of writing a monitor component to read init's state and modify its configuration works fine for simple cases, but it has performance penalties that can range from mild to severe. The configuration parsing overhead grows with the number of child components, which isn't ideal, but the real show-stopper that I see is dynamic quota management. Each quota upgrade has to wait for a new report, and reports happen at regular intervals, usually every second. Thus multiple quota upgrades, e.g. from opening one or more new web browser windows/tabs can take several seconds. With an init library, such upgrades could be handled very quickly. I can handle this particular issue myself as soon as it becomes a high enough priority.
This is extremely interesting for me because I reached a similar conclusion while creating the user interface for the sculpt manager. In fact, I almost started working on the "librarization" of init in summer. I ultimately decided to defer this work to give it appropriate attention instead of rushing it.
Thanks for encouraging me. I will definitely put the "librarization" of init on the road map!
*Hardware Support*
Aside from a desktop environment, hardware support is another likely show-stopper for new users. I've been working on this myself, and hope to be able to make Genode/Sculpt usable on my new desktop computer in 2019. I accept that support for my hardware is mostly my issue, but improvements in debugging tools and documentation would help a lot with this.
I wonder, would any of these improvements be roadmap-worthy?
*Debugging*
As for debugging tools, I suggest building both remote (serial and/or (W)LAN) and on-screen GDB component debugging scenarios. This would help with many other goals, and might also help make Genode more attractive to developers.
This problem can probably be addressed in two stages. At the first stage, we should do a better job with documenting our existing time-tested debugging and development workflows that are used within Genode Labs. At the second stage, it may be cool to find a way of integrating our GDB monitor easily in Sculpt.
*Package Management*
I'd like to second and add to what ttcoder said about package management. It would be very helpful to provide a way for Sculpt users to discover and add depots. This would increase usability, and would encourage developers outside Genode Labs to write and port more software for Genode.
Currently, depots don't provide package or launcher lists. Ideally, I would like to see Sculpt have an option to fetch the latest launchers from the repositories selected by the user, allowing them to easily discover components. This would also fix the issue I've seen with Sculpt launchers expecting package versions only available locally on the build machine. Currently, when I modify a component used by Sculpt and update its recipe version, Sculpt expects my new version to be available from depot.genode.org. If a launcher list is fetched from the Genode Labs depot (possibly with a correct default launcher list provided in the Sculpt image), this problem will be solved.
These are all very good points. I definitely plan to improve Sculpt in this respect. It is not as simple as importing pre-defined launchers along with (potentially untrusted) depot content though. The launchers should always remain under conscious control by the user. Imagine a malicious depot provider that tricks the user into adding its depot along with a launcher that hands out the user's config fs to a package provided by the malicious depot provider. This could take over the whole system.
I think we can get a convenient experience with a graphical user interface that allows the user to select a package from a given depot provider and then interactively construct a launcher. I'll try this out in the next version of Sculpt. Let's see how you'll like it. ;-)
*Performance*
All current Genode scenarios (AFAIK), including Sculpt, run almost everything on a single hardware thread. An SMP balancing component would dramatically increase performance (and likely responsiveness) on modern multi-core CPUs. If nobody else gets around to it first, I'll probably write such a component, but if anyone else here more familiar with Genode's scheduling mechanisms is interested enough, they can probably write a much better implementation than I would.
That is certainly an interesting topic. For Sculpt, a good first step would be making init's affinity-managing feature available to the deploy configuration. This would give the user an easy way to explicitly assign workload to CPUs. Even though I see the balancing as a subsequent step, you may already start experimenting.
*Summary*
In short, I'd like to see the following things on the 2019 roadmap:
- Rework VFS and File_system interfaces to support partial writes, as
needed by FIFOs 2. Interactive local and remote GDB debugging scenarios for real hardware 3. Depot and Launcher discovery for Sculpt 4. SMP balancing component 5. Improved documentation
That's a very good and realistic list. Thank you for breaking down your requirements to these tangible goals.
We don't all see eye-to-eye on how a desktop environment should operate, so I'm particularly interested in getting help with (1) so I can make more progress on writing my own, and with (3) so I can make it easily available so others can try it out and give me feedback.
I wholeheartedly agree and look forward to this phase of experimentation.
Cheers Norman
Hi Cedric,
your enthusiasm is a beautiful sight. I find your perspective as a newcomer immensely interesting. Thank you very much for joining the discussion. :-)
I'll now 'do my part' with Genode though: will open a micro-blog in a few days about my technical forays into Genode, my mistakes and misunderstandings and how I solve them.. So that others will save time. Including some additional thoughts on this email thread -- to help keep this message short(er than a novel). Once I have more exciting news items in store, I'll give my micro-blog wider circulation, on the haiku forums (they don't have a gigantic readorship, but they are very enthusiastic early-adopter types so likely worth the effort). Waiting until I announce "our flagship app runs on Genode" will carry more weight.
That sounds fantastic. Just out of curiosity, how does the genodians.org idea that I mentioned in my reply to Guide resonate with you?
This might relate to a similar topic I have in mind these days: if e.g. the genode team does not come up with an "official" desktop but leaves it to the community to do (several) implementations, we might end up with the fragmentation that is prevalent in the Linux world: there are many "distros", which I hear are more or less (sometimes "less") compatible with each other.
The way this decision making difficulty has always been addressed in the Be/Haiku world is something like, "choose reasonable defaults", "allow some (limited) easy customisations", and "keep power-user customisation accessible but under the hood".
I agree that we should learn from the experience with the fragmented Linux world and the much more coherent Haiku world.
We ultimately should strive for a agreed-upon official way. Right now, however, I feel that it's still too early to take informed decisions. So may we take 2019 as the year for desktop experimentation?
Concrete ideas for "spreading the word" on packages:
The single biggest issue I have with the 'depot' system and packages in Genode, is the lack of an introductory paragraph, explaining what each package is about. Small potatoes, right? Power users probably know software names by heart in their sleep? But there are lots of intermediate-style users who don't. It does make a difference.
Thank you for making this point.
Regarding the need to "get the word out" re. the existence of packages, and the fact that people's efforts tend to remain below-the-radar:
There should be a central depot manager, with a very easy way to add third-party repos (depots).
E.g. in the example I am familiar with, https://www.haiku-os.org/docs/userguide/en/applications/haikudepot.html one just has to invoke Tools > Manage Depots, add a new depot URL, and you may immediately download (and run) whatever is in that depot.
Those URLs are not 'auto discovered' though, one still has to read up about them e.g. in a discussion forum: someone would announce "I've created my own depot with my own app builds, here it is, use at your own risk". Then it's up to you to copy-paste the depot's address in the depot manager. Either that or, using a terminal, I'd type "pkgman add http...some-repo-here", and pkgman downloads the metadata and from now on "pkgman search foo" will return results also from that newly added repo (depot).
This is actually very similar to what I had in mind for Sculpt.
Qt seems to have become almost an industry standard.. But the documentation alone is dozens of MB big; at least that's how big it shows in the haiku depots here:
[...]
For that, and for other reasons I guess it will not be used by Genode as its "native" API of course. Looking at the nitpicker code seems to confirm that in spades.
In my reply to Guido, I described my personal stance. In short, your impression is correct.
To my (biased) eyes, the Be/Haiku API (organized in "kits") compares favorably to what I have seen in my coding life (incl. Java: the JDK is so broad and powerful.. but also of a more difficult access). There are some imperfections (BMessenger in the app kit, some aspects of the InterfaceKit, the Media Kit) that should obviously be fixed by whoever uses that API as inspiration. But it could indeed provide some inspiration to Genode. Developer should be able to tap classes like BButton, BMenu, BListView, automatic font-sensitive layout and the like. Modernized, cleaned up of deprecated calls, wrapped in namespaces instead of polluting the top namespace of course. Reading the source code of demo/scout and of sculpt/ I see lots of familiar things (repaint() is what I call Invalidate(), Window and NitPicker_Connection seem to be what I call a BWindow ..etc) but there's a big gap to fill.
[...]
Anyway, work will start soon on 'porting' the kits to Genode..
Wow! That is intriguing.
Will do some micro-blogging about it, post screenshots (still haven't found how to do a screen grab in Sculpt BTW.. now realizing maybe this is un-implemented on purpose, as it poses a security risk?) and the like, in case it's of value.
Indeed. However, we have a rough plan to enable features like screenshots and screen recording. Maybe, these are worth considering for the road map?
- I've gone "cross platform", my apps can be compiled against both Haiku and Genode with little if-def'ing, thanks to adding a
compatibility layer 'bridging' the userland API I'm used to and which (to me) is close to perfection.
- The systems we are selling are running TT-CC on Genode, customers are happy.
- I've started using Genode for non-coding day to day tasks (reading the news with arora? relaxing with Midwinter on dosbox ?).
- I've made some progress toward using Genode as a "self hosted" coding system.
- I've convinced at least a few in the Haiku community to take a look at Genode, and that working together could be a win-win rather
than a competition or a zero sum game.
- Tapping that "reservoir" of early adopter types proves to be a good idea, bringing developer and user resources to built momentum (to
both OSes).
- I wrote some code and uploaded it to chiselapp.com (probably MIT-licensed) that can be useful to others for day to day use of Genode
(lots of potential for writing apps, with the right SDK.. With the Be/Haiku API I can build a skeleton app in a day; maybe others would follow suit ..etc).
That reads like a very exciting adventure. :-)
Cheers Norman
Congratulations on a highly successful year!
Thanks for this opportunity to chime in. There's a lot to process here; for brevity's sake I will only share a few initial thoughts at this time.
On 12/23/18 2:46 PM, Norman Feske wrote:
Hallo everyone,
hereby, I'd like to follow up on our tradition to discuss Genode's road map on the mailing list. Let's take the turn of the year to recapture the events of 2018 and make plans for the next twelve months. Please feel welcome to chime in! By mid of January, I'll finalize Genode's official road map and would like to take your input into account.
[snip]
For me personally, this ride was certainly the most rewarding period in Genode's history so far. Now, when looking at the result, I am overwhelmed about what we achieved together! Whenever I have the chance to showing off Sculpt running on my laptop, Sculpt doesn't fail to impress.
I completely agree with this assessment. I think the Year Of Sculpt has tied enough loose ends together that the framework has reached the "plateau" phase, and the focus now shifts to building upon it.
That said, during the course of the year, this positive sense of achievement eventually blended with the rather dull feeling that our hard work remains largely unnoticed outside the inner circle of Genode enthusiasts. The public at large remains quite indifferent (e.g., I was unable to capture the interest of FOSDEM to feature a talk about Sculpt OS on their main track). For a long time, I tricked myself into believing that once we overcome all technical road blocks, Genode will eventually become widely recognized automatically. There was always a technical challenge to take on. With Sculpt, we have reached a point where this excuse doesn't hold anymore. There is no technical piece missing.
I agree that there is no technical piece missing. But as an old refugee from the Amiga, etc., I'm painfully aware that the best technology doesn't always get the mind- or market-share. (The FOSDEM news is surprising, though.)
This leaves me with the question of how to make Genode relevant at a larger scale? Since this is not a technical question, I admittedly struggle to find a good answer. When thinking of the overall leitmotif for 2019, I always come back to this question.
Plan for 2019
I see three directions to help Genode to become more widely recognized.
[snip]
- Widening the audience of Sculpt OS
Sculpt works well, but it is arguably still too hard to use for non-technical users and it lacks a lot of software that users take for granted today. Consequently, we should improve its ease of use. In particular, I'd love to explore the more consequent use of Sculpt's component graph as an intuitive user interface for both the composition and the configuration of components. The ultimate goal would be to eliminate the need for editing any text files. With respect to software availability, I have high hopes in the Genode SDK that we introduced in Genode 18.11. Let's follow this route further.
I don't have any big ideas on this subject, but I just want to support the idea of focusing on the Component Graph. It is a powerful (and unique) feature, and I can't wait to use it for configuration also.
- Fostering the community spirit around our project
Our project enjoys a healthy community of users and contributors. But the community is rather scattered and the various groups work pretty much in isolation. As a result, the work of one group is often invisible to the others, not to speak of any public visibility. Could we possibly establish an instrument that would help each participant to gain more visibility and thereby help the community at large to become more relevant? I'll come up with a concrete proposal for such an instrument soon.
This should really help us hit critical mass. It sounds like some interesting (and unexpected) ideas are percolating already!
One technical element may be a feature of Sculpt that would allow the user to browse the software depots provided by various community members and install/deploy packages by just a few clicks. This would vastly ease the discoverability of the available software and highlight the roles of the respective developers.
I think this is a really important idea also.
[snip]
(I need to put more thought into the questions at the end, so I'll respond to those separately.)
As a final thought, I'm really looking forward to what both Genode Labs and the community do in the upcoming year, and to being part of it!
Happy Holidays!
John J. Karcher devuser@alternateapproach.com
Dear list,
To reflect on this year's progress, its been a pleasure to watch the Genode desktop evolve from a ramshackle init configuration into Sculpt and the launcher system. The lack of interest from the outside world is disappointing, but I blame a pessimistic zeitgeist rather than the lameness of Genode.
As for concrete goals for next year, I have one laptop that I use exclusively with Sculpt, but I also have a NixOS VPS that I use as my always-on OS. Therefore I would like a headless Sculpt that I can use remotely. For this to happen I would imagine a nested sculpt_manager so that I could share the same machine with friends, a Curses-like implementation of menu_view, and some secure networking tunneling.
Its my perception that the greatest impedement to new users and contributers is the singular C++ system API. Perhaps additional Sculpt shells could be implemented in interpreted languages such as Python or Lisp. I've known a few "power-users" that would appreciate booting into their favorite language intepreter, but couldn't care less about how to use Bash. Python seems the surest choice, the digital artists I know work with Python and there was a time when I used IPython daily.
As for my personal roadmap, in the past few weeks I've begun work on a distributed non-hierarchal storage layer. The intention is to support a subset of the de facto file-system semantics with quick composition using the formalism of set logic. This is a pivot of my roadmap from a year ago and reuses code and ideas from another storage project of mine, so progress is swift. The use-case is distributed and redundant storage for things like my permanent music collection as well as the package depot.
Best wishes for next year, Emery
Hi Emery,
As for concrete goals for next year, I have one laptop that I use exclusively with Sculpt, but I also have a NixOS VPS that I use as my always-on OS. Therefore I would like a headless Sculpt that I can use remotely. For this to happen I would imagine a nested sculpt_manager so that I could share the same machine with friends, a Curses-like implementation of menu_view, and some secure networking tunneling.
headless Sculpt is a tempting idea. AFAIK, Christian and Sebastian already took steps into the same direction. I'm looking forward to see what you will come up with.
Its my perception that the greatest impedement to new users and contributers is the singular C++ system API. Perhaps additional Sculpt shells could be implemented in interpreted languages such as Python or Lisp. I've known a few "power-users" that would appreciate booting into their favorite language intepreter, but couldn't care less about how to use Bash. Python seems the surest choice, the digital artists I know work with Python and there was a time when I used IPython daily.
Support for languages other than C++ seems to be crucial for a wider adoption. This was the primary motivation behind the added JAVA support. Also, Johannes' port of Python3 could be leveraged. Language-wise, I sense interest in Go and Javascript (V8) from potential Genode users.
As another idea, with GDC having just become official part of GCC, D could be supported quite easily with the next update of Genode's tool chain. Since I sympathize with D for a long time, I would really like that. It would give me a good excuse to pick up this language more. ;-)
[1] https://gcc.gnu.org/viewcvs/gcc?view=revision&revision=265573
With the goal of Genode adoption, I think the most sensible approach would be supporting such languages on top of Genode's POSIX layer. This way, we could satisfy most use cases without the need for bindings to Genode's native API (which would be a lot of work, for each language).
When speaking of functional languages, you mentioned lisp. But given your work on MirageOS, wouldn't OCAML be a more obvious choice?
As for my personal roadmap, in the past few weeks I've begun work on a distributed non-hierarchal storage layer. The intention is to support a subset of the de facto file-system semantics with quick composition using the formalism of set logic. This is a pivot of my roadmap from a year ago and reuses code and ideas from another storage project of mine, so progress is swift. The use-case is distributed and redundant storage for things like my permanent music collection as well as the package depot.
Is this a topic you'd like to see reflected on the official road map?
Thanks Emery for the fantastic year of sculpting we had together!
Cheers Norman
On Wednesday, December 26, 2018 11:52:47 AM CET, Norman Feske wrote:
With the goal of Genode adoption, I think the most sensible approach would be supporting such languages on top of Genode's POSIX layer. This way, we could satisfy most use cases without the need for bindings to Genode's native API (which would be a lot of work, for each language).
When speaking of functional languages, you mentioned lisp. But given your work on MirageOS, wouldn't OCAML be a more obvious choice?
Porting MirageOS required almost no work to be done in OCaml, but I'm getting some requests for native bindings to things like framebuffer, so I will look into porting the OCaml runtime or reusing the runtime from Mirage.
I mentioned Lisp because I'm interested in using a minimalist language as a shell and to build GUIs from reports. I'm also looking a bit at REBOL and Smalltalk, the critical factor being low effort and low maintenance over features and performance.
As for my personal roadmap, in the past few weeks I've begun work on a distributed non-hierarchal storage layer. ...
Is this a topic you'd like to see reflected on the official road map?
I would like to see storage on the roadmap, but in a broad sense. Better safety when closing Block subsystems, and perhaps a "blessed" API for key-value storage.
Looking forward to next years projects, Emery
On a more practical, less experimental note, I would like to see more work on the SDK, in documentation and integration with third party build systems, and would like to try to build Genode with Clang. Both for the benefit of Cedric, and to define a Clang target that does not use segment registers for TLS.
Cheers, Emery
On 12/26/18 5:52 AM, Norman Feske wrote:
Hi Emery,
As for concrete goals for next year, I have one laptop that I use exclusively with Sculpt, but I also have a NixOS VPS that I use as my always-on OS. Therefore I would like a headless Sculpt that I can use remotely. For this to happen I would imagine a nested sculpt_manager so that I could share the same machine with friends, a Curses-like implementation of menu_view, and some secure networking tunneling.
headless Sculpt is a tempting idea. AFAIK, Christian and Sebastian already took steps into the same direction. I'm looking forward to see what you will come up with.
Headless Sculpt is on my wish list for the year also. In the interest of reducing duplicated work, could you tell us what Christian and Sebastian are working on more specifically?
I was thinking along the lines of a VNC server. (I was able to port the "libvncserver" library without any code changes, but since I'm still working on the application half, I don't know how well it works yet.)
Thanks,
John J. Karcher devuser@alternateapproach.com
Hi John,
sorry that I left your previous posting unanswered. Thank you for following the discussion so closely and for actively participating.
Headless Sculpt is on my wish list for the year also. In the interest of reducing duplicated work, could you tell us what Christian and Sebastian are working on more specifically?
In addition to using Sculpt on our laptops, we longed for a long-running Genode system that carries real-world work loads with a designated uptime of many weeks.
The basis is Sculpt with a custom NIC-router configuration and an automatically deployed variant of the "noux-system" package. In contrast to the regular noux-system package, which hosts a noux session in graphical window, this variant makes the noux session remotely available via SSH. Since the noux-system subsystem has the config fs, report fs, and the Genode partition mounted, the whole system can be remotely inspected and sculpted. E.g., new network services can be added by modifying /config/deploy and config/nic_router on the fly.
As an example for real-world workload, we spawn one or more virtual machines as Genode build servers. Each of those VMs is remotely accessible via SSH also.
If you like, you can have a look at Christian's topic branch [1]. But I'm afraid that it is not too useful without proper documentation, yet.
[1] https://github.com/chelmuth/genode/commits/18.11-plus-buildbot
I was thinking along the lines of a VNC server. (I was able to port the "libvncserver" library without any code changes, but since I'm still working on the application half, I don't know how well it works yet.)
That's great to know! The integration of VNC is not on our radar yet. But I can see how nicely it would complement the scenario. If the VNC server provides a framebuffer and input service, we could easily combine it with a dedicated instance of nitpicker+wm - each of those hosted as subsystems within Sculpt's runtime. Then arbitrary interactive applications could be routed to this GUI stack and thereby become remotely accessible. That would rock! :-)
Cheers Norman
On 12/27/18 11:11 AM, Norman Feske wrote:
Hi John,
[snip]
I was thinking along the lines of a VNC server. (I was able to port the "libvncserver" library without any code changes, but since I'm still working on the application half, I don't know how well it works yet.)
That's great to know! The integration of VNC is not on our radar yet. But I can see how nicely it would complement the scenario. If the VNC server provides a framebuffer and input service, we could easily combine it with a dedicated instance of nitpicker+wm - each of those hosted as subsystems within Sculpt's runtime. Then arbitrary interactive applications could be routed to this GUI stack and thereby become remotely accessible. That would rock! :-)
I hadn't thought of that angle, but that would be powerful!
The library does most of the hard work, so all the custom code has to do is provide a framebuffer and handle input events.
For the initial iteration, I am trying to attach to the main framebuffer, but once that is working, it should be trivial to set it up as you describe.
Thanks!
John J. Karcher devuser@alternateapproach.com
I would like to second the journaling filesystem request, but add it toward the end of the priority list, considering its difficulty (which seems rather high, considering the lack of cross-platform support for most filesystems). My own research into this suggests that most filesystem drivers are integrated more tightly into the OS kernels than ideal. FUSE seems like it should solve this issue, except there don't seem to be any complete, actively-maintained FUSE implementations for mainstream journaling filesystems. That said, ext4 support would definitely be nice.
On Thu, Dec 27, 2018 at 6:30 PM John J. Karcher < devuser@alternateapproach.com> wrote:
On 12/27/18 11:11 AM, Norman Feske wrote:
Hi John,
[snip]
I was thinking along the lines of a VNC server. (I was able to port the "libvncserver" library without any code changes, but since I'm still working on the application half, I don't know how well it works yet.)
That's great to know! The integration of VNC is not on our radar yet. But I can see how nicely it would complement the scenario. If the VNC server provides a framebuffer and input service, we could easily combine it with a dedicated instance of nitpicker+wm - each of those hosted as subsystems within Sculpt's runtime. Then arbitrary interactive applications could be routed to this GUI stack and thereby become remotely accessible. That would rock! :-)
I hadn't thought of that angle, but that would be powerful!
The library does most of the hard work, so all the custom code has to do is provide a framebuffer and handle input events.
For the initial iteration, I am trying to attach to the main framebuffer, but once that is working, it should be trivial to set it up as you describe.
Thanks!
John J. Karcher devuser@alternateapproach.com
Genode users mailing list users@lists.genode.org https://lists.genode.org/listinfo/users
On 12/23/18 2:46 PM, Norman Feske wrote:
Hallo everyone,
hereby, I'd like to follow up on our tradition to discuss Genode's road map on the mailing list. Let's take the turn of the year to recapture the events of 2018 and make plans for the next twelve months. Please feel welcome to chime in! By mid of January, I'll finalize Genode's official road map and would like to take your input into account.
[snip]
Please don't hesitate to share your thoughts. You may consider the following questions:
- What are your ambitions for 2018?
I have goals in three basic categories:
1. Desktop. This has two parts. The easy one is to switch to Genode as my VirtualBox host. But I also would like to have the VirtualBox Guest Additions for using Genode within a VM.
Would this qualify for the roadmap?
If not, I am starting to work on it - it should make a great learning experience (and hopefully a useful blog series), because the various facets touch so many different parts of the system. But I will be slow, so if others are interested, I will be happy to learn from them instead! ;^)
2. Network Appliance. I would like to run an always-on system (probably a Raspberry Pi or something similar) for a small mail server (dovecot, etc.), a file server, and a streaming audio server. This should mainly be a matter of porting the appropriate software.
In order to run this machine headless, I am (slowly) porting/writing a VNC server.
3. Tablet. This is part of a longer-term goal, but I would love to have Genode running in some form on an open-hardware tablet if possible. (I don't even care if it's useful at this stage.)
I know there are efforts toward building mobile front-ends under Qt, but I would be even more interested in working toward a native UI front-end. I really feel that Genode would be a great fit for tablets.
I know tablet-related work will not make the roadmap for 2019, but would it be possible to agree on a semi-official tablet hardware target for the next year or two, for those of us who are interested in pursuing this?
- Which areas of Genode would you like to see improved? How would you possibly contribute to these improvements?
- GUI system configuration (e.g., adding configuration features to the Component Graph, better discoverability for Depot packages), as you already described
- I would feel more comfortable with my VM host running on a journaling FS (e.g., ext3 / ext4). Do others feel the same way?
- Which technical topics do you find tempting?
Almost everything. (This works both for me and against me!) Despite having mentioned ported software several times, my main interests are in exploring the native Genode APIs, proper component design, and verifiable code techniques (e.g., Ada/SPARK, etc.). I'd also like to start experimenting with RISC-V.
- If you imagine a Genode-based system one year in the future, how would it look like?
Good question. I would see it very similar to today's, but with more applications (ported and native), along with the beefed-up GUI configuration tools mentioned earlier.
- Do you have further ideas that would help making Genode relevant at a larger scale than today?
Sorry for the boring answer, but I agree that there aren't really any technical barriers any more, so I think your ideas on highlighting Genode-based products/solutions and Genodians.org are the right direction.
Thanks!
John J. Karcher devuser@alternateapproach.com
Hi John,
- What are your ambitions for 2018?
I have goals in three basic categories:
- Desktop. This has two parts. The easy one is to switch to Genode as
my VirtualBox host. But I also would like to have the VirtualBox Guest Additions for using Genode within a VM.
Would this qualify for the roadmap?
the topic fits well with the overall theme that is emerging. I would probably speak more generally about fostering the integration of Genode with existing protocols and infrastructure. E.g., Emery's recent work with 9p [1] would also fall in this category. Also, the use of Xpra would be fitting [2].
[1] https://github.com/ehmry/ninep [2] https://github.com/zorodc/genode-world/commits/xpra
That said, I would not go as far as making definite promises.
Speaking of the Guest Additions for Genode, I can imagine a nice picture where Sculpt could offer a VBox shared folder as storage option in addition to the detected AHCI/NVMe devices and the RAM fs. This way, the user could host the Sculpt storage in a plain directory on a non-Genode host system and deploy Sculpt within VirtualBox by accessing those files. The integration would be quite seamless.
If not, I am starting to work on it - it should make a great learning experience (and hopefully a useful blog series), because the various facets touch so many different parts of the system. But I will be slow, so if others are interested, I will be happy to learn from them instead! ;^)
Cool!
- Tablet. This is part of a longer-term goal, but I would love to have
Genode running in some form on an open-hardware tablet if possible. (I don't even care if it's useful at this stage.)
I know there are efforts toward building mobile front-ends under Qt, but I would be even more interested in working toward a native UI front-end. I really feel that Genode would be a great fit for tablets.
I know tablet-related work will not make the roadmap for 2019, but would it be possible to agree on a semi-official tablet hardware target for the next year or two, for those of us who are interested in pursuing this?
From my point of view, the most sensible choice would be a device based
on the NXP i.MX SoC. This SoC is used by the Librem phone [3]. So there may be potential of synergies. More importantly, i.MX comes with proper official documentation and Genode already has a number of drivers for several i.MX peripherals.
[3] https://puri.sm/products/librem-5/
Cheers Norman
On 1/3/19 7:54 AM, Norman Feske wrote:
Hi John,
- What are your ambitions for 2018?
I have goals in three basic categories:
- Desktop. This has two parts. The easy one is to switch to Genode as
my VirtualBox host. But I also would like to have the VirtualBox Guest Additions for using Genode within a VM.
Would this qualify for the roadmap?
the topic fits well with the overall theme that is emerging. I would probably speak more generally about fostering the integration of Genode with existing protocols and infrastructure. E.g., Emery's recent work with 9p [1] would also fall in this category. Also, the use of Xpra would be fitting [2].
[1] https://github.com/ehmry/ninep [2] https://github.com/zorodc/genode-world/commits/xpra
Oh dear, those are interesting also. It's getting hard to keep up with it all! ;^)
That said, I would not go as far as making definite promises.
Speaking of the Guest Additions for Genode, I can imagine a nice picture where Sculpt could offer a VBox shared folder as storage option in addition to the detected AHCI/NVMe devices and the RAM fs. This way, the user could host the Sculpt storage in a plain directory on a non-Genode host system and deploy Sculpt within VirtualBox by accessing those files. The integration would be quite seamless.
Yes, that would be very convenient.
I haven't thought this all the way through, but do you have any thoughts on handling mounted directories along side partitions, e.g., in Leitzentrale? I assume there would have to be some changes, but how major would they be?
-- John J. Karcher devuser@alternateapproach.com
Hi folks,
here are my three cents: I am very proud of what our team achieved last year. Sculpt is up and running. I did setup the long running version and it has not crashed so far and even if it did, I would see it as a chance to mature Genode. We are humans after all and we do make mistakes, but thanks to the various kernels, and thus different runtime behavior, we are able to trigger bugs very fast - enabling us to fix them quickly. There is more robustness now than I ever imagined.
Otherwise, I see two concurrent paths: 1. POSIX compliance and 2. The Genode way. For porting software (1) is very convenient. But when writing new applications I would like to see more of (2). Most likely we will have to traverse both paths.
Besides doing Java and getting it ready to work properly, I would like to rework the whole audio stack. That would be the third iteration, but sometimes things emerge very slowly. So, if anyone here has experiences with low latency audio, a good book or homepage to read, or just knows how to design this stuff ... let me know.
Even tough Norman's talk got rejected from the FOSDEM organizers, we do enjoy a very successful run at the Microkernel devroom [1], we will organize it this year, where we actually had to reject more talks than I am comfortable with. But there are only eight hours ...
I think by now we have a very small but fine community,
Sebastian
[1]: https://fosdem.org/2019/schedule/track/microkernels_and_component_based_os [2]: https://en.wikipedia.org/wiki/Genode
Hi Sebastian,
nice to see you joining the discussion!
I agree that the audio topic certainly deserves more attention. But before re-approaching the topic again, I'd like to stress the need for proper measurement tools that help us to interpret the observed behavior correctly instead relying on assumptions as an aid.
I am thinking of a graphical front end for Genode's tracing feature that
1. can be readily deployed in Sculpt, 2. lists all components and their threads, 3. allows the user to select one or multiple threads from the list, 4. records a trace (by pressing a button), and 5. shows the timeline visually.
I'd love to have such an application at hand, also for many other performance-related developments. And I think that implementing such a tool using Qt seems quite doable.
Sebastian, would you support prioritizing the creation of such a tool over the actual audio work?
Cheers Norman
On 28.12.18 16:02, Sebastian Sumpf wrote:
Hi folks,
here are my three cents: I am very proud of what our team achieved last year. Sculpt is up and running. I did setup the long running version and it has not crashed so far and even if it did, I would see it as a chance to mature Genode. We are humans after all and we do make mistakes, but thanks to the various kernels, and thus different runtime behavior, we are able to trigger bugs very fast - enabling us to fix them quickly. There is more robustness now than I ever imagined.
Otherwise, I see two concurrent paths: 1. POSIX compliance and 2. The Genode way. For porting software (1) is very convenient. But when writing new applications I would like to see more of (2). Most likely we will have to traverse both paths.
Besides doing Java and getting it ready to work properly, I would like to rework the whole audio stack. That would be the third iteration, but sometimes things emerge very slowly. So, if anyone here has experiences with low latency audio, a good book or homepage to read, or just knows how to design this stuff ... let me know.
Even tough Norman's talk got rejected from the FOSDEM organizers, we do enjoy a very successful run at the Microkernel devroom [1], we will organize it this year, where we actually had to reject more talks than I am comfortable with. But there are only eight hours ...
I think by now we have a very small but fine community,
Sebastian
Genode users mailing list users@lists.genode.org https://lists.genode.org/listinfo/users
Hey Norman,
On 1/3/19 2:07 PM, Norman Feske wrote:
Hi Sebastian,
nice to see you joining the discussion!
I agree that the audio topic certainly deserves more attention. But before re-approaching the topic again, I'd like to stress the need for proper measurement tools that help us to interpret the observed behavior correctly instead relying on assumptions as an aid.
I am thinking of a graphical front end for Genode's tracing feature that
- can be readily deployed in Sculpt,
- lists all components and their threads,
- allows the user to select one or multiple threads from the list,
- records a trace (by pressing a button), and
- shows the timeline visually.
I'd love to have such an application at hand, also for many other performance-related developments. And I think that implementing such a tool using Qt seems quite doable.
Sebastian, would you support prioritizing the creation of such a tool over the actual audio work?
Of course, I think the need for performance analyzing tools, may it be throughput or latency measurements, becomes more important with each passing year. Especially when looking at the growing complexity of the overall Sculpt system.
So, I am with you on this one,
Sebasitan
On 28.12.18 16:02, Sebastian Sumpf wrote:
Hi folks,
here are my three cents: I am very proud of what our team achieved last year. Sculpt is up and running. I did setup the long running version and it has not crashed so far and even if it did, I would see it as a chance to mature Genode. We are humans after all and we do make mistakes, but thanks to the various kernels, and thus different runtime behavior, we are able to trigger bugs very fast - enabling us to fix them quickly. There is more robustness now than I ever imagined.
Otherwise, I see two concurrent paths: 1. POSIX compliance and 2. The Genode way. For porting software (1) is very convenient. But when writing new applications I would like to see more of (2). Most likely we will have to traverse both paths.
Besides doing Java and getting it ready to work properly, I would like to rework the whole audio stack. That would be the third iteration, but sometimes things emerge very slowly. So, if anyone here has experiences with low latency audio, a good book or homepage to read, or just knows how to design this stuff ... let me know.
Even tough Norman's talk got rejected from the FOSDEM organizers, we do enjoy a very successful run at the Microkernel devroom [1], we will organize it this year, where we actually had to reject more talks than I am comfortable with. But there are only eight hours ...
I think by now we have a very small but fine community,
Sebastian
Genode users mailing list users@lists.genode.org https://lists.genode.org/listinfo/users
Hi,
My main observation is that every aspect of modifying of and integrating with Genode should be made as easy as possible. Due to the nature of Genode there are many things that are different from commonly known solutions and I think especially those parts should be made more easy to handle. Below I enumerate those that I find important.
Norman Feske norman.feske@genode-labs.com writes:
Please don't hesitate to share your thoughts. You may consider the following questions:
- What are your ambitions for 2018?
Recently I've purchased RPI 3B+ - it is my first arm device and currently all aspects of making Genode work on it are on my radar.
I've seen comments from Alexander Weidinger about his attempts and looked briefly at his work and he started from copying all rpi specific code to rpi2 version. Given that until now there is about eight variants of Raspberry Pi I felt that this shouldn't be the way to go. So I started checking how Linux is handling different raspberry pi boards (and other ARM variants) and found out about device tree and its usage. So my first ambition is to integrate device tree usage into Genode. I think that work could make much EASIER integration of devices with Genode.
Unfortunately my attempts currently are a seqence of different problems/surprises which I try to overcome one step after another and it isn't going as smooth as I expected. Currenlty I'm forced to work with assembly code after almost 25 years (and it was for 80286). I'll write more about it in another thread along with questions that maybe someone will be able to answer.
- Which areas of Genode would you like to see improved? How would you possibly contribute to these improvements?
Bare metal
In my current work with RPI (I'm attempting to make base-hw work) it would be very useful to have some very early debugging facility - probably serial output. Given that currently I'm not even able to successfully execute code from crt0.s I'd like to see it (this output) initialized there although maybe I want too much. It took me two days (due to my lack of experience) to have any diagnostics from that code - I can enable and disable leds attached to gpio pins :-). So early output from initialization enabled in some standard way (build flag?).
Porting
Writing initialization files that define access to all resources and routing rules is something completely Genode specific and requires good understanding design principles behind capabilities and resource providers, etc. Even though I think I understand concepts well I find it hard to analyze all dependencies (at least it was hard for me in Sculpt). I think that for "typical" ported software it can require only some of "standard" resources and configuration could be made by clicking options what given application is allowed to access. Something similar to privileges on current phones. I know that Genode solution is better but only internally and currently is more complicated. Maybe this could become part of SDK. Maybe even SDK should be "generated" for concrete list of required resources and consequently APIs?
Inspectability
It should be easy to inspect running system internals. At least after enabling some debug mode if that can make system slower or less safe. I know that you know that it is important but I wanted to emphasize that.
Sources
I find difficult to find relevant places in sources when I look for something. Currently I heavily use find with grep:
find ~/projects/genode/genode \ -path "*/.git/*" -prune -o \ -path "*/depot/*" -prune -o \ -path "*/contrib/*" -prune -o \ -path "*/build/*" -prune -o \ -path "*.run" -prune -o \ -type f -exec egrep --color -nH -e 'pattern' {} +
but it would make life much easier if there was some integration with some IDE (maybe there is) that would provide some notion of project where files would be automatically included to project if they are used to build concrete target. I'd like to be able to limit search only to files used e.g. for "run/log for rpi with kernel hw". Although in this case I have no idea how to achieve this :-) Some additional output from make as build system is built on top of that?
Knowledge base
Although there is a great book and greate articles about different design desicions are made and informations about each release it's hard to quickly find informations about what is currently working/tested, what is supposed to work and what not and why.
I'd like to have some overview on state of particular configurations as some matrix with functions, boards and kernels.
High level examples:
* support for rpi is only for base-hw and foc, but not for Nova. * virtualization is supported only on Nova (I think)
Low level:
* real case I stumbled some time ago - tried some filesystem tests and for linux target there where errors for 'unlink' (if I correctly remember) that it is not implemented. In such case I'd like to know if it is not implemented because nobody did that yet or there are some obstacles that make it hard or mayby dangeroues (it was definitely about deleting so maybe). I know that such detailed information cannot be accurate if it was maintained by a human (probably I've messed this sentence) but it could be somehow assembled from results of automatic tests that should be made public (maybe they are but I just haven't found it) and marked with some status about expected results for each configuration.
- Which technical topics do you find tempting?
Currently ARM devices as it is completely new subject for me.
- Do you have further ideas that would help making Genode relevant at a larger scale than today?
For enthusiasts: make changes that would allow to contribute more quickly - allow to make doing everything easier :-)
For business I think it is important to find some niches where Genode could shine. Probably that should be use cases:
* not very complex - because otherwise initial cost and risk would be too high.
* where security is important.
Additionally I think that some comparison of efficiency of some workloads (e.g. against linux) would be needed for business to consider using Genode as a base for solutions. I'd consider efficiency problems as a high risk of using Genode if I wasn't presented some results. This information would be useful for everyone although I don't know if current results wouldn't be somewhat discouraging. I hope not.
Tomasz Gajewski
Hi Tomasz,
I'm working on implementing a base-hw kernel on a Cortex A53 platform (not Rpi) which is what the Rpi 3x is based on. So, that means a significant amount of work to convert from the existing Genode code supporting the armv7 architecture to support the armv8 architecture. I'm focused initially on only the aarch64 instruction set, but will later add support for aarch32. This is a retirement project for me so my progress is not driven by any great need and is slow. After about six months of casual working I'm at the stage of debugging a virtual memory translation table enhancements need to support 64-bit memory (a new Level-0 table in the page table code in lpae.h). The means all the assembly code changes required have been made (and about 12 other areas of change) and I clean compile a from a log run script.
For debugging at this level, you really only have two choices: Either delve into the ARM hardware debugging facilites (CoreSight) or insert small pieces of code to poke variable values or a trace number into a currently unused region of memory. Code like:
using ull = unsigned long long; ull volatile * debug_ptr = (ull *)0x91000600; *debug_ptr = MAX_ENTRIES; debug_ptr += 0x01; *debug_ptr = BLOCK_SIZE_LOG2; debug_ptr += 0x01;
if your booting into uboot to start up, you can inspect the memory region through uboots' md command. Breakpoints can be implemented with the assembly code line: asm volatile("hlt #01\n"); . Objdump gives you the assembly code listing if you need it.
I sincerely hope Genode does not implement a version of the Linux DT. It is an error prone approach developed many years ago in the embedded world. There are more elegant, syntax checkable, configuration description approaches available today.
Let me know if I can help with any compile/link issues you have, I probably came across most of them in my work to-date. I'm using a Linaro tool chain for aarch64. There is a separate one for aarch32 which I don't currently use.
Bob Stewart
On 1/2/19 5:22 AM, Tomasz Gajewski wrote:
Hi,
My main observation is that every aspect of modifying of and integrating with Genode should be made as easy as possible. Due to the nature of Genode there are many things that are different from commonly known solutions and I think especially those parts should be made more easy to handle. Below I enumerate those that I find important.
Norman Feske norman.feske@genode-labs.com writes:
Please don't hesitate to share your thoughts. You may consider the following questions:
- What are your ambitions for 2018?
Recently I've purchased RPI 3B+ - it is my first arm device and currently all aspects of making Genode work on it are on my radar.
I've seen comments from Alexander Weidinger about his attempts and looked briefly at his work and he started from copying all rpi specific code to rpi2 version. Given that until now there is about eight variants of Raspberry Pi I felt that this shouldn't be the way to go. So I started checking how Linux is handling different raspberry pi boards (and other ARM variants) and found out about device tree and its usage. So my first ambition is to integrate device tree usage into Genode. I think that work could make much EASIER integration of devices with Genode.
Unfortunately my attempts currently are a seqence of different problems/surprises which I try to overcome one step after another and it isn't going as smooth as I expected. Currenlty I'm forced to work with assembly code after almost 25 years (and it was for 80286). I'll write more about it in another thread along with questions that maybe someone will be able to answer.
- Which areas of Genode would you like to see improved? How would you possibly contribute to these improvements?
Bare metal
In my current work with RPI (I'm attempting to make base-hw work) it would be very useful to have some very early debugging facility - probably serial output. Given that currently I'm not even able to successfully execute code from crt0.s I'd like to see it (this output) initialized there although maybe I want too much. It took me two days (due to my lack of experience) to have any diagnostics from that code - I can enable and disable leds attached to gpio pins :-). So early output from initialization enabled in some standard way (build flag?).
Porting
Writing initialization files that define access to all resources and routing rules is something completely Genode specific and requires good understanding design principles behind capabilities and resource providers, etc. Even though I think I understand concepts well I find it hard to analyze all dependencies (at least it was hard for me in Sculpt). I think that for "typical" ported software it can require only some of "standard" resources and configuration could be made by clicking options what given application is allowed to access. Something similar to privileges on current phones. I know that Genode solution is better but only internally and currently is more complicated. Maybe this could become part of SDK. Maybe even SDK should be "generated" for concrete list of required resources and consequently APIs?
Inspectability
It should be easy to inspect running system internals. At least after enabling some debug mode if that can make system slower or less safe. I know that you know that it is important but I wanted to emphasize that.
Sources
I find difficult to find relevant places in sources when I look for something. Currently I heavily use find with grep:
find ~/projects/genode/genode \ -path "*/.git/*" -prune -o \ -path "*/depot/*" -prune -o \ -path "*/contrib/*" -prune -o \ -path "*/build/*" -prune -o \ -path "*.run" -prune -o \ -type f -exec egrep --color -nH -e 'pattern' {} +
but it would make life much easier if there was some integration with some IDE (maybe there is) that would provide some notion of project where files would be automatically included to project if they are used to build concrete target. I'd like to be able to limit search only to files used e.g. for "run/log for rpi with kernel hw". Although in this case I have no idea how to achieve this :-) Some additional output from make as build system is built on top of that?
Knowledge base
Although there is a great book and greate articles about different design desicions are made and informations about each release it's hard to quickly find informations about what is currently working/tested, what is supposed to work and what not and why.
I'd like to have some overview on state of particular configurations as some matrix with functions, boards and kernels.
High level examples:
- support for rpi is only for base-hw and foc, but not for Nova.
- virtualization is supported only on Nova (I think)
Low level:
- real case I stumbled some time ago - tried some filesystem tests and for linux target there where errors for 'unlink' (if I correctly remember) that it is not implemented. In such case I'd like to know if it is not implemented because nobody did that yet or there are some obstacles that make it hard or mayby dangeroues (it was definitely about deleting so maybe). I know that such detailed information cannot be accurate if it was maintained by a human (probably I've messed this sentence) but it could be somehow assembled from results of automatic tests that should be made public (maybe they are but I just haven't found it) and marked with some status about expected results for each configuration.
- Which technical topics do you find tempting?
Currently ARM devices as it is completely new subject for me.
- Do you have further ideas that would help making Genode relevant at a larger scale than today?
For enthusiasts: make changes that would allow to contribute more quickly - allow to make doing everything easier :-)
For business I think it is important to find some niches where Genode could shine. Probably that should be use cases:
not very complex - because otherwise initial cost and risk would be too high.
where security is important.
Additionally I think that some comparison of efficiency of some workloads (e.g. against linux) would be needed for business to consider using Genode as a base for solutions. I'd consider efficiency problems as a high risk of using Genode if I wasn't presented some results. This information would be useful for everyone although I don't know if current results wouldn't be somewhat discouraging. I hope not.
Tomasz Gajewski
Genode users mailing list users@lists.genode.org https://lists.genode.org/listinfo/users
Hi,
On 02.01.2019 11:22, Tomasz Gajewski wrote:
Recently I've purchased RPI 3B+ - it is my first arm device and currently all aspects of making Genode work on it are on my radar.
I agree that better support for the Raspberry Pi would be useful, considering how easy and affordable it is to get one of these devices and to start experimenting with it. Maybe we could also provide an SD card image with Sculpt and some showcase applications and games for interested users to try out.
On a different note, there will probably be a tool chain update planned for this year and this would also be a good opportunity to improve its compatibility with GNU Autotools based configuration of ported software, for example by conforming to the established target-triplet form (x86_64-genode-gcc instead of genode-x86-gcc), by supporting the '-l' linker flag or by knowing where to find the correct linker script by default.
Christian
Hello,
thanks Norman for launching this enjoyable discussion (and keeping up the tradition). Reading the postings helped much to get in proper reflective mood to wrap up the past months and organize my ideas for 2019.
On Sun, Dec 23, 2018 at 08:46:48PM +0100, Norman Feske wrote:
The Year of Sculpt
Looking back at the past year I'm very proud of our achievements. Not only that our team at Genode Labs conducts its work on Sculpt every day, the OS is also mature enough to be useful for enthusiasts. Also, each step on the Sculpt timeline widened the horizon of what could be achieved with this approach in the future.
As the previous posters, I organize my ideas for 2019 into the suggested categories, because I'm confident they make up a perfect frame for our planning.
- Widening the audience of Sculpt OS
In this category I see two important aspects, which were mentioned in the discussion already.
Documentation
We should keep up the series of articles about Sculpt that we founded in 2018 and not only update the development progress but also document the Sculpt OS as a whole (starting from our talk slides). Ideally, the Sculpt documentation will evolve into an up-to-date book about this specific Genode OS accompanying the Foundations.
Applications
As we can't implement all required applications natively using the Genode API by ourselves, the porting effort of existing applications must be reduced. The first take of the Genode SDK was released in November and we should intensify our efforts in this regard. I see the SDK primarily as environment for POSIX applications that integrate seamlessly into the system by means of the libc and the VFS. Additional SDK modules could be added on top of the base SDK, for example, a Genode Qt SDK, which integrates with the Qt Creator/qmake/cmake easily.
Christian Prochaska also mentioned a tool-chain update with additional effort for better autotool support. Maybe we should merge the tool chain into the SDK? Also, we may consider musl as (Linux-compatible) C runtime for relevant platforms, namely x86-32/64, ARMv7/8, and RISC-V. The Linux compatibility of the libc may help to enable more applications which lack universal UNIX support or suffer from poor FreeBSD adaptation.
Further, I'd like to support the proposition that the enablement of more language runtimes attracts a broader diversity of developers.
- Fostering the community spirit around our project
I'll definitely have my part on genodians.org. Therefore, I plan to document progress on the network-appliance projects mentioned below and other stuff about Genode. Furthermore, I wonder if Genode could become more visible on open source gatherings beyond FOSDEM or even conferences/exhibitions and are all ears for suggestions how to achieve this.
- Marketing of Genode-based products
Even if "products" is a rather heavy term, I plan to push Genode-based deployments to address tasks in our networking environment at Genode Labs. I see the Sculpt runtime as a key to easily develop appliances for a broad range of applications like Sebastian's build server, secured NAS, IPv4 router/firewall, or even John Karcher's always-on mail/file/audio-streaming server. This goal involves many improvements and adaptions of components (e.g., NIC router) and Sculpt (e.g., support for head-less scenarios that even lack a framebuffer).
I'm looking forward to an exciting Year 2019!
Regards
Hi guys,
It's very interesting and motivating to read all your ideas regarding the Roadmap 2019 and see the enthusiasm spread throughout the community! So, here's what I came up with:
Implementation
* I'm currently experimenting with SPARK on Genode and really like to go deeper into if, and if yes, how and where SPARK could be beneficial in Genodes main repositories.
Testing
* Doing nightly tests with Sculpt * More starvation/resource-leak/bad-client tests for critical servers
Documentation
* Finding native speakers to translate the Wikipedia article to further languages
* IMO it's an issue that not every component brings a _comprehensive_ README with information like the whole user interface description, a complete list of services needed and provided and real-live examples.
* Furthermore, I would suggest the integration of a mechanism in the Sculpt component graph that lets you access a README for a specific component with merely one click. I know there's the foundations book but I have the feeling that this inline help would be much more convenient in a lot of cases.
* It would be nice to have some info in the READMEs more systematic, like config attributes (XML Schemes) or services required/provided. This information could be used by development tools to support programmers as well as in the integration tools like the component graph to guide users.
* To address a point that Tomasz brought in regarding working/tested documentation: How about also giving information about that in a systematic way in READMEs? This way, in systems like Sculpt we could, for instance, mark software according to its development state when listed in the package manager or in the "Add"-menu of the component graph. It would also enable filtering software lists and thereby ease, for instance, browsing software cross-platform.
* I really like the idea of genodians.org and think we should definitely keep trying to connect more an more Genode-related media maintained by users to give them a kickoff in tooling and publicity.
Development
* I strongly share Normans feelings about the importance of the SDK and a way to browse/install software conveniently in Sculpt.
Cheers, Martin
Hello everyone,
thanks to all participants in our vivid road-map discussion!
I have now condensed the discussion into the official road map:
https://genode.org/about/road-map
Even though I left out many details, I hope that the result still captures our common ambitions well.
Cheers Norman
-
Bob Stewart -
Christian Helmuth -
Christian Prochaska -
Emery Hemingway -
Guido Witmond -
John J. Karcher -
Martin Stein -
Nobody III -
Norman Feske -
Sebastian Sumpf -
Tomasz Gajewski -
ttcoder＠netcourrier.com