On 12/24/21 07:37, ttcoder@netcourrier.com wrote:

My thoughts in no particular order :

After several years, I still can't get over how modular Genode is and how take-no-prisoners no-compromise its policy is re. clean design and best practices. Seeing that kind of dedication keeps me motivated when I feel tired IRL.

I feel exactly the same way.

If I get a chance before the end of 2022, I'll port (or help with porting) some packages, including easy ones like 1) fossil or 2) jam, but especially the fledgling-but-already-awesome 3) V-lang language. Could help with day to day developer life, who knows.

As to my pet project, in the next couple weeks I'm about to wrap up the one-before-last 'ticket' that's a pre-requisite before my software runs on Genode, so it looks like 2022 (can't believe it's 2022 already... time flies) will be the year I can resume selling my software and can scale back the "odd jobs" (freelance website programming etc), which will be nice for sure.

That's very exciting news - please keep us informed!

Not to mention, it'll open the perspective of an 'alternative' desktop for Genode -- though I dare not say I'll get to that before 2023, given my velocity history <g>.

This is very intriguing also. If you need an experimental/alpha tester for this, you know who to ask. :^)

Thanks!

John J. Karcher devuser@alternateapproach.com

Gesendet: Freitag, 24. Dezember 2021 um 14:47 Uhr Von: "Uwe" geno.de@public-files.de An: users@lists.genode.org Betreff: Aw: Roadmap 2022

...

Gesendet: Donnerstag, 23. Dezember 2021 um 19:05 Uhr Von: "Norman Feske" norman.feske@genode-labs.com An: users@lists.genode.org Betreff: Roadmap 2022

Besides the Pinephone, I am planning to simplify and solidify Genode's base framework by gradually removing complexity (like C++ exceptions [7]), increasing the strictness of the coding style (like the aftermath of [8]), and attending the most-neglected corners of our issue tracker [9].

[7] https://genodians.org/nfeske/2021-11-26-attempt-no-exceptions

Please do not remove exceptions. https://www.stroustrup.com/P0976-the-evils-of-paradigms.pdf At least leave in enough to test the handling of them regularly by using them. The reason for this request is that exceptions are the only open types in c++. That means I could add to them as an afterthought. That may be needed in mutiplexers, that could be configured with additional cases.

If you need to change between error return and exceptions consider https://ideone.com/m2ZfHN the lippincott function.

...

[8] https://genodians.org/nfeske/2021-12-07-wconversion [9] https://github.com/genodelabs/genode/issues

...

In mid of January, I am going to update the official road map.

Cheers, Norman

-- Dr.-Ing. Norman Feske Genode Labs

https://www.genode-labs.com · https://genode.org

Genode Labs GmbH · Amtsgericht Dresden · HRB 28424 · Sitz Dresden Geschäftsführer: Dr.-Ing. Norman Feske, Christian Helmuth

---

Genode users mailing list users@lists.genode.org https://lists.genode.org/listinfo/users

---

Genode users mailing list users@lists.genode.org https://lists.genode.org/listinfo/users

Hi Alexander,

it in interesting to learn more about the context of your work with Go.

You said that you are not a Go programmer yourself. But to you happen to have users of your Go runtime to get their feedback?

Like namespaces based isolation (read: ability to have same names/id’s/etc in different domains for objects and anything provided by the Genode to user apps, together with additional related API). At least for app snapshotting, migration and persistency this is «the must». They are not so necessary for containers themselves, there are support of some platforms without it, as well without dedicated layered FS (unions and similar like auFS/btrfs/zfs/etc - while it is good to have it).

I think the two aspects OS-level virtualization and snapshotting/persistency should best be looked at separately.

Regarding OS-level virtualization, Genode's protection domains already provide the benefit of being light-weight - like namespaces when compared to virtual machines - while providing much stronger isolation. Each Genode component has its private capability space after all with no sharing by default. Hence, OS-level virtualization on Genode comes down to hosting two regular Genode sub systems side by side.

The snaphotting/persistency topic is not yet covered. But I see a rather clear path towards it, at least for applications based on Genode's libc. In fact, the libc already has the ability to replicate the state of its application as part of the fork mechanism. Right now, this mechanism is only used internally. But it could be taken as the basis for, e.g., serializing the application state into snapshot file. Vice versa, similar to how a forked process obtains its state from the forking process, the libc could support the ability to import a snapshot file at startup. All this can be implemented in the libc without changing Genode's base framework.

That being said, there is an elephant in the room, namely how POSIX threads fit into the picture. How can the state of a multi-threaded application be serialized in a consistent way? That would be an interesting topic to research.

These are just my thoughts from the top of my head. I'm looking forward to see your steps into this direction.

Cheers Norman

Hi Norman, thanks for answer. Some thoughts below.

it in interesting to learn more about the context of your work with Go.

couple years ago I do have a project to implement docker support on some new microkernel OS. as a starting point I need to try this ontop of Genode (because project owners do not have OS sources available for me that time). Initially I think that I need to implement integration of partitioning with containers, like having single container per OS partition. Finally I need to support only a set of containers inside single OS partition with provided by OS linux emulation layer. Later I found that main problem was not in fixing kernel, drivers /etc - problem that all docker support was implemented using golang. So, I need to port a couple millions LOC written on Go (AKA docker support) , and start from porting runtime of golang itself (another 1.2m LOC on golang and C, which touch ALL available syscalls/services/etc of underlying OS, and requires good understanding of small differences between different OS APIs). I have this work half-done for genode and then switches back to main OS (where I finished later everything - port of runtime and port of docker inside single partition).

In this moment I returned back from old project and want to finish undone work for genode as a testbed for initial idea of integration of docker and OS partition in 1 <-> 1, probably using libc port. I do not have formal customers for it, just my curiosity.

You said that you are not a Go programmer yourself. But to you happen to have users of your Go runtime to get their feedback?

About users - not sure, recently I publish my patches and do not have any feedback yet. Go actively used by the developers, I hope that it will be easy to bring some application software to Genode (e.g. different handling of http staff). Anyway, current lack of customers will not stop me from the second part of my research.

I have to compile and run inside genode docker support code - couple millions golang lines which heavily use system OS API including posix and dialects.

So, I consider to have "go build» to run natively inside genode inside qemu. First step was to have TCP support inside integrated with golang - done. Next will be gccgo native (non-cross) support.

...

Like namespaces based isolation (read: ability to have same names/id’s/etc in different domains for objects and anything provided by the Genode to user apps, together with additional related API). At least for app snapshotting, migration and persistency this is «the must». They are not so necessary for containers themselves, there are support of some platforms without it, as well without dedicated layered FS (unions and similar like auFS/btrfs/zfs/etc - while it is good to have it).

I think the two aspects OS-level virtualization and snapshotting/persistency should best be looked at separately.

Regarding OS-level virtualization, Genode's protection domains already provide the benefit of being light-weight - like namespaces when compared to virtual machines - while providing much stronger isolation. Each Genode component has its private capability space after all with no sharing by default. Hence, OS-level virtualization on Genode comes down to hosting two regular Genode sub systems side by side.

General note. Initially, when in swsoft/virtuozzo/parallels we do create a container-based Os virtualisation (we call it "virtual environment" in 2000), we assume 3 main pillars (having in mind that we want to use it as a base for hosting in hostile environment with open unlimited access from Internet to containers):

1. namespace virtualisation not only to isolate resources but to be able to have the same pid and related resources in different containers (for Unix we think about emulation of init process with pre-defined pid 1, at least)

2. file system virtualisation to allow COW and transparent sharing of the same files (e.g. executable for apache between 100’th of containers instances) to preserve kernel memory and objects space (oppose to VM where you can’t able to share efficiently files and data structures between different VM instances) - key to containers high scalability and performance , and for docker it also a key for "encapsulation of changes" paradigm. Sharing using single kernel instance is a wide paradigm - it allow to optimise kernel structure allocation, resources sharing, single instance of memory allocator/etc.

3. ALL resources limitations on per-container base (we call this userbeancounters) which prevent any attempts to make DoS attack from one container to another or to the host.

every container initially should be like remotely-accessible complete instance of linux with root access and init process, but without ability to load own device drivers. we do implement this first for linux, later for FreeBSD/Solaris (partially), Windows (based on hot patching technique and their Terminal Server), consider mach/MacOS Darwin (just experiments). for linux and Windows it was a commercial-grade implementation and still used by millions of customers.

Now all these features (may be except file systems while zfs/brtfs/overlayfs/etc have something similar) are became a part of most of commercial OS available on the mass market. IMHO they can be cheaply implemented from the very beginning of OS kernels development - everything was in place, except understanding why this is necessary outside of simple testing environments for developers. May be it is also a time for Genode to think into this direction?

returning to Genode.

reasons for existence of namespaces (ns) is not only isolation, it is a bit wider. One thing is an ability to disallow «manual construction» of objects ids/handles/capabilities/references/etc to access something which should not be visible at all.

For example, in ns isolated containers I should not be able to send signal to arbitrary process by name (pid in our case) even if it exists in the kernel. Or vice versa - use some pre-defined processes ids to do something (e.g. unix like to attach orphans to pid 1, and later try to enumerate them, this should be emulated somehow during user-level software port, e.g. for linux docker this is important).

in case of genode probably I can created and keep capability (with data pointer inside) and perform some operations with it, if I do store it somewhere. if this capability be virtualised then we will have additional level of control over it (by creation of pre-defined caps and explicit level of limitations even if it is intentionally shared during initialisation procedure which could be a part of legacy software being ported to genode).

For better understanding - use case. imagine that you want to port application which use 3-d party library which do init some exotic file descriptors. good example is a docker itself - when you exec process inside docker container you typically don’t want to have your main process opened descriptors including stdin/out (typically it’s achieved via CLOEXEC flag - but lets consider its absence, you can just don’t know about descriptors existence). Technically it is a code in your application while it was initialised by 3-d party library linked to it, and you do not have easy way to control it.

ns implementation has simple rules to maintain group isolation, and it is not considered as unnecessary even in linux kernel with their own capabilities set. I think that namespace is a convent way to handle legacy-related question and it worth to have in Genode level where you already have wrappers around native low level kernel calls.

And, for snapshotting (see comment below) this is a must - I need to re-create all objects with the same id even if they do exists in other threads/sessions/processes because they could be stored in «numerical form» inside user thread memory.

as for file system like overlayfs - not sure, I assume that it is possible to port some known fs into genode, while it is not a first priority task (Windows docker does not have it).

for resource counting and limitations - I do not tackle this topic at all for genode.

The snaphotting/persistency topic is not yet covered. But I see a rather clear path towards it, at least for applications based on Genode's libc. In fact, the libc already has the ability to replicate the state of its application as part of the fork mechanism. Right now, this mechanism is only used internally. But it could be taken as the basis for, e.g., serializing the application state into snapshot file. Vice versa, similar to how a forked process obtains its state from the forking process, the libc could support the ability to import a snapshot file at startup. All this can be implemented in the libc without changing Genode's base framework.

That being said, there is an elephant in the room, namely how POSIX threads fit into the picture. How can the state of a multi-threaded application be serialized in a consistent way? That would be an interesting topic to research.

I think we can follow the ideas developed for CRIU patch for linux [1], no need to invent something too complex: It can freeze a running container (or an individual application) and checkpoint its state to disk. The data saved can be used to restore the application and run it exactly as it was during the time of the freeze. Using this functionality, application or container live migration, snapshots, remote debugging, and many other things are now possible.

in short, they do utilise existent linux kernel sys calls like ptrace and add very small subset of absent ones to enumerate process-related objects [2]. This does not mean that you need to have ptrace - it just used as a kind of auxillary interface to obtain info about processes, it can be implemented in different ways.

to stop (freeze) a set of related processes (tree) even with POSIX they use feature (can be considered as a part of ns virtualisation) known as cgroups [3]: The freezer allows the checkpoint code to obtain a consistent image of the tasks by attempting to force the tasks in a cgroup into a quiescent state. Once the tasks are quiescent another task can walk /proc or invoke a kernel interface to gather information about the quiesced tasks. Checkpointed tasks can be restarted later should a recoverable error occur. This also allows the checkpointed tasks to be migrated between nodes in a cluster by copying the gathered information to another node and restarting the tasks there.

Seems that similar to ns and cgroup features should be a first part/base of checkpoint/restore implementation. OF course, part of serialisation related to fork/libc, as you mention, also could be an another pillar.

In general, I think that to implement snapshotting we need 1. freeze set of threads (or make them COW, e.g. for memory changes) 2. enumerate threads 3. enumerate related objects/states (e.g. file descriptors/pipes/etc) 4. enumerate virt mem areas, and related «shared resources» between threads 5. enumerate network stack/sockets states (a bit different beast) 5. dump everything

for restore we need not only create some objects with the same numerical values of id (even same memory layout), we need to have an api to force every object to have the same content/state and related security/ns, and force (restore) «sharing» and parent/child relations if any of object between different threads/processes/sessions/etc

related to this topic: we also need to be able to bring some drivers to the same state, because during checkpointing/restore we assume external connections in the known state (e.g. imagine video drivers and application which draw on the screen. content of their video memory is a part of its state while it is not stored in application). Probably it is related to restartable drivers feature (and related fault tolerance questions).

Note: By the way, now of the key problem of CRIU patch in this moment is inability to restore graphical screen for X/etc. We can restore related sockets, while protocol parts themselves which need to be repeated are not known when you order checkpoint.I think there are no people now available who know real X protocol details necessary for the operations… but this is different story not directly related to genode questions.

[1] https://criu.org/Main_Page [2] https://criu.org/Checkpoint/Restore [3] https://www.kernel.org/doc/Documentation/cgroup-v1/freezer-subsystem.txt

Sincerely, Alexander

Hi Johannes,

Gesendet: Donnerstag, 06. Januar 2022 um 13:51 Uhr Von: "Johannes Schlatow" johannes.schlatow@genode-labs.com An: users@lists.genode.org Betreff: Re: Roadmap 2022

Hi Alexander, Thus, when thinking about running a container on Genode, I noticed we have most ingredients already in stock since a Genode component is a sandboxed process with its resource quota and local namespace.

Regarding the file system virtualisation, we have the VFS and can even host a shared VFS in a dedicated server component. I'm not sure about a copy-on-write feature, though.

In my (current) point of view, enabling containerised workloads on Genode probably requires three ingredients:

1. Implementing additional VFS plugins for mounting container images, overlays, and cow functionality.
2. Adding missing plugins for special file nodes in devfs, sysfs or procfs. This highly depends on what the particular container process expects, though.
3. Implementing a container runtime for Genode that sets up a sub-init to launch the container process with the appropriate VFS and helper components according to the container configuration.

I think noux already implements all that is needed to run containers. Only the file format is different. So you need a program to translate between these. It could even be integrated into the download tool. So the Server sends the canonical format an the tool stores in translated form.

Cheers Johannes

Hi Johannes,

As far as I understood, a container is basically a filesystem image with some configuration of how to set things up. The container runtime will read the configuration and prepare everything depending on the target system before it launches the process defined by the container.

yes, except the fact that if we want all infrastructure being applicable (all these container management tools like kubernetis/etc) then we need to support appropriate API on the host level which, in turn, requires 1. golang support 2. some containertization features like controllable communications between containers 3. cow on file system base (optional).

After that, the started container is merely a standard process that has been encapsulated with namespaces, cgroup and other isolation mechanisms. The process performs syscalls just like a non-containerised process would do.

definitely so. while there are something «around» process - e.g. a way to execute process inside existing container, error handling/etc

Thus, when thinking about running a container on Genode, I noticed we have most ingredients already in stock since a Genode component is a sandboxed process with its resource quota and local namespace.

partially (mostly) yes, while some infrastructure around are missing

Regarding the file system virtualisation, we have the VFS and can even host a shared VFS in a dedicated server component. I'm not sure about a copy-on-write feature, though.

main idea of sharing was scalability. What happens if we will try to run the same executable (e.g. Apache with 10M code) in 2 different containers ? if this is VM like containers then with big probability we will not share the code pages (e.g. for Windows some of the DLL code pages do contain variables, so, even for the same executable pages content could be different). imagine you have 100 containers with apache. they will eat 10m x 100 = 1GB of ram just for code pages. while could only 10m

Another problem is a memory distribution. imagine that you have, for example, kernel object descriptor of 25 bytes, and a lot of them. if you have single os image then you have single memory allocator (if this is linux- slab/slub/etc) when you could store object instances related to different containers on the same memory page.

if you have own copy of everything - again, you will not just only inefficiently waste the kernel memory space for unused tails, but also will spend memory bandwidth/etc.

if we want to share effectively files they should be visible with the same «inode» (or similar, depending upon a file system) then instance of file system should be visible from every container via single FS instance. it should handle COW as well.

While again, FS is not first priority, e.g. widows version of native (non-linux) containers do not have any dedicated FS (use just slow layers onto of NTFS).

In my (current) point of view, enabling containerised workloads on Genode probably requires three ingredients:

1. Implementing additional VFS plugins for mounting container images,

overlays, and cow functionality.

not 100% necessary, you can run (slowly!) just tar, this is supported by docker

2. Adding missing plugins for special file nodes in devfs, sysfs or

procfs. This highly depends on what the particular container process expects, though.

this require to have an answer for question: what model you want to utilize? if linux one - then yes, /proc /sys /dev and so on. But, my porting experience show that this is tricky way - if you try to pretend to be linux - you need to have full size emulation of linux facitities, api/etc. Every time when you try to make something it force you to use linux tricky way. as for golang runtime I choose NetBSD as a base as less advanced, adding some features from other systems. E.g. it could use some exotic TCP or fcntl options to force container to do something.

IMHO there are no simple answer, always pro/con trade-off.

3. Implementing a container runtime for Genode that sets up a sub-init

to launch the container process with the appropriate VFS and helper components according to the container configuration.

again, same question like above. typically you could use something like tinit (tiny init) for such purposes, while it is not mandatory and for many apps it will work without. but you need to understand what will be with child processes inside container, who will own them after death of parent (or this should not happens and you can use app itself as pseudo init).

Re. 3., I'm uncertain whether this is best approached from scratch or by porting existing runtimes such as runc or crun. The downside of the

it is too big to be rewritten. I do consider both ways and found that for me easier to port, I have really huge porting experience.

latter approach is that it requires us to provide all the Linux management interfaces such as cgroup, namespaces, etc. and map these to Genode sub-init configuration. Parsing the container configuration and applying the appropriate actions directly seems more natural to me at the moment.

imho it is easier to tailor (cut) appropriate code from existing golang sources for different platforms then to write it from the scratch.

@Alexander: What do you think are the major road blocks for running a first container image on Genode?

it depends upon definition of «run». for me to run docker container on my first project on some new OS I ported 3 coupled apps including runc, containerd and docker cli. they are connected via unix sockets. then you have api to create/delete/manipulate containers.

I don’t see any principal roadblocks while think that having some compatible with linux API for ns and cgroups will help a lot while again -non mandatory.

we need ti invent the way how to manipulate set of processes (signals are virtually absent in terms of Unix on Genode), connect and reconnect pipes (unix sockets), and run chroot() - like (anything like this available on Genode?)

I assume a lot of work related to tuning/utlization of some features used by container manipulation (e.g. file descriptor flags, options related to exec/fork and similar) while there are Windows containers without them definitely can exists (they use native Windows API).

but all problems probably solvable. IMHO. may be we need to implement something inside libc port as well.

Alexander

Hi, Johannes, thanks for ideas sharing.

...

definitely so. while there are something «around» process - e.g. a way to execute process inside existing container, error handling/etc

In my view, a container maps in Genode to a subsystem (comprising a VFS server and other infrastructure). Hence, starting a process inside an existing container should be pretty straightforward.

…

I believe sharing code pages in Genode would be a matter of the parent component who sets up the children's address spaces. Currently, we use the sandbox library for this.

I assume that this sharing implemented on the edge between file system and page cache (at least this is true for Linux/unix and Windows)

in that case we need to have single VFS server with own cache/page mapping for files being shared between different instance of containers (subsystems), not only for children’s? is it true for current implementation of [single VFS+FS server] <=> [[multiple subsystems]]?

...

if we want to share effectively files they should be visible with the same «inode» (or similar, depending upon a file system) then instance of file system should be visible from every container via single FS instance. it should handle COW as well.

I think this is exactly what a VFS server component does. It provides a File_system service to which multiple components can connect.

do you have an example of implementation of combination of VFS+FS server and a set of subsystems (at least 2) connected to the single server instance?

...

...

...

3. Implementing a container runtime for Genode that sets up a

sub-init to launch the container process with the appropriate VFS and helper components according to the container configuration.

again, same question like above. typically you could use something like tinit (tiny init) for such purposes, while it is not mandatory and for many apps it will work without. but you need to understand what will be with child processes inside container, who will own them after death of parent (or this should not happens and you can use app itself as pseudo init).

Sorry, I was not crystal clear in my terminology. By "sub-init", I meant Genode's init component that we use for spawning subsystems. Honestly, I haven't spent any thought on multi-process containers. I had the impression that most commonly a container merely runs a single process, i.e. does not spawn new processes on its own.

this is not exactly true. while initially containers was developed with such an idea, later it became more complex

imagine build container - it run make inside (which fork gcc which in turn fork cpp then cc1 then as then ls and may be ar/ranlib/objcopy/etc) and if you have make -j4 - then make will run 4 parallel compilation (if Makefile allows). they must use the same file system instance (volume) to process intermediate files like .c -> .i -> .s -> .o -> .out...

returning back to genode and subsystems. how it is implemented in this moment , e.g. how make (native) can run inside genode noux? probably it use libc fork()/exec()/etc together with pthreads? do the processes (threads in genode terminology) share something bu default after start? can I run a bunch of «processes» inside genode in single subsystem which share some services from outside (like VFS+FS)?

more interesting question - do they share single swapping to disk service if need? or every subsystem has own pager with own page file?

I think that if I will have examples of implementation of these features in the way which will be suitable for genode subsystem-per-container model then we can have docker on genode relatively fast.

Sincerely, Alexander

Hi Johannes,

...

in that case we need to have single VFS server with own cache/page mapping for files being shared between different instance of containers (subsystems), not only for children’s? is it true for current implementation of [single VFS+FS server] <=> [[multiple subsystems]]?

I'm afraid you lost me. In Genode, a file system is accessed via a File_system session. This session provides an API for typical file/directory operations (open/create, symlink, watch, move). File content is transferred via a packet stream (cf. Genode Foundations Book). A VFS server would access e.g. a persistent file system and deliver its contents to its own clients, which could be separate subsystems. I see two places for caching here: First, the VFS server could cache some file content so that it can be delivered to multiple clients without transferring it from the block device multiple times. Second, the clients can perform their own (local) caching. Since I'm not familiar with the internals implementation though, I don't know to what extend such mechanisms are already implemented.

another question here is a way how to provide an access rights (different for different clients) which will use the same FS server. Do you have something like ACL applied to file system?

or it just borrowed from, e.g. ext2 implementation (need to provide /etc/passwd and /etc/groups together with chmod/chown as separate files and utils)? I see in ssh_server.run some inline implementation of similar files...

Do you have own or external auth mechanism, like LDAP server/Radius/etc?

Sincerley, Alexander

Hi Johannes, thanks for clarification

...

Do you have something like ACL applied to file system?

The VFS server has no notion of users or ACL, yet it is possible to provide different parts of the VFS to different clients/sessions. This is achieved by specifying a <policy>, which sets the root directory for the session and whether write operations are permitted. This basically provides per-directory access control.

...

Do you have own or external auth mechanism, like LDAP server/Radius/etc?

Natively, there is no notion of users in Genode. Instead, access control is conducted on a per-session basis. On the one hand, the init component takes care of routing a particular session request to a certain child component (or parent). On the other hand, the child providing the service may further allow the specification of session policies (as mentioned above) so that different clients receive different permissions.

I want to integrate Genode + low level OS (e.g. nova or sel4) to be integrated with existing environments related to containers. Most of this things do assume some kind of per-user control.

may be I can ask my question in different format - what is the best way of such integration??

Simplistic approach I saw in the implementation of ssh_server.run - they just create «inline» plain text fake user+password (with non-fake crypto key).

in standard unix/linux/etc during container creation I use some credentials for docker and for files access simultaneously. docker suggest to keep it outside (while can hold inside) [1]: ... Credentials store The Docker Engine can keep user credentials in an external credentials store, such as the native keychain of the operating system. Using an external store is more secure than storing credentials in the Docker configuration file.

To use a credentials store, you need an external helper program to interact with a specific keychain or external store. Docker requires the helper program to be in the client’s host $PATH.

This is the list of currently available credentials helpers and where you can download them from:

• D-Bus Secret Service: https://github.com/docker/docker-credential-helpers/releases • Apple macOS keychain: https://github.com/docker/docker-credential-helpers/releases • Microsoft Windows Credential Manager: https://github.com/docker/docker-credential-helpers/releases • pass: https://github.com/docker/docker-credential-helpers/releases …

To implement docker container I need to answer for both questions : - what access control and credentials I will use for underlaying file system (it can generate endless problems if treat in wrong way - like failed scripts execution/etc), and - how access control info should be provided to docker itself (at least in form of root/non root users, or keychains/etc), see [1].

Note: In theory we can have 3-d question to be answered - "how container will store secure 3-d party data", but it is different from first two above and can be answered by applications later. As I understand encrypted block storage vbe is a movement to this direction?

So, my question is: I do not like an idea to keep event for tests «a+rwx» mode for files and plain text user/passwords stored in run files. Are there anything better than that available for prototype, or I need to keep this unsecure approach in this moment for both file system and container permissions?

Note: I understand that internally system based on Genode will be significantly more secure by itself. Anyway we need to consider the whole system, including external clients used by users to access and manage the system.

[1] https://docs.docker.com/engine/reference/commandline/login/

Sincerely, Alexander

Hello Alexander,

Gesendet: Sonntag, 02. Januar 2022 um 15:12 Uhr Von: "Alexander Tormasov via users" users@lists.genode.org An: "Genode users mailing list" users@lists.genode.org Cc: "Alexander Tormasov" a.tormasov@innopolis.ru Betreff: Re: Roadmap 2022

In this moment for me somehow obvious that to support things above I need a support in Genode some parts of generic virtualisation. Like namespaces based isolation (read: ability to have same names/id’s/etc in different domains for objects and anything provided by the Genode to user apps, together with additional related API). At least for app snapshotting, migration and persistency this is «the must». They are not so necessary for containers themselves, there are support of some platforms without it, as well without dedicated layered FS (unions and similar like auFS/btrfs/zfs/etc - while it is good to have it).

Note: I suspect that having namespaces virtualisation on the kernel level will give some additional advantages for Genode even in terms of security. Same like happens in Linux with our proposals related to openvz/namespaces/userbeancounters (after some time it became clear that they are necessary for modern OS). This is relatively cheap in implementation and overhead points of view. Did you considers this option as a part of Genode future?

I think your mental image of genode is at odds with reality. Genode is not another Linux (or comparable OS). There are no processes. The most process-like entity in genode is the protection domain(PD). But the PD is already virtualized. That makes it like a container with only one process. To share a namespace between two PDs you have to explicitely configure that sharing. There are libraries that help you with that and contain even some default naming of capabilities. But mostly you will have to setup the sharing of a namespace by copying capabilities between PDs.

Alexander

...

23 дек. 2021 г., в 21:05, Norman Feske norman.feske@genode-labs.com написал(а):

Dear Genode community,

it is the time of the year again to reflect and make plans for the foreseeable future. Hereby, I'd like to kick off our traditional brainstorming about Genode's road map for the year ahead of us.

---

Genode users mailing list users@lists.genode.org https://lists.genode.org/listinfo/users

Hi Christian,

my main work topics of 2021 have been the enabling of multimedia support in the Falkon web browser, the tool chain update and as of latest the ongoing optimization of audio support for VirtualBox 6.

your contributions in 2021 had been nothing short of amazing. I still remember my eyes popping out watching you debug a race condition on Genode/ARM64 in JIT-compiled code generated by Chromium from a JavaScript blob delivered by youtube.com. What a super-human level of patience! :-)

I'm pretty impressed by Falkon's multimedia abilities on Genode already, even though some optimization is still needed and something I intend to continue working on in 2022, especially with the planned execution on the PinePhone in mind. More PinePhone-related personal work topics could involve porting of the 'Morph' mobile web browser to Genode and exploring the possibilities of porting QML-based Sailfish OS or Ubuntu Touch applications and making it possible to build those with Goa.

There is so much in your plans to look forward to. I think the latter point has a lot of potential from a community perspective because - if it works out as anticipated - it would nicely bridge the gap between app developers and the Genode-based phone. So our platform won't need to stay insular.

Cheers Norman

Hi Stefan,

Nevertheless, this line of work prevented me to achieve the unification of the platform driver API across different architectures in time. Although it is almost accomplished, the original goal was to finish it in 21.05. Inwardly, I dreamed of having a working Wifi card in Sculpt on the MNT Reform 2 at this time, which is not the case.

there should be no reason for any regrets. Plans change, in your case for very good reasons. Your work on DDE-Linux evolved into a much more profound project than I imagined, and I'm super happy and proud about it. Also, I think that our vision of the role and design of the platform driver was too uncertain to make definite a-priory estimations. The design that emerged over the past year vastly exceeds the old - in many places ad-hoc - nature of the traditional platform driver.

I encourage you to keep up following the goal to make Genode on the MNT-Reform fully capable, including Wifi. This is of course out of selfishness. I love the MNT-Reform laptop. I'd love it even more with wireless connectivity on Sculpt :-)

I would not call them road blocks, but while analyzing the requirements of a re-newed platform driver implementation, I was wondering whether we still need to support ancient kernels like Pistachio, or the old Fiasco. Apart from the dependencies on quite old hardware of these kernels, a lot of workloads are not possible to run on them. We have to ignore the nightly kernel faults etc. of especially Pistachio. Therefore, I would suggest to retire these kernels within 2022.

This line of reasoning is certainly valid because those old kernels won't play any practical role for real-world Genode scenarios.

However, I don't share the sentiment to retire them because I still get a lot of value from the diversity of kernels, when compared to the costs of maintenance. Let me share what those values are:

- When working on the base-framework infrastructure, I'm forced to apply all changes in a way that is workable with all the different kernels. This (arguably artificial) problem challenges me to re-evaluate changes repeatedly and from different perspectives. The repetition has become some kind of Zen when working on the base framework. Sometimes, I stop applying a change mid-way for the better because applying one idea 8 times makes one critically evaluate the idea 8 times in a row. ;-)

In your case - looking at IRQ support on PIC vs. IOAPIC - the (useless at surface level) support for plain PIC-based systems is additional work. But if we manage to cover it in an elegant way, the result will be more general and sustainable. So I invite you to accept this challenge.

- The diversity of kernels fuzzes our system and uncovers issues - often related to generic code - that would remain unknown to us without the broad spectrum of hardware/kernel combinations regularly exercised. Yes, we see kernel assertions triggered by a few tests that won't ever be fixed. But on the other hand, when assessing a regression, the cross-kernel test results often allow me to cross- correlate the behavior, and form a mental model of what could be wrong. E.g., if I see a test failing on all kernels that use s mapping database, I can form a hypothesis.

- During the development of low-level features - the fork mechanism of the libc comes in mind - it is best to start with the simplest kernel possible. E.g., on OKL4 I can readily use the Genode::raw mechanism, and can easily track capabilities across PDs because they are just global numbers. Such developments would be far more complicated on a real capability-based kernel.

- There are still some areas where we haven't yet learned all the lessons the kernels can teach us. Specifically, there still exist unexplained performance anomalies between kernels, e.g., when looking at network throughput. The performance tends to be influenced by details of the kernels scheduling. We should strive to refine Genode such that those anomalies eventually disappear, making the framework as deterministic as possible against the whims of the various kernel schedulers. Until that point is reached, it is good to keep the anomalies clearly visible and reproducible.

Other (sometimes surprising) effects occur because of different approaches when comes to cache attributes, or the choice of time source (PIT as scheduing timer).

- I actually used the L4/Fiasco kernel debugger in 2021. Not a strong argument, but still worth noting.

- Being the one with most time invested refining the base framework over the past year, I found the costs of maintenance of the legacy kernels pretty much negligible compared the grand scope of changes. I'd be fine with kicking this can further down the road.

So my position to keep the kernels supported is not merely some weird nostalgic attachment that I feel for those relics, but I see actual technical value of keeping them around.

Cheers Norman

Hello Johannes,

thank you for sharing your experience and plans. Both facets, the FPGA line of work and the tracing topic are exciting. Especially the latter fits perfectly into the recurring theme of optimization and deep performance/latency analysis. It could play a vital role to get the browser running at acceptable performance on the Pinephone for example. I also have QoS challenges - in particluar low-latency audio processing - in the back of my head for 2022. Holistic and precise event tracing would be a godsend.

Regarding the FPGA topic, I am going to familiarise myself with the Xilinx tools in order to build custom bitstreams. In particular, I'd like to investigate solutions for guarding DMA via custom FPGA logic. The main idea is to emulate the register interface of DMA-capable devices in the FPGA and having a SystemMMU-like access control mechanism controlled by the platform driver.

That's a cool idea, especially as IOMMUs don't seem to be commonplace in the ARM world yet.

By the end of the year, I envision having sophisticated and easy-to-use tracing tools for Genode that we are able to routinely use for debugging and performance analysis.

On the FPGA topic, custom programmable logic will integrate nicely into Genode and there will be corresponding documentation for how to augment a resource-limited embedded Genode system with custom hardware accelerators.

What a beautiful outlook! :-)

Cheers Norman

Hello Colin,

thank you for sharing your experiences with us.

On Tue, Jan 04, 2022 at 07:12:36PM -0500, Colin Parker wrote:

So, encouraged, by one of this year's themes, I started trying to port the linux ath9k driver. Unfortunately, I did not experience the same success as others. For example the new lx_emul/lx_kit is conceptually very nice, and I was excited about it, but I'm running into snags because I don't think it's completely implemented yet for x86. It's also a little sad to me that there is no way to take what is, under linux, a platform indepedent USB driver, and make a platform indepedent Genode component, because the porting process seems to depend on the architecture. I guess the actionable step here is for me to finish porting the driver, but it would be great to know what the plans for intel are, if the dde_linux is under development now, or if I need to hack it myself to make it work under x86.

You're right, currently the new dde_linux/lx_kit was developed for ARM64 only. Nevertheless, beginning from today we start to update our x86 driver base including framebuffer, usb host and wifi driver to use the new approach, and concurrently update the Linux kernel source to a recent version. We are confident to have first working results at the end of March. Then when an updated version of the Intel WiFi driver is running successfully that is using the new dde_linux/lx_kit, it might be a good time to step in and port the Atheros card.

Best regards Stefan

Best wishes to all for the new year, CP

---

Genode users mailing list users@lists.genode.org https://lists.genode.org/listinfo/users

Hi John,

On 13.01.22 05:18, John J. Karcher wrote:

On 12/23/21 13:05, Norman Feske wrote:

1. Get Sculpt running as a daily driver (preferably on Spunky ;^) ). The

VBox 6 update should make this transition easier for me.  I will be asking for advice on this topic as I go along.

It's cool to read that you're interested in using Spunky! But just as a side note: Spunky is still limited to the feature set of base-hw and that means no hardware virtualization on x86 so far. This is especially important when it comes to you plan of using Vbox6.

Cheers, Martin

Hi John

There is a clear motivation to add virtualization support to base-hw because there are, AFAIK, several people at Genode Labs who would like to use Sculpt on base-hw as their daily desktop system.

However, so far, I think none of us has started implementing it because other projects were more pressing. Maybe this will change in 2022 - at least it was mentioned during the roadmap discussions - but it's not set in stone.

Cheers, Martin

On 30.01.22 06:25, John J. Karcher wrote:

Thanks for the tip - I either didn't know that, or knew it and forgot.

Just curious, are there any plans to add virtualization support to base-hw (and Spunky) in the foreseeable future?

In either case, that means I now have two separate goals: using Sculpt as a daily driver, and running Sculpt/Spunky . . . just because I like it. ;^)

Thanks!

John J. Karcher   devuser@alternateapproach.com

---

Genode users mailing list users@lists.genode.org https://lists.genode.org/listinfo/users