Correction: "Mail" is probably a quite bad example, as it relies heavily on extended attributes. There are probably many candidates all over the difficulty spectrum (from easy to hard) that would make more sense to port. Unless someone came up with a hack to wrap xattr's into something else, ouch.

Cedric

On 31.03.2019 23:08, ttcoder@netcourrier.com wrote:

Correction: "Mail" is probably a quite bad example, as it relies heavily on extended attributes. There are probably many candidates all over the difficulty spectrum (from easy to hard) that would make more sense to port. Unless someone came up with a hack to wrap xattr's into something else, ouch.

Cedric

xattrs? So, Haiku uses Linux terminology? OS/2 calls them "EA's". What are extended attributes needed for in Mail applications? Indexing? Storing keywords? OS/2 mostly uses EA's to store object properties for desktop objects, and also to emulate POSIX file attributes. And also icons, keywords for search, tokenized images (i.e., bytecode) for scripts written in REXX language etc. In OS/2, EA's are limited in size to at least, 64 KB. With JFS, they theoretically could be larger, but for compatibility with older HPFS file system, the limit is set to 64 KB. IIRC, in ext2fs, max. extended attribute size is only 4 KB, which could be too little for some purposes.  In Sculpt, ext2 is used as current file system, so, its limits apply. (this is for Linux ext2 driver, not sure about a NetBSD driver in rump kernel). On other file sysytems, EA support may be missing, but it can be emulated. For example, on FAT, OS/2 uses a file called "ea data. sf" in the root directory, which contains EA's of all files on the partition in special format. All file metainfo is accessed from a directory entry (on FAT), or its inode on most other file systems. So, on FAT, a directory entry contains a 16-bit handle to an EA stored in "ea data. sf" file. Though, in FAT32 direntry, the two bytes used to reference a EA, are reused for high word of first cluster number of the file. So, for implementation of EA's in FAT32, two bits of another reserved byte of a direntry is reused. (one bit means "this file has EA's", another means "the EA's are critical", i.e., they are mandatory to be existed when opening the file). So, EA's on FAT32 are laying around near the file itself, in the same directory, in a hidden file. The disadvantage is that EA files are laying around everywhere, like a trash. So, if a file system doesn't support EA's, they can be emulated like this.

If I correctly understood, there is no EA/xattr support in Genode VFS, so far. But for my project (which is "OS/2 personality" on top of Genode), I plan to make a server (like Genode's VFS server) supporting OS/2-like IFS (Installable File Systems), like plugins. They, of course, support EA's. But then only OS/2 applications will be able to take advantage of using EA's. Serving OS/2 file systems to Genode applications is planned too. You can go similar way.

---

Genode users mailing list users@lists.genode.org https://lists.genode.org/listinfo/users

Norman Feske wrote:

Hello Valery,

thanks for the background info about extended attributes.

...

mandatory to be existed when opening the file). So, EA's on FAT32 are laying around near the file itself, in the same directory, in a hidden file. The disadvantage is that EA files are laying around everywhere, like a trash. So, if a file system doesn't support EA's, they can be emulated like this.

I admittedly have not much clue about file systems. To me as a user, features like this (this also goes for the strange "resource forks" of old Mac OS) look like glorified directories. They seem to be just more complicated, less transparent, and cause friction when moving data across file systems. Given my poor background, please take the following with a grain of salt.

Yes, also WinNT goes even further: it supports files having multiple streams of data. Resource fork from MacOS or EA can be implemented as streams too. Only Resource fork stores multiple metainfo in one stream, and EA's can be implemented both ways: as a single stream, or multiple streams.

I wonder, in situations where meta data has to contain a lot of domain knowledge, wouldn't sqlite be a good way to organize it?

Probably yes, though, sqlite files are just tables with fixed number of fields. Storing EA in SQLite or any other relational database should require a data schema with multiple tables, which may be too complex, because a file can have any number of EA's, whereas a table contains a fixed number of fields. So, it may require a complicated data schema.

In situations where meta data are simple annotations to files (like storing the icon for a file), why can't both the file and the meta data go to into a directory? This is the way Risc OS (and also today's Mac OS, AFAIK) deal with the situation.

They could be not simple annotations/text data, but they also could be any small binary data attached to a file (icons being a good example).

Directory could be too small to fit all EA's. Usually, if EA's are small enough, file systems like HPFS or JFS can store them in inode/fnode inline. But if they are not fit into inode, they are put into separate streams of sectors. Directory usually only contains a reference to EA's (a first sector/block/cluster number of EA stream, or a handle to a particular EA, or just some bits which denote that EA is present.). Not sure about RISCOS, but AFAIK, MacOS uses a separate stream of data to store its resource forks. Directories ideally should only contain a mapping between inode number and a file name. All standard metadata goes into an inode. But if it's not fit into inode, it goes into separate stream. Inode size is usually limited.

Hence, there is currently no plan to extend Genode's file-system session with extended attributes.

Yes, I of course, understand the wish to keep the interfaces and architecture simpler.

Valery's remark about an emulation scheme looks good to bridge the gap. This way, such a feature can be implemented within the libc where the libc would rely on some conventions. E.g., when asked for the extended attributes for a certain 'file', the libc may - under the hood - read the meta data from a file called '.xa/file'. Yes, this would spill the '.xa' directories here and there but it leaves the complexity out of base mechanisms like the VFS or the file-system session.

Yes, the EA/xattr store can be moved to a separate subdirectories of each current subdirectory (like in svn working copies.). This seems to be a better idea than to have EA's laying around along with their files. So, they will not pollute the usual disk contents. Yes, it looks be possible to have EA support on a libc level. Usually, EA's are returned along with standard file metainfo when setting or retrieving FileInfo/PathInfo or searching files. So, e.g., a file search retrieves a set of records, per each file. These records may contain a standard metainfo (inode contents), as well as EA contents. Also, when opening files, "open" syscall could create EA's for that file. Pointer to EA data is specified as a separate syscall parameter. Though, the open/fopen calls don't contain such parameters, but a native syscall does. Creating a FS-independent wrapper on libc level looks possible, though I fear that it could be inefficient and slow. In a file search example, if a FS returned a set of records containing file metadata, then libc wrapper should process this set of records, and add EA data to each record. This looks to be working, but it could be slow. Usually, a FS does the same with a single step: usually both, metadata contained in inode, and EA's, a returned at once. For speeding things up, the FS doesn't check each file for EA presence, instead, a flag is used, which specifies, whether file has EA's. If not, the EA lookup is just omitted.

WBR, valery

Cheers Norman

Nobody III wrote:

I suppose we could use this mechanism, which we already use for ioctl, for several other file attributes as well, e.g. timestamps.

Implement getting or setting the extended attributes via an ioctl? Seems to be a logical approach. But then this should be done in libc implementation of the "ioctl" function. So, some part of "ioctl" code is implemented in each FS driver, and some part of implementation resides in libc itself. So, this is done without calling the part that resides in the FS driver, if FS doesn't support EA's, so they are then emulated in the libc itself (maybe, with a special libc plugin). But apart from setting/getting EA values, EA's are also returned by file searching syscalls. On libc level it is done by readdir/glob functions. So the libc plugin for EA support should also hook on readdir() function. But standard readdir() doesn't seems to support EA's. Unfortunately, I'm not aware how EA access is usually done via libc in Linux or Haiku. In OS/2 the common practice seems to be using required API's/syscalls directly, without libc. There should be two cases 1) FS doesn't support EA's, so they are emulated with hooking into functions like ioctl or readdir (or extended readdir version) 2) FS supports EA's. Then the libc plugin should call the code in the FS driver immediately.

We could put it all under a directory named .attrs-(filename).

Yes, but it's better to do like Norman suggested, i.e., one subdir with files in it. Each file stores EA's of a single file in parent directory. EA's are usually stored as a single stream of data. Storing EA's of each file in separate subdir should require many dirs, which will pollute the current directory the same way as EA files (as I described for FAT32 case).

As for performance, the filesystem drivers could implement attribute access as separate files while still using attrs/xattrs as the underlying storage mechanism.

Usually EA's are returned along with other file metainfo as one structure of data, containing all the metainfo. There could be just one EA returned in this structure, or a list of EA's at once. So if use "ioctl" for returning such info, it, probably should return all metadata as one structure with multiple fields, not just a single attribute or extended attribute.

The main issue with this is when attributes are copied before the files. However, if we make the attributes transparent in libc, this shouldn't be much of an issue, especially if we guarantee that the attributes come after the files in the dirent listing. Overall, I think this could be a relatively clean way to implement both regular and extended attributes. Any other thoughts on this?

I don't see a big issue in that is EA file stored in a direntry before file itself, or after it. Of course, it should be easier to find EA's right after the file itself, but this relies on the FS implementation. I.e., we'll need to rewrite each FS so that it will store the EA files that way. But I think, it's a bad idea to rewrite every single FS. If we want to create a EA emulation layer on top of each FS, it should not rely on special way of implementing each FS. And there should be emulation only in case the FS does not support EA's. If it does support them, then emulation should be switched off.

On Tue, Apr 2, 2019, 7:54 AM Valery V. Sedletski via users <users@lists.genode.org mailto:users@lists.genode.org> wrote:

Norman Feske wrote:
> Hello Valery,
>
> thanks for the background info about extended attributes.
>
>> mandatory to be existed when opening the file). So, EA's on
FAT32 are
>> laying around near the file itself, in the same directory, in a
hidden
>> file. The disadvantage is that EA files are laying around
everywhere,
>> like a trash. So, if a file system doesn't support EA's, they
can be
>> emulated like this.
> I admittedly have not much clue about file systems. To me as a user,
> features like this (this also goes for the strange "resource
forks" of
> old Mac OS) look like glorified directories. They seem to be
just more
> complicated, less transparent, and cause friction when moving data
> across file systems. Given my poor background, please take the
following
> with a grain of salt.
Yes, also WinNT goes even further: it supports files having multiple
streams of data. Resource fork from MacOS or EA can be implemented
as streams too. Only Resource fork stores multiple metainfo in one
stream, and EA's can be implemented both ways: as a single stream,
or multiple streams.
> I wonder, in situations where meta data has to contain a lot of
domain
> knowledge, wouldn't sqlite be a good way to organize it?
Probably yes, though, sqlite files are just tables with fixed number
of fields. Storing EA in SQLite or any other relational database
should
require a data schema with multiple tables, which may be too complex,
because a file can have any number of EA's, whereas a table contains
a fixed number of fields. So, it may require a complicated data
schema.
> In situations where meta data are simple annotations to files (like
> storing the icon for a file), why can't both the file and the
meta data
> go to into a directory? This is the way Risc OS (and also
today's Mac
> OS, AFAIK) deal with the situation.
They could be not simple annotations/text data, but they also could
be any small binary data attached to a file (icons being a good
example).

Directory could be too small to fit all EA's. Usually, if EA's are
small
enough, file systems like HPFS or JFS can store them in inode/fnode
inline. But if they are not fit into inode, they are put into separate
streams of sectors. Directory usually only contains a reference to
EA's
(a first sector/block/cluster number of EA stream, or a handle to a
particular EA, or just some bits which denote that EA is present.).
Not sure about RISCOS, but AFAIK, MacOS uses a separate stream
of data to store its resource forks. Directories ideally should only
contain a mapping between inode number and a file name. All standard
metadata goes into an inode. But if it's not fit into inode, it
goes into
separate stream. Inode size is usually limited.
> Hence, there is currently no plan to extend Genode's file-system
session
> with extended attributes.
Yes, I of course, understand the wish to keep the interfaces and
architecture simpler.
> Valery's remark about an emulation scheme looks good to bridge
the gap.
> This way, such a feature can be implemented within the libc
where the
> libc would rely on some conventions. E.g., when asked for the
extended
> attributes for a certain 'file', the libc may - under the hood -
read
> the meta data from a file called '.xa/file'. Yes, this would
spill the
> '.xa' directories here and there but it leaves the complexity out of
> base mechanisms like the VFS or the file-system session.
Yes, the EA/xattr store can be moved to a separate subdirectories
of each current subdirectory (like in svn working copies.). This
seems to be a better idea than to have EA's laying around along
with their files. So, they will not pollute the usual disk contents.
Yes, it looks be possible to have EA support on a libc level.
Usually, EA's are returned along with standard file metainfo
when setting or retrieving FileInfo/PathInfo or searching files.
So, e.g., a file search retrieves a set of records, per each file.
These
records may contain a standard metainfo (inode contents), as well
as EA contents. Also, when opening files, "open" syscall could create
EA's for that file. Pointer to EA data is specified as a separate
syscall
parameter. Though, the open/fopen calls don't contain such
parameters, but a native syscall does. Creating a FS-independent
wrapper on libc level looks possible, though I fear that it could
be inefficient and slow. In a file search example, if a FS returned
a set of records containing file metadata, then libc wrapper should
process this set of records, and add EA data to each record. This
looks to be working, but it could be slow. Usually, a FS does the same
with a single step: usually both, metadata contained in inode, and
EA's,
a returned at once. For speeding things up, the FS doesn't check
each file for EA presence, instead, a flag is used, which specifies,
whether file has EA's. If not, the EA lookup is just omitted.

WBR,
valery

 > Cheers
 > Norman

_______________________________________________
Genode users mailing list
users@lists.genode.org <mailto:users@lists.genode.org>
https://lists.genode.org/listinfo/users

---

Genode users mailing list users@lists.genode.org https://lists.genode.org/listinfo/users