My personal wishlist

[Valery V. Sedletski]

Nobody III wrote:
> I suppose we could use this mechanism, which we already use for ioctl, 
> for several other file attributes as well, e.g. timestamps.
Implement getting or setting the extended attributes via an ioctl? Seems 
to be a logical approach. But then this should be done in libc 
implementation of the "ioctl" function. So, some part of "ioctl" code is 
implemented in each FS driver, and some part of implementation resides 
in libc itself. So, this is done without calling the part that resides 
in the FS driver, if FS doesn't support EA's, so they are then emulated 
in the libc itself (maybe, with a special libc plugin). But apart from 
setting/getting EA values, EA's are also returned by file searching 
syscalls. On libc level it is done by readdir/glob functions. So the 
libc plugin for EA support should also hook on readdir() function. But 
standard readdir() doesn't seems to support EA's. Unfortunately, I'm not 
aware how EA access is usually done via libc in Linux or Haiku. In OS/2 
the common practice seems to be using required API's/syscalls directly, 
without libc. There should be two cases 1) FS doesn't support EA's, so 
they are emulated with hooking into functions like ioctl or readdir (or 
extended readdir version) 2) FS supports EA's. Then the libc plugin 
should call the code in the FS driver immediately.
> We could put it all under a directory named .attrs-(filename).
>
Yes, but it's better to do like Norman suggested, i.e., one subdir with 
files in it. Each file stores EA's of a single file in parent directory. 
EA's are usually stored as a single stream of data. Storing EA's of each 
file in separate subdir should require many dirs, which will pollute the 
current directory the same way as EA files (as I described for FAT32 case).
> As for performance, the filesystem drivers could implement attribute 
> access as separate files while still using attrs/xattrs as the 
> underlying storage mechanism.
Usually EA's are returned along with other file metainfo as one 
structure of data, containing all the metainfo. There could be just one 
EA returned in this structure, or a list of EA's at once. So if use 
"ioctl" for returning such info, it, probably should return all metadata 
as one structure with multiple fields, not just a single attribute or 
extended attribute.
> The main issue with this is when attributes are copied before the 
> files. However, if we make the attributes transparent in libc, this 
> shouldn't be much of an issue, especially if we guarantee that the 
> attributes come after the files in the dirent listing. Overall, I 
> think this could be a relatively clean way to implement both regular 
> and extended attributes. Any other thoughts on this?
>
I don't see a big issue in that is EA file stored in a direntry before 
file itself, or after it. Of course, it should be easier to find EA's 
right after the file itself, but this relies on the FS implementation. 
I.e., we'll need to rewrite each FS so that it will store the EA files 
that way. But I think, it's a bad idea to rewrite every single FS. If we 
want to create a EA emulation layer on top of each FS, it should not 
rely on special way of implementing each FS. And there should be 
emulation only in case the FS does not support EA's. If it does support 
them, then emulation should be switched off.
> On Tue, Apr 2, 2019, 7:54 AM Valery V. Sedletski via users 
> <users at lists.genode.org <mailto:users at lists.genode.org>> wrote:
>
>     Norman Feske wrote:
>     > Hello Valery,
>     >
>     > thanks for the background info about extended attributes.
>     >
>     >> mandatory to be existed when opening the file). So, EA's on
>     FAT32 are
>     >> laying around near the file itself, in the same directory, in a
>     hidden
>     >> file. The disadvantage is that EA files are laying around
>     everywhere,
>     >> like a trash. So, if a file system doesn't support EA's, they
>     can be
>     >> emulated like this.
>     > I admittedly have not much clue about file systems. To me as a user,
>     > features like this (this also goes for the strange "resource
>     forks" of
>     > old Mac OS) look like glorified directories. They seem to be
>     just more
>     > complicated, less transparent, and cause friction when moving data
>     > across file systems. Given my poor background, please take the
>     following
>     > with a grain of salt.
>     Yes, also WinNT goes even further: it supports files having multiple
>     streams of data. Resource fork from MacOS or EA can be implemented
>     as streams too. Only Resource fork stores multiple metainfo in one
>     stream, and EA's can be implemented both ways: as a single stream,
>     or multiple streams.
>     > I wonder, in situations where meta data has to contain a lot of
>     domain
>     > knowledge, wouldn't sqlite be a good way to organize it?
>     Probably yes, though, sqlite files are just tables with fixed number
>     of fields. Storing EA in SQLite or any other relational database
>     should
>     require a data schema with multiple tables, which may be too complex,
>     because a file can have any number of EA's, whereas a table contains
>     a fixed number of fields. So, it may require a complicated data
>     schema.
>     > In situations where meta data are simple annotations to files (like
>     > storing the icon for a file), why can't both the file and the
>     meta data
>     > go to into a directory? This is the way Risc OS (and also
>     today's Mac
>     > OS, AFAIK) deal with the situation.
>     They could be not simple annotations/text data, but they also could
>     be any small binary data attached to a file (icons being a good
>     example).
>
>     Directory could be too small to fit all EA's. Usually, if EA's are
>     small
>     enough, file systems like HPFS or JFS can store them in inode/fnode
>     inline. But if they are not fit into inode, they are put into separate
>     streams of sectors. Directory usually only contains a reference to
>     EA's
>     (a first sector/block/cluster number of EA stream, or a handle to a
>     particular EA, or just some bits which denote that EA is present.).
>     Not sure about RISCOS, but AFAIK, MacOS uses a separate stream
>     of data to store its resource forks. Directories ideally should only
>     contain a mapping between inode number and a file name. All standard
>     metadata goes into an inode. But if it's not fit into inode, it
>     goes into
>     separate stream. Inode size is usually limited.
>     > Hence, there is currently no plan to extend Genode's file-system
>     session
>     > with extended attributes.
>     Yes, I of course, understand the wish to keep the interfaces and
>     architecture simpler.
>     > Valery's remark about an emulation scheme looks good to bridge
>     the gap.
>     > This way, such a feature can be implemented within the libc
>     where the
>     > libc would rely on some conventions. E.g., when asked for the
>     extended
>     > attributes for a certain 'file', the libc may - under the hood -
>     read
>     > the meta data from a file called '.xa/file'. Yes, this would
>     spill the
>     > '.xa' directories here and there but it leaves the complexity out of
>     > base mechanisms like the VFS or the file-system session.
>     Yes, the EA/xattr store can be moved to a separate subdirectories
>     of each current subdirectory (like in svn working copies.). This
>     seems to be a better idea than to have EA's laying around along
>     with their files. So, they will not pollute the usual disk contents.
>     Yes, it looks be possible to have EA support on a libc level.
>     Usually, EA's are returned along with standard file metainfo
>     when setting or retrieving FileInfo/PathInfo or searching files.
>     So, e.g., a file search retrieves a set of records, per each file.
>     These
>     records may contain a standard metainfo (inode contents), as well
>     as EA contents. Also, when opening files, "open" syscall could create
>     EA's for that file. Pointer to EA data is specified as a separate
>     syscall
>     parameter. Though, the open/fopen calls don't contain such
>     parameters, but a native syscall does. Creating a FS-independent
>     wrapper on libc level looks possible, though I fear that it could
>     be inefficient and slow. In a file search example, if a FS returned
>     a set of records containing file metadata, then libc wrapper should
>     process this set of records, and add EA data to each record. This
>     looks to be working, but it could be slow. Usually, a FS does the same
>     with a single step: usually both, metadata contained in inode, and
>     EA's,
>     a returned at once. For speeding things up, the FS doesn't check
>     each file for EA presence, instead, a flag is used, which specifies,
>     whether file has EA's. If not, the EA lookup is just omitted.
>
>     WBR,
>     valery
>
>      > Cheers
>      > Norman
>
>     _______________________________________________
>     Genode users mailing list
>     users at lists.genode.org <mailto:users at lists.genode.org>
>     https://lists.genode.org/listinfo/users
>
>
>
> _______________________________________________
> Genode users mailing list
> users at lists.genode.org
> https://lists.genode.org/listinfo/users