Hi,
currently we're working with the TrustZone VMM example from Genode based on Genode 15.05 on i.MX53 QSB.
We have a shared buffer in normal world, which we want to use for cross-world communication.
Currently the buffer is allocated using mmap MAP_SHARED | MAP_ANON.
We're having strange issues/problems/inconsistencies when accessing the memory range in Genode (secure world) after normal world has written to it.
We read that uncached memory should be used for this, or the cache lines have to be flushed/invalidated. However, apparently we can't get either of these options working.
So our questions as follows:
1) We use mmap(NULL, <size>, PROT_READ|PROT_WRITE, MAP_SHARED|MAP_ANON, -1, 0) to allocate the buffer in normal world (Linux). Is there a better way to allocate uncached memory?
2) We've tried __cpuc_flush_user_all(), __cpuc_flush_kern_all() in order to invalidate all cache lines (later maybe only the affected cache lines once this works in general). Is this correct, to do this in normal world (i.e. Linux)?
3) At the moment we assume that Genode doesn't have to invalidate/flush its cache lines because as it is the VMM it should be aware of the fact that the cache might be stale.
Any help is appreciated. A short hint: Currently our idea is to send an hypercall to the Genode VMM after Linux has written to the buffer, so Genode will pretty quickly access the memory after it has been written to from Linux.
Best regards, Stefan.
Hi,
I'm not sure whether my comments can help you or not.
I also asked a similar question to the community. The related articles are as follows: http://sourceforge.net/p/genode/mailman/message/34341641/
I modified my source code referring to the above link.
Based on the answers from the community, I used the "dma_alloc_coherent()" as follows in the device driver in the Normal World:
/* variables for buffer */ static dma_addr_t genblk_buf_phys; static void * genblk_buf
*genblk_buf* = dma_alloc_coherent(NULL, 1024, *&genblk_buf_phys*, GFP_KERNEL);
/* send smc instruction to initialize buffer */ smc_blk(instruction_id, *genblk_buf_phys*, 1024);
In the Secure World, tz_vmm was modified to handle this as follows:
/* variables for buffer */ void * _buf; Genode::size_t _buf_size; addr_t buf_base; addr_t buf_top, ram_top, buf_off; Ram * ram; bool buf_err;
buf_base = (addr_t)_vm->state()->r1; _buf_size = (Genode::size_t)_vm->state()->r2; buf_top = buf_base + _buf_size; ram = _vm->ram(); ram_top = ram->base() + ram->size();
buf_err = buf_top <= buf_base; buf_err |= buf_base < ram->base(); buf_err |= buf_top >= ram_top; if (buf_err) { PERR("blk: illegal buffer constraints"); break; } buf_off = buf_base - ram->base(); *_buf* = (void *)(ram->local() + buf_off);
After this, *_buf* is used to refer the shared buffer in the Secure World, and *genblk_buf* is used in the Normal World.
I hope my humble comments can help you.
Jaemin Park.
On Thu, Dec 10, 2015 at 1:15 AM, Stefan Brenner <brenner@...290...> wrote:
Hi,
currently we're working with the TrustZone VMM example from Genode based on Genode 15.05 on i.MX53 QSB.
We have a shared buffer in normal world, which we want to use for cross-world communication.
Currently the buffer is allocated using mmap MAP_SHARED | MAP_ANON.
We're having strange issues/problems/inconsistencies when accessing the memory range in Genode (secure world) after normal world has written to it.
We read that uncached memory should be used for this, or the cache lines have to be flushed/invalidated. However, apparently we can't get either of these options working.
So our questions as follows:
- We use mmap(NULL, <size>, PROT_READ|PROT_WRITE, MAP_SHARED|MAP_ANON,
-1, 0) to allocate the buffer in normal world (Linux). Is there a better way to allocate uncached memory?
- We've tried __cpuc_flush_user_all(), __cpuc_flush_kern_all() in order
to invalidate all cache lines (later maybe only the affected cache lines once this works in general). Is this correct, to do this in normal world (i.e. Linux)?
- At the moment we assume that Genode doesn't have to invalidate/flush
its cache lines because as it is the VMM it should be aware of the fact that the cache might be stale.
Any help is appreciated. A short hint: Currently our idea is to send an hypercall to the Genode VMM after Linux has written to the buffer, so Genode will pretty quickly access the memory after it has been written to from Linux.
Best regards, Stefan.
-- M.Sc. Stefan Brenner Institute of Operating Systems and Computer Networks Distributed Systems Group TU Braunschweig
www: https://www.ibr.cs.tu-bs.de/users/brenner mail: brenner@...290...
genode-main mailing list genode-main@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/genode-main
Hi Stefan,
Am 09.12.2015 um 17:15 schrieb Stefan Brenner:
- We use mmap(NULL, <size>, PROT_READ|PROT_WRITE, MAP_SHARED|MAP_ANON,
-1, 0) to allocate the buffer in normal world (Linux). Is there a better way to allocate uncached memory?
The only way I found, when working at Genode's TrustZone VMM on USB Armory, was by allocating DMA memory. As far as I know, Linux doesn't provide allocating non-cached DMA regions in the userland. However, you can do this inside the kernel. You may have a look at the initialization of genblk_buf in my para-virtualized block driver [1]. The resulting address is used for shared RAM between secure and normal world. The secure back end of the driver can be found in [2].
- We've tried __cpuc_flush_user_all(), __cpuc_flush_kern_all() in order
to invalidate all cache lines (later maybe only the affected cache lines once this works in general). Is this correct, to do this in normal world (i.e. Linux)?
I do not know whether this is sufficient, but recently there was a post with the same idea [3] and maybe the author got it working at the end. In the corresponding thread I also suggested a general design for memory synchronization caused from within the VMM. This is useful when not having non-cached memory _and_ the VMM is the transfer initiator. But I assume that the latter is not the case in your scenario.
- At the moment we assume that Genode doesn't have to invalidate/flush
its cache lines because as it is the VMM it should be aware of the fact that the cache might be stale.
I'm afraid I do not understand what you mean. A RAM region shared between the TrustZone worlds must always be located in non-secure RAM. In the tz_vmm, the whole non-secure RAM is mapped as IOMEM [4], which is - using ARM-speak - tagged as "Device Memory" and thus non-cached. So, yes, the tz_vmm must not flush caches as it should always use these non-cached mappings.
Any help is appreciated. A short hint: Currently our idea is to send an hypercall to the Genode VMM after Linux has written to the buffer, so Genode will pretty quickly access the memory after it has been written to from Linux.
This is exactly what is done in the para-virtualized block driver mentioned above after a write request has been set-up in the buffer [5].
Cheers, Martin
[1] https://github.com/m-stein/linux/blob/genode_hw_usb_armory_tz_vmm/drivers/bl...
[2] https://github.com/genodelabs/genode/blob/master/repos/os/src/server/tz_vmm/...
[3] http://sourceforge.net/p/genode/mailman/message/34667551/
[4] https://github.com/genodelabs/genode/blob/master/repos/os/src/server/tz_vmm/...
[5] https://github.com/m-stein/linux/blob/genode_hw_usb_armory_tz_vmm/drivers/bl...
-
Jaemin Park -
Martin Stein -
Stefan Brenner