Virtualization with trustzone into mx53

[Stefan Kalkowski]

Hi Pablo,

On 12/16/2013 05:06 PM, panton wrote:
> Hi,
> 
> I am having some unexpected behavior accessing from Genode VMM to guest 
> VM RAM.
> 
> 
>> Assuming, you've put the corresponding memory
>> region's address into register r3 within your para-virtualization code
>> in the guest VM. Then the following procedure will give you the memory
>> region within the VMM object:
>>
>>   addr_t phys_addr = _vm->va_to_pa(_vm->state()->r3);  // Only if r3 
>> has a virtual address
>>   addr_t virt_addr = _vm->ram()->va(phys_addr);
> 
> 
> Following the indication Stefan gave me, I am able to access to linux 
> guest's memory by passing virtual address of desired memory region into 
> registers (_vm->state()->r3), it also could be performed by passing 
> physical addr instead of virtual addr and removing first instruction.
> 
> Problem is when you access to calculated virtual address's content 
> (virt_addr), it is not always correct. First time I run my example app 
> everything works fine, but next times, I gets unexpected values. I ckeck 
> the physical address is right and the virtual address is just:
> 
>   virt_addr = _local + (phys - _base);
> 
> I really don't know why that happen, the loaded memory sometime is the 
> good one and sometimes is not. The MMU implementation used on VMM works 
> fine, since it gives correct physical address from virtual one. The 
> address passes into the register is also correct. Any idea of what could 
> be happening??? Probably, I am missing some concepts of how mmu works, 
> but I do not understand why it just fail in some cases.

If you observe differences between VMM and VM when accessing the same
physical memory region, there are two possibilities. First: the cache of
the non-secure side isn't flushed, or second the secure side is using
the non-secure side's RAM portion through the cache. As both side's
cache entries are handled independently of each other, the secure side
doesn't see non-secure cache entries, and vice versa.

On the secure side, we've enforced that the non-secure side's RAM
doesn't pollute the cache, by marking the corresponding page table
entries as being non-cacheable.

On the VM's side, you've to ensure to either mark the corresponding
memory region, you want to share with the secure side, as being
non-cacheable memory (like it's done for I/O memory), or by selectively
flushing that memory area within the cache, before doing a VMM call.

Best regards
Stefan

> 
> Thanks in advanced.
> 
> Best regards,
> Pablo Anton.
> 
> El 10.12.2013 11:49, Stefan Kalkowski escribió:
>> Hi Pablo,
>>
>> On 12/09/2013 04:31 PM, panton wrote:
>>> Hi,
>>>
>>> I am a little confused about creating a share memory between a Non
>>> Secure and Secure worlds. As far as I understood, memory region should
>>> be placed in Non Secure Side (Linux in tz_vmm example), when a change 
>>> of
>>> context is done you can send the addresses of that regions using cpu
>>> registers and looking into vm_state struct. Then, from Secure side 
>>> must
>>> be a method to access to that region but I do not find the good way to
>>> configure Genode to perform that access. Is there any example of how
>>> that works?
>>
>> actually, the whole main memory used by the non-secure side is 
>> available
>> in the virtual machine monitor of our small example. Before booting, it
>> is used to put the kernel image, and initramfs into it. After that, it
>> can be used to reproduce processing of the virtual machine, for 
>> instance
>> you might walk the page-tables of the VM to reconstruct pointers in the
>> VM's registers etc.
>>
>> Of course, you can use the shared main memory to transfer data between
>> VMM and VM too. However, you would have to either transfer the physical
>> address of the corresponding memory region via the VM's registers, or
>> implement an appropriated software MMU to translate the VM's virtual
>> address in the register into a physical address that can be located by
>> the VMM. Luckily, there is already a simplified software MMU
>> implementation within the VMM example code
>> ('os/src/server/tz_vmm/include/mmu.h'). Once you've a physical address
>> of the memory region laying in the main memory of the VM, you've of
>> course to translate again that physical address to the position in the
>> address space of the VMM. Assuming, you've put the corresponding memory
>> region's address into register r3 within your para-virtualization code
>> in the guest VM. Then the following procedure will give you the memory
>> region within the VMM object:
>>
>>   addr_t phys_addr = _vm->va_to_pa(_vm->state()->r3);
>>   addr_t virt_addr = _vm->ram()->va(phys_addr);
>>
>> I hope this is what you're looking for.
>>
>> Regards
>> Stefan
>>
>>>
>>> Regards,
>>> Pablo Anton.
>>>
>>> El 02.12.2013 16:19, Stefan Kalkowski escribió:
>>>> Hi,
>>>>
>>>> On 12/02/2013 03:48 PM, panton wrote:
>>>>> Hi Stefan,
>>>>>
>>>>> I know this is not a question about genode but I am having troubles
>>>>> compiling linux image for tz-vmm. Did you use genode toolchain?? 
>>>>> Could
>>>>> I
>>>>> ask you the config options file you used for compiling (I was not 
>>>>> able
>>>>> to get it from linux image)??
>>>>
>>>> I didn't used the Genode toolchain, but the Codesourcery ARM cross
>>>> compiler for Linux (Sourcery G++ Lite 2009q1-203). As the Genode
>>>> toolchain isn't used to compile a Linux system, it misses certain
>>>> defines you need when compiling the Linux kernel.
>>>>
>>>> The adapted kernel configuration file is part of the Linux fork I've
>>>> mentioned in my previous mail. Here is the concrete file:
>>>>
>>>> https://github.com/skalk/linux/blob/imx53-tz/arch/arm/configs/imx5_android_tz_defconfig
>>>>
>>>> Regards
>>>> Stefan
>>>>
>>>>>
>>>>> Thanks you in advance.
>>>>>
>>>>> Best regards.
>>>>> Pablo Anton.
>>>>>
>>>>> El 28.11.2013 11:07, Stefan Kalkowski escribió:
>>>>>> Hi Pablo,
>>>>>>
>>>>>> On 11/28/2013 10:52 AM, panton wrote:
>>>>>>> Hi Stefan,
>>>>>>>
>>>>>>> Thank you very much for your detailed explication. I am really 
>>>>>>> lucky
>>>>>>> beacause it seems you are working right now on that matter (last
>>>>>>> genode/staging branch update was yesterday :)).
>>>>>>>
>>>>>>>> well, this depends. There are trustzone.cc files for different
>>>>>>>> platforms/configurations. In general, that file only contains
>>>>>>>> additional
>>>>>>>> kernel initialization routines needed. For instance, configuring
>>>>>>>> IRQs
>>>>>>>> to
>>>>>>>> be "secure", or "non-secure", or configure TrustZone specific
>>>>>>>> devices
>>>>>>>> that can be configured in supervisor mode only.
>>>>>>>> For configurations where no TrustZone is supported, or used that
>>>>>>>> file
>>>>>>>> contains an empty initialization indeed. I assume you've found 
>>>>>>>> that
>>>>>>>> file.
>>>>>>>
>>>>>>> Now I am able to see files on ./base-hw/src/core/imx53/trustzone.
>>>>>>>
>>>>>>>> To sum it up, the non-secure guest has to behave cooperatively, 
>>>>>>>> or
>>>>>>>> it
>>>>>>>> will fail. Trap-and-emulate doesn't work in general. Therefore,
>>>>>>>> some
>>>>>>>> lightweight form of para-virtualization of the guest OS is 
>>>>>>>> needed.
>>>>>>>
>>>>>>> So, I assume the linux image on
>>>>>>> http://genode.org/files/images/imx53_qsb/linux_trustzone.bin is 
>>>>>>> not
>>>>>>> a
>>>>>>> normal linux image for imx53_loco but neither a L4Linux (since the
>>>>>>> example is running without Fiasco.OC). Is there any repository 
>>>>>>> where
>>>>>>> we
>>>>>>> could take that linux code?
>>>>>>>
>>>>>>
>>>>>> Sure, branches including the changes for Versatile Express, and
>>>>>> i.MX53
>>>>>> QSB/Tablet can be found on Github too:
>>>>>>
>>>>>>   git at ...116...:skalk/linux.git
>>>>>>
>>>>>> The branches are titled 'vexpress-tz', and 'imx53-tz'
>>>>>>
>>>>>>>
>>>>>>> About the example I only can say "Great work".
>>>>>>
>>>>>> Thanks, that's music to my ears.
>>>>>>
>>>>>>> I was able to run it on
>>>>>>> hardware. Some little details that could help people:
>>>>>>>
>>>>>>>> After that, do a 'make run/tz_vmm' in the build directory. The
>>>>>>>> resulting
>>>>>>>> image is located in 'var/run/tz_vmm/uImage'
>>>>>>>
>>>>>>> The resulting image is on elf, if you want to run it with uboot 
>>>>>>> you
>>>>>>> should create a valid uImage using mkimage tool.
>>>>>>>
>>>>>>
>>>>>> If you add a '--target uboot' to the RUN_OPT environment variable,
>>>>>> the
>>>>>> uImage is built automatically. Just add the following to your
>>>>>> 'etc/build.conf' (I've missed that in the previous mail):
>>>>>>
>>>>>>   RUN_OPT = --target uboot
>>>>>>
>>>>>>> Now, I go to play!
>>>>>>
>>>>>> Good luck, and
>>>>>> best regards
>>>>>> Stefan
>>>>>>
>>>>>>>
>>>>>>> Best regards
>>>>>>> Pablo Antón.
>>>>>>>
>>>>>>>
>>>>>>> El 27.11.2013 22:47, Stefan Kalkowski escribió:
>>>>>>>> Hi Pablo,
>>>>>>>>
>>>>>>>> On 11/26/2013 06:11 PM, panton wrote:
>>>>>>>>> Hello,
>>>>>>>>>
>>>>>>>>> I want to create a virtualized system with Genode in my imx53
>>>>>>>>> board.
>>>>>>>>> At
>>>>>>>>> the moment, I am able to run Genode on the board with base-foc 
>>>>>>>>> and
>>>>>>>>> also
>>>>>>>>> directly on hardware (base-hw). Thus, my plan was to run a linux
>>>>>>>>> kernel
>>>>>>>>> as a child node and hopefully run it into trustzone NS. Looking
>>>>>>>>> into
>>>>>>>>> Genode code I found base-hw/include/vm_session that seems to 
>>>>>>>>> deal
>>>>>>>>> with
>>>>>>>>> virtualization. Furthermore there is a trustzone.cc into
>>>>>>>>> base-hw/src/core.., but without real useful code.
>>>>>>>>
>>>>>>>> well, this depends. There are trustzone.cc files for different
>>>>>>>> platforms/configurations. In general, that file only contains
>>>>>>>> additional
>>>>>>>> kernel initialization routines needed. For instance, configuring
>>>>>>>> IRQs
>>>>>>>> to
>>>>>>>> be "secure", or "non-secure", or configure TrustZone specific
>>>>>>>> devices
>>>>>>>> that can be configured in supervisor mode only.
>>>>>>>> For configurations where no TrustZone is supported, or used that
>>>>>>>> file
>>>>>>>> contains an empty initialization indeed. I assume you've found 
>>>>>>>> that
>>>>>>>> file.
>>>>>>>>
>>>>>>>>>
>>>>>>>>> I was wondering if there are implemented virtualization
>>>>>>>>> capabilities
>>>>>>>>> using trustzone or even without trustzone?
>>>>>>>>
>>>>>>>> In fact, there is some kind of virtualization support using
>>>>>>>> TrustZone.
>>>>>>>> Although, TrustZone isn't designed to be a virtualization 
>>>>>>>> solution.
>>>>>>>> Therefore, the "guest" needs to be aware what devices it is 
>>>>>>>> allowed
>>>>>>>> to
>>>>>>>> use. In contrast to the CPU (including MMU, Caches, etc.), there 
>>>>>>>> is
>>>>>>>> no
>>>>>>>> support to virtualize physical memory (including memory mapped 
>>>>>>>> I/O)
>>>>>>>> for
>>>>>>>> the non-secure "guest". That means guest physical addresses are 
>>>>>>>> in
>>>>>>>> fact
>>>>>>>> physical addresses. Dependent on the concrete platform, it might 
>>>>>>>> be
>>>>>>>> possible to deny non-secure access to certain physical memory
>>>>>>>> regions,
>>>>>>>> like I/O memory of certain devices, and then use a trap-emulate
>>>>>>>> approach. But in general this approach isn't guaranteed to work.
>>>>>>>> Dependent on where the unit, which controls peripheral device
>>>>>>>> memory
>>>>>>>> (if
>>>>>>>> existent at all), is located in the bus hierarchy of the SoC, it
>>>>>>>> might
>>>>>>>> provoke asynchronous external data-aborts in the CPU core, 
>>>>>>>> instead
>>>>>>>> of
>>>>>>>> synchronous ones. Thereby, it is impossible to recover the state,
>>>>>>>> in
>>>>>>>> which the protection fault was raised.
>>>>>>>> To sum it up, the non-secure guest has to behave cooperatively, 
>>>>>>>> or
>>>>>>>> it
>>>>>>>> will fail. Trap-and-emulate doesn't work in general. Therefore,
>>>>>>>> some
>>>>>>>> lightweight form of para-virtualization of the guest OS is 
>>>>>>>> needed.
>>>>>>>>
>>>>>>>> On ARM platforms, apart from the TrustZone "virtualization", 
>>>>>>>> Genode
>>>>>>>> includes support of L4Linux, a para-virtualized Linux for the
>>>>>>>> Fiasco.OC
>>>>>>>> kernel. ARM's virtualization extensions aren't supported yet, but
>>>>>>>> we'll
>>>>>>>> investigate it certainly.
>>>>>>>>
>>>>>>>>> If so, it would be great to
>>>>>>>>> have an example of how to use it.
>>>>>>>>
>>>>>>>> A working basic example is available on Genode's current staging
>>>>>>>> branch,
>>>>>>>> and will be available in Genode's upcoming release 13.11, that 
>>>>>>>> will
>>>>>>>> be
>>>>>>>> announced this week.
>>>>>>>> The example should work out of the box for ARM's Versatile 
>>>>>>>> Express
>>>>>>>> Coretile A9x4, and Freescale's i.MX53 Quickstart board. You'll 
>>>>>>>> have
>>>>>>>> to
>>>>>>>> create a build directory for 'hw_imx53'. After creating the build
>>>>>>>> directory, you've to adapt the 'etc/specs.conf' file, and add the
>>>>>>>> following SPEC variable:
>>>>>>>>
>>>>>>>>   SPECS += trustzone
>>>>>>>>
>>>>>>>> After that, do a 'make run/tz_vmm' in the build directory. The
>>>>>>>> resulting
>>>>>>>> image is located in 'var/run/tz_vmm/uImage'. The example scenario
>>>>>>>> starts
>>>>>>>> Genode's hw kernel, core, init, and the virtual machine monitor,
>>>>>>>> which
>>>>>>>> will boot Linux with a small busybox initramfs on the non-secure
>>>>>>>> side.
>>>>>>>>
>>>>>>>> A more sophisticated example, which runs on the i.MX53 SABRE 
>>>>>>>> tablet
>>>>>>>> only, can be found on this topic branch:
>>>>>>>>
>>>>>>>> https://github.com/skalk/genode/tree/i.MX53_tablet_demo
>>>>>>>>
>>>>>>>> That example include virtual touchscreen support for the 
>>>>>>>> non-secure
>>>>>>>> guest, so that you can interact with the secure Genode system, 
>>>>>>>> and
>>>>>>>> the
>>>>>>>> non-secure Android guest side-by-side.
>>>>>>>>
>>>>>>>>>
>>>>>>>>> Thanks in advance for answers!
>>>>>>>>
>>>>>>>> You're welcome.
>>>>>>>>
>>>>>>>> Best Regards
>>>>>>>> Stefan
>>>>>>>>
>>>>>>>>>
>>>>>>>>> Best regards,
>>>>>>>>> Pablo Anton
>>>>>>>>>
>>>>>>>>> ------------------------------------------------------------------------------
>>>>>>>>> Rapidly troubleshoot problems before they affect your business.
>>>>>>>>> Most
>>>>>>>>> IT
>>>>>>>>> organizations don't have a clear picture of how application
>>>>>>>>> performance
>>>>>>>>> affects their revenue. With AppDynamics, you get 100% visibility
>>>>>>>>> into
>>>>>>>>> your
>>>>>>>>> Java,.NET, & PHP application. Start your 15-day FREE TRIAL of
>>>>>>>>> AppDynamics Pro!
>>>>>>>>> http://pubads.g.doubleclick.net/gampad/clk?id=84349351&iu=/4140/ostg.clktrk
>>>>>>>>> _______________________________________________
>>>>>>>>> Genode-main mailing list
>>>>>>>>> Genode-main at lists.sourceforge.net
>>>>>>>>> https://lists.sourceforge.net/lists/listinfo/genode-main
>>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>> Stefan Kalkowski
>>>>>>>> Genode Labs
>>>>>>>>
>>>>>>>> http://www.genode-labs.com/ · http://genode.org/
>>>>>>>>
>>>>>>>> ------------------------------------------------------------------------------
>>>>>>>> Rapidly troubleshoot problems before they affect your business.
>>>>>>>> Most
>>>>>>>> IT
>>>>>>>> organizations don't have a clear picture of how application
>>>>>>>> performance
>>>>>>>> affects their revenue. With AppDynamics, you get 100% visibility
>>>>>>>> into
>>>>>>>> your
>>>>>>>> Java,.NET, & PHP application. Start your 15-day FREE TRIAL of
>>>>>>>> AppDynamics Pro!
>>>>>>>> http://pubads.g.doubleclick.net/gampad/clk?id=84349351&iu=/4140/ostg.clktrk
>>>>>>>> _______________________________________________
>>>>>>>> Genode-main mailing list
>>>>>>>> Genode-main at lists.sourceforge.net
>>>>>>>> https://lists.sourceforge.net/lists/listinfo/genode-main
>>>>>>>
>>>>>>> ------------------------------------------------------------------------------
>>>>>>> Rapidly troubleshoot problems before they affect your business. 
>>>>>>> Most
>>>>>>> IT
>>>>>>> organizations don't have a clear picture of how application
>>>>>>> performance
>>>>>>> affects their revenue. With AppDynamics, you get 100% visibility
>>>>>>> into
>>>>>>> your
>>>>>>> Java,.NET, & PHP application. Start your 15-day FREE TRIAL of
>>>>>>> AppDynamics Pro!
>>>>>>> http://pubads.g.doubleclick.net/gampad/clk?id=84349351&iu=/4140/ostg.clktrk
>>>>>>> _______________________________________________
>>>>>>> Genode-main mailing list
>>>>>>> Genode-main at lists.sourceforge.net
>>>>>>> https://lists.sourceforge.net/lists/listinfo/genode-main
>>>>>>>
>>>>>>
>>>>>> --
>>>>>> Stefan Kalkowski
>>>>>> Genode Labs
>>>>>>
>>>>>> http://www.genode-labs.com/ · http://genode.org/
>>>>>>
>>>>>> ------------------------------------------------------------------------------
>>>>>> Rapidly troubleshoot problems before they affect your business. 
>>>>>> Most
>>>>>> IT
>>>>>> organizations don't have a clear picture of how application
>>>>>> performance
>>>>>> affects their revenue. With AppDynamics, you get 100% visibility 
>>>>>> into
>>>>>> your
>>>>>> Java,.NET, & PHP application. Start your 15-day FREE TRIAL of
>>>>>> AppDynamics Pro!
>>>>>> http://pubads.g.doubleclick.net/gampad/clk?id=84349351&iu=/4140/ostg.clktrk
>>>>>> _______________________________________________
>>>>>> Genode-main mailing list
>>>>>> Genode-main at lists.sourceforge.net
>>>>>> https://lists.sourceforge.net/lists/listinfo/genode-main
>>>>>
>>>>> ------------------------------------------------------------------------------
>>>>> Rapidly troubleshoot problems before they affect your business. Most
>>>>> IT
>>>>> organizations don't have a clear picture of how application
>>>>> performance
>>>>> affects their revenue. With AppDynamics, you get 100% visibility 
>>>>> into
>>>>> your
>>>>> Java,.NET, & PHP application. Start your 15-day FREE TRIAL of
>>>>> AppDynamics Pro!
>>>>> http://pubads.g.doubleclick.net/gampad/clk?id=84349351&iu=/4140/ostg.clktrk
>>>>> _______________________________________________
>>>>> Genode-main mailing list
>>>>> Genode-main at lists.sourceforge.net
>>>>> https://lists.sourceforge.net/lists/listinfo/genode-main
>>>>>
>>>>
>>>> --
>>>> Stefan Kalkowski
>>>> Genode Labs
>>>>
>>>> http://www.genode-labs.com/ · http://genode.org/
>>>>
>>>> ------------------------------------------------------------------------------
>>>> Rapidly troubleshoot problems before they affect your business. Most 
>>>> IT
>>>> organizations don't have a clear picture of how application 
>>>> performance
>>>> affects their revenue. With AppDynamics, you get 100% visibility into
>>>> your
>>>> Java,.NET, & PHP application. Start your 15-day FREE TRIAL of
>>>> AppDynamics Pro!
>>>> http://pubads.g.doubleclick.net/gampad/clk?id=84349351&iu=/4140/ostg.clktrk
>>>> _______________________________________________
>>>> Genode-main mailing list
>>>> Genode-main at lists.sourceforge.net
>>>> https://lists.sourceforge.net/lists/listinfo/genode-main
>>>
>>> ------------------------------------------------------------------------------
>>> Sponsored by Intel(R) XDK
>>> Develop, test and display web and hybrid apps with a single code base.
>>> Download it for free now!
>>> http://pubads.g.doubleclick.net/gampad/clk?id=111408631&iu=/4140/ostg.clktrk
>>> _______________________________________________
>>> Genode-main mailing list
>>> Genode-main at lists.sourceforge.net
>>> https://lists.sourceforge.net/lists/listinfo/genode-main
>>>
>>
>> --
>> Stefan Kalkowski
>> Genode Labs
>>
>> http://www.genode-labs.com/ · http://genode.org/
>>
>> ------------------------------------------------------------------------------
>> Sponsored by Intel(R) XDK
>> Develop, test and display web and hybrid apps with a single code base.
>> Download it for free now!
>> http://pubads.g.doubleclick.net/gampad/clk?id=111408631&iu=/4140/ostg.clktrk
>> _______________________________________________
>> Genode-main mailing list
>> Genode-main at lists.sourceforge.net
>> https://lists.sourceforge.net/lists/listinfo/genode-main
> 
> ------------------------------------------------------------------------------
> Rapidly troubleshoot problems before they affect your business. Most IT 
> organizations don't have a clear picture of how application performance 
> affects their revenue. With AppDynamics, you get 100% visibility into your 
> Java,.NET, & PHP application. Start your 15-day FREE TRIAL of AppDynamics Pro!
> http://pubads.g.doubleclick.net/gampad/clk?id=84349831&iu=/4140/ostg.clktrk
> _______________________________________________
> Genode-main mailing list
> Genode-main at lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/genode-main
> 

-- 
Stefan Kalkowski
Genode Labs

http://www.genode-labs.com/ · http://genode.org/