Hello Guys,
I am experimenting with NOVA Microkernel and trying to run some code in guest mode.
I am attaching test program which usage NOVA syscall binding interface to create test program. Objective of this test program to run cpuid and outb instruction in guest mode on intel machine. Here, I am launching intel vcpu in real mode paging disable with all necessary register setup properly in vcpu startup portal handler.
Once you download this code, run make and modify run.sh to point NOVA hypervisor.
I want to discuss two problem.
Problem -1: This test program runs (guest mode code) cpuid instruction but outb/inb instruction never runs in guest mode. I see that for outb instruction gives IO portal invoked again and again. First, I delegate IO ports from Kernel in root pd. My vcpu execution context is created in root pd. I believe that IO ports should be accessible from guest mode after delegating IO ports from kernel to guest. Secondly, same code for AMD runs if I set proper AMD vcpu registers in startup handler.
Problem -2 As Intel process supports unrestricted guest mode; Advantage of that, I can run real mode instruction in vCPU context with paging disable without doing any instruction interpretations.
I set Unrestricted guest mode control bit in VMCS control registers, though intel vCPU never runs guest in unrestricted mode. But When I experiments this with KVM it works. My question is, does NOVA kernel supports to run intel cpu in unrestricted guest mode? or is there any trick to do that so. Please help me if I am missing something.
Any help regarding these two issues would be very helpful.
Regards Avinash Singh
On Thu, 7 Nov 2013 20:35:18 +0530 avinash singh (AS) wrote:
Hi Avinash,
AS> I want to discuss two problem. AS> AS> Problem -1: AS> This test program runs (guest mode code) cpuid instruction but outb/inb AS> instruction never runs in guest mode. I see that for outb instruction AS> gives IO portal invoked again and again. First, I AS> delegate IO ports from Kernel in root pd. My vcpu execution context AS> is created in root pd. I believe that IO ports should be accessible AS> from guest mode after delegating IO ports from kernel to guest. AS> Secondly, same code for AMD runs if I set proper AMD vcpu registers in AS> startup handler.
NOVA distinguishes between I/O ports being accessible in host mode or guest mode. If you delegate an I/O CRD into a PD, the ports will be accessible in host mode only, unless you also set the G-bit in the typed item. Only if you set the G-bit during the delegation will the I/O ports be directly accessible in guest mode without causing a VM exit. See spec. 4.6.2.2
AS> Problem -2 AS> As Intel process supports unrestricted guest mode; Advantage of that, AS> I can run real mode instruction in vCPU context with paging disable AS> without doing any instruction interpretations. AS> AS> I set Unrestricted guest mode control bit in VMCS control registers, AS> though intel vCPU never runs guest in unrestricted mode. But When I AS> experiments this with KVM it works.
A user-level VMM should never have to deal with the unrestricted guest control bit. In fact, that bit is under the control of the hypervisor and will be forced to 0 or 1, depending on HW support, irrespective of what the user-level VMM wants.
AS> My question is, does NOVA kernel supports to run intel cpu in AS> unrestricted guest mode? or is there any trick to do that so. Please AS> help me if I am missing something.
NOVA automatically enables unrestricted guest if the CPU supports it. The feature is supported on processors based on the Westmere or newer microarchitectures. Unrestricted guest will be disabled, if the user forces vTLB using the command line.
AS> Any help regarding these two issues would be very helpful.
Can you enable TRACE_VMX in include/stdio.h (move it out of the #ifdef) and post the output of the microhypervisor, in particular the line that talks about VMCS, EPT, URG, VPID, etc.?
Cheers, Udo
Hi Udo,
Thanks for your response.
Please find my response below under your reply starting with Avinash.
On Fri, Nov 8, 2013 at 3:11 AM, Udo Steinberg <udo@...121...> wrote:
On Thu, 7 Nov 2013 20:35:18 +0530 avinash singh (AS) wrote:
Hi Avinash,
AS> I want to discuss two problem. AS> AS> Problem -1: AS> This test program runs (guest mode code) cpuid instruction but
outb/inb
AS> instruction never runs in guest mode. I see that for outb instruction AS> gives IO portal invoked again and again. First, I AS> delegate IO ports from Kernel in root pd. My vcpu execution context AS> is created in root pd. I believe that IO ports should be accessible AS> from guest mode after delegating IO ports from kernel to guest. AS> Secondly, same code for AMD runs if I set proper AMD vcpu registers in AS> startup handler.
NOVA distinguishes between I/O ports being accessible in host mode or
guest
mode. If you delegate an I/O CRD into a PD, the ports will be accessible
in
host mode only, unless you also set the G-bit in the typed item. Only if
you
set the G-bit during the delegation will the I/O ports be directly accessible in guest mode without causing a VM exit. See spec. 4.6.2.2
Avinash: ->> I do understand that without setting G bit in typed item resource wont be available for guest to access.
In My code, I am launching vcPU in protected mode with paging disabled with 'nonpt' command line option. I guest IO_ACCESS_PORTAL invoked for outb instruction as expected. In vcpu IO access portal handler, I delegate IO ports to guest. Please see below code in file main.cc line number:293
270 static void 271 vcpu_ioaccess_handler(void) 272 { 273 Utcb *utcb = reinterpret_cast<Utcb *>(ECHO_THREAD_UTCB); 274 275 out()->text("IOACCES invoked \n"); 276 277 out()->text("IP "); 278 out()->hex(utcb->ip); 279 out()->text("\n"); 280 281 out()->text("dx "); 282 out()->hex(utcb->dx); 283 out()->text("\n"); 284 285 static mword_t prev_dx = ~0; 286 if (prev_dx == utcb->dx) { 287 out()->text("IOACCESS: Duplicate Request...\n"); 288 HALT(); 289 } 290 prev_dx = utcb->dx; 291 292 utcb->set_msg_word(0); 293 * if (! utcb->append_item(Io_crd(0, 16), 0, true, true)) {* 294 } 295 296 reply(reinterpret_cast<void *>(event_ec_stack_top())); 297 } 298
However, I also modified my test program to delegate IO ports upfront before launching vcpu. But I still get vcpu_ioaccess_handler invoked. Please see below,
enum { ORDER_64k = 16, 341 MAP_FROM_KERNEL = 1, 342 HOTSPOT = 0, 343 }; 344 utcb_echo->crd_rcv = Io_crd(0, ORDER_64k); 345 utcb_main->set_msg_word(0); 346 347 bool success = utcb_main->append_item(Io_crd(0, ORDER_64k), HOTSPOT, MAP_FROM_KERNEL, true); 348 if(!success) 349 *(unsigned long *) (~0UL -6) = 0;
I am not sure what is wrong here. Because same logic works for AMD process. Please could you just see main.cc if something is I am doing wrong there. Goal is to run cpuid ad oubt instruction in guest mode with paging disabled and protected mode bit on.
AS> Problem -2 AS> As Intel process supports unrestricted guest mode; Advantage of that, AS> I can run real mode instruction in vCPU context with paging disable AS> without doing any instruction interpretations. AS> AS> I set Unrestricted guest mode control bit in VMCS control registers, AS> though intel vCPU never runs guest in unrestricted mode. But When I AS> experiments this with KVM it works.
A user-level VMM should never have to deal with the unrestricted guest control bit. In fact, that bit is under the control of the hypervisor and will be forced to 0 or 1, depending on HW support, irrespective of what
the
user-level VMM wants.
Avinash>> You are right this feature depends on processor .
AS> My question is, does NOVA kernel supports to run intel cpu in AS> unrestricted guest mode? or is there any trick to do that so. Please AS> help me if I am missing something.
NOVA automatically enables unrestricted guest if the CPU supports it. The feature is supported on processors based on the Westmere or newer microarchitectures. Unrestricted guest will be disabled, if the user
forces
vTLB using the command line.
Avinash>> I was trying to run same test program with paging disabled and in real mode on Core i7 processor in qemu-kvm environment. does this combination works in running guest in unrestricted mode on top of NOVA?
AS> Any help regarding these two issues would be very helpful.
Can you enable TRACE_VMX in include/stdio.h (move it out of the #ifdef)
and
post the output of the microhypervisor, in particular the line that talks about VMCS, EPT, URG, VPID, etc.?
Avinash>> I will be sending trac_VMX ouput below,
NOVA Microhypervisor v6-d5b54fa (x86_32): Nov 7 2013 22:59:09 [gcc 4.7.3]
[ 0] Warning IA32_FEATURE_CONTROL not set which ok only with qemu/kvm. [ 0] VMCS:0x017ef000 REV:0x11e57ed0 EPT:0 URG:0 VNMI:1 VPID:0 [ 0] CORE:0:0:0 6:f:b:0 [1] Intel(R) Core(TM)2 Duo CPU T7700 @ 2.40GHz Hello world All Portal created! runcpuid addr 0x02001000 eip is initialized 0x00001000 cs 0x02000000 npt fault handler .. host_adddr 0x01001000 got cpuid intercept! IOACCES invoked IP 0x00001007 dx 0x000003f8 IOACCES invoked IP 0x00001007 dx 0x000003f8 IOACCESS: Duplicate Request... [ 0] Killed EC:0xc0010300 SC:0xc0011180 V:0xd CS:0x1b EIP:0x1001717 CR2:0x0 ERR:0x0 (PT not found) [ 0] Killed EC:0xc0010400 SC:0xc0011180 V:0x1e CR0:0x10021 CR3:0x0 CR4:0x0 (IPC Abort)
Cheers, Udo
On Fri, 8 Nov 2013 12:30:10 +0530 avinash singh (AS) wrote:
AS> However, I also modified my test program to delegate IO ports upfront AS> before launching vcpu. But I still get vcpu_ioaccess_handler invoked. AS> Please see below,
In line 347, there is the following piece of code:
bool success = utcb_main->append_item(Io_crd(0, ORDER_64k), HOTSPOT, MAP_FROM_KERNEL, false);
I added "false" at the end to make the difference explicit. When running that piece of code I'm getting the following output:
NOVA Microhypervisor v6-d5b54fa (x86_32): Sep 5 2013 16:18:28 [gcc 4.9.0]
[ 0] CORE:0:0:0 6:3c:3:1 [10] Intel(R) Core(TM) i7-4770 CPU @ 3.40GHz [ 3] CORE:0:3:0 6:3c:3:1 [10] Intel(R) Core(TM) i7-4770 CPU @ 3.40GHz [ 1] CORE:0:1:0 6:3c:3:1 [10] Intel(R) Core(TM) i7-4770 CPU @ 3.40GHz [ 7] CORE:0:3:1 6:3c:3:1 [10] Intel(R) Core(TM) i7-4770 CPU @ 3.40GHz [ 5] CORE:0:1:1 6:3c:3:1 [10] Intel(R) Core(TM) i7-4770 CPU @ 3.40GHz [ 4] CORE:0:0:1 6:3c:3:1 [10] Intel(R) Core(TM) i7-4770 CPU @ 3.40GHz [ 6] CORE:0:2:1 6:3c:3:1 [10] Intel(R) Core(TM) i7-4770 CPU @ 3.40GHz [ 2] CORE:0:2:0 6:3c:3:1 [10] Intel(R) Core(TM) i7-4770 CPU @ 3.40GHz Hello world All Portal created! runcpuid addr 0x02001000 eip is initialized 0x00001000 cs 0x02000000 npt fault handler .. host_adddr 0x01001000 got cpuid intercept! IOACCES invoked IP 0x00001007 dx 0x000003f8 IOACCES invoked IP 0x00001007 dx 0x000003f8 IOACCESS: Duplicate Request... [ 0] Killed EC:0xc0011a00 SC:0xc0013500 V:0xd CS:0x1b EIP:0x1001717 CR2:0x0 ERR:0x0 (PT not found) [ 0] Killed EC:0xc0011b00 SC:0xc0013500 V:0x1e CR0:0x10031 CR3:0x0 CR4:0x0 (IPC Abort)
As expected, I/O accesses cause VM exits because the G-bit has not been set.
So then I've changed line 347 as follows:
bool success = utcb_main->append_item(Io_crd(0, ORDER_64k), HOTSPOT, MAP_FROM_KERNEL, true);
and I'm getting the following output:
NOVA Microhypervisor v6-d5b54fa (x86_32): Sep 5 2013 16:18:28 [gcc 4.9.0]
[ 0] CORE:0:0:0 6:3c:3:1 [10] Intel(R) Core(TM) i7-4770 CPU @ 3.40GHz [ 4] CORE:0:0:1 6:3c:3:1 [10] Intel(R) Core(TM) i7-4770 CPU @ 3.40GHz [ 2] CORE:0:2:0 6:3c:3:1 [10] Intel(R) Core(TM) i7-4770 CPU @ 3.40GHz [ 6] CORE:0:2:1 6:3c:3:1 [10] Intel(R) Core(TM) i7-4770 CPU @ 3.40GHz [ 3] CORE:0:3:0 6:3c:3:1 [10] Intel(R) Core(TM) i7-4770 CPU @ 3.40GHz [ 5] CORE:0:1:1 6:3c:3:1 [10] Intel(R) Core(TM) i7-4770 CPU @ 3.40GHz [ 7] CORE:0:3:1 6:3c:3:1 [10] Intel(R) Core(TM) i7-4770 CPU @ 3.40GHz [ 1] CORE:0:1:0 6:3c:3:1 [10] Intel(R) Core(TM) i7-4770 CPU @ 3.40GHz Hello world All Portal created! runcpuid addr 0x02001000 eip is initialized 0x00001000 cs 0x02000000 npt fault handler .. host_adddr 0x01001000 got cpuid intercept! Egot cpuid intercept! [ 0] Killed EC:0xc0012b00 SC:0xc0014500 V:0xc CR0:0x10031 CR3:0x0 CR4:0x0 (PT not found)
The EC is killed due to an unhandled HLT VM exit, which is expected. As you can see, there are no I/O exits and everything works as expected. Also note that the 'E' from startup.s line 46 is being printed to serial line in front of the CPUID output.
AS> I am not sure what is wrong here. Because same logic works for AMD process. AS> Please could you just see main.cc if something is I am doing wrong there. AS> Goal is to run cpuid ad oubt instruction in guest mode with paging disabled AS> and protected mode bit on.
I believe the problem is that you run NOVA as a nested hypervisor under KVM and if KVM is not virtualizing VMX correctly, then you get these kind of problems. Can you try running your tests on bare hardware (as I did) to see if they work correctly there?
AS> > AS> My question is, does NOVA kernel supports to run intel cpu in AS> > AS> unrestricted guest mode? or is there any trick to do that so. Please AS> > AS> help me if I am missing something. AS> > AS> > NOVA automatically enables unrestricted guest if the CPU supports it. The AS> > feature is supported on processors based on the Westmere or newer AS> > microarchitectures. Unrestricted guest will be disabled, if the user AS> AS> [ 0] VMCS:0x017ef000 REV:0x11e57ed0 EPT:0 URG:0 VNMI:1 VPID:0 AS> [ 0] CORE:0:0:0 6:f:b:0 [1] Intel(R) Core(TM)2 Duo CPU T7700 @ 2.40GHz
First of all, the VMCS revision identifier looks really odd. Secondly, the virtual CPU claims not to support EPT or unrestricted guest, as you can see from the EPT and URG fields both showing a 0. This means KVM is not emulating EPT/URG support and thus NOVA cannot enable it. Again, running the test on bare hardware may be a better idea, because you rule out KVM as a potential source of bugs.
Let me know what else you find out.
Cheers, Udo
Hi Udo,
Thanks for your reply. When I run my test cases on real physical hardware I can run it through properly and also run in unrestricted guest mode. Looks like Nested VT on Qemu_KVM environment is buggy. Anyway thanks for looking into this problem and giving nice thought.
Regards Avinash.
On Mon, Nov 11, 2013 at 5:27 PM, Udo Steinberg <udo@...121...> wrote:
On Fri, 8 Nov 2013 12:30:10 +0530 avinash singh (AS) wrote:
AS> However, I also modified my test program to delegate IO ports upfront AS> before launching vcpu. But I still get vcpu_ioaccess_handler invoked. AS> Please see below,
In line 347, there is the following piece of code:
bool success = utcb_main->append_item(Io_crd(0, ORDER_64k), HOTSPOT, MAP_FROM_KERNEL, false);
I added "false" at the end to make the difference explicit. When running that piece of code I'm getting the following output:
NOVA Microhypervisor v6-d5b54fa (x86_32): Sep 5 2013 16:18:28 [gcc 4.9.0]
[ 0] CORE:0:0:0 6:3c:3:1 [10] Intel(R) Core(TM) i7-4770 CPU @ 3.40GHz [ 3] CORE:0:3:0 6:3c:3:1 [10] Intel(R) Core(TM) i7-4770 CPU @ 3.40GHz [ 1] CORE:0:1:0 6:3c:3:1 [10] Intel(R) Core(TM) i7-4770 CPU @ 3.40GHz [ 7] CORE:0:3:1 6:3c:3:1 [10] Intel(R) Core(TM) i7-4770 CPU @ 3.40GHz [ 5] CORE:0:1:1 6:3c:3:1 [10] Intel(R) Core(TM) i7-4770 CPU @ 3.40GHz [ 4] CORE:0:0:1 6:3c:3:1 [10] Intel(R) Core(TM) i7-4770 CPU @ 3.40GHz [ 6] CORE:0:2:1 6:3c:3:1 [10] Intel(R) Core(TM) i7-4770 CPU @ 3.40GHz [ 2] CORE:0:2:0 6:3c:3:1 [10] Intel(R) Core(TM) i7-4770 CPU @ 3.40GHz Hello world All Portal created! runcpuid addr 0x02001000 eip is initialized 0x00001000 cs 0x02000000 npt fault handler .. host_adddr 0x01001000 got cpuid intercept! IOACCES invoked IP 0x00001007 dx 0x000003f8 IOACCES invoked IP 0x00001007 dx 0x000003f8 IOACCESS: Duplicate Request... [ 0] Killed EC:0xc0011a00 SC:0xc0013500 V:0xd CS:0x1b EIP:0x1001717 CR2:0x0 ERR:0x0 (PT not found) [ 0] Killed EC:0xc0011b00 SC:0xc0013500 V:0x1e CR0:0x10031 CR3:0x0 CR4:0x0 (IPC Abort)
As expected, I/O accesses cause VM exits because the G-bit has not been set.
So then I've changed line 347 as follows:
bool success = utcb_main->append_item(Io_crd(0, ORDER_64k), HOTSPOT, MAP_FROM_KERNEL, true);
and I'm getting the following output:
NOVA Microhypervisor v6-d5b54fa (x86_32): Sep 5 2013 16:18:28 [gcc 4.9.0]
[ 0] CORE:0:0:0 6:3c:3:1 [10] Intel(R) Core(TM) i7-4770 CPU @ 3.40GHz [ 4] CORE:0:0:1 6:3c:3:1 [10] Intel(R) Core(TM) i7-4770 CPU @ 3.40GHz [ 2] CORE:0:2:0 6:3c:3:1 [10] Intel(R) Core(TM) i7-4770 CPU @ 3.40GHz [ 6] CORE:0:2:1 6:3c:3:1 [10] Intel(R) Core(TM) i7-4770 CPU @ 3.40GHz [ 3] CORE:0:3:0 6:3c:3:1 [10] Intel(R) Core(TM) i7-4770 CPU @ 3.40GHz [ 5] CORE:0:1:1 6:3c:3:1 [10] Intel(R) Core(TM) i7-4770 CPU @ 3.40GHz [ 7] CORE:0:3:1 6:3c:3:1 [10] Intel(R) Core(TM) i7-4770 CPU @ 3.40GHz [ 1] CORE:0:1:0 6:3c:3:1 [10] Intel(R) Core(TM) i7-4770 CPU @ 3.40GHz Hello world All Portal created! runcpuid addr 0x02001000 eip is initialized 0x00001000 cs 0x02000000 npt fault handler .. host_adddr 0x01001000 got cpuid intercept! Egot cpuid intercept! [ 0] Killed EC:0xc0012b00 SC:0xc0014500 V:0xc CR0:0x10031 CR3:0x0 CR4:0x0 (PT not found)
The EC is killed due to an unhandled HLT VM exit, which is expected. As you can see, there are no I/O exits and everything works as expected. Also note that the 'E' from startup.s line 46 is being printed to serial line in front of the CPUID output.
AS> I am not sure what is wrong here. Because same logic works for AMD process. AS> Please could you just see main.cc if something is I am doing wrong there. AS> Goal is to run cpuid ad oubt instruction in guest mode with paging disabled AS> and protected mode bit on.
I believe the problem is that you run NOVA as a nested hypervisor under KVM and if KVM is not virtualizing VMX correctly, then you get these kind of problems. Can you try running your tests on bare hardware (as I did) to see if they work correctly there?
AS> > AS> My question is, does NOVA kernel supports to run intel cpu in AS> > AS> unrestricted guest mode? or is there any trick to do that so. Please AS> > AS> help me if I am missing something. AS> > AS> > NOVA automatically enables unrestricted guest if the CPU supports it. The AS> > feature is supported on processors based on the Westmere or newer AS> > microarchitectures. Unrestricted guest will be disabled, if the user AS> AS> [ 0] VMCS:0x017ef000 REV:0x11e57ed0 EPT:0 URG:0 VNMI:1 VPID:0 AS> [ 0] CORE:0:0:0 6:f:b:0 [1] Intel(R) Core(TM)2 Duo CPU T7700 @ 2.40GHz
First of all, the VMCS revision identifier looks really odd. Secondly, the virtual CPU claims not to support EPT or unrestricted guest, as you can see from the EPT and URG fields both showing a 0. This means KVM is not emulating EPT/URG support and thus NOVA cannot enable it. Again, running the test on bare hardware may be a better idea, because you rule out KVM as a potential source of bugs.
Let me know what else you find out.
Cheers, Udo
On Mon, 11 Nov 2013 22:01:27 +0530 avinash singh (AS) wrote:
AS> Thanks for your reply. When I run my test cases on real physical hardware AS> I can run it through properly and also run in unrestricted guest mode. AS> Looks like Nested VT on Qemu_KVM environment is buggy. Anyway thanks for AS> looking into this problem and giving nice thought.
Hi Avinash,
regarding running test cases, you could talk to Alexander Böttcher, who has created a unit test framework for Genode. Another option is the novaboot project by Michal Sojka (https://github.com/wentasah/novaboot).
What has worked well for us in the past is a combination of the novaboot script and the Pulsar PXE boot loader to boot test cases on bare HW on a nightly basis.
Cheers, Udo
On 13.11.2013 00:12, Udo Steinberg wrote:
regarding running test cases, you could talk to Alexander Böttcher, who has created a unit test framework for Genode.
Just to be fair, the unit test framework was already there when I joined Genode.
I mainly added support to the framework to boot the unit test cases on bare HW for x86 and various ARM boards.
For x86 we are actually now using the mentioned Pulsar PXE boot loader to run tests on all x86 flavours of Genode (Fiasco.OC, NOVA, OKL4, Pistachio, Fiasco/L4) every night.
So, Pulsar PXE is definitely a good choice to ease your life.
project by Michal Sojka (https://github.com/wentasah/novaboot).
Appreciated novaboot really during my active work on NOVA/NUL and still a good choice.
Cheers,
Alex.
-
Alexander Boettcher -
avinash singh -
Udo Steinberg