VMWare and UDEREF

[spender]

I'm making a sticky post for this, as some users may get bit by a recent necessary change to PaX's UDEREF (made a few days ago). Previously, UDEREF would detect the VMWare I/O backdoor, and if detected, would disable UDEREF for the guest. This was to avoid a conflict with VMWare that would cause incredibly poor performance on the guest. There now exist options available through the main interface of VMWare to choose to specifically use hardware-based virtualization for VM guests. The use of these options have no conflicts with UDEREF, so disabling UDEREF at startup in these cases was unnecessary.

If you're booting a VM guest with UDEREF enabled and the "Preferred mode" under Virtual Machine Settings -> Hardware -> Processors -> Execution Mode is set to "Automatic" or "Binary Translation", you need to:
Append "pax_nouderef" to your kernel command line
Check the "Disable acceleration for binary translation" option
If the second operation listed here is not done, there's some bug in VMWare's binary translation that causes the same poor performance even with UDEREF enabled but pax_nouderef used

Alternatively, you choose one of the two below options at some performance cost to keep UDEREF without the significant impact of it with binary translation:
Choose "Intel VT-x or AMD-v" as the "Preferred mode"
Choose "Intel VT-x/EPT or AMD-V/RVI" as the Preferred mode"

-Brad

[coderx]

"removed the vmware auto-detection code that disabled UDEREF on boot"
so why it is removed ?

[elazar]

I noticed that the kernel won't boot if paravirtualization is enabled on the VM(and in the kernel).

[PaX Team]

I noticed that the kernel won't boot if paravirtualization is enabled on the VM(and in the kernel).

do you have CONFIG_CC_STACKPROTECTOR enabled in your .config? i get a very early boot failure with it, but paravirt/VMI works otherwise (minus the usual features like KERNEXEC/UDEREF). can you also test a vanilla kernel and report this upstream if it is indeed a problem with SSP and paravirt/VMI?

[elazar]

root@box:/usr/src/linux-2.6.31-grsec# cat .config | grep CONFIG_CC
CONFIG_CC_OPTIMIZE_FOR_SIZE=y
# CONFIG_CC_STACKPROTECTOR is not set

[PaX Team]

root@box:/usr/src/linux-2.6.31-grsec# cat .config | grep CONFIG_CC
CONFIG_CC_OPTIMIZE_FOR_SIZE=y
# CONFIG_CC_STACKPROTECTOR is not set

ok, so i'll need more information about the boot failure, possibly vmlinux and bzImage and any boot logs/screenshots you can get.

[elazar]

I am getting looping reboots with 2.6.31.1 using grsecurity-2.1.14-2.6.31.1-200910012153, though nothing useful is displayed on the screen. System.map, bzImage etc. are at http://drop.io/qi0g5ga/asset/2-6-31-1-grsec-gz. This is a virtual machine running on ESXi 4, build 181792 on a PowerEdge T300(12gb RAM, Core2 Duo E6305), the VM has both processors, NX, and 768MB allocated, paravirtualization is enabled and CPU/MMU Virtualization is set to automatic. I will test a vanilla kernel in the morning.

elazar

[PaX Team]

System.map, bzImage etc. are at http://drop.io/qi0g5ga/asset/2-6-31-1-grsec-gz.

can you upload it in a format i can decode? looks like this is a concatenation of all files, i don't exactly feel like figuring out the file boundaries by hand

[elazar]

Uhg, my bad, this: http://drop.io/qi0g5ga/asset/2-6-31-1-grsec-tar-gz should work

[elazar]

Vanilla 2.6.31.3 works fine, and I get looping reboots with the latest 2.6.31.3 patch.

Edit:
I'm seeing this in the virtual machine's event log:

Message from virtualhost: *** Virtual
machine kernel stack fault (hardware reset) ***
The virtual machine just suffered a stack fault in
kernel mode. On a real computer, this would
amount to a reset of the processor. It can be
caused by an incorrect configuration of the
virtual machine, a bug in the operating system,
or a problem in the VMware ESX software. Press
OK to reboot virtual machine or Cancel to shut it
down.
info
10/12/2009 5:22:31 PM
User

Edit 2:
I never posted my binutils/gcc version(s):
GNU ld (Linux/GNU Binutils) 2.18.50.0.9.20080822
gcc version 4.3.3 (GCC)

[elazar]

Disabling uderef via no_paxuderef=1 allows 2.6.31 to boot fine when paravirt is enabled on the VM itself, though 2.6.31.3 still triggers a kernel mode stack fault no matter how the VM is configured, it will only boot if I compile without paravirt enabled in the kernel.

[PaX Team]

Disabling uderef via no_paxuderef=1

it's pax_nouderef and it's an 'off-only' switch, it doesn't take arguments.

though 2.6.31.3 still triggers a kernel mode stack fault no matter how the VM is configured, it will only boot if I compile without paravirt enabled in the kernel.

i've been working on getting KERNEXEC to work with VMI for the past few days and it can boot into userland here albeit still dies randomly on certain page table operations. without KERNEXEC it should have always been fine though.

[elazar]

it's pax_nouderef and it's an 'off-only' switch, it doesn't take arguments.

Woops, forgot the no part. Actually, pax_nouderef on its own did not work, pax_nouderef=1 worked.

i've been working on getting KERNEXEC to work with VMI for the past few days and it can boot into userland here albeit still dies randomly on certain page table operations. without KERNEXEC it should have always been fine though.

I can boot 2.6.31.4 with pax_nouderef, paravirt compiled in and enabled on the VM but various processes(udev, bash, syslogd etc) die with general protection faults at boot time. It does not work any other way. 2.6.31 works, but with paravirt disabled on the VM itself.

I really appreciate the time that you have been putting in to this. If you need an ESXi box to test on, please PM me.

Thanks!

[PaX Team]

Woops, forgot the no part. Actually, pax_nouderef on its own did not work, pax_nouderef=1 worked.

hmm, that must be a bug, the code doesn't need or check for any extra arguments and i've been using it under vmware workstation as such.

I can boot 2.6.31.4 with pax_nouderef, paravirt compiled in and enabled on the VM but various processes(udev, bash, syslogd etc) die with general protection faults at boot time. It does not work any other way. 2.6.31 works, but with paravirt disabled on the VM itself.

was this with KERNEXEC on or off? if it was on, can you try without?

I really appreciate the time that you have been putting in to this. If you need an ESXi box to test on, please PM me.

i don't think i'll need it as i get the GPFs here as well, although their random nature makes me think that it may no longer be my fault but something else, but it's very hard to debug as it's the hypervisor that simulates a GPF on certain page table operations.

[elazar]

Disabling KERNEXEC did the trick