vmware conflict in stable / test patch + 3/4.X.X kernels

Discuss usability issues, general maintenance, and general support issues for a grsecurity-enabled system.

Moderators: spender, PaX Team

vmware conflict in stable / test patch + 3/4.X.X kernels

Postby nixor » Mon Jun 01, 2015 12:30 am

Hi Guys !

Hope you got a spare minute or so to advise on this mystery so far:

- OS: Debian 8

Problem: vmware workstation 11.1 rebooting host instantly the moment any guest is started

Tried: (custom and automatic config of grsec, recompiled vmware modules with each kernel)

- kernel 3.14.40 with grsec patch = reboot
- kernel 3.14.43 with grsec patch = reboot
- kernel 3.2.69 with grsec patch = reboot
- kernel 4.01 with grsec test patch = reboot
- stock kernel 3.16.X without grsec patch = works

-> After the above, i disabled grsecurity and recompiled the kernels from the above list but still result = reboot

So at this point i thought what makes the stock kernel 3.16.x work and all the others don't ? The only thing
i came up with is that despite grsecurity was off the patch was applied to a load of files. So what i did is unpack the
3.14.43 vanilla kernel again but this time did not apply the grsec patch, guess what ? vmware workstation worked ! No reboot occured
when starting any guest !

Now here is where i'm stuck, it seems that the grsec patch when applied and even if it is turned "off" for some reason is causing
the reboot issue.

Kindly advise what this means or what is left to check ? As i really want the grsec patch and not just a vanilla kernel.

Thank you guys !
Last edited by nixor on Mon Jun 01, 2015 7:41 pm, edited 2 times in total.
nixor
 
Posts: 5
Joined: Mon Jun 01, 2015 12:13 am

Re: Possible conflict in stable / test patch + 3.X.X kernels

Postby PaX Team » Mon Jun 01, 2015 9:29 am

you could try to use the PaX patch alone and/or boot with nopcid to see if that makes a difference. other than that, it'd take some non-trivial debugging to find out the root cause, i'm afraid i won't have time for this anytime soon.
PaX Team
 
Posts: 2310
Joined: Mon Mar 18, 2002 4:35 pm

Re: Possible conflict in stable / test patch + 3.X.X kernels

Postby nixor » Mon Jun 01, 2015 7:04 pm

PaX Team wrote:you could try to use the PaX patch alone and/or boot with nopcid to see if that makes a difference. other than that, it'd take some non-trivial debugging to find out the root cause, i'm afraid i won't have time for this anytime soon.


So far i tried nopcid and pax_nouderef but didn't work out, starting any guest in vmware causes an instant reboot. Didn't find anything in the logs as it reboots instantly.
Not sure what could cause this kind of instant reboot by simply trying to start a guest.

If you could make a rough guess without debugging, is there anything i can try to disable / undo done by the patch ? (while grsec / pax are still "off" in the config)

Or is there any way i could gather some data before the instant reboot ? Right now i can't find anything in the logs.
nixor
 
Posts: 5
Joined: Mon Jun 01, 2015 12:13 am

Re: vmware conflict in stable / test patch + 3/4.X.X kernels

Postby spender » Mon Jun 01, 2015 8:30 pm

Add a serial port to the VM, have it log to a file. Make sure "quiet" is removed from the kernel commandline and add console=ttyS0. Did you try the PaX patch only, from https://grsecurity.net/~paxguy1/ ?

-Brad
spender
 
Posts: 2185
Joined: Wed Feb 20, 2002 8:00 pm
Location: VA, USA

Re: vmware conflict in stable / test patch + 3/4.X.X kernels

Postby nixor » Mon Jun 01, 2015 8:49 pm

spender wrote:Add a serial port to the VM, have it log to a file. Make sure "quiet" is removed from the kernel commandline and add console=ttyS0. Did you try the PaX patch only, from https://grsecurity.net/~paxguy1/ ?

-Brad


Thank you, i'm going to try the pax patch alone and will update here how it went. Btw do you mean removing the quiet from the host kernel commandline and adding console=ttyS0 ? Or the guest ? Normally when starting a guest i see the vmware bios screen flash by but the system reboots so fast that the vmware bios screen doesn't even show up.
nixor
 
Posts: 5
Joined: Mon Jun 01, 2015 12:13 am

Re: vmware conflict in stable / test patch + 3/4.X.X kernels

Postby nixor » Mon Jun 01, 2015 9:44 pm

OK, results:

- serialport to the VM logging to a file = no data, file doesn't even get created
- applied pax patch alone to vanilla 3.14.43 = same behaviour, instant reboot of host when a guest starts
- no info found in kern.log and other showing what happens

NOTE: pax wasn't even enabled in the config, only applied the patch

I'm guessing that something in the pax patch modifies another "something" causing these instant reboots when a guest is started :(
nixor
 
Posts: 5
Joined: Mon Jun 01, 2015 12:13 am

Re: vmware conflict in stable / test patch + 3/4.X.X kernels

Postby PaX Team » Tue Jun 02, 2015 6:53 am

PaX has several non-configurable changes, some of them affect low-level CPU specific things (my guess would be the changes to the per-cpu GDT) which is what probably makes vmware trigger a triple fault. debugging this would require setting up nested virtualization and catching the first guest's kernel (where the vmware 'host' code runs) as it's about to trigger the triple fault.
PaX Team
 
Posts: 2310
Joined: Mon Mar 18, 2002 4:35 pm

Re: vmware conflict in stable / test patch + 3/4.X.X kernels

Postby nixor » Tue Jun 02, 2015 12:20 pm

PaX Team wrote:PaX has several non-configurable changes, some of them affect low-level CPU specific things (my guess would be the changes to the per-cpu GDT) which is what probably makes vmware trigger a triple fault. debugging this would require setting up nested virtualization and catching the first guest's kernel (where the vmware 'host' code runs) as it's about to trigger the triple fault.


OK, would this work, all 64 bits (never tried this kind of setup btw) :

- Host debian 8 with vmware workstation 11.1 (stock kernel, no grsec/pax patch)
-> install as outer guest KVM hypervisor / (enable option virtualize intel VT-x/EPT or AMD-V/RV for this guest) + grsec/pax kernel
-> install inner guest with debian 8 stock kernel

Not sure if the above is the right setup but appreciate any advise before i go set it up, thank you !
nixor
 
Posts: 5
Joined: Mon Jun 01, 2015 12:13 am

Re: vmware conflict in stable / test patch + 3/4.X.X kernels

Postby PaX Team » Tue Jun 02, 2015 4:42 pm

the outer guest must run vmware, not kvm as that's what we want to debug after all ;). i'm not sure that vmware inside vmware works on linux, you may also try vmware inside kvm (the only important bit is that the outer guest should run vmware and a grsec kernel).
PaX Team
 
Posts: 2310
Joined: Mon Mar 18, 2002 4:35 pm

Re: vmware conflict in stable / test patch + 3/4.X.X kernels

Postby hlein » Fri Sep 02, 2016 12:13 am

I'm having what appears to be the same issue with 4.4.x and 4.5.x kernels. A host running Gentoo's hardened-sources, or vanilla + PAX patches, and vmware workstation: start a VM and the host system reboots.

I filed details at https://bugs.gentoo.org/show_bug.cgi?id=592614 , and found this thread while putting that together.

I'm game to try the vmception or other experiments.
hlein
 
Posts: 1
Joined: Wed Mar 25, 2015 6:04 pm


Return to grsecurity support