Kernel panic with 2.6.24 under Xen, take two

Discuss and suggest new grsecurity features

Kernel panic with 2.6.24 under Xen, take two

Postby xstasi » Thu Feb 28, 2008 6:33 am

Hello,

My previous post was quite messed up, and the research i did about it was proven wrong by further tests.
I would like to update you on the situation.

Maybe it's better to start from the beginning, so eventual readers/developers won't have to search through the past topics.

Facts:
Linux 2.6.24 vanilla comes with builtin support for being a Xen domU (guest), and since the grsec patch only seems to conflict with Xen host (dom0) support, I've tried to make a grsec kernel run under Xen.

Brief tech explaination:
For those who don't know how Xen works: Xen Hypervisor takes a Linux image, fills a struct with some informations, and then "jumps" to its entry point and begins to execute the code.
In the Linux guest code, the (x86) entry part first checks if the Xen Hypervisor is compatible, and then loads itself and boots normally.

Other facts of interest:
Linux 2.6.24 vanilla boots without any problem under Xen. Tested, works.

Symptoms:
The guest grsec kernel *crashes* badly, calling ud2a (the "just crash" instruction) right after entering its entry point.
The hypervisor sees that the guest kernel has gone ud2a, and terminates it.
End of the story, no printk nor anything (it doesn't even get past the entry..), so no debugging.

Analysis and research:
Since the Xen Hypervisor logs a few addresses when the kernel crashes, i took the EIP addres and checked what was there under gdb (gdb vmlinux)
There I found the line where it crashed, and that line is:
Code: Select all
 BUG_ON(memcmp(xen_start_info->magic, "xen-3", 5) != 0);

As I said before, xen_start_info is the structure that the Hypervisor fills before loading the kernel, and that line does the compatibility check i was speaking about earlier.

BUG_ON() macro checks the condition it gets and calls the infamous ud2a if the condition is true (hence, in this case, xen_start_info->magic's first 5 letters are NOT "xen-3")

At first, my thought was that the structure was not filled correctly for some reason (my first idea was about an offset issue), but then, i have eventually succeded in debugging the kernel in the very moment it was crashing (via the Xen's provided gdbserver, and a remote gdb client), and i found out that in the address where the magic was expected.... There it was.
But, what surprised me was that the second argument to memcmp() was *not* "xen-3".
The struct was ok, while the static string was not.
This looks like a problem with the linker or something like that.... Unfortunately, at this point I don't have any experience or knowledge to further research on the issue.

How to reproduce the problem:

Set up xen, make sure Debian's (or your distro) default kernel boots and works as a Xen guest.
Fetch a vanilla 2.6.24, configure it with Debian's guest configuration (/boot/config-version), patch with grsec, make oldconfig and disable anything that grsec carries with it.
This is my .config, in case you wonder
objdump --strip-unneeded vmlinux vmlinux-stripped
gzip -9 vmlinux-stripped
mv vmlinux-stripped.gz /boot/vmlinuz-xen-grsec

And then use that image as the Xen domU kernel.


I hope I have given you all the data you may need to solve the problem.

Thank you in advance for your attention,

xstasi


PS: Xen's paravirtualization host support only works, Debian speaking, under etch (2.6.18 kernel series)
xstasi
 
Posts: 13
Joined: Tue Feb 19, 2008 10:09 am

Re: Kernel panic with 2.6.24 under Xen, take two

Postby PaX Team » Thu Feb 28, 2008 8:49 pm

xstasi wrote:objdump --strip-unneeded vmlinux vmlinux-stripped
can you send me your non-working vmlinux and vmlinux-stripped images?
PaX Team
 
Posts: 2310
Joined: Mon Mar 18, 2002 4:35 pm

Re: Kernel panic with 2.6.24 under Xen, take two

Postby PaX Team » Sun Mar 02, 2008 8:26 pm

xstasi wrote:At first, my thought was that the structure was not filled correctly for some reason (my first idea was about an offset issue), but then, i have eventually succeded in debugging the kernel in the very moment it was crashing (via the Xen's provided gdbserver, and a remote gdb client), and i found out that in the address where the magic was expected.... There it was.
But, what surprised me was that the second argument to memcmp() was *not* "xen-3".
The struct was ok, while the static string was not.
btw, i just compiled a domU kernel with latest PaX and the disasm is correct, so i'm all the more interested in your vmlinux images. in addition you can always put a printk before the BUG_ON to see what exactly was passed as xen_start_info->magic.
PaX Team
 
Posts: 2310
Joined: Mon Mar 18, 2002 4:35 pm

Re: Kernel panic with 2.6.24 under Xen, take two

Postby xstasi » Mon Mar 03, 2008 4:36 am

PaX Team wrote:btw, i just compiled a domU kernel with latest PaX and the disasm is correct, so i'm all the more interested in your vmlinux images. in addition you can always put a printk before the BUG_ON to see what exactly was passed as xen_start_info->magic.


Sorry for being late in replying..
I can put printk, but since the kernel doesn't run, Xen doesn't even initialize its console, so it would be useless.
I've uploaded both vmlinux and vmlinux-stripped in a nice bzip2 archive on my website, you can take them here:

http://www.agrassi.org/vmlinux-and-stripped.tar.bz2

I hope you have fun debugging it.... :^)

Cheers

xstasi
xstasi
 
Posts: 13
Joined: Tue Feb 19, 2008 10:09 am

Re: Kernel panic with 2.6.24 under Xen, take two

Postby PaX Team » Mon Mar 03, 2008 7:42 am

xstasi wrote:I can put printk, but since the kernel doesn't run, Xen doesn't even initialize its console, so it would be useless.
doesn't the early printk facility work at this point?
I hope you have fun debugging it.... :^)
there's not much to debug here as the disasm is correct, so it must be the xen/dom0 side which doesn't pass the proper information somehow. which xen dom0 did you use (preferably with a url to the sources ;-)?
PaX Team
 
Posts: 2310
Joined: Mon Mar 18, 2002 4:35 pm

Re: Kernel panic with 2.6.24 under Xen, take two

Postby xstasi » Mon Mar 03, 2008 9:33 am

I'm using etch's default Xen kernel ( http://packages.debian.org/etch/linux-i ... -5-xen-686 ), with a 3.2 Xen Hypervisor, backported from sid ( http://packages.debian.org/sid/xen-hype ... 3.2-1-i386 )

If you want, i can upload my debian hypervisor backports for etch...
xstasi
 
Posts: 13
Joined: Tue Feb 19, 2008 10:09 am

Re: Kernel panic with 2.6.24 under Xen, take two

Postby xstasi » Mon Mar 03, 2008 9:35 am

PaX Team wrote:there's not much to debug here as the disasm is correct, so it must be the xen/dom0 side which doesn't pass the proper information somehow.


I think we're missing something....
If we debug the vmlinux, "xen-3" is at his place, but obviously the struct is empty.
If we debug the running kernel, the struct is filled correctly, but "xen-3" isn't there.
I'm really clueless...
xstasi
 
Posts: 13
Joined: Tue Feb 19, 2008 10:09 am

Re: Kernel panic with 2.6.24 under Xen, take two

Postby PaX Team » Mon Mar 03, 2008 5:56 pm

xstasi wrote:I think we're missing something....
If we debug the vmlinux, "xen-3" is at his place, but obviously the struct is empty.
If we debug the running kernel, the struct is filled correctly, but "xen-3" isn't there.
I'm really clueless...
so if "xen-3" isn't there (which is the read-only section of the kernel fwiw), what is there? can you dump some memory from that address? i also wonder, does xen relocate the kernel? if it does, maybe some addresses aren't properly relocated (i've seen and fixed such bugs in vanilla already).
PaX Team
 
Posts: 2310
Joined: Mon Mar 18, 2002 4:35 pm

Re: Kernel panic with 2.6.24 under Xen, take two

Postby xstasi » Wed Mar 05, 2008 5:47 am

Indeed, xen likes to relocate, he likes it a lot.
Here follow two debugging sessions on a 2.6.24 patched with your lastest grsec, one on the vmlinux, the other on the (not so) running kernel, from the remote debugger.

Quoted code is taken from arch/x86/xen/enlighten.c
My asm/code comments are just guessings, correct me if i'm wrong

compiled vmlinux:
Code: Select all
# gdb -q vmlinux
(no debugging symbols found)
Using host libthread_db library "/lib/tls/i686/cmov/libthread_db.so.1".
(gdb) disass xen_start_kernel
Dump of assembler code for function xen_start_kernel:
0xc021c8c8 <xen_start_kernel+0>:        push   %esi
0xc021c8c9 <xen_start_kernel+1>:        push   %ebx
0xc021c8ca <xen_start_kernel+2>:        sub    $0x18,%esp
0xc021c8cd <xen_start_kernel+5>:        mov    0xc068f8a0,%eax                    # mov xen_start_info address into eax
0xc021c8d2 <xen_start_kernel+10>:       test   %eax,%eax                          #  if(!xen_start_info)
0xc021c8d4 <xen_start_kernel+12>:       je     0xc021cac2 <xen_start_kernel+506>  # return
0xc021c8da <xen_start_kernel+18>:       mov    $0x5,%ecx                          # push '5' argument
0xc021c8df <xen_start_kernel+23>:       mov    $[b]0xc05e97ac[/b],%edx            # Also push "xen-3"
0xc021c8e4 <xen_start_kernel+28>:       call   0xc04e0fcb <memcmp>                # And call memcmp()
0xc021c8e9 <xen_start_kernel+33>:       test   %eax,%eax                          # If arguments match
0xc021c8eb <xen_start_kernel+35>:       je     0xc021c8f1 <xen_start_kernel+41>   # Go forth
0xc021c8ed <xen_start_kernel+37>:       ud2a                                      # Otherwise, make kernel crash
{snip}
---Type <return> to continue, or q <return> to quit---q
Quit
(gdb) x/s 0xc05e97ac
0xc05e97ac <kallsyms_token_index+4088>:  "xen-3"   # The readonly string is there
(gdb) x/x 0xc068f8a0
0xc068f8a0 <xen_start_info>:    0x00000000         # Obviously, the structure is not
(gdb) q



Now follows...
Running (crashed) kernel:
Code: Select all
(gdb) target remote 127.0.0.1:9999
Remote debugging using 127.0.0.1:9999
[New thread 0]
[Switching to thread 0]
0xc021c8ed in xen_start_kernel ()
warning: shared library handler failed to enable breakpoint
Failed to read a valid object file image from memory.
(gdb) disass xen_start_kernel
{disass skipped, code and addresses are the same as above}
(gdb) x/x 0xc068f8a0
0xc068f8a0 <xen_start_info>:    0xc12cf000           # Struct beginning now contains a pointer...
(gdb) x/s 0xc12cf000
0xc12cf000:      "xen-3.0-x86_32p"                   # And the magic is there. Xen put it in the right place.
(gdb) x/x 0xc05e97ac                                 # Now check the static string address.....
0xc05e97ac <kallsyms_token_index+4088>: 0x00000000
(gdb)
0xc05e97b0 <kallsyms_token_index+4092>: 0x00000000
(gdb)
0xc05e97b4 <kallsyms_token_index+4096>: 0x00000000
(gdb)                                                # Nothing but 0xzeroes.


You may also be interested in this, still from the (not so) running kernel:
Code: Select all
(gdb) x/10000x &kallsyms_token_index

This dumps thousands of 0x0.

I hope i've given you all the info you may need...

Cheers

xstasi
xstasi
 
Posts: 13
Joined: Tue Feb 19, 2008 10:09 am

Re: Kernel panic with 2.6.24 under Xen, take two

Postby xstasi » Thu Mar 06, 2008 5:00 am

A small update:

I've tried to patch a vanilla 2.6.24 with only the PaX patch, instead of the complete grsecurity bundle, and... it boots normally.
So the problem seems to be in the rest of the patch

I hope this information is somehow useful

Cheers

xstasi
xstasi
 
Posts: 13
Joined: Tue Feb 19, 2008 10:09 am

Re: Kernel panic with 2.6.24 under Xen, take two

Postby PaX Team » Thu Mar 06, 2008 8:14 am

xstasi wrote:Now follows...
Running (crashed) kernel:
Code: Select all
(gdb) x/x 0xc05e97ac                                 # Now check the static string address.....
0xc05e97ac <kallsyms_token_index+4088>: 0x00000000
(gdb)
0xc05e97b0 <kallsyms_token_index+4092>: 0x00000000
(gdb)
0xc05e97b4 <kallsyms_token_index+4096>: 0x00000000
(gdb)                                                # Nothing but 0xzeroes.
this suggests that a large chunk of the kernel's image hasn't been properly loaded (.rodata and probably following sections too). i have no idea how Xen does it and how it could result in such a failure, i think it's time to ask the Xen devs...
PaX Team
 
Posts: 2310
Joined: Mon Mar 18, 2002 4:35 pm

Re: Kernel panic with 2.6.24 under Xen, take two

Postby PaX Team » Thu Mar 06, 2008 8:19 am

xstasi wrote:I've tried to patch a vanilla 2.6.24 with only the PaX patch, instead of the complete grsecurity bundle, and... it boots normally.
So the problem seems to be in the rest of the patch
thing is, spender's part doesn't at all have any low-level changes that could result in this (e.g., no linker script tweaks or low-level setup code changes), so it's either a bad merge on his part (but i didn't see anything in the interdiff) or you just got lucky... btw, does enabling/disabling PaX features affect the result (both when using grsec and plain PaX)?
PaX Team
 
Posts: 2310
Joined: Mon Mar 18, 2002 4:35 pm

Re: Kernel panic with 2.6.24 under Xen, take two

Postby xstasi » Mon Mar 10, 2008 12:57 pm

Turns out that i was wrong in my last report:

It is actually PaX that breaks up the kernel
I'm really clueless...
xstasi
 
Posts: 13
Joined: Tue Feb 19, 2008 10:09 am

Re: Kernel panic with 2.6.24 under Xen, take two

Postby cormander » Mon Mar 10, 2008 6:22 pm

I'll add my two cents here.

I've built the 2.6.24.3 kernel from source with and without the grsecurity patch applied... they both boot the machine. But when the machine is running a xen 3.0.2 kernel (CentOS 5 default xen RPMs for dom0) the vanilla kernel works as a guest (domU) and the grsecurity kernel doesn't, whether or not it has any options enabled.

Code: Select all
[root@localhost ~]# xm create /dev/null kernel=/boot/vmlinuz-2.6.24.3-grsec name=ramdisk vcpus=1 memory=64 extra=console=hvc0 -c
Using config file "/dev/null".
Started domain ramdisk


No more output. When I do this with the vanilla kernel, it attaches me to the console and shows me the boot output up until the panic where it doesn't have an initrd (because I didn't specify one in this example). But the point is, the grsec kernel doesn't even make it to the point where the console is attached for the very beginning of the boot output. Seems as if xen is killing it before it ever loads.

The xend.log from the above generates:

Code: Select all
[2008-03-10 14:47:07 xend.XendDomainInfo 3053] DEBUG (XendDomainInfo:200) XendDomainInfo.create(['vm', ['name', 'ramdisk'], ['memory', 64], ['vcpus', 1], ['image', ['linux', ['kernel', '/boot/vmlinuz-2.6.24.3-grsec'], ['args', 'console=hvc0']]]])
[2008-03-10 14:47:07 xend.XendDomainInfo 3053] DEBUG (XendDomainInfo:306) parseConfig: config is ['vm', ['name', 'ramdisk'], ['memory', 64], ['vcpus', 1], ['image', ['linux', ['kernel', '/boot/vmlinuz-2.6.24.3-grsec'], ['args', 'console=hvc0']]]]
[2008-03-10 14:47:07 xend.XendDomainInfo 3053] DEBUG (XendDomainInfo:411) parseConfig: result is {'shadow_memory': None, 'start_time': None, 'uuid': None, 'on_crash': None, 'on_reboot': None, 'localtime': None, 'image': ['linux', ['kernel', '/boot/vmlinuz-2.6.24.3-grsec'], ['args', 'console=hvc0']], 'on_poweroff': None, 'bootloader_args': None, 'cpus': None, 'name': 'ramdisk', 'backend': [], 'vcpus': 1, 'cpu_weight': None, 'features': None, 'vcpu_avail': None, 'memory': 64, 'device': [], 'bootloader': None, 'cpu': None, 'maxmem': None}
[2008-03-10 14:47:07 xend.XendDomainInfo 3053] DEBUG (XendDomainInfo:1349) XendDomainInfo.construct: None
[2008-03-10 14:47:07 xend 3053] DEBUG (balloon:127) Balloon: 549376 KiB free; need 2048; done.
[2008-03-10 14:47:07 xend.XendDomainInfo 3053] DEBUG (XendDomainInfo:1397) XendDomainInfo.initDomain: 49 1.0
[2008-03-10 14:47:07 xend.XendDomainInfo 3053] DEBUG (XendDomainInfo:1436) _initDomain:shadow_memory=0x0, maxmem=0x40, memory=0x40.
[2008-03-10 14:47:07 xend 3053] DEBUG (balloon:127) Balloon: 549376 KiB free; need 65536; done.
[2008-03-10 14:47:07 xend 3053] INFO (image:136) buildDomain os=linux dom=49 vcpus=1
[2008-03-10 14:47:07 xend 3053] DEBUG (image:201) domid          = 49
[2008-03-10 14:47:07 xend 3053] DEBUG (image:202) memsize        = 64
[2008-03-10 14:47:07 xend 3053] DEBUG (image:203) image          = /boot/vmlinuz-2.6.24.3-grsec
[2008-03-10 14:47:07 xend 3053] DEBUG (image:204) store_evtchn   = 1
[2008-03-10 14:47:07 xend 3053] DEBUG (image:205) console_evtchn = 2
[2008-03-10 14:47:07 xend 3053] DEBUG (image:206) cmdline        =  console=hvc0
[2008-03-10 14:47:07 xend 3053] DEBUG (image:207) ramdisk        =
[2008-03-10 14:47:07 xend 3053] DEBUG (image:208) vcpus          = 1
[2008-03-10 14:47:07 xend 3053] DEBUG (image:209) features       =
[2008-03-10 14:47:07 xend.XendDomainInfo 3053] DEBUG (XendDomainInfo:756) Storing VM details: {'shadow_memory': '0', 'uuid': '22cdb64a-6413-0b74-74bd-6a630ea23afa', 'on_reboot': 'restart', 'start_time': '1205182027.94', 'on_poweroff': 'destroy', 'name': 'ramdisk', 'xend/restart_count': '0', 'vcpus': '1', 'vcpu_avail': '1', 'memory': '64', 'on_crash': 'restart', 'image': "(linux (kernel /boot/vmlinuz-2.6.24.3-grsec) (args 'console=hvc0'))", 'maxmem': '64'}
[2008-03-10 14:47:07 xend.XendDomainInfo 3053] DEBUG (XendDomainInfo:791) Storing domain details: {'console/ring-ref': '164685', 'console/port': '2', 'name': 'ramdisk', 'console/limit': '1048576', 'vm': '/vm/22cdb64a-6413-0b74-74bd-6a630ea23afa', 'domid': '49', 'cpu/0/availability': 'online', 'memory/target': '65536', 'store/ring-ref': '164686', 'store/port': '1'}
[2008-03-10 14:47:07 xend.XendDomainInfo 3053] DEBUG (XendDomainInfo:988) XendDomainInfo.handleShutdownWatch
[2008-03-10 14:47:07 xend 3053] DEBUG (DevController:143) Waiting for devices vif.
[2008-03-10 14:47:07 xend 3053] DEBUG (DevController:143) Waiting for devices usb.
[2008-03-10 14:47:07 xend 3053] DEBUG (DevController:143) Waiting for devices vbd.
[2008-03-10 14:47:07 xend 3053] DEBUG (DevController:143) Waiting for devices irq.
[2008-03-10 14:47:07 xend 3053] DEBUG (DevController:143) Waiting for devices vkbd.
[2008-03-10 14:47:08 xend 3053] DEBUG (DevController:143) Waiting for devices vfb.
[2008-03-10 14:47:08 xend 3053] DEBUG (DevController:143) Waiting for devices pci.
[2008-03-10 14:47:08 xend 3053] DEBUG (DevController:143) Waiting for devices ioports.
[2008-03-10 14:47:08 xend 3053] DEBUG (DevController:143) Waiting for devices tap.
[2008-03-10 14:47:08 xend 3053] DEBUG (DevController:143) Waiting for devices vtpm.
[2008-03-10 14:47:08 xend 3053] INFO (XendDomain:370) Domain ramdisk (49) unpaused.
[2008-03-10 14:47:08 xend.XendDomainInfo 3053] WARNING (XendDomainInfo:923) Domain has crashed: name=ramdisk id=49.
[2008-03-10 14:47:08 xend.XendDomainInfo 3053] DEBUG (XendDomainInfo:1566) XendDomainInfo.destroyDomain(49)
[2008-03-10 14:47:08 xend.XendDomainInfo 3053] DEBUG (XendDomainInfo:200) XendDomainInfo.create(['domain', ['domid', 49], ['uuid', '22cdb64a-6413-0b74-74bd-6a630ea23afa'], ['vcpus', 1], ['vcpu_avail', 1], ['cpu_weight', 1.0], ['memory', 64], ['shadow_memory', 0], ['maxmem', 64], ['features', ''], ['name', 'ramdisk'], ['on_poweroff', 'destroy'], ['on_reboot', 'restart'], ['on_crash', 'restart'], ['image', ['linux', ['kernel', '/boot/vmlinuz-2.6.24.3-grsec'], ['args', 'console=hvc0']]], ['state', '----c-'], ['shutdown_reason', 'crash'], ['cpu_time', 6.6023000000000001e-05], ['online_vcpus', 1], ['up_time', '0.0964879989624'], ['start_time', '1205182027.94'], ['store_mfn', 164686], ['console_mfn', 164685]])
[2008-03-10 14:47:08 xend.XendDomainInfo 3053] DEBUG (XendDomainInfo:306) parseConfig: config is ['domain', ['domid', 49], ['uuid', '22cdb64a-6413-0b74-74bd-6a630ea23afa'], ['vcpus', 1], ['vcpu_avail', 1], ['cpu_weight', 1.0], ['memory', 64], ['shadow_memory', 0], ['maxmem', 64], ['features', ''], ['name', 'ramdisk'], ['on_poweroff', 'destroy'], ['on_reboot', 'restart'], ['on_crash', 'restart'], ['image', ['linux', ['kernel', '/boot/vmlinuz-2.6.24.3-grsec'], ['args', 'console=hvc0']]], ['state', '----c-'], ['shutdown_reason', 'crash'], ['cpu_time', 6.6023000000000001e-05], ['online_vcpus', 1], ['up_time', '0.0964879989624'], ['start_time', '1205182027.94'], ['store_mfn', 164686], ['console_mfn', 164685]]
[2008-03-10 14:47:08 xend.XendDomainInfo 3053] DEBUG (XendDomainInfo:411) parseConfig: result is {'shadow_memory': 0, 'start_time': 1205182027.9400001, 'uuid': '22cdb64a-6413-0b74-74bd-6a630ea23afa', 'on_crash': 'restart', 'on_reboot': 'restart', 'localtime': None, 'image': ['linux', ['kernel', '/boot/vmlinuz-2.6.24.3-grsec'], ['args', 'console=hvc0']], 'on_poweroff': 'destroy', 'bootloader_args': None, 'cpus': None, 'name': 'ramdisk', 'backend': [], 'vcpus': 1, 'cpu_weight': 1.0, 'features': '', 'vcpu_avail': 1, 'memory': 64, 'device': [], 'bootloader': None, 'cpu': None, 'maxmem': 64}
[2008-03-10 14:47:08 xend.XendDomainInfo 3053] DEBUG (XendDomainInfo:1349) XendDomainInfo.construct: None
[2008-03-10 14:47:08 xend 3053] DEBUG (balloon:127) Balloon: 549376 KiB free; need 2048; done.
[2008-03-10 14:47:08 xend.XendDomainInfo 3053] DEBUG (XendDomainInfo:1397) XendDomainInfo.initDomain: 50 1.0
[2008-03-10 14:47:08 xend.XendDomainInfo 3053] DEBUG (XendDomainInfo:1436) _initDomain:shadow_memory=0x0, maxmem=0x40, memory=0x40.
[2008-03-10 14:47:08 xend 3053] DEBUG (balloon:127) Balloon: 549376 KiB free; need 65536; done.
[2008-03-10 14:47:08 xend 3053] INFO (image:136) buildDomain os=linux dom=50 vcpus=1
[2008-03-10 14:47:08 xend 3053] DEBUG (image:201) domid          = 50
[2008-03-10 14:47:08 xend 3053] DEBUG (image:202) memsize        = 64
[2008-03-10 14:47:08 xend 3053] DEBUG (image:203) image          = /boot/vmlinuz-2.6.24.3-grsec
[2008-03-10 14:47:08 xend 3053] DEBUG (image:204) store_evtchn   = 1
[2008-03-10 14:47:08 xend 3053] DEBUG (image:205) console_evtchn = 2
[2008-03-10 14:47:08 xend 3053] DEBUG (image:206) cmdline        =  console=hvc0
[2008-03-10 14:47:08 xend 3053] DEBUG (image:207) ramdisk        =
[2008-03-10 14:47:08 xend 3053] DEBUG (image:208) vcpus          = 1
[2008-03-10 14:47:08 xend 3053] DEBUG (image:209) features       =
[2008-03-10 14:47:08 xend.XendDomainInfo 3053] DEBUG (XendDomainInfo:756) Storing VM details: {'shadow_memory': '0', 'uuid': '22cdb64a-6413-0b74-74bd-6a630ea23afa', 'on_reboot': 'restart', 'start_time': '1205182028.23', 'on_poweroff': 'destroy', 'name': 'ramdisk', 'vcpus': '1', 'vcpu_avail': '1', 'memory': '64', 'on_crash': 'restart', 'image': "(linux (kernel /boot/vmlinuz-2.6.24.3-grsec) (args 'console=hvc0'))", 'maxmem': '64'}
[2008-03-10 14:47:08 xend.XendDomainInfo 3053] DEBUG (XendDomainInfo:791) Storing domain details: {'console/ring-ref': '163917', 'console/port': '2', 'name': 'ramdisk', 'console/limit': '1048576', 'vm': '/vm/22cdb64a-6413-0b74-74bd-6a630ea23afa', 'domid': '50', 'cpu/0/availability': 'online', 'memory/target': '65536', 'store/ring-ref': '163918', 'store/port': '1'}
[2008-03-10 14:47:08 xend.XendDomainInfo 3053] WARNING (XendDomainInfo:923) Domain has crashed: name=ramdisk id=50.
[2008-03-10 14:47:08 xend.XendDomainInfo 3053] ERROR (XendDomainInfo:1825) VM ramdisk restarting too fast (0.234522 seconds since the last restart).  Refusing to restart to avoid loops.
[2008-03-10 14:47:08 xend.XendDomainInfo 3053] DEBUG (XendDomainInfo:1557) XendDomainInfo.destroy: domid=50
[2008-03-10 14:47:08 xend.XendDomainInfo 3053] DEBUG (XendDomainInfo:1566) XendDomainInfo.destroyDomain(50)


The output for the vanilla kernel is pretty much exactly the same (except the kernel name in the output) up until the first reference to this line:

Code: Select all
[2008-03-10 14:47:08 xend.XendDomainInfo 3053] WARNING (XendDomainInfo:923) Domain has crashed: name=ramdisk id=49.


Which doesn't happen for the vanilla kernel, because it doesn't crash. Looks like xen sees the crash, tries again, crashes again, and stops: "Refusing to restart to avoid loops."

I just finished recompiling the kernel again with just your latest PAX patch... still the same problem.

Is there any way to pull the PAX patch out of the grsecurity patch and see if it works that way? I divied up the grsecurity patch into a .patch file for each thing it touches, and removed any of them that reference PAX... but then grsecurity options disappear from the menuconfig... so it appears that grsecurity depends (at least in part) on PAX existing. If there's a quick and dirty way to do this, let me know.

I know that PAX is a big part of grsecurity, and ripping out PAX won't solve this issue for us, but it might help to narrow down the issue. Also, if I could at the very least have a domU kernel with the TPE, RBAC system, chroot hardening, etc and no PAX, it would be a lot better than nothing.
cormander
 
Posts: 154
Joined: Tue Jan 29, 2008 12:51 pm

Re: Kernel panic with 2.6.24 under Xen, take two

Postby PaX Team » Mon Mar 10, 2008 6:47 pm

cormander wrote:Is there any way to pull the PAX patch out of the grsecurity patch and see if it works that way?
you can interdiff PaX and grsec to get the grsec-only parts + some config stuff still related to PaX that you'll have to remove manually (or just not bother, i think it won't cause a problem).
PaX Team
 
Posts: 2310
Joined: Mon Mar 18, 2002 4:35 pm

Next

Return to grsecurity development