2.6.32.11+ with KERNEXEC and CPU's lacking NX capability

[ef--]

Hello,

grsecurity-2.1.14-2.6.32.11-201004071936 and above when configured with KERNEXEC enabled on older p4 & xeon cpu's either lacking NX support or having NX support set to disabled in BIOS results in fairly frequent seemingly random application crashed with the kernel logging "corrupted page table at adress xxxxx ":

May 1 18:15:34 hostname kernel: which: Corrupted page table at address b399c810
May 1 18:15:34 hostname kernel: *pdpt = 000000003616d001 *pde = 000000007fca8025
May 1 18:15:34 hostname kernel: Bad pagetable: 000d [#1] SMP
May 1 18:15:34 hostname kernel: last sysfs file: /sys/devices/pci0000:00/0000:00:1e.0/0000:03:01.0/host7/target7:0:0/7:0:0:0/block/sdf/removable
May 1 18:15:34 hostname kernel: Modules linked in:
May 1 18:15:34 hostname kernel:
May 1 18:15:34 hostname kernel: Pid: 3715, comm: which Not tainted (2.6.32.12-grsec #1)
May 1 18:15:34 hostname kernel: EIP: 0073:[<5399c810>] EFLAGS: 00010212 CPU: 1
May 1 18:15:34 hostname kernel: EAX: 00000000 EBX: 00000000 ECX: 00000000 EDX: 00000000
May 1 18:15:34 hostname kernel: ESI: 00000000 EDI: 00000000 EBP: 00000000 ESP: 5fe31be0
May 1 18:15:34 hostname kernel: DS: 007b ES: 007b FS: 0000 GS: 0000 SS: 007b
May 1 18:15:34 hostname kernel: Process which (pid: 3715, ti=f671c000 task=f6d35560 task.ti=f671c000)
May 1 18:15:34 hostname kernel:
May 1 18:15:34 hostname kernel: EIP: [<5399c810>] SS:ESP 007b:5fe31be0
May 1 18:15:34 hostname kernel: ---[ end trace 913b802e946143a3 ]---

May 1 19:23:14 hostname kernel: expr: Corrupted page table at address acb65810
May 1 19:23:14 hostname kernel: *pdpt = 0000000036199001 *pde = 000000007fca8025
May 1 19:23:14 hostname kernel: Bad pagetable: 000d [#2] SMP
May 1 19:23:14 hostname kernel: last sysfs file: /sys/devices/pci0000:00/0000:00:1e.0/0000:03:00.0/host4/target4:0:0/4:0:0:0/block/sdc/removable
May 1 19:23:14 hostname kernel: Modules linked in:
May 1 19:23:14 hostname kernel:
May 1 19:23:14 hostname kernel: Pid: 5129, comm: expr Tainted: G D (2.6.32.12-grsec #1)
May 1 19:23:14 hostname kernel: EIP: 0073:[<4cb65810>] EFLAGS: 00010212 CPU: 0
May 1 19:23:14 hostname kernel: EAX: 00000000 EBX: 00000000 ECX: 00000000 EDX: 00000000
May 1 19:23:14 hostname kernel: ESI: 00000000 EDI: 00000000 EBP: 00000000 ESP: 5b9fdea0
May 1 19:23:14 hostname kernel: DS: 007b ES: 007b FS: 0000 GS: 0000 SS: 007b
May 1 19:23:14 hostname kernel: Process expr (pid: 5129, ti=f63ac000 task=f6e663a0 task.ti=f63ac000)
May 1 19:23:14 hostname kernel:
May 1 19:23:14 hostname kernel: EIP: [<4cb65810>] SS:ESP 007b:5b9fdea0
May 1 19:23:14 hostname kernel: ---[ end trace 913b802e946143a4 ]---

May 1 19:31:55 hostname kernel: uname: Corrupted page table at address b0381810
May 1 19:31:55 hostname kernel: *pdpt = 000000003671d001 *pde = 000000007fca8025
May 1 19:31:55 hostname kernel: Bad pagetable: 000d [#3] SMP
May 1 19:31:55 hostname kernel: last sysfs file: /sys/devices/pci0000:00/0000:00:1e.0/0000:03:00.0/host3/target3:0:0/3:0:0:0/block/sdb/removable
May 1 19:31:55 hostname kernel: Modules linked in:
May 1 19:31:55 hostname kernel:
May 1 19:31:55 hostname kernel: Pid: 5949, comm: uname Tainted: G D (2.6.32.12-grsec #1)
May 1 19:31:55 hostname kernel: EIP: 0073:[<50381810>] EFLAGS: 00010212 CPU: 0
May 1 19:31:55 hostname kernel: EAX: 00000000 EBX: 00000000 ECX: 00000000 EDX: 00000000
May 1 19:31:55 hostname kernel: ESI: 00000000 EDI: 00000000 EBP: 00000000 ESP: 5ce99900
May 1 19:31:55 hostname kernel: DS: 007b ES: 007b FS: 0000 GS: 0000 SS: 007b
May 1 19:31:55 hostname kernel: Process uname (pid: 5949, ti=f63a4000 task=f6d04000 task.ti=f63a4000)
May 1 19:31:55 hostname kernel:
May 1 19:31:55 hostname kernel: EIP: [<50381810>] SS:ESP 007b:5ce99900
May 1 19:31:55 hostname kernel: ---[ end trace 913b802e946143a5 ]---

System cpu usage also shows a drastic increase up to 30-40% on and off on a otherwise basically idle system. On older systems that has NX capable cpu's but defaulted to having it disabled in BIOS the problem goes away when enabling it there with no change to kernel. On systems lacking NX capability altogether the problem goes away when disabling KERNEXEC in configuration so im assuming its the software emulation of the NX bit that is somehow causing it in more recent grsecurity versions.

Last patch known to not displaying the behavior: grsecurity-2.1.14-2.6.32.9-201002231820

Recent patches tested that all displays the same behavior (any patches between 2.6.32.9 and 2.6.32.11 not tested):
grsecurity-2.1.14-2.6.32.11-201004071936.patch
grsecurity-2.1.14-2.6.32.12-201004292005
grsecurity-2.1.14-2.6.32.12-201005012055.patch

config at: http://temp.tgk.net/config
vmlinux at: http://temp.tgk.net/vmlinux
bzImage at: http://temp.tgk.net/bzImage
binutils version: 2.18.50.0.9.20080822

[PaX Team]

looks like i broke SEGMEXEC somehow (that's what takes over PAGEEXEC when NX isn't available), i'll take a look.

[specs]

I haven't seen anything yet on a singlecore VIA processor (i386).

[ef--]

Update:

crashes with "Corrupted page table at address" also seems to happen on NX capable machines. Although far less frequent, once or twice in a couple of days as compared to several times within an hour after booting for the machines lacking NX support.

May 3 01:16:20 otherhost kernel: java: Corrupted page table at address 6f8df000
May 3 01:16:20 otherhost kernel: *pdpt = 0000000029eb8001 *pde = 16b74f053b2833c3
May 3 01:16:20 otherhost kernel: Bad pagetable: 000f [#1] SMP
May 3 01:16:20 otherhost kernel: last sysfs file: /sys/devices/pci0000:00/0000:00:02.0/0000:01:00.3/0000:05:02.0/host0/target0:0:0/0:0:0:0/block/sda/stat
May 3 01:16:20 otherhost kernel: Modules linked in:
May 3 01:16:20 otherhost kernel:
May 3 01:16:20 otherhost kernel: Pid: 5632, comm: java Not tainted (2.6.32.12-grsec #1) S5000PSL
May 3 01:16:20 otherhost kernel: EIP: 0073:[<b6caffe4>] EFLAGS: 00210202 CPU: 7
May 3 01:16:20 otherhost kernel: EAX: 00000001 EBX: 6f8df000 ECX: 00001000 EDX: 00000003
May 3 01:16:20 otherhost kernel: ESI: 0805c4f0 EDI: 6f8df000 EBP: b6a7cff8 ESP: b6a7cfb0
May 3 01:16:20 otherhost kernel: DS: 007b ES: 007b FS: 0000 GS: 0033 SS: 007b
May 3 01:16:20 otherhost kernel: Process java (pid: 5632, ti=eb2d4000 task=f68f9c80 task.ti=eb2d4000)
May 3 01:16:20 otherhost kernel:
May 3 01:16:20 otherhost kernel: EIP: [<b6caffe4>] SS:ESP 007b:b6a7cfb0
May 3 01:16:20 otherhost kernel: ---[ end trace ba54260057c6548a ]---
May 3 01:16:22 otherhost kernel: java: Corrupted page table at address 6f91f000
May 3 01:16:22 otherhost kernel: *pdpt = 000000001b8ec001 *pde = ecad1adca1333b0d
May 3 01:16:22 otherhost kernel: Bad pagetable: 000f [#2] SMP
May 3 01:16:22 otherhost kernel: last sysfs file: /sys/devices/pci0000:00/0000:00:02.0/0000:01:00.3/0000:05:02.0/host0/target0:0:0/0:0:0:0/block/sda/stat
May 3 01:16:22 otherhost kernel: Modules linked in:
May 3 01:16:22 otherhost kernel:
May 3 01:16:22 otherhost kernel: Pid: 5704, comm: java Tainted: G D (2.6.32.12-grsec #1) S5000PSL
May 3 01:16:22 otherhost kernel: EIP: 0073:[<b6cf1fe4>] EFLAGS: 00210202 CPU: 5
May 3 01:16:22 otherhost kernel: EAX: 00000001 EBX: 6f91f000 ECX: 00001000 EDX: 00000003
May 3 01:16:22 otherhost kernel: ESI: 0805c4e0 EDI: 6f91f000 EBP: b6abeff8 ESP: b6abefb0
May 3 01:16:22 otherhost kernel: DS: 007b ES: 007b FS: 0000 GS: 0033 SS: 007b
May 3 01:16:22 otherhost kernel: Process java (pid: 5704, ti=f3c08000 task=f4a351f0 task.ti=f3c08000)
May 3 01:16:22 otherhost kernel:
May 3 01:16:22 otherhost kernel: EIP: [<b6cf1fe4>] SS:ESP 007b:b6abefb0
May 3 01:16:22 otherhost kernel: ---[ end trace ba54260057c6548b ]---

[PaX Team]

crashes with "Corrupted page table at address" also seems to happen on NX capable machines. Although far less frequent, once or twice in a couple of days as compared to several times within an hour after booting for the machines lacking NX support.

hmm, this is worse than i thought . can you turn off PAX_PER_CPU_PGD and try with such a config? you'll have to edit security/Kconfig and remove the select line from KERNEXEC first.

[ef--]

Compiled a kernel with KERNEXEC support but without PAX_PER_CPU_PGD and tried it on one of the machines lacking NX support since it occured much more frequent there. So far its been running it a little over 2 hours without any "Corrupted page table at" program crashes. So either it made it go away or at least made it occur far less frequent.

[ef--]

Update:

System has now been up 14 hours without any signs of program crashes.

[ef--]

Since it seems to be running fine when compiled without PAX_PER_CPU_PGD what are the performance&security implications of running such a configuration? i.e is it a viable workaround for the time being?

[specs]

I'd like to add a question:
Why do I only see the PAX_PER_CPU_PGD with AMD64 (not with ARCH=i386)?

Even with AMD64 te option is not visible using "make menuconfig".

[PaX Team]

Since it seems to be running fine when compiled without PAX_PER_CPU_PGD what are the performance&security implications of running such a configuration? i.e is it a viable workaround for the time being?

yes, everything will still work. i'm also about to test the fix so hopefully this won't have to last long.

[PaX Team]

Why do I only see the PAX_PER_CPU_PGD with AMD64 (not with ARCH=i386)?

because you didn't enable PAE/KERNEXEC on i386 .

Even with AMD64 te option is not visible using "make menuconfig".

it's because it's not something end-users should be concerned with, it's automatically selected for certain PaX features (that are user selectable). the kernel is full of such options btw, normally it's used to mark specific feature implementations in the code that are needed for other features but are nice to have explicitly marked for easier maintenance (say, PAE and HIGHMEM64G on i386).

[specs]

Why do I only see the PAX_PER_CPU_PGD with AMD64 (not with ARCH=i386)?

because you didn't enable PAE/KERNEXEC on i386 .

Which brings me to the next question:
Does Pax rely on PAE for NX?
If I follow your hint I would need PAE for KERNEXEC. For PAE I would need 64GB. And PAE seems to be needed for NX-suport in the vanilla kernel (pae and nx are supported in one i386 SMP kernel).

(It seems the reason PAE is not enabled is the lack of the 64GB-option, KERNEXEC is selected.)

[PaX Team]

Does Pax rely on PAE for NX?
If I follow your hint I would need PAE for KERNEXEC.

PaX uses the CPU provided NX bit whenever possible, but whether you actually need it for a given PaX feature or not, it depends. in particular, i386/KERNEXEC does *not* rely on the NX bit, instead it uses segmentation to accomplish what otherwise the NX bit would be used for (but the kernel can still use the NX bit, it's just not relevant for KERNEXEC/i386).

For PAE I would need 64GB. And PAE seems to be needed for NX-suport in the vanilla kernel (pae and nx are supported in one i386 SMP kernel).

you can enable PAE without the 64G support, you just need to enable the embedded option first then you can select PAE separately.

[PaX Team]

i'm also about to test the fix so hopefully this won't have to last long.

the latest test patch should fix this, can you guys test it please?

[ef--]

Still occurs on grsecurity-2.1.14-2.6.32.13-201005151340 with PAX_PER_CPU_PGD, shortly after boot:

memory: Corrupted page table at address b399c810
*pdpt = 0000000036771001 *pde = 000000007fcc3025
Bad pagetable: 000d [#1] SMP
last sysfs file: /sys/devices/pci0000:00/0000:00:1e.0/0000:03:01.0/host9/target9:0:0/9:0:0:0/block/sdh/removable
Modules linked in:

Pid: 5877, comm: memory Not tainted (2.6.32.13-grsec #1)
EIP: 0073:[<5399c810>] EFLAGS: 00010212 CPU: 0
EAX: 00000000 EBX: 00000000 ECX: 00000000 EDX: 00000000
ESI: 00000000 EDI: 00000000 EBP: 00000000 ESP: 5eaf5470
DS: 007b ES: 007b FS: 0000 GS: 0000 SS: 007b
Process memory (pid: 5877, ti=f66ae000 task=f6e8eac0 task.ti=f66ae000)

EIP: [<5399c810>] SS:ESP 007b:5eaf5470
---[ end trace 7ae7197c7a9fff5a ]---

Edit: same 2.1.14-2.6.32.13-201005151340 _without_ PAX_PER_CPU_PGD still seems to rune fine after a couple of hours just as previous verions