[SOLVED] Linux-3.13.* crash when loading modules.

Discuss usability issues, general maintenance, and general support issues for a grsecurity-enabled system.

[SOLVED] Linux-3.13.* crash when loading modules.

Postby jacekalex » Tue Feb 04, 2014 8:14 pm

Hi

When I try to load any module randomly this message appears:
Code: Select all
[  116.962213] BUG: unable to handle kernel paging request at ffffffff7b031050
[  116.962679] IP: [<ffffffffa0043f58>] ffffffffa0043f58
[  116.963002] PGD 1946067 PUD 0
[  116.963002] Thread overran stack, or stack corrupted
[  116.963002] Oops: 0000 [#1] PREEMPT SMP
[  116.963002] Modules linked in: cx88_alsa(+) cx88xx tveeprom btcx_risc videobuf_dma_sg videobuf_core slhc
[  116.963002] CPU: 0 PID: 391 Comm: modprobe Not tainted 3.13.1-gr4 #2
[  116.963002] Hardware name: Gigabyte Technology Co., Ltd. P43-ES3G/P43-ES3G, BIOS F14 08/23/2010
[  116.963002] task: ffff8800db93ab20 ti: ffff8800db93b130 task.ti: ffff8800db93b130
[  116.963002] RIP: 0010:[<ffffffffa0043f58>]  [<ffffffffa0043f58>] ffffffffa0043f58
[  116.963002] RSP: 0018:ffff8800dab5ba98  EFLAGS: 00010292
[  116.963002] RAX: ffffffffdafeff50 RBX: ffff880119ee0098 RCX: 0000000000000006
[  116.963002] RDX: 00000000dafeff50 RSI: ffffffffa00443d0 RDI: ffff880119ee0000
[  116.963002] RBP: ffff8800dab5bb08 R08: ffff8800daa03390 R09: 0000000000000000
[  116.963002] R10: 0000000000000000 R11: ffff880119ee0098 R12: ffff880119ee0000
[  116.963002] R13: ffffffffa0041068 R14: 0000000000000000 R15: ffffffffa00443d0
[  116.963002] FS:  000002afff8d7700(0000) GS:ffff88011fc00000(0000) knlGS:0000000000000000
[  116.963002] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[  116.963002] CR2: ffffffff7b031050 CR3: 0000000001930000 CR4: 00000000000007f0
[  116.963002] Stack:
[  116.963002]  0000000000000001 0000000000000202 ffff8800dab5bac8 ffffffff8190b185
[  116.963002]  ffff880119ee0098 0000000000000004 ffff8800dab5bb08 ffffffff8153e4fc
[  116.963002]  ffff8800dab5bb18 ffff880119ee0098 ffff880119ee0000 ffffffffa0041068
[  116.963002] Call Trace:
[  116.963002]  [<ffffffff8190b185>] ? _raw_spin_unlock_irqrestore+0x23/0x3f
[  116.963002]  [<ffffffff8153e4fc>] ? __pm_runtime_resume+0x5d/0x73
[  116.963002]  [<ffffffffa00443d0>] ? cx88_audio_pci_tbl+0x20/0x60 [cx88_alsa]
[  116.963002]  [<ffffffff8145b7ab>] pci_device_probe+0x84/0xeb
[  116.963002]  [<ffffffff81533655>] ? driver_probe_device+0x1e3/0x1e3
[  116.963002]  [<ffffffff81533529>] driver_probe_device+0xb7/0x1e3
[  116.963002]  [<ffffffff815336c8>] __driver_attach+0x73/0x9d
[  116.963002]  [<ffffffff81533655>] ? driver_probe_device+0x1e3/0x1e3
[  116.963002]  [<ffffffff81531947>] bus_for_each_dev+0x72/0xad
[  116.963002]  [<ffffffff81533061>] driver_attach+0x24/0x2f
[  116.963002]  [<ffffffff81532c4a>] bus_add_driver+0xf3/0x1de
[  116.963002]  [<ffffffffa0044a03>] ? .LC9+0x1b9/0x2e6 [cx88_alsa]
[  116.963002]  [<ffffffff81533dc6>] driver_register+0x8c/0xc9
[  116.963002]  [<ffffffff8145b8e0>] __pci_register_driver+0x5a/0x68
[  116.963002]  [<ffffffffa0046000>] ? 0xffffffffa0045fff
[  116.963002]  [<ffffffffa0046039>] cx88_audio_pci_driver_init+0x39/0x1686 [cx88_alsa]
[  116.963002]  [<ffffffff81000339>] do_one_initcall+0x9a/0x129
[  116.963002]  [<ffffffff8110fadd>] load_module+0x1d70/0x20d1
[  116.963002]  [<ffffffff8110c51b>] ? copy_module_from_fd+0x12b/0x12b
[  116.963002]  [<ffffffffa0044fd9>] ? __param_enable+0x469/0x478 [cx88_alsa]
[  116.963002]  [<ffffffffa0044c30>] ? __param_enable+0xc0/0x478 [cx88_alsa]
[  116.963002]  [<ffffffffa0046048>] ? cx88_audio_pci_driver_init+0x48/0x1686 [cx88_alsa]
[  116.963002]  [<ffffffff81110039>] SyS_finit_module+0x5b/0x77
[  116.963002]  [<ffffffff8190c989>] tracesys+0xda/0xdf
[  116.963002] Code: 66 66 66 66 90 55 48 89 e5 41 57 41 56 41 55 41 54 49 89 fc 53 48 83 ec 48 8b 15 6c d9 ff ff 83 fa 1f 0f 8f f2 03 00 00 48 63 c2 <80> b8 00 11 04 a0 00 75 13 ff c2 41 be fe ff ff ff 89 15 49 d9
[  116.963002] RIP  [<ffffffffa0043f58>] ffffffffa0043f58
[  116.963002]  RSP <ffff8800dab5ba98>
[  116.963002] CR2: ffffffff7b031050
[  116.963002] ---[ end trace 40186657584e0913 ]---

it strives for different widths, eg slhc, pppox, cx88-alsa, nvidia, regardless of whether they are loaded by modprobe or by udev, or /etc/init.d/modules service.

OS:
Gentoo x86_64
Code: Select all
gcc version 4.7.3 (Gentoo Hardened 4.7.3-r1 p1.3, pie-0.5.5)

Vanilla-kernel 3.13.0, 3.13.1.
Grsec patch - all for Linux-3.13.x.

Linux-3.13.1 without grsec error does not occur.

Cheers
8)
Last edited by jacekalex on Thu Feb 20, 2014 5:17 pm, edited 1 time in total.
jacekalex
 
Posts: 39
Joined: Tue Jan 11, 2011 2:16 pm

Re: Linux-3.13.* crash when loading modules.

Postby spender » Wed Feb 05, 2014 8:21 am

This is a known issue involving the new RANDSTRUCT plugin. I've just recently been able to reproduce it and hope to resolve it soon. In the meantime you can disable GRKERNSEC_RANDSTRUCT.

Thanks,
-Brad
spender
 
Posts: 2185
Joined: Wed Feb 20, 2002 8:00 pm

Re: Linux-3.13.* crash when loading modules.

Postby Dwokfur » Wed Feb 05, 2014 10:10 pm

Dear Spender,

I think I also hit this bug upon enabling RANDSTRUCT option in 3.13.0-hardened.
Now I disabled the option and successfully booted 3.13.1-hardened.
The bug only affected the server, not the laptop.

I would be happy to give another try to RANDSTRUCT as soon as you can sort it out. I wonder how I can figure out the next time I should give a try to RANDSTRUCT. Because the option sounds great.

Thanks:
Dw.
Dwokfur
 
Posts: 99
Joined: Tue Jun 08, 2004 10:07 am

Re: Linux-3.13.* crash when loading modules.

Postby spender » Thu Feb 06, 2014 1:07 am

Hi,

It should finally be fixed in the patch just uploaded.

Thanks,
-Brad
spender
 
Posts: 2185
Joined: Wed Feb 20, 2002 8:00 pm

Re: Linux-3.13.* crash when loading modules.

Postby jacekalex » Thu Feb 06, 2014 12:04 pm

spender wrote:This is a known issue involving the new RANDSTRUCT plugin. I've just recently been able to reproduce it and hope to resolve it soon. In the meantime you can disable GRKERNSEC_RANDSTRUCT.

Thanks,
-Brad

Turning off the RANDSTRUCT helped. :)

Thanks
8)
jacekalex
 
Posts: 39
Joined: Tue Jan 11, 2011 2:16 pm

Re: Linux-3.13.* crash when loading modules.

Postby Dwokfur » Wed Feb 12, 2014 4:59 pm

spender wrote:It should finally be fixed in the patch just uploaded.

Hi Brad,

I've recently booted 3.13.2-hardened-r2. It includes grsecurity patch from February 9th.
I've re-enabled RANDSTRUCT, but also enabled RANDSTRUCT_PERFORMANCE. The machine boots fine with this new kernel and config. No crash.
Next time I try without RANDSTRUCT_PERFORMANCE.

Thx: Dw.
Dwokfur
 
Posts: 99
Joined: Tue Jun 08, 2004 10:07 am

Re: [fixed] Linux-3.13.* crash when loading modules.

Postby jacekalex » Mon Feb 17, 2014 1:48 am

Hi

jacekalex wrote:Hi

When I try to load any module randomly this message appears:
Code: Select all
[  116.962213] BUG: unable to handle kernel paging request at ffffffff7b031050
[  116.962679] IP: [<ffffffffa0043f58>] ffffffffa0043f58
[  116.963002] PGD 1946067 PUD 0
[  116.963002] Thread overran stack, or stack corrupted
[  116.963002] Oops: 0000 [#1] PREEMPT SMP
[  116.963002] Modules linked in: cx88_alsa(+) cx88xx tveeprom btcx_risc videobuf_dma_sg videobuf_core slhc
[  116.963002] CPU: 0 PID: 391 Comm: modprobe Not tainted 3.13.1-gr4 #2
[  116.963002] Hardware name: Gigabyte Technology Co., Ltd. P43-ES3G/P43-ES3G, BIOS F14 08/23/2010
[  116.963002] task: ffff8800db93ab20 ti: ffff8800db93b130 task.ti: ffff8800db93b130
[  116.963002] RIP: 0010:[<ffffffffa0043f58>]  [<ffffffffa0043f58>] ffffffffa0043f58
[  116.963002] RSP: 0018:ffff8800dab5ba98  EFLAGS: 00010292
[  116.963002] RAX: ffffffffdafeff50 RBX: ffff880119ee0098 RCX: 0000000000000006
[  116.963002] RDX: 00000000dafeff50 RSI: ffffffffa00443d0 RDI: ffff880119ee0000
[  116.963002] RBP: ffff8800dab5bb08 R08: ffff8800daa03390 R09: 0000000000000000
[  116.963002] R10: 0000000000000000 R11: ffff880119ee0098 R12: ffff880119ee0000
[  116.963002] R13: ffffffffa0041068 R14: 0000000000000000 R15: ffffffffa00443d0
[  116.963002] FS:  000002afff8d7700(0000) GS:ffff88011fc00000(0000) knlGS:0000000000000000
[  116.963002] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[  116.963002] CR2: ffffffff7b031050 CR3: 0000000001930000 CR4: 00000000000007f0
[  116.963002] Stack:
[  116.963002]  0000000000000001 0000000000000202 ffff8800dab5bac8 ffffffff8190b185
[  116.963002]  ffff880119ee0098 0000000000000004 ffff8800dab5bb08 ffffffff8153e4fc
[  116.963002]  ffff8800dab5bb18 ffff880119ee0098 ffff880119ee0000 ffffffffa0041068
[  116.963002] Call Trace:
[  116.963002]  [<ffffffff8190b185>] ? _raw_spin_unlock_irqrestore+0x23/0x3f
[  116.963002]  [<ffffffff8153e4fc>] ? __pm_runtime_resume+0x5d/0x73
[  116.963002]  [<ffffffffa00443d0>] ? cx88_audio_pci_tbl+0x20/0x60 [cx88_alsa]
[  116.963002]  [<ffffffff8145b7ab>] pci_device_probe+0x84/0xeb
[  116.963002]  [<ffffffff81533655>] ? driver_probe_device+0x1e3/0x1e3
[  116.963002]  [<ffffffff81533529>] driver_probe_device+0xb7/0x1e3
[  116.963002]  [<ffffffff815336c8>] __driver_attach+0x73/0x9d
[  116.963002]  [<ffffffff81533655>] ? driver_probe_device+0x1e3/0x1e3
[  116.963002]  [<ffffffff81531947>] bus_for_each_dev+0x72/0xad
[  116.963002]  [<ffffffff81533061>] driver_attach+0x24/0x2f
[  116.963002]  [<ffffffff81532c4a>] bus_add_driver+0xf3/0x1de
[  116.963002]  [<ffffffffa0044a03>] ? .LC9+0x1b9/0x2e6 [cx88_alsa]
[  116.963002]  [<ffffffff81533dc6>] driver_register+0x8c/0xc9
[  116.963002]  [<ffffffff8145b8e0>] __pci_register_driver+0x5a/0x68
[  116.963002]  [<ffffffffa0046000>] ? 0xffffffffa0045fff
[  116.963002]  [<ffffffffa0046039>] cx88_audio_pci_driver_init+0x39/0x1686 [cx88_alsa]
[  116.963002]  [<ffffffff81000339>] do_one_initcall+0x9a/0x129
[  116.963002]  [<ffffffff8110fadd>] load_module+0x1d70/0x20d1
[  116.963002]  [<ffffffff8110c51b>] ? copy_module_from_fd+0x12b/0x12b
[  116.963002]  [<ffffffffa0044fd9>] ? __param_enable+0x469/0x478 [cx88_alsa]
[  116.963002]  [<ffffffffa0044c30>] ? __param_enable+0xc0/0x478 [cx88_alsa]
[  116.963002]  [<ffffffffa0046048>] ? cx88_audio_pci_driver_init+0x48/0x1686 [cx88_alsa]
[  116.963002]  [<ffffffff81110039>] SyS_finit_module+0x5b/0x77
[  116.963002]  [<ffffffff8190c989>] tracesys+0xda/0xdf
[  116.963002] Code: 66 66 66 66 90 55 48 89 e5 41 57 41 56 41 55 41 54 49 89 fc 53 48 83 ec 48 8b 15 6c d9 ff ff 83 fa 1f 0f 8f f2 03 00 00 48 63 c2 <80> b8 00 11 04 a0 00 75 13 ff c2 41 be fe ff ff ff 89 15 49 d9
[  116.963002] RIP  [<ffffffffa0043f58>] ffffffffa0043f58
[  116.963002]  RSP <ffff8800dab5ba98>
[  116.963002] CR2: ffffffff7b031050
[  116.963002] ---[ end trace 40186657584e0913 ]---

it strives for different widths, eg slhc, pppox, cx88-alsa, nvidia, regardless of whether they are loaded by modprobe or by udev, or /etc/init.d/modules service.

OS:
Gentoo x86_64
Code: Select all
gcc version 4.7.3 (Gentoo Hardened 4.7.3-r1 p1.3, pie-0.5.5)

Vanilla-kernel 3.13.0, 3.13.1.
Grsec patch - all for Linux-3.13.x.

Linux-3.13.1 without grsec error does not occur.

Cheers
8)


spender wrote:Hi,

It should finally be fixed in the patch just uploaded.


In Poland we have a proverb that
"History repeats itself" ;)

Grsecurity patch-3.0-3.13.3-201402152204.patch has the same problem.
RANDSTRUCT causes random crashes loading kernel modules.

Thanks
8)
jacekalex
 
Posts: 39
Joined: Tue Jan 11, 2011 2:16 pm

Re: [fixed] Linux-3.13.* crash when loading modules.

Postby spender » Mon Feb 17, 2014 9:50 am

Start with a clean build environment (make mrproper), make sure CONFIG_FRAME_POINTER is enabled, then make O=targetdir and send me (or make available) the entire contents of that targetdir (.config, vmlinux, modules, all generated files). No one else has run into your issue, so we'll need a lot of information in order to be able to debug it. In your original post, you also mentioned that the problem does not occur with grsecurity disabled. Did you see if the problem goes away if only RANDSTRUCT is disabled?

-Brad
spender
 
Posts: 2185
Joined: Wed Feb 20, 2002 8:00 pm

Re: [fixed] Linux-3.13.* crash when loading modules.

Postby jacekalex » Thu Feb 20, 2014 4:27 am

I'm sorry, it's not the same error, my mistake.
At the time of starting the majority of the modules loaded properly, but while the RANDSTRUCT and RANDSTRUCT_PERFORMACE there is a problem with loading modules for PPPoE.
It's one of these modules:
Code: Select all
2 pppoe 11330
pppox 2690 1 pppoe
ppp_generic 26179 6 pppoe, pppox
slhc 5209 1 ppp_generic


Linux-3.13.2 & grsecurity-3.0-3.13.2-201402062224.patch
Code: Select all
/boot/config-3.13.2-gr1:CONFIG_GRKERNSEC_RANDSTRUCT=y
/boot/config-3.13.2-gr1:CONFIG_GRKERNSEC_RANDSTRUCT_PERFORMANCE=y

This kernel works fine.
http://pastebin.com/raw.php?i=ewBjnWVK


Linux-3.13.3 & grsecurity-3.0-3.13.3-201402152204.patch
Code: Select all
/boot/config-3.13.3-gr2:# CONFIG_GRKERNSEC_RANDSTRUCT is not set

This kernel works fine without RANDSTUCT
http://pastebin.com/raw.php?i=4E5TuQqs

Linux-3.13.3 & grsecurity-3.0-3.13.3-201402152204.patch - randstruct enable:
Code: Select all
/boot/config-3.13.3-gr2-randstruct:CONFIG_GRKERNSEC_RANDSTRUCT=y
/boot/config-3.13.3-gr2-randstruct:CONFIG_GRKERNSEC_RANDSTRUCT_PERFORMANCE=y

This kernel has kernel_panic or silent crash kernel.

They occur two possible situations occurring almost at random:
Automatic start system modules for PPPoE are loaded automatically by the pppd daemon, and apparently loaded correctly, but the attempt of any network connection causes kernel-panic.
Image
I noticed it only when the system starts switched to the root shell, took command pppd daemon
Code: Select all
pon <provider>.

The connection has been raised, but when I then sent a ping on any server crash that followed the system was functioning then just sysrq.
Image

This is the problematic kernel image:
http://jacekalex.sh.dug.net.pl/linux-3. ... x86.tar.xz

Thanks
8)
jacekalex
 
Posts: 39
Joined: Tue Jan 11, 2011 2:16 pm

Re: [fixed] Linux-3.13.* crash when loading modules.

Postby PaX Team » Thu Feb 20, 2014 6:58 am

can you remove __randomize_layout from struct neigh_table in include/net/neighbour.h and see if it gets any further?
PaX Team
 
Posts: 2310
Joined: Mon Mar 18, 2002 4:35 pm

Re: [fixed] Linux-3.13.* crash when loading modules.

Postby jacekalex » Thu Feb 20, 2014 5:12 pm

PaX Team wrote:can you remove __randomize_layout from struct neigh_table in include/net/neighbour.h and see if it gets any further?


It helped. :)

I do not know whether the partition works fine on 100%, but it looks like it helped (before the latest patch also was the problem).

Linux 3.13.3
grsecurity-3.0-3.13.3-201402192252.patch
Code: Select all
grep -iA30 neigh_table /usr/src/linux/include/net/neighbour.h | grep randomize


Code: Select all
grep RANDSTRUCT /usr/src/linux/.config
CONFIG_GRKERNSEC_RANDSTRUCT=y
CONFIG_GRKERNSEC_RANDSTRUCT_PERFORMANCE=y


Thanks
8)
jacekalex
 
Posts: 39
Joined: Tue Jan 11, 2011 2:16 pm

Re: [SOLVED] Linux-3.13.* crash when loading modules.

Postby PaX Team » Thu Feb 20, 2014 5:28 pm

thanks, it'll be fixed properly in the next patch (it's actually a bug in linux, some code from '98 ;)).
PaX Team
 
Posts: 2310
Joined: Mon Mar 18, 2002 4:35 pm


Return to grsecurity support

cron