Page 1 of 1

[SOLVED] Linux-3.13.* crash when loading modules.

PostPosted: Tue Feb 04, 2014 8:14 pm
by jacekalex
Hi

When I try to load any module randomly this message appears:
Code: Select all
[  116.962213] BUG: unable to handle kernel paging request at ffffffff7b031050
[  116.962679] IP: [<ffffffffa0043f58>] ffffffffa0043f58
[  116.963002] PGD 1946067 PUD 0
[  116.963002] Thread overran stack, or stack corrupted
[  116.963002] Oops: 0000 [#1] PREEMPT SMP
[  116.963002] Modules linked in: cx88_alsa(+) cx88xx tveeprom btcx_risc videobuf_dma_sg videobuf_core slhc
[  116.963002] CPU: 0 PID: 391 Comm: modprobe Not tainted 3.13.1-gr4 #2
[  116.963002] Hardware name: Gigabyte Technology Co., Ltd. P43-ES3G/P43-ES3G, BIOS F14 08/23/2010
[  116.963002] task: ffff8800db93ab20 ti: ffff8800db93b130 task.ti: ffff8800db93b130
[  116.963002] RIP: 0010:[<ffffffffa0043f58>]  [<ffffffffa0043f58>] ffffffffa0043f58
[  116.963002] RSP: 0018:ffff8800dab5ba98  EFLAGS: 00010292
[  116.963002] RAX: ffffffffdafeff50 RBX: ffff880119ee0098 RCX: 0000000000000006
[  116.963002] RDX: 00000000dafeff50 RSI: ffffffffa00443d0 RDI: ffff880119ee0000
[  116.963002] RBP: ffff8800dab5bb08 R08: ffff8800daa03390 R09: 0000000000000000
[  116.963002] R10: 0000000000000000 R11: ffff880119ee0098 R12: ffff880119ee0000
[  116.963002] R13: ffffffffa0041068 R14: 0000000000000000 R15: ffffffffa00443d0
[  116.963002] FS:  000002afff8d7700(0000) GS:ffff88011fc00000(0000) knlGS:0000000000000000
[  116.963002] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[  116.963002] CR2: ffffffff7b031050 CR3: 0000000001930000 CR4: 00000000000007f0
[  116.963002] Stack:
[  116.963002]  0000000000000001 0000000000000202 ffff8800dab5bac8 ffffffff8190b185
[  116.963002]  ffff880119ee0098 0000000000000004 ffff8800dab5bb08 ffffffff8153e4fc
[  116.963002]  ffff8800dab5bb18 ffff880119ee0098 ffff880119ee0000 ffffffffa0041068
[  116.963002] Call Trace:
[  116.963002]  [<ffffffff8190b185>] ? _raw_spin_unlock_irqrestore+0x23/0x3f
[  116.963002]  [<ffffffff8153e4fc>] ? __pm_runtime_resume+0x5d/0x73
[  116.963002]  [<ffffffffa00443d0>] ? cx88_audio_pci_tbl+0x20/0x60 [cx88_alsa]
[  116.963002]  [<ffffffff8145b7ab>] pci_device_probe+0x84/0xeb
[  116.963002]  [<ffffffff81533655>] ? driver_probe_device+0x1e3/0x1e3
[  116.963002]  [<ffffffff81533529>] driver_probe_device+0xb7/0x1e3
[  116.963002]  [<ffffffff815336c8>] __driver_attach+0x73/0x9d
[  116.963002]  [<ffffffff81533655>] ? driver_probe_device+0x1e3/0x1e3
[  116.963002]  [<ffffffff81531947>] bus_for_each_dev+0x72/0xad
[  116.963002]  [<ffffffff81533061>] driver_attach+0x24/0x2f
[  116.963002]  [<ffffffff81532c4a>] bus_add_driver+0xf3/0x1de
[  116.963002]  [<ffffffffa0044a03>] ? .LC9+0x1b9/0x2e6 [cx88_alsa]
[  116.963002]  [<ffffffff81533dc6>] driver_register+0x8c/0xc9
[  116.963002]  [<ffffffff8145b8e0>] __pci_register_driver+0x5a/0x68
[  116.963002]  [<ffffffffa0046000>] ? 0xffffffffa0045fff
[  116.963002]  [<ffffffffa0046039>] cx88_audio_pci_driver_init+0x39/0x1686 [cx88_alsa]
[  116.963002]  [<ffffffff81000339>] do_one_initcall+0x9a/0x129
[  116.963002]  [<ffffffff8110fadd>] load_module+0x1d70/0x20d1
[  116.963002]  [<ffffffff8110c51b>] ? copy_module_from_fd+0x12b/0x12b
[  116.963002]  [<ffffffffa0044fd9>] ? __param_enable+0x469/0x478 [cx88_alsa]
[  116.963002]  [<ffffffffa0044c30>] ? __param_enable+0xc0/0x478 [cx88_alsa]
[  116.963002]  [<ffffffffa0046048>] ? cx88_audio_pci_driver_init+0x48/0x1686 [cx88_alsa]
[  116.963002]  [<ffffffff81110039>] SyS_finit_module+0x5b/0x77
[  116.963002]  [<ffffffff8190c989>] tracesys+0xda/0xdf
[  116.963002] Code: 66 66 66 66 90 55 48 89 e5 41 57 41 56 41 55 41 54 49 89 fc 53 48 83 ec 48 8b 15 6c d9 ff ff 83 fa 1f 0f 8f f2 03 00 00 48 63 c2 <80> b8 00 11 04 a0 00 75 13 ff c2 41 be fe ff ff ff 89 15 49 d9
[  116.963002] RIP  [<ffffffffa0043f58>] ffffffffa0043f58
[  116.963002]  RSP <ffff8800dab5ba98>
[  116.963002] CR2: ffffffff7b031050
[  116.963002] ---[ end trace 40186657584e0913 ]---

it strives for different widths, eg slhc, pppox, cx88-alsa, nvidia, regardless of whether they are loaded by modprobe or by udev, or /etc/init.d/modules service.

OS:
Gentoo x86_64
Code: Select all
gcc version 4.7.3 (Gentoo Hardened 4.7.3-r1 p1.3, pie-0.5.5)

Vanilla-kernel 3.13.0, 3.13.1.
Grsec patch - all for Linux-3.13.x.

Linux-3.13.1 without grsec error does not occur.

Cheers
8)

Re: Linux-3.13.* crash when loading modules.

PostPosted: Wed Feb 05, 2014 8:21 am
by spender
This is a known issue involving the new RANDSTRUCT plugin. I've just recently been able to reproduce it and hope to resolve it soon. In the meantime you can disable GRKERNSEC_RANDSTRUCT.

Thanks,
-Brad

Re: Linux-3.13.* crash when loading modules.

PostPosted: Wed Feb 05, 2014 10:10 pm
by Dwokfur
Dear Spender,

I think I also hit this bug upon enabling RANDSTRUCT option in 3.13.0-hardened.
Now I disabled the option and successfully booted 3.13.1-hardened.
The bug only affected the server, not the laptop.

I would be happy to give another try to RANDSTRUCT as soon as you can sort it out. I wonder how I can figure out the next time I should give a try to RANDSTRUCT. Because the option sounds great.

Thanks:
Dw.

Re: Linux-3.13.* crash when loading modules.

PostPosted: Thu Feb 06, 2014 1:07 am
by spender
Hi,

It should finally be fixed in the patch just uploaded.

Thanks,
-Brad

Re: Linux-3.13.* crash when loading modules.

PostPosted: Thu Feb 06, 2014 12:04 pm
by jacekalex
spender wrote:This is a known issue involving the new RANDSTRUCT plugin. I've just recently been able to reproduce it and hope to resolve it soon. In the meantime you can disable GRKERNSEC_RANDSTRUCT.

Thanks,
-Brad

Turning off the RANDSTRUCT helped. :)

Thanks
8)

Re: Linux-3.13.* crash when loading modules.

PostPosted: Wed Feb 12, 2014 4:59 pm
by Dwokfur
spender wrote:It should finally be fixed in the patch just uploaded.

Hi Brad,

I've recently booted 3.13.2-hardened-r2. It includes grsecurity patch from February 9th.
I've re-enabled RANDSTRUCT, but also enabled RANDSTRUCT_PERFORMANCE. The machine boots fine with this new kernel and config. No crash.
Next time I try without RANDSTRUCT_PERFORMANCE.

Thx: Dw.

Re: [fixed] Linux-3.13.* crash when loading modules.

PostPosted: Mon Feb 17, 2014 1:48 am
by jacekalex
Hi

jacekalex wrote:Hi

When I try to load any module randomly this message appears:
Code: Select all
[  116.962213] BUG: unable to handle kernel paging request at ffffffff7b031050
[  116.962679] IP: [<ffffffffa0043f58>] ffffffffa0043f58
[  116.963002] PGD 1946067 PUD 0
[  116.963002] Thread overran stack, or stack corrupted
[  116.963002] Oops: 0000 [#1] PREEMPT SMP
[  116.963002] Modules linked in: cx88_alsa(+) cx88xx tveeprom btcx_risc videobuf_dma_sg videobuf_core slhc
[  116.963002] CPU: 0 PID: 391 Comm: modprobe Not tainted 3.13.1-gr4 #2
[  116.963002] Hardware name: Gigabyte Technology Co., Ltd. P43-ES3G/P43-ES3G, BIOS F14 08/23/2010
[  116.963002] task: ffff8800db93ab20 ti: ffff8800db93b130 task.ti: ffff8800db93b130
[  116.963002] RIP: 0010:[<ffffffffa0043f58>]  [<ffffffffa0043f58>] ffffffffa0043f58
[  116.963002] RSP: 0018:ffff8800dab5ba98  EFLAGS: 00010292
[  116.963002] RAX: ffffffffdafeff50 RBX: ffff880119ee0098 RCX: 0000000000000006
[  116.963002] RDX: 00000000dafeff50 RSI: ffffffffa00443d0 RDI: ffff880119ee0000
[  116.963002] RBP: ffff8800dab5bb08 R08: ffff8800daa03390 R09: 0000000000000000
[  116.963002] R10: 0000000000000000 R11: ffff880119ee0098 R12: ffff880119ee0000
[  116.963002] R13: ffffffffa0041068 R14: 0000000000000000 R15: ffffffffa00443d0
[  116.963002] FS:  000002afff8d7700(0000) GS:ffff88011fc00000(0000) knlGS:0000000000000000
[  116.963002] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[  116.963002] CR2: ffffffff7b031050 CR3: 0000000001930000 CR4: 00000000000007f0
[  116.963002] Stack:
[  116.963002]  0000000000000001 0000000000000202 ffff8800dab5bac8 ffffffff8190b185
[  116.963002]  ffff880119ee0098 0000000000000004 ffff8800dab5bb08 ffffffff8153e4fc
[  116.963002]  ffff8800dab5bb18 ffff880119ee0098 ffff880119ee0000 ffffffffa0041068
[  116.963002] Call Trace:
[  116.963002]  [<ffffffff8190b185>] ? _raw_spin_unlock_irqrestore+0x23/0x3f
[  116.963002]  [<ffffffff8153e4fc>] ? __pm_runtime_resume+0x5d/0x73
[  116.963002]  [<ffffffffa00443d0>] ? cx88_audio_pci_tbl+0x20/0x60 [cx88_alsa]
[  116.963002]  [<ffffffff8145b7ab>] pci_device_probe+0x84/0xeb
[  116.963002]  [<ffffffff81533655>] ? driver_probe_device+0x1e3/0x1e3
[  116.963002]  [<ffffffff81533529>] driver_probe_device+0xb7/0x1e3
[  116.963002]  [<ffffffff815336c8>] __driver_attach+0x73/0x9d
[  116.963002]  [<ffffffff81533655>] ? driver_probe_device+0x1e3/0x1e3
[  116.963002]  [<ffffffff81531947>] bus_for_each_dev+0x72/0xad
[  116.963002]  [<ffffffff81533061>] driver_attach+0x24/0x2f
[  116.963002]  [<ffffffff81532c4a>] bus_add_driver+0xf3/0x1de
[  116.963002]  [<ffffffffa0044a03>] ? .LC9+0x1b9/0x2e6 [cx88_alsa]
[  116.963002]  [<ffffffff81533dc6>] driver_register+0x8c/0xc9
[  116.963002]  [<ffffffff8145b8e0>] __pci_register_driver+0x5a/0x68
[  116.963002]  [<ffffffffa0046000>] ? 0xffffffffa0045fff
[  116.963002]  [<ffffffffa0046039>] cx88_audio_pci_driver_init+0x39/0x1686 [cx88_alsa]
[  116.963002]  [<ffffffff81000339>] do_one_initcall+0x9a/0x129
[  116.963002]  [<ffffffff8110fadd>] load_module+0x1d70/0x20d1
[  116.963002]  [<ffffffff8110c51b>] ? copy_module_from_fd+0x12b/0x12b
[  116.963002]  [<ffffffffa0044fd9>] ? __param_enable+0x469/0x478 [cx88_alsa]
[  116.963002]  [<ffffffffa0044c30>] ? __param_enable+0xc0/0x478 [cx88_alsa]
[  116.963002]  [<ffffffffa0046048>] ? cx88_audio_pci_driver_init+0x48/0x1686 [cx88_alsa]
[  116.963002]  [<ffffffff81110039>] SyS_finit_module+0x5b/0x77
[  116.963002]  [<ffffffff8190c989>] tracesys+0xda/0xdf
[  116.963002] Code: 66 66 66 66 90 55 48 89 e5 41 57 41 56 41 55 41 54 49 89 fc 53 48 83 ec 48 8b 15 6c d9 ff ff 83 fa 1f 0f 8f f2 03 00 00 48 63 c2 <80> b8 00 11 04 a0 00 75 13 ff c2 41 be fe ff ff ff 89 15 49 d9
[  116.963002] RIP  [<ffffffffa0043f58>] ffffffffa0043f58
[  116.963002]  RSP <ffff8800dab5ba98>
[  116.963002] CR2: ffffffff7b031050
[  116.963002] ---[ end trace 40186657584e0913 ]---

it strives for different widths, eg slhc, pppox, cx88-alsa, nvidia, regardless of whether they are loaded by modprobe or by udev, or /etc/init.d/modules service.

OS:
Gentoo x86_64
Code: Select all
gcc version 4.7.3 (Gentoo Hardened 4.7.3-r1 p1.3, pie-0.5.5)

Vanilla-kernel 3.13.0, 3.13.1.
Grsec patch - all for Linux-3.13.x.

Linux-3.13.1 without grsec error does not occur.

Cheers
8)


spender wrote:Hi,

It should finally be fixed in the patch just uploaded.


In Poland we have a proverb that
"History repeats itself" ;)

Grsecurity patch-3.0-3.13.3-201402152204.patch has the same problem.
RANDSTRUCT causes random crashes loading kernel modules.

Thanks
8)

Re: [fixed] Linux-3.13.* crash when loading modules.

PostPosted: Mon Feb 17, 2014 9:50 am
by spender
Start with a clean build environment (make mrproper), make sure CONFIG_FRAME_POINTER is enabled, then make O=targetdir and send me (or make available) the entire contents of that targetdir (.config, vmlinux, modules, all generated files). No one else has run into your issue, so we'll need a lot of information in order to be able to debug it. In your original post, you also mentioned that the problem does not occur with grsecurity disabled. Did you see if the problem goes away if only RANDSTRUCT is disabled?

-Brad

Re: [fixed] Linux-3.13.* crash when loading modules.

PostPosted: Thu Feb 20, 2014 4:27 am
by jacekalex
I'm sorry, it's not the same error, my mistake.
At the time of starting the majority of the modules loaded properly, but while the RANDSTRUCT and RANDSTRUCT_PERFORMACE there is a problem with loading modules for PPPoE.
It's one of these modules:
Code: Select all
2 pppoe 11330
pppox 2690 1 pppoe
ppp_generic 26179 6 pppoe, pppox
slhc 5209 1 ppp_generic


Linux-3.13.2 & grsecurity-3.0-3.13.2-201402062224.patch
Code: Select all
/boot/config-3.13.2-gr1:CONFIG_GRKERNSEC_RANDSTRUCT=y
/boot/config-3.13.2-gr1:CONFIG_GRKERNSEC_RANDSTRUCT_PERFORMANCE=y

This kernel works fine.
http://pastebin.com/raw.php?i=ewBjnWVK


Linux-3.13.3 & grsecurity-3.0-3.13.3-201402152204.patch
Code: Select all
/boot/config-3.13.3-gr2:# CONFIG_GRKERNSEC_RANDSTRUCT is not set

This kernel works fine without RANDSTUCT
http://pastebin.com/raw.php?i=4E5TuQqs

Linux-3.13.3 & grsecurity-3.0-3.13.3-201402152204.patch - randstruct enable:
Code: Select all
/boot/config-3.13.3-gr2-randstruct:CONFIG_GRKERNSEC_RANDSTRUCT=y
/boot/config-3.13.3-gr2-randstruct:CONFIG_GRKERNSEC_RANDSTRUCT_PERFORMANCE=y

This kernel has kernel_panic or silent crash kernel.

They occur two possible situations occurring almost at random:
Automatic start system modules for PPPoE are loaded automatically by the pppd daemon, and apparently loaded correctly, but the attempt of any network connection causes kernel-panic.
Image
I noticed it only when the system starts switched to the root shell, took command pppd daemon
Code: Select all
pon <provider>.

The connection has been raised, but when I then sent a ping on any server crash that followed the system was functioning then just sysrq.
Image

This is the problematic kernel image:
http://jacekalex.sh.dug.net.pl/linux-3. ... x86.tar.xz

Thanks
8)

Re: [fixed] Linux-3.13.* crash when loading modules.

PostPosted: Thu Feb 20, 2014 6:58 am
by PaX Team
can you remove __randomize_layout from struct neigh_table in include/net/neighbour.h and see if it gets any further?

Re: [fixed] Linux-3.13.* crash when loading modules.

PostPosted: Thu Feb 20, 2014 5:12 pm
by jacekalex
PaX Team wrote:can you remove __randomize_layout from struct neigh_table in include/net/neighbour.h and see if it gets any further?


It helped. :)

I do not know whether the partition works fine on 100%, but it looks like it helped (before the latest patch also was the problem).

Linux 3.13.3
grsecurity-3.0-3.13.3-201402192252.patch
Code: Select all
grep -iA30 neigh_table /usr/src/linux/include/net/neighbour.h | grep randomize


Code: Select all
grep RANDSTRUCT /usr/src/linux/.config
CONFIG_GRKERNSEC_RANDSTRUCT=y
CONFIG_GRKERNSEC_RANDSTRUCT_PERFORMANCE=y


Thanks
8)

Re: [SOLVED] Linux-3.13.* crash when loading modules.

PostPosted: Thu Feb 20, 2014 5:28 pm
by PaX Team
thanks, it'll be fixed properly in the next patch (it's actually a bug in linux, some code from '98 ;)).