kernel BUG at mm/mmap.c:1683

[linkfanel]

I ran into a kernel BUG when restarting a crashed Iceweasel.

Code: Select all

[945426.297697] kernel BUG at mm/mmap.c:1683!
[945426.297699] invalid opcode: 0000 [#1]
[945426.297701] Modules linked in: tun bitrev michael_mic arc4 ecb blkcipher cryptomgr ieee80211_crypt_tkip af_packet radeon drm fan ipv6 ehci_hcd uhci_hcd usbcore pcmcia yenta_socket rsrc_nonstatic pcmcia_core nls_iso8859_1 nls_cp850 vfat fat nls_utf8 ntfs nls_base reiserfs dm_snapshot dm_mirror dm_mod hdaps fbcon crc32 font bitblit softcursor radeonfb fb fb_ddc i2c_algo_bit cfbcopyarea cfbimgblt cfbfillrect 8250_pci snd_intel8x0 snd_intel8x0m snd_ac97_codec ac97_bus snd_pcm_oss snd_mixer_oss thermal ipw2200 8250_pnp ieee80211 ieee80211_crypt snd_pcm firmware_class 8250 serial_core evdev video snd_timer battery ac processor e1000 button thinkpad_acpi hwmon backlight nvram i2c_i801 intel_agp agpgart output psmouse rtc snd i2c_core soundcore snd_page_alloc unix
[945426.297750] CPU:    0
[945426.297751] EIP:    0060:[<000480cb>]    Not tainted VLI
[945426.297752] EFLAGS: 00010246   (2.6.23.8-grsec #1)
[945426.297755] eax: cecd09f8   ebx: 4683a000   ecx: cecd09f8   edx: cecd09f8
[945426.297757] esi: 4683a000   edi: cecd0aa8   ebp: cecd0aa8   esp: cc40fe90
[945426.297760] ds: 0068   es: 0068   fs: 0000  gs: 0033  ss: 0068
[945426.297763] Process firefox-bin (pid: 19961, ti=cc40e000 task=d64f9a90 task.ti=cc40e000)
[945426.297765] Stack: cecd09f8 cecd0aa8 cecd0aa8 0004919d 4683a000 a683a000 4683a000 00000000
[945426.297771]        00000000 00000000 00000000 00000000 4683a000 cc40ff90 cecd0aa8 0004a7fd
[945426.297775]        4683b000 00100075 d949cea0 00000000 0004683a 00000000 d38d0e00 00100073
[945426.297781] Call Trace:
[945426.297804]  [<0004919d>] <0> [<0004a7fd>] <0> [<00100075>] <0> [<0004683a>] <0> [<00100073>] <0> [<00100075>] <0> [<0004acb6>] <0> [<00100075>] <0> [<00013053>] <0> [<0002161e>] <0> [<00003cfc>] <0> [<00003cd6>] <0> =======================
[945426.298138] Code: 29 f0 39 c3 75 36 8b 41 44 3b 42 44 74 04 0f 0b eb fe 8b 41 3c 3b 42 3c 75 f4 8b 4a 14 31 cf 81 e7 dd df ef df 74 af 0f 0b eb fe <0f> 0b eb fe 90 0f 0b eb fe 0f 0b eb fe 0f 0b eb fe 0f 0b eb fe
[945426.298176] EIP: [<000480cb>]  SS:ESP 0068:cc40fe90

It happened on a 2.6.23.8 patched with grsecurity-2.1.11-2.6.23.8-200711230831. According to my sources, obviously PaX-related. It doesn't seem that I can reproduce it.

[PaX Team]

It happened on a 2.6.23.8 patched with grsecurity-2.1.11-2.6.23.8-200711230831. According to my sources, obviously PaX-related.

can you send me your vmlinux (if you no longer have it then bzImage) and System.map please?

It doesn't seem that I can reproduce it.

maybe you can check your browser history to see what sites you visited around that time? are you using flash/etc? did you disable any PaX flags on firefox?

[linkfanel]

can you send me your vmlinux (if you no longer have it then bzImage) and System.map please?

You can get them at http://www.linkfanel.net/stuff/vmlinuz-2.6.23.8-grsec and http://www.linkfanel.net/stuff/System.m ... 23.8-grsec

maybe you can check your browser history to see what sites you visited around that time? are you using flash/etc? did you disable any PaX flags on firefox?

So I was checking https://bugzilla.mozilla.org/show_bug.cgi?id=399293 with Flash enabled and everything. I shut down my browser properly, and I reset the binary's flags with paxctl -zex. I started the browser, with a single tab opened, and logged to Yahoo! Mail. Expectedly, it was killed by PaX. I disabled MPROTECT with paxctl -m, and I restarted the browser. The alert box proposing me to resume the crashed session opened up, and I clicked on ok. It went back to Yahoo! Mail, and after a few seconds froze. Then I noticed the BUG on all my terminals. `ps faux` would hang too. I rebooted and everything was fine after that.

[PaX Team]

So I was checking https://bugzilla.mozilla.org/show_bug.cgi?id=399293 with Flash enabled and everything. I shut down my browser properly, and I reset the binary's flags with paxctl -zex. I started the browser, with a single tab opened, and logged to Yahoo! Mail. Expectedly, it was killed by PaX. I disabled MPROTECT with paxctl -m, and I restarted the browser. The alert box proposing me to resume the crashed session opened up, and I clicked on ok. It went back to Yahoo! Mail, and after a few seconds froze. Then I noticed the BUG on all my terminals. `ps faux` would hang too. I rebooted and everything was fine after that.

i've been trying to reproduce this ever since but to no avail, Yahoo Mail and any other flash content works fine here. so to make some forward progress, i'll need an strace log of a crashing firefox session, preferably under the just released 2.6.24 kernel (there's a PaX test patch already at the usual place). since the plain strace logs would probably be too big, you can filter only the following: strace -f -e trace=open,mmap2,mprotect,munmap . i'll also need the corresponding kernel BUG message.

[Xaid]

Hi,

I can reproduce the exact problem using mplayer and any .wmv/.asf file (since I'm guessing that makes it using the win32codecs). I have tested with both 2.6.23 and 2.6.24 (with the latest test9 pax patch) and I see the problem in both. 2.6.20 does not have the same issue.

I am running a Gentoo Hardened system and the error message that i see is the following (I am running 2.6.24 with the latest pax patch):

------------[ cut here ]------------
kernel BUG at mm/mmap.c:1686!
invalid opcode: 0000 [#1] PREEMPT SMP
Modules linked in: snd_seq_midi snd_emu10k1_synth snd_emux_synth snd_seq_virmidi snd_seq_midi_event snd_seq_midi_emul snd_seq w83627hf hwmon_vid snd_emu10k1 snd_rawmidi snd_ac97_codec ac97_bus snd_pcm snd_seq_device snd_timer snd_page_alloc snd_util_mem snd_hwdep snd

Pid: 16606, comm: mplayer Not tainted (2.6.24 #4)
EIP: 0060:[<c0858a1d>] EFLAGS: 00010246 CPU: 0
EIP is at pax_find_mirror_vma+0x87/0x97
EAX: a0400fff EBX: 00401000 ECX: f6b97580 EDX: 00000000
ESI: 0052d000 EDI: 00100077 EBP: 00506000 ESP: f5ae3e94
DS: 0068 ES: 0068 FS: 00d8 GS: 0033 SS: 0068
Process mplayer (pid: 16606, ti=f5ae2000 task=f7077540 task.ti=f5ae2000)
Stack: f69f4380 f6b97580 00506000 c0858ef7 00000000 00000105 60506000 00000000
f6b97580 00507000 00506000 f69f4380 c08591e1 00000000 00506000 f6b97898
f69f4380 00506000 00001000 f5ae3f38 c085910c f69f43b4 00507000 00000073
Call Trace:
[<c0858ef7>] split_vma+0x26/0x222
[<c08591e1>] __do_munmap+0x9c/0x131
[<c085910c>] do_munmap+0x19/0x52
[<c08586ef>] mmap_region+0x41a/0x534
[<c09066b2>] __copy_to_user_ll+0x19/0x24
[<c0858177>] do_mmap_pgoff+0x273/0x35f
[<c0807d62>] sys_mmap2+0x73/0x9d
[<c0802d32>] syscall_call+0x7/0xb
=======================
Code: 39 41 3c 75 2e 33 7a 14 89 d0 81 e7 dd df ef df 75 25 5b 5e 5f c3 8b 51 54 31 c0 85 d2 74 f3 0f 0b eb fe 0f 0b eb fe 0f 0b eb fe <0f> 0b eb fe 0f 0b eb fe 0f 0b eb fe 0f 0b eb fe 83 ec 18 89 74
EIP: [<c0858a1d>] pax_find_mirror_vma+0x87/0x97 SS:ESP 0068:f5ae3e94
---[ end trace a10f0aada312b36b ]---

My kernel config can be found at http://rafb.net/p/KVMSNP99.html

Thanks

Edit: Changed config URL to rafb.net

[PaX Team]

I can reproduce the exact problem using mplayer and any .wmv/.asf file (since I'm guessing that makes it using the win32codecs). I have tested with both 2.6.23 and 2.6.24 (with the latest test9 pax patch) and I see the problem in both. 2.6.20 does not have the same issue.

thanks for the info, -test11 should fix this.

[Xaid]

Hi,
I can confirm that -test11 fixes the problem. I will continue running this and will report back if I run across any issues.

Thank you for the quick response

[linkfanel]

The BUG had happened to me again randomly a couple of times, sometimes while I was away. However, I just eventually came across a web page that triggers it in a reproducible way. Go to http://www.cam4.com/ and pick one (NSFW, and yeah they must be ugly and annoying, I couldn't see anyway). I guess it's flash-related again.

I tried the newer patches and I expected that it would be fixed, but it isn't. The BUG still occurs with:
- Linux 2.6.23.12 + grsecurity-2.1.11-2.6.23.9-200712101800.patch
- Linux 2.6.24 + pax-linux-2.6.24-test9.patch
- Linux 2.6.24 + pax-linux-2.6.24-test11.patch
- Linux 2.6.24.2 + grsecurity-2.1.11-2.6.24.2-200802131840

Here is a trace of the last one:

Code: Select all

[  114.828688] kernel BUG at mm/mmap.c:1721!
[  114.828690] invalid opcode: 0000 [#1]
[  114.828692] Modules linked in: michael_mic arc4 ecb blkcipher cryptomgr ieee80211_crypt_tkip af_packet radeon drm fan ipv6 nls_iso8859_1 nls_cp850 vfat fat nls_utf8 ntfs nls_base reiserfs dm_snapshot dm_mirror dm_mod hdaps input_polldev fbcon crc32 font bitblit softcursor radeonfb fb fb_ddc i2c_algo_bit cfbcopyarea cfbimgblt cfbfillrect 8250_pci 8250_pnp snd_intel8x0 snd_intel8x0m snd_ac97_codec ac97_bus snd_pcm_oss snd_mixer_oss snd_pcm snd_timer thinkpad_acpi hwmon ipw2200 snd nvram i2c_i801 8250 serial_core rtc psmouse ieee80211 ieee80211_crypt evdev soundcore snd_page_alloc i2c_core video backlight e1000 firmware_class battery ac output thermal button intel_agp agpgart processor unix
[  114.828729]
[  114.828732] Pid: 2403, comm: firefox-bin Not tainted (2.6.24.2-grsec #1)
[  114.828734] EIP: 0060:[<0004ac49>] EFLAGS: 00010246 CPU: 0
[  114.828737] EAX: d60210b0 EBX: 44eb0000 ECX: d60210b0 EDX: d60210b0
[  114.828739] ESI: 44eb0000 EDI: dc3cff90 EBP: d6021108 ESP: dc3cfe90
[  114.828742]  DS: 0068 ES: 0068 FS: 0000 GS: 0033 SS: 0068
[  114.828745] Process firefox-bin (pid: 2403, ti=dc3ce000 task=dc3fb540 task.ti=dc3ce000)
[  114.828747] Stack: d60210b0 d6021108 dc3cff90 0004bd1d 00000000 44eb1000 44eb0000 0004d5c9
[  114.828752]        00000001 00100073 00000000 00000000 44eb0000 dc3cff90 d6021108 0004d45d
[  114.828757]        44eb1000 00100075 dc2f7de0 00000000 00044eb0 00000000 dc3e6a80 00100073
[  114.828762] Call Trace:
[  114.828783]  [<0004bd1d>] <0> [<0004d5c9>] <0> [<00100073>] <0> [<0004d45d>] <0> [<00100075>] <0> [<00044eb0>] <0> [<00100073>] <0> [<00100075>] <0> [<0004d914>] <0> [<00100075>] <0> [<0001c629>] <0> [<0001346f>] <0> [<00003d26>] <0> =======================
[  114.829256] Code: 39 c3 75 3e 8b 41 44 3b 42 44 74 06 0f 0b eb fe 66 90 8b 41 3c 3b 42 3c 75 f2 8b 4a 14 31 cf 81 e7 dd df ef df 74 a5 0f 0b eb fe <0f> 0b eb fe 8d 76 00 0f 0b eb fe 0f 0b eb fe 0f 0b eb fe 0f 0b
[  114.829294] EIP: [<0004ac49>]  SS:ESP 0068:dc3cfe90
[  114.829299] ---[ end trace 7d103f02eab9cd04 ]---


[linkfanel]

Still happens with Linux 2.6.24.2 + grsecurity-2.1.11-2.6.24.2-200802131959.

[PaX Team]

The BUG had happened to me again randomly a couple of times, sometimes while I was away. However, I just eventually came across a web page that triggers it in a reproducible way. Go to http://www.cam4.com/ and pick one (NSFW, and yeah they must be ugly and annoying, I couldn't see anyway).

oh boy, i wish i hadn't seen this either , but in any case, i reproduced one problem at least, so -test18 is waiting for you. i don't know if it will fix your particular bug, but it might fix the one that others have seen back in .23 with a funny NULL deref in the context switch code. if it still fails for you, i'll need that you enable kernel symbols (disable grsec's symbol hiding first) and also your .config plus the strace i asked for some time ago.

[linkfanel]

It's still not fixed. I tried with Linux 2.6.24.2 + pax-linux-2.6.24.2-test26, and it gave:

Code: Select all

[  271.632743] kernel BUG at mm/mmap.c:1715!
[  271.632745] invalid opcode: 0000 [#1]
[  271.632747] Modules linked in: michael_mic arc4 ecb blkcipher cryptomgr ieee80211_crypt_tkip af_packet radeon drm fan ipv6 nls_iso8859_1 nls_cp850 vfat fat nls_utf8 ntfs nls_base reiserfs dm_snapshot dm_mirror dm_mod hdaps input_polldev fbcon crc32 font bitblit softcursor radeonfb fb fb_ddc i2c_algo_bit cfbcopyarea cfbimgblt cfbfillrect 8250_pci 8250_pnp snd_intel8x0 snd_intel8x0m snd_ac97_codec ac97_bus snd_pcm_oss snd_mixer_oss snd_pcm snd_timer snd thinkpad_acpi hwmon nvram 8250 serial_core rtc psmouse evdev soundcore snd_page_alloc i2c_i801 i2c_core ipw2200 ieee80211 ieee80211_crypt firmware_class e1000 battery ac video backlight output thermal button processor intel_agp agpgart unix
[  271.632784]
[  271.632787] Pid: 2731, comm: firefox-bin Not tainted (2.6.24.2 #1)
[  271.632790] EIP: 0060:[<0004bf29>] EFLAGS: 00210246 CPU: 0
[  271.632798] EIP is at pax_find_mirror_vma+0x99/0xc0
[  271.632801] EAX: d292e5d8 EBX: 44d65000 ECX: d292e5d8 EDX: d292e5d8
[  271.632803] ESI: 44d65000 EDI: d29dff70 EBP: d292e630 ESP: d29dfe70
[  271.632806]  DS: 0068 ES: 0068 FS: 0000 GS: 0033 SS: 0068
[  271.632809] Process firefox-bin (pid: 2731, ti=d29de000 task=d29619f0 task.ti=d29de000)
[  271.632811] Stack: d292e5d8 d292e630 d29dff70 0004cf5d 00000001 d29619f0 44d65000 00000001
[  271.632817]        d2960030 d8361140 00000000 00000000 44d65000 d29dff70 d292e630 0004e61d
[  271.632822]        44d66000 00100075 d2936840 00000000 00044d65 00000000 d2969700 00100073
[  271.632827] Call Trace:
[  271.632848]  [<0004cf5d>] vma_merge+0x5d/0x380
[  271.632918]  [<0004e61d>] mprotect_fixup+0xed/0x380
[  271.632928]  [<00100075>] vt_ioctl+0x1125/0x1980
[  271.632946]  [<00044d65>] isolate_lru_pages+0x55/0x1c0
[  271.632962]  [<00100073>] vt_ioctl+0x1123/0x1980
[  271.632989]  [<00023bb3>] ptrace_notify+0x63/0x70
[  271.633001]  [<00100075>] vt_ioctl+0x1125/0x1980
[  271.633016]  [<0004eade>] sys_mprotect+0x22e/0x410
[  271.633026]  [<00100075>] vt_ioctl+0x1125/0x1980
[  271.633226]  [<00003d26>] syscall_call+0x7/0xb
[  271.633409]  =======================
[  271.633411] Code: 39 c3 75 3e 8b 41 44 3b 42 44 74 06 0f 0b eb fe 66 90 8b 41 3c 3b 42 3c 75 f2 8b 42 14 31 c7 81 e7 dd df ef df 74 a5 0f 0b eb fe <0f> 0b eb fe 8d 76 00 0f 0b eb fe 0f 0b eb fe 0f 0b eb fe 0f 0b
[  271.633449] EIP: [<0004bf29>] pax_find_mirror_vma+0x99/0xc0 SS:ESP 0068:d29dfe70


The strace ends on:

Code: Select all

[pid  2731] mprotect(0x44d65000, 4096, PROT_READ|PROT_EXEC <unfinished ...>


MPROTECT is disabled on the binary. The complete strace is available at [url]removed by PaX Team[/url] and the .config at http://www.linkfanel.net/stuff/config-2.6.24.2-pax_prue.

[PaX Team]

MPROTECT is disabled on the binary. The complete strace is available at [url]removed by PaX Team[/url] and the .config at http://www.linkfanel.net/stuff/config-2.6.24.2-pax_prue.

thanks for the trace, i'm working on it. in the meantime i removed the url 'cos the trace leaks the path of your mozilla dir, next time better send such info in mail .

[PaX Team]

test27 should fix this, can you test it? (interdiff from test26 applies to grsec as well)

[linkfanel]

test27 should fix this, can you test it? (interdiff from test26 applies to grsec as well)

Yes it's fixed now. Thanks, good job!