grsec: halting the system due to suspicious kernel crash

[spender]

Can you change your second printk so that it makes sure each dereference inside dentry isn't on NULL? Like:

Code: Select all

printk(KERN_ALERT "do_handle_create dentry->d_name.name=%.16s dentry->d_inode = %p, dentry->d_inode->i_sb = %p\n",
          dentry->d_name.name, dentry->d_inode, dentry->d_inode ? dentry->d_inode->i_sb : NULL);


It seems both d_inode and d_sb must be null here.

BTW, are you able to reproduce this on our 3.14.23 or 3.17.2 kernels?

-Brad

[Orfheo]

I understand what you asking for, not sure I may be able to accomplish it today Brad.

For the kernels, this is the list of what I've available on my hardened gentoo

Code: Select all

* sys-kernel/hardened-sources
     Available versions:
     (3.2.61-r2) 3.2.61-r2^bs
     (3.2.61-r5) 3.2.61-r5^bs
     (3.2.62-r1) 3.2.62-r1^bs
     (3.2.63-r1) ~3.2.63-r1^bs
     (3.2.63-r2) ~3.2.63-r2^bs
     (3.2.63-r3) ~3.2.63-r3^bs
     (3.2.63-r4) ~3.2.63-r4^bs
     (3.2.63-r6) ~3.2.63-r6^bs
     (3.14.12-r2) 3.14.12-r2^bs
     (3.14.15) 3.14.15^bs
     (3.14.17-r1) 3.14.17-r1^bs
     (3.14.19) ~3.14.19^bs
     (3.14.19-r1) ~3.14.19-r1^bs
     (3.14.20-r1) ~3.14.20-r1^bs
     (3.14.21) ~3.14.21^bs
     (3.14.22) ~3.14.22^bs
     (3.15.5-r2) 3.15.5-r2^bs
     (3.15.8) 3.15.8^bs
     (3.15.10-r1) 3.15.10-r1^bs
     (3.16.3) ~3.16.3^bs
     (3.16.3-r1) ~3.16.3-r1^bs
     (3.16.4-r1) ~3.16.4-r1^bs
     (3.16.5) ~3.16.5^bs
     (3.17.1) ~3.17.1^bs


Make your bet.

Orfheo.

[spender]

The 3.14.22 and 3.17.1 should be fine to test.

-Brad

[Orfheo]

Guess you would like to know.

I had kernel 3.14.5-hardened-r2 already installed (I always keep
couple of old kernels around, just in case)

Code: Select all

Linux alecto 3.14.5-hardened-r2 #1 SMP PREEMPT Sat Jun 28 20:01:06 CEST 2014 i686 Intel(R) Xeon(TM) CPU 2.80GHz GenuineIntel GNU/Linux


Found the time to reboot my firewall and running my simple
testcase.

No panic, no weird messages, the directory /sys/fs/cgroup/openrc/acct get
created without a glitch.

Code: Select all

alecto ~ # ls -l /sys/fs/cgroup/openrc/acct
total 0
-rw-r--r-- 1 root root 0 Nov  2 19:48 cgroup.clone_children
-rw-r--r-- 1 root root 0 Nov  2 19:48 cgroup.procs
-rw-r--r-- 1 root root 0 Nov  2 19:48 notify_on_release
-rw-r--r-- 1 root root 0 Nov  2 19:48 tasks


May I make an educated guess? The "control files" inside the directory
get automatically created by the kernel. May have them something to
do with the panic?

Did you add some news grsecurity kernel configuration defines
that have something to do with cgroups filesystem?

I usually "copy" kernel configuration from nearby versions: I may have missed
a new key

Going to test the testcase against 3.17.1. I will report as soon as possible.

Orfheo.

[Orfheo]

Tested 3.17.1, same crash.

Here a screen dump of the panic trace



It looks quite the same to me.

Orfheo.

[Orfheo]

You were completely right

dentry->d_inode = (nil), dentry->d_inode->i_sb = (nil)

both null, using just the one line of code you sent.

Here is the complete Oops output

Code: Select all

[  196.291856] do_handle_create dentry->d_name.name=acct dentry->d_inode =    (nil), dentry->d_inode->i_sb =    (nil)
[  196.293290] BUG: unable to handle kernel NULL pointer dereference at 00000028
[  196.293388] IP: [<c1593a5d>] c1593a5d
[  196.293534] *pdpt = 0000000033727001 *pde = 0000000000000000
[  196.293703] Oops: 0000 [#1] PREEMPT SMP
[  196.294089] Modules linked in: vmw_vsock_vmci_transport vsock vmw_balloon vmw_vmci
[  196.294370] CPU: 1 PID: 5275 Comm: mkdir Not tainted 3.15.10-hardened-r1 #2
[  196.294399] Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 04/15/2011
[  196.294508] task: f65e5940 ti: f65e5da8 task.ti: f65e5da8
[  196.294622] EIP: 0060:[<c1593a5d>] EFLAGS: 00010246 CPU: 1
[  196.294645] EAX: 00000000 EBX: f514f780 ECX: 00000003 EDX: 80000003
[  196.294668] ESI: f3722820 EDI: 00000000 EBP: f35c7e68 ESP: f35c7e4c
[  196.294692]  DS: 0068 ES: 0068 FS: 00d8 GS: 0033 SS: 0068
[  196.294725] CR0: 80050033 CR2: 00000028 CR3: 33726000 CR4: 000006f0
[  196.294835] Stack:
[  196.294952]  c1722d78 f514f7a4 00000000 00000000 f3722820 f514f780 00000000 f35c7e78
[  196.294988]  c11be109 f514f780 00000002 f35c7ea0 c10ed7fb ffffff9c 59388449 01ed6800
[  196.295012]  f5678510 f6324680 59388449 59388433 593881b4 f35c7eb4 c10ed83d ffffff9c
[  196.295048] Call Trace:
[  196.296185]  [<c11be109>] gr_handle_create+0x66/0x90
[  196.296251]  [<c10ed7fb>] SyS_mkdirat+0x8f/0xbf
[  196.296268]  [<c10ed83d>] SyS_mkdir+0x12/0x14
[  196.296289]  [<c1599e98>] syscall_call+0x7/0x7
[  196.296313]  [<c10ed1fa>] ? user_path_at_empty+0x65/0x82
[  196.296329]  [<c10551e8>] ? get_parent_ip+0xb/0x31
[  196.296338]  [<c159c857>] ? preempt_count_add+0x6a/0x7c
[  196.296342]  [<c10551e8>] ? get_parent_ip+0xb/0x31
[  196.296346]  [<c159c857>] ? preempt_count_add+0x6a/0x7c
[  196.296350]  [<c15998a6>] ? _raw_spin_unlock+0x10/0x21
[  196.296359]  [<c10f7f78>] ? mntput_no_expire+0x25/0xec
[  196.296364]  [<c10f805d>] ? mntput+0x1e/0x20
[  196.296372]  [<c10e8b2f>] ? path_put+0x15/0x18
[  196.296384]  [<c11d5ff9>] ? debug_smp_processor_id+0x12/0x15
[  196.296396]  [<c1008af3>] ? pax_randomize_kstack+0x2d/0x6e
[  196.296401]  [<c1599eb3>] ? restore_all_pax+0x7/0x7
[  196.296791] Code: c3 55 89 e5 57 56 53 89 c6 89 d3 8b 52 20 85 d2 74 05 8b 42 1c eb 02 31 c0 50 52 ff 73 1c 68 78 2d 72 c1 e8 77 da ff ff 8b 43 20 <8b> 78 28 89 d8 e8 6d 89 c2 ff 89 c1 89 fa 89 f0 e8 ca fd ff ff
[  196.296979] EIP: [<c1593a5d>] do_handle_create.isra.10+0x2a/0x4a SS:ESP 0068:f35c7e4c
[  196.297020] CR2: 0000000000000028
[  196.297393] ---[ end trace 7027911e9a3bb7eb ]---
[  196.297567] note: mkdir[5275] exited with preempt_count 2


And note

dentry->d_name.name=acct

pointing to the correct directory name "acct", from my test mkdir command.

Anthing else I can do?

Orfheo.

[spender]

Can you install inotifywatch and run:
inotifywatch -r /sys/fs/cgroup/openrc

and then perform the mkdir with RBAC disabled?

If this is an upstream bug in the kernfs code (which cgroup recently ported to, and for which mkdir support is a recent addition), the notify code should trigger the same oops.

-Brad

[Orfheo]

Nope.

Installed sys-fs/inotify-tools-3.14 and run

Code: Select all

alecto account # inotifywatch -r /sys/fs/cgroup/openrc
Establishing watches...
Finished establishing watches, now collecting statistics.


with RBAC disabled, as you asked, under my current 3.15.10-hardened-r1
kernel.

My simple test command "mkdir -p /sys/fs/cgroup/openrc/acct" got
executed without any panic, oops or weird dmesg message

The directory was created with its control files

Code: Select all

ls -l /sys/fs/cgroup/openrc/acct/
total 0
-rw-r--r-- 1 root root 0 Nov  3 17:16 cgroup.clone_children
-rw-r--r-- 1 root root 0 Nov  3 17:16 cgroup.procs
-rw-r--r-- 1 root root 0 Nov  3 17:16 notify_on_release
-rw-r--r-- 1 root root 0 Nov  3 17:16 tasks


Orfheo.

P.S. EDIT.
Sorry forgot to add the output of inotifywatch after stopping

Code: Select all

Establishing watches...
Finished establishing watches, now collecting statistics.
^Ctotal  close_nowrite  open  create  filename
9      4              4     1       /sys/fs/cgroup/openrc/
2      1              1     0       /sys/fs/cgroup/openrc/acct/


It looks it worked and got the events, isn't?

[spender]

Hi Orfheo,

It looks like kernfs doesn't initialize the dentry fully on an mkdir, unlike all other filesystems. It seems to defer this initialization to a subsequent lookup. I've pinged upstream to see if this is intended -- if it is then I'll have to disable fine-grained access control checks on kernfs as it doesn't act like a normal filesystem.

-Brad

[Orfheo]

Not being a kernel developer, not sure I fully understand the issue Brad,
but I think I picture what you mean.

Let me know how this problem evolve: I solved it going
around the bug. My virtual firewall is simple enough to allow such
a solution: just routing packets at the kernel level and rotating
some logs.

But I would like to be sure I will not miss the version where
is definitely closed in a clean way.

Hope in the meantime I was of some help and I didn't
waste your time.

Orfheo.

[spender]

Hi Orfheo,

Not a waste of time at all, thanks again for your help! I'll update this thread when we have a resolution.

-Brad

[Orfheo]

Had some thoughts about what you wrote Brad.

If I understand what the kernel is doing, the directory creation is not considered complete until the control files are in place.
This would make sense for a filesystem specialized in process and task control.

You probably right, the kernel doesn't update the directory entry until it consider the directory complete.

From the other side the kernel should have some way, should call a "user exit", using an old Big Blue definition, when the job is finished: inotifywatch works correctly, as we verified.

I guess grsecurity doesn't have any reason to look "inside" such a directory, the kernel knows what is doing and grsecurity has no reasons or right to interfere with such a process bounded inside the kernel domain.

I picture in the future other specialized control filesystem will probably have some role in the linux kernel life.

Wouldn't be a good idea to change grsecurity to handle in the correct way this schema?

Just curiosity Brad, if it sounds confused or if you don't have time, throw my question in your trash ;-)

Orfheo.

[timbgo]

Had some thoughts about what you wrote Brad.
...[snip]...
Just curiosity Brad, if it sounds confused or if you don't have time, throw my question in your trash ;-)

Orfheo.

It's that these people, spender and PaX Team, are too busy. They are respectful, but developing is... oh, you know it, you found what the cause was with your issue...
I envy you, but in a nice way, not the evil way, for your programming skills.
It was nice reading your report.

I'll be adding a not more, next, on the syslog-ng saga, the latest edisode that I posted previous to Orfheo's report, because that saga seems to have a little more clearer indications now, as to its causes.

[timbgo]

Possibly related to this issue (say maybe it's tripping something under some of the code that grsec is to patch onto the kernel, and making grsec look bad (just guessing, I'm not a dev, but I have seen bad games against grsec before)...

[Possibly related to this issue], esp. my not being able to find anything whatsoever in the logs on the issues that I had, and which made me a little angry, see here:

(same topic)
viewtopic.php?f=3&t=3709&start=15#p14457

is what I described I still have in my system despite the very careful air-gapped near-reinstall:

Syslog-ng from Delay in Logging to Broken Pipe and no Loggin
https://forums.gentoo.org/viewtopic-t-1001994.html

EDIT START 2014-10-25 I was thinking, and hope it's not too late to say it here: what could that be? The September version of syslog-ng made huge delays in logs, and even lost logging for hours, and made Grsecurity possibly misbehave...

Did that syslog-ng version 3.5.6 have issues in SELinux hardened kernel? In non-hardened kernel?

If I were able to dedicate serious time to the issue, I'd search for bugs with the syslog-ng maintainer, but no time. Not that capable... So just pointing to the issue...
EDIT END

Let's see if anything clarifies on the issue...

That is the complete post from the Page 2 of this topic halting the system due to suspicious kernel crash.

I am sick, and in fever, have to post quickly, will try and remember and trim it when I'm well.

But pls. see also previous to there what the issue was.

And this may be important:

in December, with the new version of syslog-ng, same, or similar, and worse, problems:

Syslog-ng from Delay Logging to BrokenPipe/no Logging
http://forums.gentoo.org/viewtopic-t-10 ... ml#7667574

Just like I said there:

My suspicion is: remove Grsecurity and all will be well!

This time, differently than the last time, I did get to save some logs and strange spurious errors of some kind (search for

Code: Select all

Configuration error. Please fix your configfile

Maybe some of you experts could discover what the matter here really is.

I don't konw how soon I'll be well again. So apologies upfront if I am unable to reply quickly here.

Thank you!

Miroslav Rovis
Zagreb, Croatia
http://www.CroatiaFidelis.hr

[spender]

Hi,

I will be fixing this myself it seems, as I reported to upstream that unlike every other filesystem, kernfs doesn't instantiate inodes on mkdir, but they haven't yet done anything with my bug report. I'll let you know when I have a fix.

Thanks,
-Brad