Programms start dying after a while (< 1 day)

[kreutzm]

Hello,
I'm using current Debian testing (hopefully soon stable), except of course
the kernel itself.

I'm a longterm grsecurity user (> 10 years). As long as it was possible, I
was staying on the stable patches.

Now I have to use the unstable patches and run into stability problems. The
kernel boots and operates fine for a while, i.e. all programms seem to run
as expected. However, after some uptime, programms start to segfault. After
a reboot (or earlier) they ran fine. If I do not reboot in this situation
then the system quickly becomes unusuable.

A typical example is a kernel compile, which essentially only works if the
machine is freshly rebooted and not many other processes have already been
run. If this is not the case, cc invocations start dying:

Jan 4 13:58:05 samd kernel: [17330.465258] cc1[29858]: segfault at ffffffff99789119 ip 00002acb822dc496 sp 00007ffe32e975a8 error 5 in libc-2.24.so[2acb8225c000+195000]
Jan 4 13:58:05 samd kernel: [17330.465278] grsec: denied resource overstep by requesting 4096 for RLIMIT_CORE against limit 0 for /usr/lib/gcc/x86_64-linux-gnu/6/cc1[cc1:29858] uid/euid:1000/1000 gid/egid:1000/1000, parent /usr/bin/x86_64-linux-gnu-gcc-6[gcc:29857] uid/euid:1000/1000 gid/egid:1000/1000
Jan 4 13:58:05 samd kernel: [17330.922227] traps: dpkg-architectu[29889] general protection ip:5618fbcd6b75 sp:7ffe87f7a150 error:0 in perl[5618fbbfd000+1e5000]
Jan 4 13:58:05 samd kernel: [17330.922247] grsec: denied resource overstep by requesting 4096 for RLIMIT_CORE against limit 0 for /usr/bin/dpkg-architecture[dpkg-architectu:29889] uid/euid:1000/1000 gid/egid:1000/1000, parent /usr/bin/make[make:29876] uid/euid:1000/1000 gid/egid:1000/1000
Jan 4 13:58:05 samd kernel: [17330.964961] traps: dpkg-architectu[29891] general protection ip:55f9234bbca4 sp:7fff88f5a168 error:0 in perl[55f9233e2000+1e5000]
Jan 4 13:58:05 samd kernel: [17330.964980] grsec: denied resource overstep by requesting 4096 for RLIMIT_CORE against limit 0 for /usr/bin/dpkg-architecture[dpkg-architectu:29891] uid/euid:1000/1000 gid/egid:1000/1000, parent /usr/bin/make[make:29876] uid/euid:1000/1000 gid/egid:1000/1000
Jan 4 13:58:05 samd kernel: [17331.015486] traps: dpkg-architectu[29894] general protection ip:55842fd92b75 sp:7ffe2b643150 error:0 in perl[55842fcb9000+1e5000]
Jan 4 13:58:05 samd kernel: [17331.015505] grsec: denied resource overstep by requesting 4096 for RLIMIT_CORE against limit 0 for /usr/bin/dpkg-architecture[dpkg-architectu:29894] uid/euid:1000/1000 gid/egid:1000/1000, parent /bin/dash[sh:29893] uid/euid:1000/1000 gid/egid:1000/1000
Jan 4 13:58:05 samd kernel: [17331.081627] traps: dpkg-architectu[29899] general protection ip:5586240054a8 sp:7ffdfc2b45c0 error:0 in perl[558623f41000+1e5000]
Jan 4 13:58:05 samd kernel: [17331.081665] grsec: denied resource overstep by requesting 4096 for RLIMIT_CORE against limit 0 for /usr/bin/dpkg-architecture[dpkg-architectu:29899] uid/euid:1000/1000 gid/egid:1000/1000, parent /usr/bin/make[make:29876] uid/euid:1000/1000 gid/egid:1000/1000
Jan 4 13:58:06 samd kernel: [17331.461214] traps: as[29970] general protection ip:7f89bd05d256 sp:7ffc7c925088 error:0 in libc-2.24.so[7f89bcf34000+195000]
Jan 4 13:58:06 samd kernel: [17331.461240] grsec: denied resource overstep by requesting 4096 for RLIMIT_CORE against limit 0 for /usr/bin/x86_64-linux-gnu-as[as:29970] uid/euid:1000/1000 gid/egid:1000/1000, parent /usr/bin/x86_64-linux-gnu-gcc-6[gcc:29968] uid/euid:1000/1000 gid/egid:1000/1000
Jan 4 13:58:06 samd kernel: [17331.486494] traps: as[29975] general protection ip:55cf4f80bee0 sp:7ffe09fa0030 error:0 in x86_64-linux-gnu-as[55cf4f7f5000+5b000]
Jan 4 13:58:06 samd kernel: [17331.486526] grsec: denied resource overstep by requesting 4096 for RLIMIT_CORE against limit 0 for /usr/bin/x86_64-linux-gnu-as[as:29975] uid/euid:1000/1000 gid/egid:1000/1000, parent /usr/bin/x86_64-linux-gnu-gcc-6[gcc:29973] uid/euid:1000/1000 gid/egid:1000/1000
Jan 4 13:58:06 samd kernel: [17331.511748] traps: as[29980] general protection ip:7f07a1905fd8 sp:7fffdc2180a8 error:0 in libc-2.24.so[7f07a17dc000+195000]
Jan 4 13:58:06 samd kernel: [17331.535530] traps: as[29985] general protection ip:555bfdb9aee0 sp:7ffda72ac000 error:0 in x86_64-linux-gnu-as[555bfdb84000+5b000]
Jan 4 13:58:06 samd kernel: [17331.578992] traps: as[29996] general protection ip:7fda6fcd52dc sp:7ffe5f91d1d8 error:0 in libc-2.24.so[7fda6fbac000+195000]
Jan 4 13:58:10 samd kernel: [17335.516059] do_general_protection: 29 callbacks suppressed
Jan 4 13:58:10 samd kernel: [17335.516062] traps: as[30765] general protection ip:7f121e3ceaa3 sp:7fff64e220b0 error:0 in libbfd-2.27.51-system.20161220.so[7f121e384000+129000]
Jan 4 13:58:10 samd kernel: [17335.878357] traps: as[30849] general protection ip:7f4b0d8dd256 sp:7ffd21beb0a8 error:0 in libc-2.24.so[7f4b0d7b4000+195000]
Jan 4 13:58:11 samd kernel: [17336.149815] traps: as[30935] general protection ip:7fc61b2da929 sp:7ffe430efa40 error:0 in ld-2.24.so[7fc61b2cc000+23000]
Jan 4 13:58:11 samd kernel: [17336.239764] traps: as[30961] general protection ip:5594995f7ee0 sp:7ffd7026de40 error:0 in x86_64-linux-gnu-as[5594995e1000+5b000]
Jan 4 13:58:11 samd kernel: [17337.134012] cc1[31017]: segfault at 8 ip 000000000103c574 sp 00007ffd0ca64f10 error 4 in cc1[400000+1509000]
Jan 4 13:58:12 samd kernel: [17337.627289] traps: as[31146] general protection ip:7f925ff7d256 sp:7fff8c274358 error:0 in libc-2.24.so[7f925fe54000+195000]
Jan 4 13:58:12 samd kernel: [17337.882856] traps: as[31209] general protection ip:7f7feeda3549 sp:7ffc20d72050 error:0 in libbfd-2.27.51-system.20161220.so[7f7feed34000+129000]
Jan 4 13:58:12 samd kernel: [17337.895914] traps: as[31215] general protection ip:7ff8f2a55256 sp:7ffee7a18428 error:0 in libc-2.24.so[7ff8f292c000+195000]
Jan 4 13:58:12 samd kernel: [17337.910684] traps: as[31221] general protection ip:5642f986cee0 sp:7ffef32983c0 error:0 in x86_64-linux-gnu-as[5642f9856000+5b000]
Jan 4 13:58:12 samd kernel: [17337.939704] traps: as[31233] general protection ip:7fbd1ce6d256 sp:7ffe933ce428 error:0 in libc-2.24.so[7fbd1cd44000+195000]
Jan 4 13:58:15 samd kernel: [17340.610733] do_general_protection: 24 callbacks suppressed
Jan 4 13:58:15 samd kernel: [17340.610735] traps: ld[31811] general protection ip:7fe82b0e4a29 sp:7ffc39f0ce60 error:0 in libbfd-2.27.51-system.20161220.so[7fe82b064000+129000]
Jan 4 13:58:15 samd kernel: [17340.939259] traps: sh[31886] general protection ip:7f1f7969a929 sp:7ffd62c69ca0 error:0 in ld-2.24.so[7f1f7968c000+23000]
Jan 4 13:58:15 samd kernel: [17341.082649] traps: as[31918] general protection ip:7fd70916299d sp:7fff6ade5f90 error:0 in libbfd-2.27.51-system.20161220.so[7fd7090f4000+129000]
Jan 4 13:58:15 samd kernel: [17341.094925] traps: as[31924] general protection ip:56259b00aee0 sp:7ffe6e89d170 error:0 in x86_64-linux-gnu-as[56259aff4000+5b000]
Jan 4 13:58:15 samd kernel: [17341.112116] traps: as[31930] general protection ip:7fab2645d6d8 sp:7ffc5d1ed1d8 error:0 in libc-2.24.so[7fab26334000+195000]
Jan 4 13:58:16 samd kernel: [17341.316120] traps: as[32007] general protection ip:7f7fd982d256 sp:7ffc67d0ce98 error:0 in libc-2.24.so[7f7fd9704000+195000]
Jan 4 13:58:16 samd kernel: [17341.576127] traps: gcc[32064] general protection ip:7fa48a66911d sp:7ffd36635a90 error:0 in libc-2.24.so[7fa48a634000+195000]
Jan 4 13:58:16 samd kernel: [17341.625542] traps: as[32082] general protection ip:7f72b2c1d256 sp:7ffdf89e0df8 error:0 in libc-2.24.so[7f72b2af4000+195000]
Jan 4 13:58:16 samd kernel: [17341.659922] traps: as[32092] general protection ip:7fbfbfbcd256 sp:7ffca37d3de8 error:0 in libc-2.24.so[7fbfbfaa4000+195000]
Jan 4 13:58:16 samd kernel: [17341.678601] traps: as[32097] general protection ip:55ce11949ee0 sp:7fff0f75cd70 error:0 in x86_64-linux-gnu-as[55ce11933000+5b000]
Jan 4 13:58:17 samd kernel: [17342.380869] grsec: denied resource overstep by requesting 4096 for RLIMIT_CORE against limit 0 for /usr/bin/x86_64-linux-gnu-as[as:32273] uid/euid:1000/1000 gid/egid:1000/1000, parent /usr/bin/x86_64-linux-gnu-gcc-6[gcc:32272] uid/euid:1000/1000 gid/egid:1000/1000
Jan 4 13:58:17 samd kernel: [17342.392558] grsec: denied resource overstep by requesting 4096 for RLIMIT_CORE against limit 0 for /usr/bin/x86_64-linux-gnu-as[as:32279] uid/euid:1000/1000 gid/egid:1000/1000, parent /usr/bin/x86_64-linux-gnu-gcc-6[gcc:32278] uid/euid:1000/1000 gid/egid:1000/1000

What I do see in my logs all the time is:
grsec: denied resource overstep by requesting .. for TARGET against limit .* for
where TARGET is one of RLIMIT_CORE, RLIMIT_MEMLOCK, RLIMIT_NICE, RLIMIT_NOFILE,
RLIMIT_STACK.

It looks like very many programms report this, starting with Xorg down to rm (in
the conditions stated above).

However, most of the time programms seem to work fine (at least from a user POV).

(Also sometimes:
Jan 24 10:38:48 samd kernel: [ 3461.480043] grsec: denied ptrace of /usr/lib/firefox-esr/firefox-esr(firefox-esr:2740) by /usr/lib/firefox-esr/firefox-esr[firefox-esr:5255] uid/euid:1001/1001 gid/egid:1001/1001, parent /usr/lib/firefox-esr/firefox-esr[firefox-esr:2740] uid/euid:1001/1001 gid/egid:1001/1001)

I used (and experienced the above) so far under the following grsecurity patches:
grsecurity-3.1-4.5.5-201605211442.patch
grsecurity-3.1-4.8.15-201612151923.patch

My impression is that this happens much less under the later kernel/grsecurity patch,
however kernel compiles are still succeeding only seldomly.

I'm going to try grsecurity-3.1-4.8.17-201701151620.patch (once I get it build).

Is there a way to track this down (or even better, avoid it)?

Thanks & Greetings

Helge

[timbgo]

Could you pls. try and use "Code" button, to get:

Hello,
...

Code: Select all

Jan  4 13:58:05 samd kernel: [17330.465258] cc1[29858]: segfault at ffffffff99789119 ip 00002acb822dc496 sp 00007ffe32e975a8 error 5 in libc-2.24.so[2acb8225c000+195000]
Jan  4 13:58:05 samd kernel: [17330.465278] grsec: denied resource overstep by requesting 4096 for RLIMIT_CORE against limit 0 for /usr/lib/gcc/x86_64-linux-gnu/6/cc1[cc1:29858] uid/euid:1000/1000 gid/egid:1000/1000, parent /usr/bin/x86_64-linux-gnu-gcc-6[gcc:29857] uid/euid:1000/1000 gid/egid:1000/1000
Jan  4 13:58:05 samd kernel: [17330.922227] traps: dpkg-architectu[29889] general protection ip:5618fbcd6b75 sp:7ffe87f7a150 error:0 in perl[5618fbbfd000+1e5000]
Jan  4 13:58:05 samd kernel: [17330.922247] grsec: denied resource overstep by requesting 4096 for RLIMIT_CORE against limit 0 for /usr/bin/dpkg-architecture[dpkg-architectu:29889] uid/euid:1000/1000 gid/egid:1000/1000, parent /usr/bin/make[make:29876] uid/euid:1000/1000 gid/egid:1000/1000
...

What I do see in my logs all the time is:

Code: Select all

grsec: denied resource overstep by requesting .. for TARGET against limit .* for
where TARGET is one of RLIMIT_CORE, RLIMIT_MEMLOCK, RLIMIT_NICE, RLIMIT_NOFILE,
RLIMIT_STACK.


...

It's hard to view your post like it is now, without the tags that make the code look like... code.
(Not promising I can give a very useful reply, but maybe someone will, if the above is met.)
Regards!

[specs]

Hello,
I'm using current Debian testing (hopefully soon stable), except of course
the kernel itself.

I'm a longterm grsecurity user (> 10 years). As long as it was possible, I
was staying on the stable patches.

Just wondering:
Years ago you submitted a bug report to the debian maintainer. Does this imply that you apply the grsecurity patch together with the debian patches?

It could be the stable kernel is nearly unpatched while the testing kernel is patched to include some new and hot hardware support otherwise unavailable for Debian.
That might explain why my kernel runs without problems on debian. I always download the kernel separately from kernel.org.

Paxctld always applies the necessary pax settings to the binaries, but gcc and as work fine without for me (as I checked with paxctl-ng).

[kreutzm]

Thanks for your reply.

I always use vanilla kernels as basis. Yes, many, many years ago I sometimes combined patches, but I no longer do this. So just
vanilla kernel + grsecurity patch.

[specs]

I hope you don't take offence on the question asked.

Sorry if I can't provide much help.
The gcc-version I currently use is the version 6.3.0 20170124 for the 201702060653 grsec-patch (debian sid).
I did have grsecurity-3.1-4.8.15-201612151923.patch installed, but it has long disappeared from my logs.
I can't remember the problems you are facing.

I could look up the gcc-versions of the previous kernels if needed, if pax-team or spender requests it.

[kreutzm]

I'm now running 3.1-4.8.17-201701151620.patch with similar results, although it appears to be less severe. I tried building 3.1-4.9.8-201702071801.patch, however, it again dies:

Feb 12 09:15:36 samd kernel: [14425.510157] ld[7995]: segfault at 561dd7e3b808 ip 00002b295a21d4b9 sp 00007ffc83b8d680 error 4 in libbfd-2.27.90-system.20170124.so[2b295a193000+129000]

$ gcc --version
gcc (Debian 6.3.0-6) 6.3.0 20170205

[spender]

This is a very unusual case. Have you confirmed (by running the same tests over multiple days) that a vanilla kernel of the same version, compiled with the same GCC, with the same config doesn't produce the problem? If the vanilla kernel works, then you could try first applying the PaX patch alone and reuse the grsec configuration (with make oldconfig). If that fails, then do a binary search of disabling PaX features until the culprit is found.

-Brad

[kreutzm]

Hello Brad,
thanks your your help. I'll try compiling a vanilla kernel and see if I find the culprit. However, given that the machine is in use and I have only limited time (and compling kernels is a pain, as it takes some time and atm usually dies somewhere inbetween) it will take some time.

I'm not sure I understand with what you mean with "apply the PaX" patch alone? I always use the grsecurity patch.

Greetings

Helge

[spender]

The latest PaX patch is currently available at https://grsecurity.net/~paxguy1/pax-lin ... est4.patch
It should apply fine to a vanilla 4.9.11 kernel.

-Brad

[kreutzm]

I've successfully built a vanilla kernel on this machine and put it under load (kernel builds etc.) and it shows the same symptoms.

Thus not grsecurity related, sorry for the noise.

I'll continue my search for the root cause.

Thanks for all the help, this thread can be closed.