higher memory pressure related hardened kernel?

Discuss usability issues, general maintenance, and general support issues for a grsecurity-enabled system.

Moderators: spender, PaX Team

higher memory pressure related hardened kernel?

Postby brainatwork » Wed Feb 15, 2017 10:44 am

Hi all

I'm wondering if any options in the PAX/GRSECURITY section are known to create a (much) higher memory pressure on the system.

Since some time (starting with kernel version 4.6.5) i'm experiencing high memory usage (up to frequent OOM) on my systems. I've taken some time to investigate this but i'm now stuck.
As an example i'm attaching the memory footprint of a example system (internal mailserver, domU on kvm) with some kernel versions. All specs are gathered right after a fresh reboot...
Script from: https://raw.githubusercontent.com/pixel ... /ps_mem.py

# zcat /proc/config.gz | grep "PAX\|GRKERNSEC"
CONFIG_PAX_PER_CPU_PGD=y
CONFIG_GRKERNSEC=y
CONFIG_GRKERNSEC_CONFIG_AUTO=y
# CONFIG_GRKERNSEC_CONFIG_CUSTOM is not set
CONFIG_GRKERNSEC_CONFIG_SERVER=y
# CONFIG_GRKERNSEC_CONFIG_DESKTOP is not set
# CONFIG_GRKERNSEC_CONFIG_VIRT_NONE is not set
CONFIG_GRKERNSEC_CONFIG_VIRT_GUEST=y
# CONFIG_GRKERNSEC_CONFIG_VIRT_HOST is not set
CONFIG_GRKERNSEC_CONFIG_VIRT_EPT=y
# CONFIG_GRKERNSEC_CONFIG_VIRT_SOFT is not set
# CONFIG_GRKERNSEC_CONFIG_VIRT_XEN is not set
# CONFIG_GRKERNSEC_CONFIG_VIRT_VMWARE is not set
CONFIG_GRKERNSEC_CONFIG_VIRT_KVM=y
# CONFIG_GRKERNSEC_CONFIG_VIRT_VIRTUALBOX is not set
# CONFIG_GRKERNSEC_CONFIG_VIRT_HYPERV is not set
CONFIG_GRKERNSEC_CONFIG_PRIORITY_PERF=y
# CONFIG_GRKERNSEC_CONFIG_PRIORITY_SECURITY is not set
CONFIG_GRKERNSEC_PROC_GID=10
CONFIG_GRKERNSEC_SYMLINKOWN_GID=100
CONFIG_PAX=y
# CONFIG_PAX_SOFTMODE is not set
# CONFIG_PAX_PT_PAX_FLAGS is not set
CONFIG_PAX_XATTR_PAX_FLAGS=y
# CONFIG_PAX_NO_ACL_FLAGS is not set
CONFIG_PAX_HAVE_ACL_FLAGS=y
# CONFIG_PAX_HOOK_ACL_FLAGS is not set
CONFIG_PAX_NOEXEC=y
CONFIG_PAX_PAGEEXEC=y
CONFIG_PAX_EMUTRAMP=y
CONFIG_PAX_MPROTECT=y
# CONFIG_PAX_MPROTECT_COMPAT is not set
# CONFIG_PAX_ELFRELOCS is not set
CONFIG_PAX_KERNEXEC=y
CONFIG_PAX_ASLR=y
CONFIG_PAX_RANDKSTACK=y
CONFIG_PAX_RANDUSTACK=y
CONFIG_PAX_RANDMMAP=y
# CONFIG_PAX_MEMORY_SANITIZE is not set
CONFIG_PAX_MEMORY_UDEREF=y
CONFIG_PAX_REFCOUNT=y
CONFIG_PAX_USERCOPY=y
# CONFIG_PAX_USERCOPY_DEBUG is not set
CONFIG_GRKERNSEC_KMEM=y
CONFIG_GRKERNSEC_IO=y
CONFIG_GRKERNSEC_BPF_HARDEN=y
CONFIG_GRKERNSEC_PERF_HARDEN=y
CONFIG_GRKERNSEC_RAND_THREADSTACK=y
CONFIG_GRKERNSEC_PROC_MEMMAP=y
CONFIG_GRKERNSEC_KSTACKOVERFLOW=y
CONFIG_GRKERNSEC_BRUTE=y
CONFIG_GRKERNSEC_HIDESYM=y
# CONFIG_GRKERNSEC_KERN_LOCKOUT is not set
CONFIG_GRKERNSEC_NO_RBAC=y
CONFIG_GRKERNSEC_ACL_HIDEKERN=y
CONFIG_GRKERNSEC_ACL_MAXTRIES=3
CONFIG_GRKERNSEC_ACL_TIMEOUT=30
CONFIG_GRKERNSEC_PROC=y
CONFIG_GRKERNSEC_PROC_USER=y
CONFIG_GRKERNSEC_PROC_ADD=y
CONFIG_GRKERNSEC_LINK=y
# CONFIG_GRKERNSEC_SYMLINKOWN is not set
CONFIG_GRKERNSEC_FIFO=y
CONFIG_GRKERNSEC_SYSFS_RESTRICT=y
# CONFIG_GRKERNSEC_ROFS is not set
CONFIG_GRKERNSEC_DEVICE_SIDECHANNEL=y
CONFIG_GRKERNSEC_CHROOT=y
CONFIG_GRKERNSEC_CHROOT_MOUNT=y
CONFIG_GRKERNSEC_CHROOT_DOUBLE=y
CONFIG_GRKERNSEC_CHROOT_PIVOT=y
CONFIG_GRKERNSEC_CHROOT_CHDIR=y
CONFIG_GRKERNSEC_CHROOT_CHMOD=y
CONFIG_GRKERNSEC_CHROOT_FCHDIR=y
CONFIG_GRKERNSEC_CHROOT_MKNOD=y
CONFIG_GRKERNSEC_CHROOT_SHMAT=y
CONFIG_GRKERNSEC_CHROOT_UNIX=y
CONFIG_GRKERNSEC_CHROOT_FINDTASK=y
CONFIG_GRKERNSEC_CHROOT_NICE=y
CONFIG_GRKERNSEC_CHROOT_SYSCTL=y
CONFIG_GRKERNSEC_CHROOT_RENAME=y
CONFIG_GRKERNSEC_CHROOT_CAPS=y
CONFIG_GRKERNSEC_CHROOT_INITRD=y
# CONFIG_GRKERNSEC_AUDIT_GROUP is not set
# CONFIG_GRKERNSEC_EXECLOG is not set
CONFIG_GRKERNSEC_RESLOG=y
CONFIG_GRKERNSEC_CHROOT_EXECLOG=y
CONFIG_GRKERNSEC_AUDIT_PTRACE=y
# CONFIG_GRKERNSEC_AUDIT_CHDIR is not set
CONFIG_GRKERNSEC_AUDIT_MOUNT=y
CONFIG_GRKERNSEC_SIGNAL=y
CONFIG_GRKERNSEC_FORKFAIL=y
CONFIG_GRKERNSEC_TIME=y
CONFIG_GRKERNSEC_PROC_IPADDR=y
CONFIG_GRKERNSEC_RWXMAP_LOG=y
CONFIG_GRKERNSEC_DMESG=y
CONFIG_GRKERNSEC_HARDEN_PTRACE=y
CONFIG_GRKERNSEC_PTRACE_READEXEC=y
CONFIG_GRKERNSEC_SETXID=y
CONFIG_GRKERNSEC_HARDEN_IPC=y
CONFIG_GRKERNSEC_HARDEN_TTY=y
# CONFIG_GRKERNSEC_TPE is not set
CONFIG_GRKERNSEC_BLACKHOLE=y
CONFIG_GRKERNSEC_NO_SIMULT_CONNECT=y
# CONFIG_GRKERNSEC_SOCKET is not set
CONFIG_GRKERNSEC_SYSCTL=y
# CONFIG_GRKERNSEC_SYSCTL_DISTRO is not set
CONFIG_GRKERNSEC_SYSCTL_ON=y
# CONFIG_GRKERNSEC_SELINUX_AVC_LOG_IPADDR is not set
CONFIG_GRKERNSEC_FLOODTIME=10
CONFIG_GRKERNSEC_FLOODBURST=6
-----------------------------------------------------------------

Code: Select all
Stock debian: #1 SMP Debian 3.16.39-1 (2016-12-30)
-----------------------------------------------------
 Private  +   Shared  =  RAM used   Program

 80.0 KiB +  34.5 KiB = 114.5 KiB   init
 64.0 KiB +  62.5 KiB = 126.5 KiB   freshclam
112.0 KiB +  41.5 KiB = 153.5 KiB   udevd
152.0 KiB +  12.5 KiB = 164.5 KiB   tail
144.0 KiB +  64.0 KiB = 208.0 KiB   fcron
116.0 KiB + 156.5 KiB = 272.5 KiB   ntpd (2)
284.0 KiB +  40.5 KiB = 324.5 KiB   getty (2)
316.0 KiB + 236.0 KiB = 552.0 KiB   log
364.0 KiB + 255.5 KiB = 619.5 KiB   dovecot
620.0 KiB +  87.5 KiB = 707.5 KiB   master
576.0 KiB + 210.5 KiB = 786.5 KiB   pickup
704.0 KiB + 214.5 KiB = 918.5 KiB   qmgr
724.0 KiB + 267.5 KiB = 991.5 KiB   config
748.0 KiB + 266.5 KiB =   1.0 MiB   tlsmgr
736.0 KiB + 396.0 KiB =   1.1 MiB   su
  1.1 MiB +  93.5 KiB =   1.2 MiB   rsyslogd
  1.1 MiB + 771.5 KiB =   1.8 MiB   anvil (2)
  1.2 MiB +   2.7 MiB =   3.8 MiB   sshd (3)
  5.7 MiB +   1.6 MiB =   7.2 MiB   bash (2)
 13.6 MiB + 279.5 KiB =  13.8 MiB   tenshi
  1.8 MiB +  25.6 MiB =  27.4 MiB   /usr/sbin/spamd
 23.0 MiB +  12.3 MiB =  35.3 MiB   /usr/sbin/amavi (3)
  3.6 MiB +  51.2 MiB =  54.8 MiB   spamd child (2)
473.9 MiB + 332.5 KiB = 474.3 MiB   clamd
---------------------------------
                        627.6 MiB
=================================

Gentoo Hardened 4.6.5
-----------------------------------------------------
 Private  +   Shared  =  RAM used   Program
 72.0 KiB + 640.0 KiB = 712.0 KiB   tail
224.0 KiB +   1.4 MiB =   1.7 MiB   fcron
288.0 KiB +   1.8 MiB =   2.1 MiB   log
820.0 KiB +   1.7 MiB =   2.5 MiB   ntpd (3)
288.0 KiB +   2.2 MiB =   2.5 MiB   dovecot
952.0 KiB +   1.9 MiB =   2.8 MiB   agetty (7)
540.0 KiB +   2.4 MiB =   2.9 MiB   systemd-udevd
412.0 KiB +   2.6 MiB =   3.0 MiB   su
864.0 KiB +   2.2 MiB =   3.1 MiB   config
552.0 KiB +   3.1 MiB =   3.6 MiB   proxymap
980.0 KiB +   2.7 MiB =   3.7 MiB   rsyslogd
624.0 KiB +   3.1 MiB =   3.8 MiB   qmgr
572.0 KiB +   3.2 MiB =   3.8 MiB   master
696.0 KiB +   3.2 MiB =   3.8 MiB   anvil (2)
588.0 KiB +   3.3 MiB =   3.8 MiB   pickup
576.0 KiB +   3.3 MiB =   3.9 MiB   trivial-rewrite
588.0 KiB +   3.3 MiB =   3.9 MiB   local
648.0 KiB +   3.6 MiB =   4.2 MiB   tlsmgr
  1.5 MiB +   3.3 MiB =   4.8 MiB   cleanup
  1.9 MiB +   2.9 MiB =   4.8 MiB   systemd
760.0 KiB +   4.5 MiB =   5.2 MiB   smtp
  2.3 MiB +   3.3 MiB =   5.6 MiB   systemd-journald
  2.0 MiB +   4.5 MiB =   6.5 MiB   sshd (3)
  1.8 MiB +   4.8 MiB =   6.5 MiB   smtpd (2)
  4.3 MiB +   3.2 MiB =   7.4 MiB   bash (2)
  3.1 MiB +  12.7 MiB =  15.7 MiB   freshclam
 13.5 MiB +   3.7 MiB =  17.2 MiB   tenshi
 72.5 MiB +   5.7 MiB =  78.2 MiB   /usr/sbin/spamd
141.5 MiB +   5.5 MiB = 147.0 MiB   /usr/sbin/amavi (3)
145.1 MiB +   2.8 MiB = 147.8 MiB   spamd child (2)
463.8 MiB +  22.9 MiB = 486.7 MiB   clamd
Warning: Shared memory is slightly over-estimated by this system
for each program, so totals are not reported.


Hardened: 4.8.17
-----------------------------------------------------
Private  +   Shared  =  RAM used   Program
 72.0 KiB + 644.0 KiB = 716.0 KiB   tail
224.0 KiB +   1.6 MiB =   1.8 MiB   fcron
144.0 KiB +   1.8 MiB =   1.9 MiB   anvil
288.0 KiB +   1.9 MiB =   2.2 MiB   log
284.0 KiB +   2.3 MiB =   2.6 MiB   dovecot
820.0 KiB +   1.8 MiB =   2.6 MiB   ntpd (3)
540.0 KiB +   2.3 MiB =   2.9 MiB   systemd-udevd
952.0 KiB +   2.0 MiB =   2.9 MiB   agetty (7)
412.0 KiB +   2.6 MiB =   3.0 MiB   su
864.0 KiB +   2.3 MiB =   3.1 MiB   config
308.0 KiB +   3.0 MiB =   3.3 MiB   systemd-journald
552.0 KiB +   3.1 MiB =   3.6 MiB   scache
576.0 KiB +   3.1 MiB =   3.7 MiB   trivial-rewrite
980.0 KiB +   2.7 MiB =   3.7 MiB   rsyslogd
552.0 KiB +   3.2 MiB =   3.8 MiB   proxymap
572.0 KiB +   3.2 MiB =   3.8 MiB   master
588.0 KiB +   3.2 MiB =   3.8 MiB   pickup
672.0 KiB +   3.2 MiB =   3.9 MiB   qmgr
796.0 KiB +   3.4 MiB =   4.1 MiB   cleanup
608.0 KiB +   3.6 MiB =   4.2 MiB   tlsmgr
  1.9 MiB +   2.9 MiB =   4.7 MiB   systemd
  1.8 MiB +   3.3 MiB =   5.1 MiB   local (3)
  1.5 MiB +   4.6 MiB =   6.1 MiB   smtp (2)
  2.0 MiB +   4.4 MiB =   6.5 MiB   sshd (3)
  2.5 MiB +   4.7 MiB =   7.2 MiB   smtpd (3)
  4.3 MiB +   3.2 MiB =   7.5 MiB   bash (2)
  2.8 MiB +  11.8 MiB =  14.7 MiB   freshclam
 13.5 MiB +   3.6 MiB =  17.1 MiB   tenshi
 72.6 MiB +   5.8 MiB =  78.4 MiB   /usr/sbin/spamd
141.7 MiB +   5.5 MiB = 147.2 MiB   /usr/sbin/amavi (3)
145.1 MiB +   2.7 MiB = 147.8 MiB   spamd child (2)
463.6 MiB +  23.0 MiB = 486.6 MiB   clamd
Warning: Shared memory is slightly over-estimated by this system
for each program, so totals are not reported.


Gentoo (non hardened): 4.9.6-gentoo-r1
-----------------------------------------------------
 Private  +   Shared  =  RAM used   Program
164.0 KiB +  96.5 KiB = 260.5 KiB   tail
296.0 KiB +  73.0 KiB = 369.0 KiB   fcron
316.0 KiB + 180.5 KiB = 496.5 KiB   log
360.0 KiB + 205.0 KiB = 565.0 KiB   dovecot
572.0 KiB + 186.0 KiB = 758.0 KiB   proxymap
552.0 KiB + 219.5 KiB = 771.5 KiB   su
596.0 KiB + 192.0 KiB = 788.0 KiB   pickup
612.0 KiB + 192.0 KiB = 804.0 KiB   master
628.0 KiB + 204.5 KiB = 832.5 KiB   trivial-rewrite
680.0 KiB + 192.0 KiB = 872.0 KiB   qmgr
580.0 KiB + 306.0 KiB = 886.0 KiB   ntpd (3)
652.0 KiB + 239.0 KiB = 891.0 KiB   tlsmgr
932.0 KiB + 164.5 KiB =   1.1 MiB   systemd-udevd
740.0 KiB + 384.5 KiB =   1.1 MiB   anvil (2)
  1.0 MiB + 209.0 KiB =   1.2 MiB   config
  1.0 MiB + 642.5 KiB =   1.6 MiB   agetty (7)
  1.6 MiB + 204.5 KiB =   1.8 MiB   systemd-journald
  2.0 MiB + 141.5 KiB =   2.2 MiB   rsyslogd
  2.9 MiB + 174.5 KiB =   3.0 MiB   systemd
  1.3 MiB +   2.3 MiB =   3.6 MiB   sshd (3)
  3.4 MiB +   1.8 MiB =   5.2 MiB   smtpd (4)
  4.6 MiB +   1.3 MiB =   5.9 MiB   bash (2)
  3.3 MiB +   4.3 MiB =   7.6 MiB   freshclam
 13.5 MiB + 440.0 KiB =  14.0 MiB   tenshi
  2.3 MiB +  24.2 MiB =  26.6 MiB   /usr/sbin/spamd
  3.8 MiB +  47.9 MiB =  51.7 MiB   spamd child (2)
 29.0 MiB +  40.5 MiB =  69.5 MiB   /usr/sbin/amavi (3)
474.7 MiB +   4.4 MiB = 479.0 MiB   clamd
---------------------------------
                        683.1 MiB
=================================
brainatwork
 
Posts: 23
Joined: Wed Aug 13, 2008 12:53 pm

Re: higher memory pressure related hardened kernel?

Postby PaX Team » Thu Feb 16, 2017 1:54 pm

nothing should cause extreme memory usage increase, there's at most some variability worth a few pages due to ASLR. can you compare the smaps files of a few processes to see which parts of the address space are so different? also please try 4.9 as 4.6 is too old for us to debug/support now.
PaX Team
 
Posts: 2310
Joined: Mon Mar 18, 2002 4:35 pm

Re: higher memory pressure related hardened kernel?

Postby brainatwork » Tue Feb 21, 2017 10:53 am

When trying to get /proc/<pid>/smaps into a GRSEC kernel i noticed this in mm/Kconfig

----
config HWPOISON_INJECT
tristate "HWPoison pages injector"
depends on MEMORY_FAILURE && DEBUG_KERNEL && PROC_FS && !GRKERNSEC
select PROC_PAGE_MONITOR
depends on !GRKERNSEC
----
This is 4.9.11-hardened sources from Gentoo.
On 4.9.6-gentoo sources PROC_PAGE_MONITOR gives me smaps files.

Unfortunately this blocks me from delivering more detailed memory informations.
Any help is appreciated.
brainatwork
 
Posts: 23
Joined: Wed Aug 13, 2008 12:53 pm


Return to grsecurity support

cron