Grsecurity breaks systemd-logind

[osteichthyes]

Somewhere in the neighborhood of the 4.8.12 patchet, I stopped being able to boot my arch linux grsecurity hardened kernel. It would get about half-way through, then just have an inconsistently flashing cursor. I've tried soft-moding PAX, tons of different compilation flags, nothing seemed to fix it. I also was unable to ever get good logs. I've finally gotten some command line dumping to get some info during boot. It appears that systemd-logind creates a tmpfs for the root user (120), mounts it, then chdirs into it. Immediately after, it unmounts the tmpfs, and chdirs back to the previous location. Systemd-logind then repeats this. It repeats thousands of times. It seems similar, but slightly different from an older bug with SELINUX: https://bugzilla.redhat.com/show_bug.cgi?id=1075835

I only have this particular issue when using gdm, when using lightdm, the system gets to the display manager, then hangs after logging in, I guess it calls logind after authenticating on the dm?

My build is a fork of the one built for Arch Linux, you can find the pkgbuild and config here: https://github.com/osteichthyes/linux-grsec

I'm in the process of testing and will post more when I have some more information.

[osteichthyes]

Since posting, I've built about 20-30 different versions of the kernel, with grsecurity turned off, turned on, pax on, pax off, numerous kernel features, etc. I've figured out that the issue is NOT grsecurity or pax, it's the tracing support in the kernel. The reason grsecurity plays into it is that grsecurity kmem disables the debug_fs and tracing_support. Tracing_support is selected by pci and stacktrace_support. I think that tracing_support is causing the boot issue. I'm not sure if it has to do with conflicts with other grsecurity flags, or not. This may be contrary to the nature of grsecurity hardening, but perhaps you could have a way to have a flag that disables all of the flags that conflict with grsecurity kmem without the actual kmem protection. I use tlp, for pstate scaling, and I need kmem off, but turning it on seems to be the only way to have a usable grsecurity kernel. Could grsecurity be conflicting with stacktrace_support or tracing_support?

[andyj]

Have you tried adding the daemon uid to the proc restrictions exception in the grsec kconfig? I have multiple systemd hosts and no issues with systemd-login. This was also helpful for me in getting gdm working on grsec kernel using wayland session. On my systems the daemon uid was 2. Here is a useful arch wiki link with other information regarding systemd-login on grsec hardened kernel, https://wiki.archlinux.org/index.php/gr ... om_.2Fproc

[osteichthyes]

I haven't. My vanilla kernel boots without issue. I don't use the proc restrictions in grsecurity, I use hidepid, which I thought did so. I'm trying it now.

[osteichthyes]

That did not fix it. I really think the only fix right now is turning on grsec_kmem, which breaks cpupower.

This is the error from a few builds ago, but here's a link to a picture:
https://drive.google.com/file/d/0B7y3E9 ... sp=sharing

[osteichthyes]

I had a working build using the exact config from the nning's repo. I then turned off GRSECURITY_KMEM and set the grsecurity switches as I wanted them (I'll push to github in a second) and it built and booted just fine. I just rebuilt with the 5 March patchset and it's once again broken. I think the periodic fixing and breaking is the result of the versioning of the grsec patchset, because even the exact arch linux with grsecurity config will only boot sometimes, yet the precompiled binary is fine.

I'm doing more testing and will report back shortly.

[osteichthyes]

I'm now confident this is a bug. I rebuilt the 5 March patch set again, with kmem on. At this point, the February 27 patch set would boot across the board. This version will not. Somewhere between the last few patch sets, something changed, breaking compatibility with gdm/systemd-logind. My previous observations about working/non-working switches seem more related to versioning of the patchset. The break started around the end of last year, around 4.8.10ish and have persisted until the 22 February patchset.

[andyj]

All my grsec patched systemd hosts have CONFIG_GKERNSEC_KMEM=y, and from at least 4.8.17+ none of them have had any problems with systemd-logind. Haven't tried with that config option disabled. Would be happy to compare versions of packages with you, I run gentoo-hardened not arch but maybe there is a systemd version difference between our setups?

Sorry your having problems, wish I could be more help.

[osteichthyes]

It seems completely unrelated to any switches. I'm now getting a new warning, instead of chdir and all that of systemd-logind, I now just get zillions of audit lines, until it overflows. I think they must be the same, but are somehow annotated differently.

I'm starting to think it pertains to versioning and/or just builds. Could the ASLR put the kernel in a place that makes it impossible for the systemd tmpfs impossible to place? That sounds wacky, but I'm almost thinking it needs to be some sort of compilation issue. Could my compiler be wonky?

[spender]

ASLR doesn't affect the placement of the kernel. Are you sure your userland isn't changing at all? Newer systemd versions etc. GRKERNSEC_KMEM disables debugfs, not sure if systemd is depending on that and that's the issue. I don't have much else to go on based on the information you've given.

-Brad

[osteichthyes]

Userland isn't changing aside from regular updates. I just booted into the vanilla kernel, then rebooted into the grsecurity kernel, and it did the same thing again.

I thought debugfs was the issue, it won't boot with and without kmem. I've got probably 30 or 40 configs I've tried. All but two fail. The two that work only work on the 22 February patchset.

I'm lost. I have never hit an issue like this with your patch. The error looks slightly different than the last picture I posted. I'll post another this evening.

[andyj]

Here are the gnome/x11/systemd packages I have installed on my grsec desktop systems with gnome 3.22, don't know if comparing the versions with what you have on your system will help any, but these package versions in gentoo work fine with grsec and systemd on all patches 4.8.17+ with kmem enabled.

[binary R ] x11-proto/xproto-7.0.29
[binary R ] x11-proto/xextproto-7.3.0
[binary R ] x11-libs/xtrans-1.3.5
[binary R ] x11-proto/inputproto-2.3.2
[binary R ] x11-libs/libxkbcommon-0.6.0
[binary R ] x11-proto/kbproto-1.0.7
[binary R ] x11-themes/hicolor-icon-theme-0.15
[binary R ] x11-proto/renderproto-0.11.1-r1
[binary R ] x11-libs/libxshmfence-1.2
[binary R ] x11-proto/xf86vidmodeproto-2.3.1-r1
[binary R ] x11-proto/recordproto-1.14.2-r1
[binary R ] x11-proto/damageproto-1.2.1-r1
[binary R ] x11-proto/compositeproto-0.4.2-r1
[binary R ] x11-proto/randrproto-1.5.0
[binary R ] x11-proto/videoproto-2.3.3
[binary R ] x11-proto/xf86bigfontproto-1.2.0-r1
[binary R ] x11-proto/presentproto-1.0
[binary R ] x11-proto/xf86driproto-2.1.1-r1
[binary R ] x11-misc/xbitmaps-1.1.1-r1
[binary R ] x11-apps/sessreg-1.1.0
[binary R ] x11-apps/rgb-1.0.6
[binary R ] x11-themes/sound-theme-freedesktop-0.8
[binary R ] x11-proto/scrnsaverproto-1.2.2-r1
[binary R ] x11-themes/gnome-backgrounds-3.20
[binary R ] x11-proto/xf86miscproto-0.9.3
[binary R ] x11-proto/dri3proto-1.0
[binary R ] x11-proto/dri2proto-2.8-r1
[binary R ] x11-proto/xineramaproto-1.2.1-r1
[binary R ] x11-proto/fontsproto-2.1.3
[binary R ] x11-misc/util-macros-1.19.1
[binary R ] x11-libs/libICE-1.0.9
[binary R ] x11-libs/libXau-1.0.8
[binary R ] x11-libs/libXdmcp-1.1.2-r1
[binary R ] x11-proto/fixesproto-5.0-r1
[binary R ] x11-libs/libfontenc-1.1.3
[binary R ] x11-libs/libpciaccess-0.13.4
[binary R ] x11-themes/gnome-icon-theme-3.12.0
[binary R ] x11-libs/libxcb-1.12
[binary R ] x11-apps/iceauth-1.0.7
[binary R ] x11-libs/libX11-1.6.4
[binary R ] x11-libs/xcb-util-renderutil-0.3.9-r1
[binary R ] x11-apps/mkfontscale-1.1.2
[binary R ] x11-libs/xcb-util-wm-0.4.1-r1
[binary R ] x11-libs/xcb-util-keysyms-0.4.0
[binary R ] x11-libs/libXfont-1.5.1
[binary R ] x11-libs/libXext-1.3.3
[binary R ] x11-libs/libXfixes-5.0.3
[binary R ] x11-libs/libXrender-0.9.10
[binary R ] x11-apps/xprop-1.2.2
[binary R ] x11-libs/libxkbfile-1.0.9
[binary R ] x11-apps/xmodmap-1.0.9
[binary R ] x11-apps/mkfontdir-1.0.7
[binary R ] x11-libs/libXi-1.7.8
[binary R ] x11-libs/libXrandr-1.5.1
[binary R ] x11-libs/libXcomposite-0.4.4-r1
[binary R ] x11-libs/libXdamage-1.1.4-r1
[binary R ] x11-libs/libXcursor-1.1.14
[binary R ] x11-libs/libXv-1.0.11
[binary R ] x11-libs/libXxf86vm-1.1.4
[binary R ] x11-apps/xkbcomp-1.3.1
[binary R ] x11-libs/libXinerama-1.1.3
[binary R ] x11-libs/libXxf86misc-1.0.3
[binary R ] x11-libs/libXScrnSaver-1.2.2-r1
[binary R ] x11-libs/libXtst-1.2.3
[binary R ] x11-misc/xkeyboard-config-2.17
[binary R ] x11-apps/xdpyinfo-1.3.2
[binary R ] x11-apps/setxkbmap-1.3.1
[binary R ] x11-libs/libXft-2.3.2
[binary R ] x11-misc/xdg-user-dirs-0.15
[binary R ] x11-libs/xcb-util-0.4.0
[binary R ] x11-libs/startup-notification-0.12-r1
[binary R ] x11-libs/xcb-util-image-0.4.0
[binary R ] x11-base/xorg-drivers-1.18-r1
[binary R ] x11-proto/glproto-1.4.17-r1
[binary R ] x11-proto/xcb-proto-1.12-r2
[binary R ] dev-libs/wayland-1.11.0
[binary R ] x11-libs/xpyb-1.3.1-r3
[binary R ] x11-libs/xcb-util-cursor-0.1.3-r1
[binary R ~] dev-libs/wayland-protocols-1.7
[binary R ] sys-apps/systemd-226-r2
[binary R ] x11-libs/libSM-1.2.2-r1
[binary R ] sys-apps/gentoo-systemd-integration-6
[binary R ] sys-process/systemd-cron-1.5.4
[binary R ] x11-libs/libXt-1.1.5
[binary R ] x11-libs/libXmu-1.1.2
[binary R ] x11-libs/libXpm-3.5.12
[binary R ] x11-apps/xauth-1.0.9-r2
[binary R ] x11-apps/xset-1.2.3
[binary R ] x11-libs/libXaw-1.0.13
[binary R ] x11-apps/xhost-1.0.7
[binary R ] x11-apps/xrdb-1.1.0
[binary R ] x11-apps/xinit-1.3.4-r1
[binary R ] gnome-base/dconf-0.26.0-r1
[binary R ] x11-misc/shared-mime-info-1.4
[ebuild R ] x11-libs/libdrm-2.4.70 VIDEO_CARDS="-intel*"
[binary R ] x11-misc/xdg-utils-1.1.1-r1
[ebuild R ] x11-libs/pixman-0.34.0 CPU_FLAGS_X86="ssse3*"
[binary R ] x11-libs/cairo-1.14.6
[binary R ] x11-base/xorg-server-1.18.4
[binary R ] x11-apps/mesa-progs-8.3.0
[binary R ] x11-drivers/xf86-video-ati-7.7.0
[binary R ] x11-drivers/xf86-input-synaptics-1.8.3
[binary R ] x11-drivers/xf86-input-evdev-2.10.3
[binary R ~] x11-libs/gdk-pixbuf-2.36.5
[binary R ~] gnome-base/gsettings-desktop-schemas-3.22.0
[binary R ] x11-libs/libxklavier-5.3
[binary R ] gnome-base/libgtop-2.34.2
[binary R ] gnome-base/gnome-menus-3.13.3-r1
[binary R ] x11-libs/pango-1.40.3
[binary R ~] x11-themes/gnome-themes-standard-3.22.2
[binary R ] gnome-base/librsvg-2.40.16
[binary R ~] x11-themes/adwaita-icon-theme-3.22.0
[binary R ] x11-libs/gtk+-2.24.31-r1
[binary R ] x11-themes/gtk-engines-adwaita-3.20.2
[binary R ] x11-libs/libnotify-0.7.6-r3
[binary R ] gnome-base/gnome-keyring-3.20.0
[binary R ~] x11-libs/gtk+-3.22.5
[binary R ] x11-misc/colord-1.3.4
[binary R ] gnome-extra/polkit-gnome-0.105-r1
[binary R ~] gnome-base/gnome-desktop-3.22.2
[binary R ] gnome-base/gconf-3.2.6-r4
[binary R ] x11-misc/xdg-user-dirs-gtk-0.10
[binary R ] x11-libs/colord-gtk-0.1.26
[binary R ] gnome-base/libgnome-keyring-3.12.0
[binary R ~] x11-libs/vte-0.46.1
[binary R ] gnome-extra/zenity-3.20.0
[binary R ] gnome-base/libgnomekbd-3.6.0
[binary R ~] app-arch/gnome-autoar-0.1.1
[binary R ] gnome-extra/gconf-editor-3.0.1-r1
[binary R ~] gnome-base/gvfs-1.30.3
[binary R ~] gnome-base/gnome-core-libs-3.22.2
[binary R ] gnome-extra/evolution-data-server-3.20.6
[binary R ~] gnome-base/gnome-settings-daemon-3.22.1
[binary R ~] x11-wm/mutter-3.22.3
[binary R ] net-wireless/gnome-bluetooth-3.20.0
[binary R ] gnome-extra/gnome-color-manager-3.20.0
[binary R ~] gnome-base/gnome-session-3.22.2
[binary R ~] gnome-base/gnome-control-center-3.22.1-r1
[binary R ~] gnome-base/gnome-shell-3.22.2
[binary R ~] gnome-base/gdm-3.22.1-r1
[binary R ~] gnome-base/nautilus-3.22.2
[binary R ] gnome-extra/nautilus-tracker-tags-1.8.3
[binary R ~] x11-terms/gnome-terminal-3.22.1
[binary R ] gnome-extra/gnome-tweak-tool-3.20.1
[binary R ~] gnome-base/gnome-light-3.22.0

Would you like me to test a kernel with kmem disabled to see if it will boot for me? So you can rule out the grsec option as the cause of your issue? Sounds to me like you are getting something from the arch rolling update that is a binary that's incompatible with the grsec patch somehow, but not really sure what that would be. Always had better luck using gentoo-hardened with grsec than arch, tried arch with grsec first before giving up and just going gentoo-hardened and everything was way easier after the move honestly.

[osteichthyes]

I've tried with and without kmem numerous times. I'll take a look at the package versions shortly

[osteichthyes]

I'm currently posting from a bootable, working grsecurity kernel, the one from the archlinux repos.

The issue is clearly not a bug, but something with my build environment. My config switches are basically identical to those in the repo, so it's not that, the issue must be in a missing dependency or something.

[osteichthyes]

Have you ever had an issue that's so inconsistent you start to feel like it may be you?

So, yesterday I realized I was missing one of the build dependencies for the kernel. I have no idea how that happened, how long it's been missing, or why it even built without an error.

Anyway, I installed the dependency, rebuilt the kernel again, rebooted and had a working custom kernel without issue. Then, I started tinkering with the configs and rebuilt, and the kernel failed to boot.
I thought I finally had the error licked and started eliminating switches to find the single switch causing the issue. I'd narrowed it down to about a dozen.

However, today I rebuilt using the config that yesterday yielded a bootable system. Today, it yielded the same systemd-logind errors.

I think I've actually been hitting two independent errors. (1) something in the userland is updating and breaking gdm, I think arch is in the process of updating the mesa drivers. I have a nasty habit of updating daily, and that may indeed be the culprit. And, (2) the missing build dep was causing a log with audit dumping tons of errors. I still don't have a bootable kernel right now. I'm going to try to latest patchset and see what happens.