I've applied the 4.1.4 patch, and compiled the kernel using gcc5. When I try and do a 'exec startx' as a regular user, after logging in, I am unable to use my keyboard and mouse in the graphical environment.
i found part of my problem was because of this:
[ 16.903401] grsec: denied use of ioperm() by /bin/vmmouse_detect[vmmouse_detect:842] uid/euid:0/0 gid/egid:0/0, parent /lib/systemd/systemd-udevd[systemd-udevd:803] uid/euid:0/0 gid/egid:0/0
and another one for ioperm() for /usr/bin/Xorg
I could make those go away by disabling CONFIG_GRKERNSEC_IO, but this surprises me because I am running the latest Xorg, and Debian unstable, and the help information for this option makes it seem like this is only necessary for very old X.
If I disable CONFIG_GRKERNSEC_IO I dont get that error, but I still can't use my keyboard/mouse in X, the only grsec related message i get is:
[ 94.945270] grsec: denied resource overstep by requesting 21 for RLIMIT_NICE against limit 0 for /usr/bin/xinit[xinit:3367] uid/euid:1000/1000 gid/egid:1000/1000, parent /usr/bin/startx[startx:2896] uid/euid:1000/1000 gid/egid:1000/1000