mmap failure with grsec 4.3.5

Discuss usability issues, general maintenance, and general support issues for a grsecurity-enabled system.

mmap failure with grsec 4.3.5

Postby benoit@scylladb.com » Wed Feb 24, 2016 11:48 am

Hi,

I am running grsecurity for personal convenience on my workstation.

While trying to run scylladb the product made by my current customer I got the following trace.

The setup is an fedora 23 lxc container running on an ubuntu 4.3.5 grsec host.

mprotect(0x348c50da000, 12288, PROT_READ) = 0
mprotect(0x348c52de000, 4096, PROT_READ) = 0
mprotect(0x1c2d000, 4096, PROT_READ) = 0
mprotect(0x348c5500000, 4096, PROT_READ) = 0
munmap(0x348c54f1000, 37083) = 0
set_tid_address(0x348c54db750) = 9874
set_robust_list(0x348c54db760, 24) = 0
rt_sigaction(SIGRTMIN, {0x348c13bdb20, [], SA_RESTORER|SA_SIGINFO, 0x348c13c8430}, NULL, 8) = 0
rt_sigaction(SIGRT_1, {0x348c13bdbb0, [], SA_RESTORER|SA_RESTART|SA_SIGINFO, 0x348c13c8430}, NULL, 8) = 0
rt_sigprocmask(SIG_UNBLOCK, [RTMIN RT_1], NULL, 8) = 0
getrlimit(RLIMIT_STACK, {rlim_cur=8192*1024, rlim_max=RLIM64_INFINITY}) = 0
statfs("/sys/fs/selinux", 0x38d24f612e0) = -1 ENOENT (No such file or directory)
statfs("/selinux", {f_type="EXT2_SUPER_MAGIC", f_bsize=4096, f_blocks=49077591, f_bfree=41930274, f_bavail=39431510, f_files=12476416, f_ffree=11989493, f_fsid={861819776, -1156077281}, f_namelen=255, f_frsize=4096, f_flags=4128}) = 0
mmap(NULL, 35184372088832, PROT_NONE, MAP_PRIVATE|MAP_ANONYMOUS|MAP_NORESERVE, -1, 0) = -1 ENOMEM (Cannot allocate memory)
rt_sigprocmask(SIG_UNBLOCK, [ABRT], NULL, 8) = 0
tgkill(9874, 9874, SIGABRT) = 0
--- SIGABRT {si_signo=SIGABRT, si_code=SI_TKILL, si_pid=9874, si_uid=0} ---
+++ killed by SIGABRT (core dumped) +++
Abandon (core dumped)

So the Scylla database is failing to mmap anonymous memory and core dumping.

Any idea why mmap is failling this way ?

Best regards

Benoît Canet
benoit@scylladb.com
 
Posts: 4
Joined: Wed Feb 24, 2016 11:43 am

Re: mmap failure with grsec 4.3.5

Postby PaX Team » Wed Feb 24, 2016 1:12 pm

does this fail on a vanilla kernel? other than that, we don't restrict mmap sizes or rlimits or overcommit ourselves, so maybe check those first.

edit: actually there's one restriction for such big sizes, if you have UDEREF enabled then the userland address space size will be reduced to 42 bits (from 47) thus a 45 bit allocation won't fit.
PaX Team
 
Posts: 2310
Joined: Mon Mar 18, 2002 4:35 pm

Re: mmap failure with grsec 4.3.5

Postby benoit@scylladb.com » Wed Feb 24, 2016 2:54 pm

Ok I will test vanilla and lxc on friday.
(I am currently on travel an jailed in the container).

and also removing uderef.

Thanks

Best regards

Benoît
benoit@scylladb.com
 
Posts: 4
Joined: Wed Feb 24, 2016 11:43 am

Re: mmap failure with grsec 4.3.5

Postby PaX Team » Wed Feb 24, 2016 6:14 pm

if you know you had UDEREF in the kernel then i'm pretty sure that's the limit you ran into.
PaX Team
 
Posts: 2310
Joined: Mon Mar 18, 2002 4:35 pm

Re: mmap failure with grsec 4.3.5

Postby benoit@scylladb.com » Fri Jul 22, 2016 6:18 am

I previously just disabled my grsecurity kernel and now a few month later I tried again with 4.6.4
while removing UDEREF.

Same behavior.
benoit@scylladb.com
 
Posts: 4
Joined: Wed Feb 24, 2016 11:43 am

Re: mmap failure with grsec 4.3.5

Postby PaX Team » Fri Jul 22, 2016 6:52 am

on amd64 KERNEXEC also reduces the userland address space size (as it's a side effect of the underlying per-cpu pgd feature).
PaX Team
 
Posts: 2310
Joined: Mon Mar 18, 2002 4:35 pm

Re: mmap failure with grsec 4.3.5

Postby benoit@scylladb.com » Fri Jul 22, 2016 8:01 am

Thanks a lot it work when removing KERNEXEC and UDEREF from the kernel config.
benoit@scylladb.com
 
Posts: 4
Joined: Wed Feb 24, 2016 11:43 am


Return to grsecurity support