I actually was stupid enough to accidently enable that option when it first was introduced to grsecurity though
actually it's not a stupid thing to enable but your userland had better be prepared for it
this is a problem with glibc which will eventually hit all distros as they upgrade (i think 2.3.3 will have the code in question). what the RedHat guys cooked up for supporting ExecShield and gcc nested function trampolines is that they introduced an ELF marking scheme (google for PT_GNU_STACK if you want the details) that in turn is 'understood' by both the(ir) kernel and ld.so. what the ld.so piece does is that when it detects the need for an executable stack in a given library (in practice that's the result of the use of the gcc nested function trampoline) it will call an internal function called make_stack_executable() that does just what its name implies. this is also the point of failure as in the typical PaX setup you have MPROTECT on and that doesn't allow something like this to happen (imagine a return-to-libc style attack to call this function then the payload...). there's also a problem with mixing gcc versions: if you have objects (libraries) produced by different versions of gcc (or other compilers/assemblers) then chances are that some of them don't have the proper markings (in a .o file that would be a .note.GNU-stack section). now when you link an executable/library from objects with such a mixed origin, the linker will mark the output with an executable PT_GNU_STACK program header (to be backwards compatible). as you can guess, that's a fast way to provoke the runtime error you ran into.
now as for fixing this mess:
1. you can take out make_stack_executable() from glibc/ld.so (that's advisable anyway for a secure setup), or just neutralize it by having it ignore the EACCES error from mprotect(). i know that Gentoo took the latter approach and it works.
2. in the not too distant past i added support for PT_GNU_STACK in PaX (i use it to enable EMUTRAMP as needed, this is about the only useful feature of this marking) but it only works when the main executable carries this mark, for libraries it won't work (which is the case that hit you). however you can explicitly mark your executables that load such shared libraries: when you link them add the -z execstack
switch to ld or -Wl,-z,execstack
to gcc (there's also an execstack
utility in the latest prelink package that can do the same). of course before starting this executable (re)marking campaign you should verify first that the library in question does actually use nested functions and wasn't marked because it was merely linked from a mixed set of object files.
3. you can disable MPROTECT in your kernel config or on the affected executables. this is the least desirable approach of course as it will open up a nice attack vector for exploits.
4. the best solution is of course to get rid of nested functions altogether, but that requires some programming effort and is a slow process.