vast amount of kernel time spending in gr_attach_curr_ip

Discuss and suggest new grsecurity features

vast amount of kernel time spending in gr_attach_curr_ip

Postby maple » Sun Sep 14, 2003 10:44 am

Hello!
I using patch 1.9.12 for 2.4.22, and, after profiling kernel I noticed that gr_attach_curr_ip leading in readprofile just after default_idle, with all grsec network options turned off

here is output of
readprofile -m /boot/System.map-2.4.22-grsec | sort -nr | head -10

289227 total 0.1998
144133 default_idle 2771.7885
85892 gr_attach_curr_ip 230.8925
7095 do_wp_page 10.6213
5295 do_fork 2.4883
3625 do_exit 4.7697
3155 schedule 2.4047
3057 release_task 6.7039
2045 do_page_fault 1.5804
1613 system_call 28.8036

why gr_attach_curr_ip is consumung CPU?

my config:
enabled only jail restrictions with chroot_findtask, dmesg, chroot_deny_sysctl turned on

Linux 2.4.22 patched with grsec 1.9.12, two AIC7XXX drives in raid0, reiserfs over raid, ext3 as root, 4G RAM, two pentium4 with ht enabled, total 4 cpus, running mostly Apache 1.3.28 & mysql 3.23.56
maple
 
Posts: 9
Joined: Sun Sep 14, 2003 10:30 am

Postby spender » Sun Sep 14, 2003 3:06 pm

The gr_attach_curr_ip function has polynomial time complexity: O(n^2) due to the algorithm having to detect whether a connection is being made locally, so that the remote IP will be carried over to the task, instead of the local IP. Currently it looks through each file descriptor of every task that has used connect during its life to see if they have a socket opened that is the other end of a connection we're about to make. I don't know of any way currently to reduce the complexity of it, and no one has been able to suggest a better algorithm.

-Brad
spender
 
Posts: 2185
Joined: Wed Feb 20, 2002 8:00 pm

Postby spender » Sun Sep 14, 2003 3:31 pm

I think I've found a way to turn it into an O(n) problem, where n is the current number of tasks (though the number of operations per iteration will be very small)...so this should reduce the time significantly. If anyone knows of any way to turn it into an O(1) problem, let me know.

-Brad
spender
 
Posts: 2185
Joined: Wed Feb 20, 2002 8:00 pm

Postby maple » Mon Sep 15, 2003 4:56 am

will be great to get improvements in this area, I think nobody likes situation when grsec consumes half of system resources in internal scanning ;)
May I ask when you planning to implement your better algorithm?
maple
 
Posts: 9
Joined: Sun Sep 14, 2003 10:30 am

Postby spender » Mon Sep 15, 2003 3:26 pm

I wrote the new code yesterday, tested it today, and committing to CVS at this moment.

-Brad
spender
 
Posts: 2185
Joined: Wed Feb 20, 2002 8:00 pm

Postby spender » Mon Sep 15, 2003 3:30 pm

Also, I would really appreciate it if you could run the profiling against the kernel in CVS and give me the results of it.

-Brad
spender
 
Posts: 2185
Joined: Wed Feb 20, 2002 8:00 pm

Postby maple » Tue Sep 16, 2003 1:12 am

ok, I will try CVS kernel soon, its production machine and better to not reboot it every five minutes. Should I consider cvs version stable?
And next bugreport, probably related to prevous. Problem with this server was not only cpu time going to /dev/null. With moderate load apache begin to respond _very_ slowly, up to 15 seconds even in local network. tcpdump shows that server often not responding to TCP SYNs. After I found gr_attach_curr_ip eating cpu, i decided to try vanilla 2.4.22. And all problems goes away, mrtg traffic grow almost twice! system time now ~8% instead of ~30 with grsec.
maple
 
Posts: 9
Joined: Sun Sep 14, 2003 10:30 am

Postby maple » Wed Sep 17, 2003 7:32 am

I rebooted to cvs kernel and it running without problems 5 hours up to now.
gr_attach_curr_ip at 8 place in readprofile, _much_ less than prevous, but still consuming cpu. Problems with slow http serving seems to disappear. Brad, thanks a lot!

543815 total 0.3756
477009 default_idle 9173.2500
7030 schedule 5.3582
6704 do_wp_page 10.0359
3972 system_call 70.9286
2695 fget 37.4306
2461 do_page_fault 1.9019
2422 .text.lock.sched 5.0669
1832 gr_attach_curr_ip 7.0462
1539 file_read_actor 6.8705
1526 do_anonymous_page 5.6940
1484 do_generic_file_read 1.3203
1325 e100intr 2.6714
1188 fput 4.8689
1175 d_lookup 4.1373
maple
 
Posts: 9
Joined: Sun Sep 14, 2003 10:30 am

Postby spender » Wed Sep 17, 2003 8:06 am

I think I can even cut this down some more, and make it an O(1) algorithm. I'll let you know when I get around to it.

-Brad
spender
 
Posts: 2185
Joined: Wed Feb 20, 2002 8:00 pm


Return to grsecurity development