Page 1 of 1

PAX: size overflow detected in function ipv6_frag_rcv net/ipv6/reassembly.c:223

PostPosted: Sun Oct 23, 2016 9:53 am
by deagol
I'm running Gentoo hardened and after updating to sys-kernel/hardened-sources-4.7.6 one of my virtual systems started crashing. Normally within one minute after boot, but sometimes it took an hours or so.
I upgraded to sys-kernel/hardened-sources-4.7.9, but it changed nothing. Nevertheless I stuck to the 4.7.9 kernel for the debugging and everything here was done on the sys-kernel/hardened-sources-4.7.9 kernel from Gentoo.

I think I'm close to the solution, but I do not know if the linux reassembly code is "broken" or the pax overflow detection triggers in a case it should not.
Now I need some help how to address and get the problem fixed properly...

For my understanding any fragmented ipv6 (tcp?) packet will trigger the pax overflow detection and panic the kernel.
I already opened a Gentoo bug for the issue. It has some more data, including two full panic logs and a packet capture of packets causing the panic: https://bugs.gentoo.org/show_bug.cgi?id=597792

According to PAX the problematic code is the second line of of the following calculation from net/ipv6/reassembly.c:
Code: Select all
   end = offset + (ntohs(ipv6_hdr(skb)->payload_len) -
                   ((u8 *)(fhdr + 1) - (u8 *)(ipv6_hdr(skb) + 1)));  // line 223 in net/ipv6/reassembly.c

Without changing the code above and just adding the first printk line from the patch below I do not see any real overflow, even if the casts are cutting off the higher bits. (And I think they are not) .
Here the debug output for two PAX Overflow reports (The Gentoo ticket has the full trace log, if needed):
Code: Select all
Oct 22 22:32:01 gandalf kernel: DDD ip6_frag_queue  offset=0; payload_len=4d8; fhdr1=f1c8a27e; ipv6_hdr=f1c8a276
Oct 22 22:32:01 gandalf kernel: DDD ip6_frag_queue  offset=4d0; payload_len=199; fhdr1=f1c8a07e; ipv6_hdr=f1c8a076


Based on my (limited) understanding I tried to work around this Pax Overflow crash, and it did indeed work:
I still get debug output from the added printk lines, but no longer PAX Overflows.

Here the patch which kind of works for me:
Code: Select all
--- /tmp/reassembly.c   2016-10-23 14:13:28.086253478 +0200
+++ net/ipv6/reassembly.c       2016-10-22 23:13:52.778959198 +0200
@@ -211,7 +211,7 @@
 {
        struct sk_buff *prev, *next;
        struct net_device *dev;
-       int offset, end;
+       int offset, end, temp;
        struct net *net = dev_net(skb_dst(skb)->dev);
        u8 ecn;
 
@@ -219,8 +219,14 @@
                goto err;
 
        offset = ntohs(fhdr->frag_off) & ~0x7;
-       end = offset + (ntohs(ipv6_hdr(skb)->payload_len) -
-                       ((u8 *)(fhdr + 1) - (u8 *)(ipv6_hdr(skb) + 1)));
+
+       printk ("DDD ip6_frag_queue  offset=%x; payload_len=%x; fhdr1=%x; ipv6_hdr=%x\n", offset, ntohs(ipv6_hdr(skb)->payload_len), (u8 *)(fhdr + 1), (u8 *)(ipv6_hdr(skb) + 1));                                                                                                                                                                     
+                                                                                                                                                                           
+       temp = (u8 *)(fhdr + 1) - (u8 *)(ipv6_hdr(skb) + 1);                                                                                                               
+       temp &= 0xff;  //This line is probably not needed and wrong!!                                                                                                                                                       
+       end = offset + (ntohs(ipv6_hdr(skb)->payload_len) - temp);                                                                                                         
+//                     ((u8 *)(fhdr + 1) - (u8 *)(ipv6_hdr(skb) + 1)));                                                                                                   
+       printk ("DDD ip6_frag_queue2 end=%x\n", end);
 
        if ((unsigned int)end > IPV6_MAXPLEN) {
                __IP6_INC_STATS(net, ip6_dst_idev(skb_dst(skb)),


I've now removed the probably incorrect line "temp &= 0xff;" from my running kernel and that also seems to work. The added printk lines still get triggered but no complains from PAX any more.
I suspect the Pax Overflow detection gets somehow confused by the u8* casts, but then I'm only guessing here.

Here are some lines with the full patch above, but without the probably invalid line "temp &= 0xff;"

Code: Select all
[  296.213127] DDD ip6_frag_queue  offset=0; payload_len=4d8; fhdr1=c557987e; ipv6_hdr=c5579876
[  296.213157] DDD ip6_frag_queue2 end=4d0
[  296.213175] DDD ip6_frag_queue  offset=4d0; payload_len=8e; fhdr1=c557967e; ipv6_hdr=c5579676
[  296.213189] DDD ip6_frag_queue2 end=556

I even removed the pax_size_overflow_report_only boot parameter and now only the debug code is triggered and PAX is fine.
Any pointers how to get that properly fixed?

Re: PAX: size overflow detected in function ipv6_frag_rcv net/ipv6/reassembly.c:223

PostPosted: Sun Oct 23, 2016 11:10 am
by PaX Team
thanks for the report, this is indeed a gcc induced intentional overflow on computing that expression. can you try the following patch:
Code: Select all
--- a/net/ipv6/reassembly.c     2016-07-26 23:40:26.032023221 +0200
+++ b/net/ipv6/reassembly.c       2016-10-23 17:06:15.393638963 +0200
@@ -219,8 +219,8 @@
                goto err;

        offset = ntohs(fhdr->frag_off) & ~0x7;
-       end = offset + (ntohs(ipv6_hdr(skb)->payload_len) -
-                       ((u8 *)(fhdr + 1) - (u8 *)(ipv6_hdr(skb) + 1)));
+       end = offset + ntohs(ipv6_hdr(skb)->payload_len);
+       end -= (u8 *)(fhdr + 1) - (u8 *)(ipv6_hdr(skb) + 1);

        if ((unsigned int)end > IPV6_MAXPLEN) {
                __IP6_INC_STATS(net, ip6_dst_idev(skb_dst(skb)),

PS: when you print out pointers, you should use the %p format string to not truncate their values.

Re: PAX: size overflow detected in function ipv6_frag_rcv net/ipv6/reassembly.c:223

PostPosted: Sun Oct 23, 2016 12:37 pm
by deagol
Yes, the patch works for me.
I've added a printk line on top of it, so I still have an indication when the path is triggered.

Debug output was triggered, but no Overflow warning :D

Code: Select all
[ 1683.355848] DDD ip6_frag_queue  offset=0; payload_len=4d8; fhdr1=ffff8800c2b1c47e; ipv6_hdr=ffff8800c2b1c476
[ 1683.355899] DDD ip6_frag_queue  offset=4d0; payload_len=e4; fhdr1=ffff8800c2b1c27e; ipv6_hdr=ffff8800c2b1c276


Thanks for your help