RANDSRC option, TIME_WAIT and FTP

[mtg]

we experienced problems when stress-testing a ftp-upload application between machines with 2.4-grsec kernels that have the "Randomized TCP source ports" (RANDSRC) option enabled.
It seems like the randomization is not aware if the random-port which it assigns is in TIME_WAIT state.
With active ftp, the client application regularly throwed "Address already in use" and the server got stuck in SYN_SENT. With passive ftp, it's vice versa.
Neither lowering "tcp_fin_timeout" nor setting "tcp_tw_reuse" or "tcp_tw_recycle" helped reliably. The problem doesn't occur with RANDSRC disabled.

Is this expected behaviour of the TCP source port randomization?