once more i got nice feedback from somebody out there who benefits from my tinkering - it's very nice to see that one's efforts aren't totally wasted.
lorenco catucci suggested that pam_recent works better for rate limiting services that don't terminate the network connection on a failed login if it offered handlers for the other pam phases too, most importantly auth.
that way, one can use pam_recent to record that somebody attempts access in an iptables' recent list, and clear the record if and only if the connection gets to the account or session stage. in this setup pam is providing the control and iptables recent match just enforces the limits.
nice idea, just a few simple changes required and the result is better and more useful than before.