The pam_tally.so PAM module is intended to denying further authentication attempts after a given count of failed authentications. pam_tally is the maintenance program for pam_tally.so: it can list and reset the accumulated counts.
See http://cvs.sourceforge.net/viewcvs.py/*checkout*/pam/Linux-PAM/modules/pam_tally/README?content-type=text%2Fplain&rev=HEAD for details. (I don't know whether this is the authoritative source though.)
One page links to pam_tally(8):