My answer is primarily Linux configuration, not really security. It addresses your aim but doesn't provide for resetting the timeout on a correct password or incrementing it while already banned. That probably has to be out of band, using some other event as a key.
This answer links to an article showing how to deal with repeat offenders once, and this shows how to perform an incrementing ban. Since just linking is bad I'll try to rephrase what it's doing.
In effect, you want to create several configuration blocks that use nearly identical names; the author uses "f2b-loop2", "f2b-loop3", and so on. The DEFAULT block is the driver.
Each loop step uses the same filter. That filter ignores all loop-named bans but continues counting the DEFAULT bans over the scantime, effectively making level determine "How many times has DEFAULT banned this IP in my scan period/findtime?". The total attempts that trigger successively higher bantimes are therefore a multiplication of DEFAULT's maxtimes and the loop level's maxtimes.
You can continue this pattern to end with a negative bantime to permanently ban.