Essentially it depends on the attacker and what resources they have available. Ordinary home-users or attackers who aren't very dedicated might be deterred by source IP address blocking, but more advanced/determined attackers have a number of ways to get round this.
The obvious one is to use one of the wide range of proxy services available on the Internet. This includes things like ToR and Anonymizer.com. Since one of the goals of these services is to mask the source IP address of the user, they will bypass the block until the attacker tries enough attempts with a given IP and gets blocked again.
As an interesting aside, the attacker could cause a partial DoS to your application by going round enough of these sites, as any of your other users who make use of those services will be blocked, once you block the IP addresses belonging to them.
On top of online services, the attacker could do things like use Wi-Fi hotspots or college/university networks to get access to more source IPs
So essentially this blocking would make things more awkward but wouldn't stop a determined attack.