As you know, REST services of mobile application are vulnerable from calling by other unauthorized users, some attacks can be done, and we don't have any prevention mechanism for them, such as dictionary attack. In web applications the basic mechanism that we can implemented is a Captcha, but for REST services of mobile application there's nothing we can do. I guess the device fingerprint can be effective solution in this way, what do you think about this?
It's VERY difficult to prevent random users from calling your endpoint. I would not recommend fingerprinting (assuming this is a general service). How is the fingerprint generated and verified? Can you ensure confidentiality, integrity and authentication of the fingerprint? What's stopping another user from analyzing the requests and sending the same fingerprint? Generally, you should assume that attackers know EVERYTHING about your endpoints and build security controls inside of your endpoints. Security by obscurity is almost always a bad idea.
Not entirely the same, but following the same concept, a great example of how client filtering failed magnificently would be the Pokemon Go application (story here). The short summary of what happened was that Pokemon Go wasn't able to limit the requests sent from just their mobile application. So, hackers starting building secondary clients that would send fake location requests back to the application, thus allowing users to jump anywhere on the map and collect Pokemon from all locations.
If your service is only used by a known set of immutable users, IP filtering would be a great technique to prevent others from calling your API.
Otherwise, you would want to build security controls within your API. (Ex. a 5 minute timeout for every 3 incorrect logins per IP, etc.)
You can't distinguish a good device fingerprint from a bad one. A suspicious fingerprint might actually be legitimate, if the user has changed device or is connecting from another network, for example. And a good fingerprint might actually be malicious, if the attacker has been able to spoof it by sending acceptable parameters. Device fingerprints can be useful to send alerts like "somebody signed in from a suspicious device or location, was that you?", but you can't use them reliably to filter traffic.
The right way to avoid remote bruteforcing is to limit the attempts, introducing a delay or a block when too many login failures are detected. This limitation is usually applied only to the IPs that are failing to authenticate. If you decided to delay or block logins in general you would introduce DoS issues for legitimate users. Provided that the passwords or the authentication keys are strong enough, limiting failed logins will make remote bruteforcing totally unfeasible.