Can the WiFi protocol recognize when a brute force or dictionary attack on it is occurring? WiFi gives the user an authentication failed message after entering an incorrect WiFi password several times.
No, it is fundamentally impossible to detect a brute force or dictionary attack when using the IEEE 802.11i (WPA2) protocol. This is due to the fact that brute force attacks do not require interacting with the router at all, only passively observing existing connections. If they observe the 4-way handshake, they can attempt to brute force it offline and use it to either authenticate or passively decrypt traffic. The router has no way to know if a client connecting to it is connecting with a key legitimately obtained or with a key that has been obtained through cracking* the 4-way handshake.
The upcoming WPA3 protocol will no longer be vulnerable to offline brute force and dictionary attacks due to its use of the Dragonfly protocol. Unlike WPA2, the encryption key is generated randomly and is distinct from the password used for authentication. Attempting to authenticate with any given password requires interacting with the router, so a brute force attack will thus require actively requesting authentication from the router, which can then rate-limit requests, log failed attempts, or outright refuse to serve a client that is repeatedly failing authentication.
* A one-way hash of a the password is sent in the clear, so all the attacker has to do is run candidate passwords through the same algorithm and compare the resulting hash to the hash of the correct password.