Antivirus software detects threats by heuristic analysis like it analyzes the commands as they are performed, monitoring for common viral activities such as replication, file overwrites, and attempts to hide the existence of the suspicious file. So can heuristic analysis detect keylogging activity?
Most systems provide APIs and hooks that allow apps to capture keystrokes (for instance for keyboard shortcuts)
A heuristic could detect the use of such facilities and flag the app for further analysis or provide means to whitelist/block the app, similar to those firewall prompts on Windows.
Modern keyloggers seem to also log the clipboard, presumably to account for paasword manager users.
As all heuristic analysis, it's subject to being subverted/bypassed by a creative keylogger developer.
The logging can also be subverted by tools like KeePass that adds bogus keystrokes when inputting passwords to confuse keylogger.
This rapidly leads to an arms race of increasing costs for both sides :)