@Forest's answer is helpful but I'd like to address the security perspective.
While there is a security risk, it's likely considered an acceptable risk, and possibly a feature, as mentioned in @BenCreasy's comment. It's clearly acceptable enough such that Xorg has been the default display server on many distributions for many years.
The primary concern seems to be that a process running as a user can log all keystrokes entered by the user from other processes, which may include passwords or other secrets. However, with a standard threat model, to get to the point where this could be used, you've already trusted and allowed the program to run as your user. It could do much nastier things than simply log keystrokes. Therefore, while an oversimplification, it's like saying "there's a security vulnerability in the system because running this malicious program gives me a virus".
However, this is not the best model, and is why Wayland attempts to solve the issue.
man xauth
for details), so other users cannot connect to your X server and grab your keypresses. So under most circumstances, a program that can access your X server to grab keys is already running locally under your user, which is why I believe Bakuriu called that scenario game over. Where does your interpretation differ?