I'm not a CLI-lover, sometimes I prefer to do some tasks in a server using a GUI rather than using the CLI (it speeds up my work), but I don't know if installing a GUI and accessing it using VNC and SSH will compromise the server security.
Every new service or program can pose a new security risk because they can have their own vulnerabilities. X11 has their own CVE's, for instance. Then when you access it over something like VNC, you introduce other potential problems and possibilities for misconfigurations.
So, yes, there are inherent risks, but nothing specific for GUIs.