If you don't want a SSH server running on your system at all, the better approach is to remove the package that provides the SSH server software. Doing so will remove it cleanly, though it is possible (depending on your distribution) that packages depend on the SSH server packages for some reason. If you just remove its configuration files, odds are good that the system will try to start it anyway, and it will start logging errors (which will clutter up your logs and ultimately might hide something actually interesting).
To do this, use your package manager to remove whichever package provides the SSH server. On Debian and Debian derivatives, you'll likely want to remove
openssh-server as seen by searching the list of packages for OpenSSH packages:
dpkg -l '*openssh*' by running
sudo apt-get remove --purge openssh-server Other systems are likely to be different, so adjust accordingly.
Removing the SSH server shouldn't break anything else as long as the package manager doesn't want to remove additional, unrelated packages; and you can always reinstall it later if you want to.