Let's say have a linux server to which only I have shell access and all other users shells are disabled. Is there any reason to be preoccupied?
A more complete answer in my opinion:
You could execute dirtycow exploit if you have:
shell access; RCE in web server or anything else; a social engineered customer support; etc
You should fully understand or at least partially understand how the exploit works to understand it's possibilities of being executed. Read up on it at: github.com/dirtycow/
Multiple versions of it hosted on that account.