Let's say have a linux server to which only I have shell access and all other users shells are disabled. Is there any reason to be preoccupied?
Dirty cow does not need shell access. It only needs the ability to execute own software. This can for example also be done by using a security problem in a web application to upload and execute remote code.
A more complete answer in my opinion:
You could execute dirtycow exploit if you have:
shell access; RCE in web server or anything else; a social engineered customer support; etc
You should fully understand or at least partially understand how the exploit works to understand it's possibilities of being executed. Read up on it at: github.com/dirtycow/
Multiple versions of it hosted on that account.