Although in theory modelines shouldn't allow you to do anything bad, sanitizing modeline functions input is hard. There have been several security issues in the past that allowed arbitrary command execution or DoS attacks. To cite a few past vulnerabilities I could find:
- CVE-2002-1377 allowed an attacker to execute an arbitrary command
- CVE-2005-2368: arbitrary command execution using glob and expand functions
- 2007 (couldn't find a CVE number): DoS through the spell checker
- CVE-2007-2438: shell command execution through writefile, feedkeys and system
- CVE-2016-1248: arbitrary code execution again using filetype, syntax and keymap
- CVE-2019-12735: one more arbitrary code execution, by breaking out of the sandbox
While I believe they have all been fixed, new vulnerabilities keep popping up and it is likely that similar vulnerabilities will be found in the future, which is why distros disable modelines by default (at least Debian and SuSE do).