Mods certainly can be used as infection vectors. A lot of it comes down to a question of trust. A mod with tens of thousands of downloads and nobody suggesting they've had any problems is likely to be OK (though still no guarantee!).
In an ideal world:
- Wherever possible, avoid mods that have an installer. There are some situations that this can't be done, but mods with installers are probably the biggest threat. A lot of users will happily click through a request for administrative privileges from an installer.
- Download mods from reputable sources. Ensure you're always downloading the mod as the creator released it, and not repacked by anyone else!
- Check comments sections for the mod, and do a quick google search with keywords like "virus".
- As per the first section of this answer, favor popular mods over ones with barely any downloads.
Sections of your question are not possible to answer with any real useful information, as there are so many different types of mod available with great differences between them. However, generalizing:
How dangerous are gaming mods, does language and file-type matter, and how can I decide these for each individual game?
How long is a piece of string? Mods can be very dangerous or not dangerous at all. File type certainly matters. Mods that just replace textures, or add new models etc are pretty unlikely to cause you any problems. Mods that use .exe installers are something to be wary of.
Does the maliciousness/exploitability depend on the mod file I am downloading, or the type of files the mod-file is modifying?
Both. If the mod you are downloading is a .exe, it's a bigger risk than if it's a .zip or .rar (generally). If the mod file just replaces some textures or config files, it's a much smaller risk than if it replaces the game executable...
Will a decent AV with signatures/heuristics pick up a malicious payload?
Hopefully, but don't rely on it.
Can mods be examined in a debugger (OllyDBG) or other tool (which type)?
Again, this would depend on what the mod is and what it does. You can certainly check .exe files in a debugger, but frankly I just wouldn't run them at all. Or run them sandboxed in a VM if you really must run them to check them in a debugger.