If you have a HSM then the pepper should be in that HSM, because that's the whole point of having a HSM.
Otherwise, the pepper is a key which should be handled as such. Having a key involves key management; it should not (and, really, must not) be hardcoded in the source. The source code is copied in multiple places: developer's machine, versioning systems, many backups... The key really is a property of the server instance (lone server or set of servers that use the same set of registered users) while the source is, conceptually, shared between many server instances.
On Windows, the DPAPI is what will make auditors most happy, although one may say that the extra protection offered by DPAPI is not, in fact, that big. If you reboot your server, it starts automatically; this means that regardless of where you put the pepper, the cold storage of the server (harddisk contents) is enough to recover the pepper. The pepper is meant to resist partial read accesses by attackers (which is why you do not put it in the database), but if the attacker can read the registry, then he has a lot of control over the machine and probably won't be much hindered by DPAPI either. If you use the registry, adjust access rights on that specific key so that only relevant users may read it (typically, the user account under which the server runs).
Using a file incurs the risk of read access through a poorly developed script that ends up dumping contents of attacker-chosen files.
On Linux there is no registry and no DPAPI, so use a configuration file. A case could be made of running the pepper processing in a distinct process accessed through some inter-process communication mechanism (say, Unix-level sockets), and run the actual server code in a chroot.
In all cases, using a pepper has its own costs: you have a key to manage, i.e. to generate securely, and keep safe. If you lose that key, then you lose the ability to verify user passwords, which can be very inconvenient. In that sense, the pepper is more critical than the server private key for its SSL certificate.