As long as things are setup correctly on the server, PHP files should be registered as scripts and the web server should have them interpreted by PHP when requested and only display the results of that interpretation.
That said, any number of issues can result in files being exposed. Some of these issues can also expose data regardless of if they are in a public folder or not. It is always important to make sure your server is properly configured to only allow the requests you need allowed. This reduces the surface area available to attack and helps avoid possible bug related issues that could result in a breach.
Is it a good idea to have a config file in a public folder? As long as the server is configured not to give out the file without processing it, it probably isn't much less secure than any other spot on the system. There is the small chance of a bug in the web server being used to prevent execution by the scripting engine, but the more likely attacks are attacks that would come from some other direction like SQL, FTP or some code injection where being in a private folder would be equally exposed.
That said, the flip side of the question is why not put it somewhere else. The most secure option would be to put it someplace that only the user that the web site's PHP instance runs as can access and deny access to the file from any other mechanism (such as the FTP user or any other publicly used users.) This is rather difficult to configure and manage however, so a decision has to be made if the additional security is necessary or not.
It's a toss up on which is best. It's a lot of extra work to manage all the paths, permissions and users to maintain that level of security. On the flip side, as long as the server is kept patched and properly configured, you should only be vulnerable to zero day exploits that attack at a very low level and can be safe against pretty much all common attacks, even with the config file in the public folder.