A GPG_AGENT is a program that when asked, (de|en)crypts the data for the caller (in this case using gpg). The 'files' you see are most probably sockets for communicating with the agent.
Once the program is closed (or locked), it can't be used, those files don't contain any private information, agents are (generally) pretty-well designed, and won't even store private material out of ram (in most cases, the OS may not allow memory locking, or if you hibernate the computer, the keys can go into disk).
In regards to your question
Also, what is the correct procedure to clear/stop the keyring agent and thereby make it forget all the passphrases it is remembering.
You don't specify which agent you are using, the basic approach is to kill the agent process, but it will probably support unloading the keys / being put into locked state. See also
How to lock a unlocked GNOME Keyring?
¹ See if the first mode character is
srw------- 1 indiajoe indiajoe 0 jun 8 21:57 agent.1234