Like any software you need to harden it to only allow the user what you intend to allow him. In your specific question it really relies on what language you have programmed your bot in, what types of commands you allow it and how it sanitizes the input from the user.
I am not familiar with any books directly targeting security of an IRC bot, however there should be many clear guidelines on how to program safely in the programming language of your choice.
It is also very relevant to read the RF1459 on how your bot should behave when communicating over IRC.
When you initially set up your IRC bot make sure that the core functionality is in place and working properly. This is the simple stuff like:
- replying to ping with pong
- join channels
Then you need to dissect these operations and make sure that they are inherently secure. Does your bot reply to a random PING from anyone? What if I supply a buffer overflow attack against your bot with a "PING AAAA x50000"? Does it try to store it in buffer and copy it to the PONG reply? Perhaps the only source allowed to make a PING to you should be the server your connected too.
When you later make the bot to accept commands from anyone you probably want to whitelist (positive security model) the commands you allow (join, kick, ban, ping, ??) and then sanitize properly the input to the bot.
The sanitation that needs to be done is very dependent on what your bot actually does. For instance:
When you design such a bot you have to try think of every possible way someone will try to exploit it. You have to be very careful and paranoid when creating it.