I was reading material on secure coding and at many places it is mentioned not to use setuid and setgid bits for privilege escalation. These a vulnerable and can lead to security issues. Instead we should run a dedicated helper process which can do the job, like launchd on osx. SO my question is how is setuid and setgid insecure ? and how can it be exploited ?
It is not the bits themselves that are insecure, but if an attacker was able to exploit a flaw in a setuid/setgid binary it would result in code execution under elevated privileges. So by reducing the number or binaries (and lines of code) that run with elevated privileges you are reducing the attack surface.
The reason a software author would use suid/guid is because some program features require root access. Take for example ping utility. In order to do all that ICMP communication, a raw socket is needed. Opening a raw socket requires root privileges. So, the program is build as a suid/guid binary, it starts by opening the raw socket, and immediately after that it drops suid and guid privileges.
Running this kind of programs is fine as long as you can trust the people or the organization who wrote the program - for example you can trust the ping utility coming with a major Linux or Unix distribution. If you have the source code you can read it yourself.
Dedicated helper processes run as root, and have the same trust issues as suid/guid binaries. Look who wrote the helper and/or read the code.