Firstly, anything you might try to do to detect a jailbroken system could be interpreted by Apple as an attempt by your app to violate the sandboxing conditions, which could lead to your app being rejected. For example, let's say you tried to detect a jailbroken OS by if (-1 != open("/dev/random", O_RDONLY)). Apple might assume you're trying to read from /dev/random and reject you.
Anything you might try to do to detect a jailbroken system could be patched by the jailbreak community. For example, they might patch their kernel to forbid your app from opening /dev/random, because they know you're only doing it to detect a jailbroken OS.
What else would you want to do? You don't trust the OS you're running on, so you give up running on it.
In short, jailbreaking is not your problem, it is Apple's problem. Allow Apple to solve their problems and concentrate on solving yours. (Incidentally I do wonder how people ever released PC software if they don't trust jailbroken iOS, when Windows and jailbroken iOS basically allow the same things to happen.)