In short: If you really care about stuff like that, go into a retail store and buy a new router that's on stock. The risk is small, but you can't easily mitigate it.
I could imagine some creep buying lots of routers, returning them and then spying on the people who bought them just for the kick of it. Or of course some evil organization. This risk is small, sure.
But I'd like to doubt the claims of the other answers about "just" resetting the router. Sure, reflashing the firmware should erase pretty much every bad thing that might be on it. But how would you do that? You can't use the web interface (that would be the first thing someone would disable/fake) and a physical button probably also just sends a signal to the current firmware, that it should reset itself. Serial transfer is also handled by the current firmware I would expect.
Unless you are going to reflash the firmware using a JTAG interface that might or might not be there (or something equivalent) then I'm pretty sure resetting isn't much better than just trusting the device to be basically ok (of course you should reset it anyway to get rid of the settings of previous owners).
And I don't know enough about JTAG to assess its security, the only thing I'm sure about is that it's less trivial to fake than the web interface/button.