Mostly, yes, this is about writing code and remembering not to do anything bad. At its core, the crypto-code writing process really is a combination of understanding and updated knowledge.
To write crypto code, you have to fully know how the code you write gets translated into assembly, then machine code, then electric signals; it can help to have some notions on how a transistor works. You do not have to develop in assembly, but you must be able to think about what your code becomes. Observing the assembly output from the compiler (assuming you work in C or something similar) is very instructive.
The point of this knowledge is that most fashionable attacks nowadays are about side channels, that feed on subtle differences in execution time or memory access pattern or power consumption of code. Programming languages are meant to provide an abstract view that hides all these details; the successful crypto coder is one which can undo that abstraction in his head as he writes the code. A crypto coder must master his programming language to the point that he could write the compiler itself (not that he will do that, be he must be able to).
Then you have to maintain yourself up-to-date with all known attack vectors, which means keeping track of all discoveries on that subject. Experimentally, I find that this very site (security.SE) and the endless ramblings of the chatroom users is a good method to pick up some information -- however, once you know that a potential attack exist, you have to dig up a bit, including reading the academic papers and abstracts, which requires, of course, skills at reading academic papers (the kind of skills that you acquire when you do a PhD).
A crypto implementation can be considered as good only insofar as it can be reviewed, so, regardless of how well you write it, you still need to document it heavily. Security cannot be tested, so some code can be considered as secure only if it can be fully understood. The "burden of proof" lies on the developer's shoulders: it is not sufficient that nobody found a way to break through the code (yet); the developer himself must explicitly demonstrate why his code is fine.
All of this is about implementation of an existing, standardized algorithm. For instance, you are writing an RSA code. The standard is very specific about where each bit should go, so your job is "only" to make a robust implementation that does not leak information about the key or the encrypted data (which basically means using random blinding, or constant-time code with a fixed memory access pattern). When considering embedded devices with an external, attacker-controlled power supply (e.g. smart cards), the problem becomes a lot more complex and can be defined only as "high-level research" (and smart card vendors employ high-level researchers exactly for that reason). For more mundane software platforms, this is doable by a lone programmer, provided that he masters his tools and knows how to use academic research.
Designing your own algorithms, or your own protocols (a protocol can be defined as an assembly of algorithms), is another can of worms. A big one.
I am not sure whether it is possible to sum up all the required knowledge for good crypto coding as a teachable set of rules. However, some people are trying. That Web site might not be the ultimate answer, but at least it contains lots of good advice and pointers, and can at least serve as a list of things to watch for. Don't try to write production-level crypto code until you can read the coding rules and at least understand what each listed problem is, where it comes from, how it could be exploited, and why you should mind it (or not) in your specific context.
Ultimately, skills at good crypto coding are built on experience so the best you can do in that respect is to begin by writing some implementations yourself, then compare them with other implementations and see in what aspects they differ and why. Hash functions like the SHA family are a good starting point.
