The W3C security instructions for this are surprisingly accurate and complete:
Authors should check the origin attribute to ensure that messages are
only accepted from domains that they expect to receive messages from.
Otherwise, bugs in the author's message handling code could be
exploited by hostile sites.
Furthermore, even after checking the origin attribute, authors should
also check that the data in question is of the expected format.
Otherwise, if the source of the event has been attacked using a
cross-site scripting flaw, further unchecked processing of information
sent using the postMessage() method could result in the attack being
propagated into the receiver.
Authors should not use the wildcard keyword (*) in the targetOrigin
argument in messages that contain any confidential information, as
otherwise there is no way to guarantee that the message is only
delivered to the recipient to which it was intended.
Authors who accept messages from any origin are encouraged to consider
the risks of a denial-of-service attack. An attacker could send a high
volume of messages; if the receiving page performs expensive
computation or causes network traffic to be sent for each such
message, the attacker's message could be multplied into a
denial-of-service attack. Authors are encouraged to employ rate
limiting (only accepting a certain number of messages per minute) to
make such attacks impractical.
The most important thing is to always validate the origin - you'd be surprised how many sites fail to do so. If the message is confidential then the targetOrigin argument must be specified.
This of course assumes that the browser implementation is correct and without holes. The major browsers have been evaluated and no issues have been found.
See also this Checklist for postMessage Security Review.