I don't know enough about C#'s serialization to know whether it is a security risk, but I can tell you this:
In Java, it is not safe to unserialize untrusted data. There are a number of subtle security pitfalls that can really screw you over. If you are guru-level, you can probably avoid the pitfalls, but an average developer probably has no clue about the dangers.
Here are some examples of subtle points:
If you want to deserialize untrusted data, you have to write special deserialization code to defend against, e.g., a malicious byte sequence that defeats your code's security invariant.
If you do any security checks in your constructor or factory methods, you have to duplicate them in special deserialization methods. If you call the SecurityManager in the constructor or factory methods, you have to replicate those calls in a special deserialization method.
There have been obscure security vulnerabilities associated with how deserialization was used in Java standard library classes. And then some more. And yet more. And even more. Are you bored yet or would you like another?
By the way, this example illustrates subtle and dangerous interactions between deserialization and the Java permission model (based upon stack inspection).
There are some crazy hairy security pitfalls associated with deserialization that (1) allow an attacker to mutate fields that should not have been mutable, or (2) allow an attacker to exploit concurrency/reentrancy bugs to break security-critical object invariants or learn secret values you shouldn't be able to learn, or (3) observe incompletely initialized objects (thereby allowing to mutate objects that were intended to be immutable).
Here are some more insane risks with Java deserialization.
If you fully understand all of that and can keep it in your head, you may be a good candidate to write secure deserialization code that can safely handle untrusted byte streams. On the other hand, if you are a mere mortal (like me) who throws up their hands in disgust at the whole thing, then it might be prudent to ensure that you never deserialize untrusted data.