It is possible to access the flash memory of a smart card. You can find some details here, or in its bibliography. For feasibility, though, you'd need a very resourceful and well-supplied attacker (State-operated, probably, or a well-endowed university research team). The most promising known technique so far has been developed by Sandia National Laboratories.
But there's another problem. The private key on the certificate is (at least it ought to be) encrypted, using the encryption chip on the card. That's why you need to enter a PIN to unlock the certificate and use it to e.g. sign a document.
So what the attacker needs to do is extract the private key in encrypted form, then run a brute force attack (which requires to know the smart card's encryption algorithm and the key-expansion protocol used by the host driver to generate the real decryption key from the PIN). It's not all that much of a complication for someone who just recovered a flash memory chunk through tunnel microscopy or ion-beam analysis, but it is an added complication all the same.
Of course, you might have a "weak security" smart card that does not encrypt/decrypt anything, and merely verifies the PIN and gives or withholds access to its certificate memory. From the outside, a customer couldn't tell the difference. And in this case, reading the memory will yield the certificate without further hassles. But I believe that at least reputable suppliers like Gemalto (Thales) employ both internal encryption and several undisclosed design tricks to make life difficult for inquisitive microscope guys, so to answer your question ("How safe..."), I'd go with very.
