In "pure X.509", it does not really matter if an extension is critical or not, because conforming implementations are supposed to honour the extensions that they recognize, be they marked critical or not. The "critical" flag is for extensions which are not standard: you make such an extension critical if it is important for security (implementations which do not understand the extension should reject the certificate), or non-critical otherwise (implementations which do not understand the extension can then safely ignore it).
There is a slight exception to this rule, with the CRL Distribution Points
extension. It has two purposes: to document where CRL can be downloaded from, and to implement scope segmentation. This latter role is active only when the extension is critical. When the extension is critical, then a CRL can be deemed to cover the certificate (i.e. to be able to tell something about its revocation status) only if the CRL contains an Issuing Distribution Point
extension, with a "distribution point" which matches one of those specified in the CRL Distribution Point
extension in the certificate. When the extension is not critical, the extension serves only in its documentation role.
In practice, you will make extensions critical or not depending on whether they can be ignored without piercing too big a hole in the whole security, and also depending on whether making the extension critical will induce existing implementations to reject the certificate for lack of support. For instance, if you use a critical Name Constraints
extension, then you risk unconditional rejection from old versions of OpenSSL (versions prior to 1.0, all of which are EOL, don't support it); but if you make it non-critical, then the same OpenSSL will ignore it. The Name Constraints
extension is a typical case of an extension which cannot be ignored safely, and thus should always be marked critical (but interoperability issues make its usage potentially problematic).
RFC 5280 lists, for each certificate extension, whether a conforming CA should make the extension critical or not. These are requirements on the CA and not on validators; a system which validates a certificate must not reject it on the basis that it includes a critical Subject Key Identifier
extension, even though RFC 5280 says (section 4.2.1.2):
Conforming CAs MUST mark this extension as non-critical.
See the RFC for the details on every extension: these are guidelines for how your CA should behave. For instance, Key Usage
is a "SHOULD be critical", Basic Constraints
is a "MAY be critical", Name Constraints
is a "MUST be critical", and so on... If you follow these rules, your security will be fine (but you may have to make some modifications for interoperability).