In "pure X.509", it does not really matter if an extension is critical or not, because conforming implementations are supposed to honour the extensions that they recognize, be they marked critical or not. The "critical" flag is for extensions which are not standard: you make such an extension critical if it is important for security (implementations which do not understand the extension should reject the certificate), or non-critical otherwise (implementations which do not understand the extension can then safely ignore it).
There is a slight exception to this rule, with the
CRL Distribution Points extension. It has two purposes: to document where CRL can be downloaded from, and to implement scope segmentation. This latter role is active only when the extension is critical. When the extension is critical, then a CRL can be deemed to cover the certificate (i.e. to be able to tell something about its revocation status) only if the CRL contains an
Issuing Distribution Point extension, with a "distribution point" which matches one of those specified in the
CRL Distribution Point extension in the certificate. When the extension is not critical, the extension serves only in its documentation role.
In practice, you will make extensions critical or not depending on whether they can be ignored without piercing too big a hole in the whole security, and also depending on whether making the extension critical will induce existing implementations to reject the certificate for lack of support. For instance, if you use a critical
Name Constraints extension, then you risk unconditional rejection from old versions of OpenSSL (versions prior to 1.0, all of which are EOL, don't support it); but if you make it non-critical, then the same OpenSSL will ignore it. The
Name Constraints extension is a typical case of an extension which cannot be ignored safely, and thus should always be marked critical (but interoperability issues make its usage potentially problematic).
RFC 5280 lists, for each certificate extension, whether a conforming CA should make the extension critical or not. These are requirements on the CA and not on validators; a system which validates a certificate must not reject it on the basis that it includes a critical
Subject Key Identifier extension, even though RFC 5280 says (section 126.96.36.199):
Conforming CAs MUST mark this extension as non-critical.
See the RFC for the details on every extension: these are guidelines for how your CA should behave. For instance,
Key Usage is a "SHOULD be critical",
Basic Constraints is a "MAY be critical",
Name Constraints is a "MUST be critical", and so on... If you follow these rules, your security will be fine (but you may have to make some modifications for interoperability).