It depends a lot on the software and SSO protocol. There are a wide variety of "keys" and "secrets" in play when dealing with SSO.
For the sake of contrasting different keys, consider your example of a smart card. Usually this is used for authenticating the user. A system would generally know about the user's public key already, and issues a challenge to the smart card which it digitally signs and returns to the sign-on system. That system can then use the previously known public key to verify the user's digital signature. Only the smart card can produce a valid digital signature because only it holds the private key.
SSL is not a form of SSO, but for the sake of conceptual understanding, the above is similar to how SSL works. There are two aspects of SSL, encryption and authentication of the remote server. Lets' just speak in terms of authentication. When you go to stackexchange.com, how do you really know that it is not a fake server pretending to be Stack Exchange? SSL involves the server holding the private key, and you are able to verify the digital signature it produces with that private key, because you hold the public key of the authority that issues the certificate(such as Verisign, which is installed in your browser). There's a lot more to it, but usually on the server side you install your private key certificate into a certificate store which is only accessible by administrators with sufficient access to the server. The actual implementation of a certificate store is dependent on the operating system we are talking about.
(While I stated SSL is not SSO, it is often used are part of the SSO process to secure communication between parties.)
In SSO, often there are 3 parties involved. The user who wants to login, the server who knows who that user is, and the server who wants to verify that user's identity(the server the user wants to log in to). Depending on the particular SSO implementation, one or both servers may store one or more shared secrets that are used in the process to allow the servers to verify that one-another are authenticate, or sometimes used as part of a key generation process to establish an encrypted connection.
I say all this to emphasize that there is not always a stored symmetric key. Some processes use asymmetric keys to generate a symmetric key on the fly, so you don't have to store a symmetric key. Some process use a shared secret as part of a hash to generate a post from the user, which prevents tampering with the post. In such an SSO scheme there may be no need for a symmetric key at all.
Where are symmetric keys stored?
Often times if you are the programmer implementing one side of an SSO process, then it is up to you how you store shared secrets and/or keys. It is often best to do so in a way the minimizes any potential access from employees. Follow the best practices of your platform. For example, if certificates are involved, then store them in the OS's certificate store and ensure only the most minimum number of employees have permissions to that certificate store. Ensure your server's are physically secure. There are a lot of best practices that assume your servers are physically secure. Unfortunately there is no perfect answer when it comes to securing keys on a server, because usually the software involved in the SSO process at some point has to access the key as part of the process, meaning the software must have permission to access the key. Keys/certificates can be encrypted with a password, but every time the server restarts you'd have to re-enter the password.
C# apps, for example, provide ways to encrypt configuration data, where you could store such keys. But the encryption uses another key stored on the local machine, so anyone with sufficient permissions to the machine, and/or physical access, can with enough effort decrypt that config and get the shared secrets.
Hardware Security Modules (HSM)
I do not know much about these, but the idea is that the key is stored in this piece of hardware, and does not expose the key through any network interface. Instead you send requests to the HSM for the steps which involve the key, and it performs whatever crypto processing is necessary and returns the result. For example, during SSL when the certificate's private key is used to generate a session key. It is usually in the form of a rack mount piece of hardware, or some sort of PCI card. The impression I get is they are only useful for certain functions, since they have to perform all of the cryptography functions, then if you are doing something custom this probably won't work for you.
Very similar in concept to a smart card, in that the key never leaves the device and performs the request crypto operation. But of course providing many more functions besides just signing, and usually with alot of additional security features as well as accelerated performance. That is based purely on what I have read about them though, so don't consider that an endorsement. In theory though they sound like a great idea and seems like mission critical operations like banks/finance use them.