Your question seems to be both about, what does Facebook do, and what could be done. I don't know what FB does, but I can try to make some good guesses at what makes a secure system. I hope that helps.
Let me try to rephrase what Facebook is doing. Sites typically support a single factor of authentication, something you know (ie: your password). Sites that support a "remember me" functionality, use the something you have factor (the "something" is the cookie in your browser). This can be weaker as anyone with access to your browser, even a remote attacker, can now log in as you. But not having to type your password every time you access a site, especially if you have a strong password, is a hassle. So Facebook is choosing a middle-ground, basically having a two-factor "remember me" feature where you need the nonce (something you have) and your PIN (something you know).
- What might the server be storing? A hash of the nonce + PIN?
It is very likely that Facebook is storing a hash of the nonce+PIN. Note that this could be a simple, secure hash instead of something like the more complex bcrypt password storage function. A hash such as SHA512 is irreversible when operated on large, cryptographically random data strings. The nonce is (presumably) such a random number.
- How might the server be validating PIN login attempts?
The server simply redoes the hash and compares it against the appropriate field in the database.
- Are there any reference implementations for a system like this that I could look at?
Not that I know of, maybe others can help.
- Assuming that the content stored in the browser is not taken by an attacker, is it possible for this system to be at least as secure as traditional username/password authentication? If not, what kind of attack would it be more vulnerable to?
If we make that really big and unfounded assumption, this system is more secure than regular uname/pwd as this has the effect of making the password the concatenation of a cryptographically secure random number, the nonce, and the PIN. Cryptographically secure random number passwords are very strong.
But, in the real world, browser data can never be assumed to be secure so the security of this mechanism boils down to the security of the PIN and nonce. That is difficult to put concretely. This is definitely more secure than simple "remember me" functionality. If your browser data is reasonably secure, this may be more secure than standard passwords. Otherwise, standard passwords may be more secure. In almost any situation, this beats weak passwords because of the something you have factor.
Some points of note:
- The server must perform the hash. If the client were to do the hash, then a stolen database table could be used as login credentials. This can't happen with the server performing the hash because the hash is irreversible.
- Extra security could be added by using a password storage function such as bcrypt instead of a standard hash. This is only needed if you use a shorter nonce. Longer nonces will provide sufficient protection against a brute-force attack, even with a simple SHA512 (or similar) being used.
- I think it makes sense to record failed attempts at PIN logins and, after say 3 failed attempts, delete the PIN login from your database and force the user to login with their password. This provides a lot of protection against a dictionary search for the PIN, even in the face of a leaked nonce, as the PIN will be rendered useless. Even if the user reuses the same number as a new PIN on their next login, they will get a new nonce that an attacker will need to steal to try more PIN possibilities.