When reading about how antivirus and sandbox work, my understanding is that an AV scans a file to see whether it matches with known viruses signatures. Sandbox can be used to obtain the behaviors of a file when it runs. However, I don't know how sandboxes can be used to detect unknown malware or not. Could you please let me know this? Thanks!
AntiVirus (AV) software operates based on the idea that you can decide what is bad, detect which programs do bad things, and kill/uninstall them. Reactive security systems like AV software require very good knowledge about the threats you're facing, or about the difference between malicious and normal behaviour. That makes them costly and imprecise.
Sandboxes operate on the idea that you cannot decide what is good or bad, but the user can decide what they choose to trust. They provide you with the ability to confine or isolate specific programs so that if they do bad things, it doesn't do any harm to things outside the sandbox. The downside of confinement systems is that there must be a logical and simple way for the system or for its users of deciding which programs get confined where, and which are also to talk to each other.
These are completely different concepts, and the two families of tools can be used in conjunction. It's the case on OSs like Windows where store applications all run within a sandbox, and AV software can be used to detect modifications to the system or to detect the installation of a known piece of malware (which might or might not be sandboxed).
From a theoretical perspective, the use of confinement transforms the problem of classifying / making sense of application behaviours into a problem of deciding what security properties hold given a set of arbitrary boundaries between untrusted actors. As far as I'm concerned there's ample experimental evidence that the former approach is doomed to fail, based on the poor performance of AV software and the role of contextuality in deciding whether any given program behaviour is malicious or not.
These are two quite different things. A little simplified:
An AV is a piece of software that can (among other things?) scan your system to identify and attempt to isolate and remove threats like viruses or other malware.
A sandbox on the other hand, is basically a context in which a piece of software can be run isolated from the rest of the world. Java applets running in a browser is a classic example, as should be Flash (though it is apparently not nearly as well isolated and safe as would be ideal): These are contexts where programs can run without having access to resources on the host machine (your PC), such as your file system, etc.
Another example can be a set of virtual machines. You could build your own "virus lab" by setting up several VM's running different OS's, then linking them together in an internal network, which will only be visible to those VM's. Now you could experiment by running malware on these machines, and seeing how it affects them, and how they affect one another, without affecting anything outside of their virtual network.
Another way to think of think of a sandbox, is as (ideally) a kind of digital software aquarium.