When reading the Difference between antivirus and sandbox?, I came up with the idea of how to integrate a sandbox with AV. When the sandbox analyzes the sample and finds that the malware, antivirus automatically called and removes the malware. How can I do this?
Actually, modern antivirus software actually already uses a sandbox. Many of them will execute suspicious files in their own built-in sandbox, all the while monitoring them for malicious behavior for a fraction of a second. If the software appears to behave itself, the program is re-executed with the sandbox disabled. Otherwise, the AV software will quarantine or delete the file. This is a common technique in heuristic antivirus software. Note that malware often has techniques to evade this detection method by running a benign task long enough for the sandbox check to pass.
Well antivirus do use sandbox's. (Most of them.) It can use a sandbox to unpack a file that is packed that it has not designed a packer for. Antivirus's also use emulation for tasks such as these. However, there are a lot of sandbox based bypasses. https://wikileaks.org/ciav7p1/cms/page_3375582.html It is not a good idea to only rely on results from running a file in a sandbox. AV's also monitor API calls etc of the program running in the sandbox...
You can design a sandbox using QEMU KVM etc... I personally used https://sandbox.libvirt.org/ to design a sandbox because I am lazy. But there do exist quite a bit of bugs for libvirt...https://www.cvedetails.com/product/20594/Redhat-Libvirt.html?vendor_id=25 But I would still use that before rolling out your own code unless you think your code will contain less bugs.
If you want to emulate a PE file use unicorn engine.
If you want to look over an open source project: https://github.com/cuckoosandbox/cuckoo