I am trying to understand how custom malware unpacking. There are packers generally used but sophisticated malware use their own packer. How do i learn about packer briefly and how they work and how to unpack them to analyze malware?
Packers are pieces of software that apply different compressing techniques. But as far as software protection and malwares are concerned, modern packers are also capable of code obsfucation, executing code through virtual environments, detecting if the program is being debugged, or run under some sort of sandbox etc. They are trying to hide the code and make its analysis very complex. There are many packers that are ready-to-use, but most serious programmers, or malware creators make some of their own, so there isn't a standard procedure for unpacking. If you are into reverse engineering and you want to learn manual unpacking, try downloading some well-known packers and start working with them through tutorials available online. Experience is everything in this domain - and a good malware analysis toolbox with offline tools!