Improving Omniunpack Alghorithm in Generic Unpacking PE File with Page Monitoring

Analysts used file signature comparison to detect malware and analyze the behavior of the executable file in the past. To prevent signature examination, the authors of the new and advanced malware used obfuscation methods to hide information, of which packaging is the most important and the most efficient one. This method encrypts and compresses the code, without harming the behavior of the original executable file and the code is obscure until it is executed. The methods that are now used to unpack these files are often methods that are specially designed for each type of packer for that file. There are other methods, such as Renovo and OmniUnpack for reopening that are known as public reopens, and actually cover the weakness of previous approaches which is the need to know the type of packager, but their main problem is finding the original point. The main entry of the program is the end of the unpacking section. Our approach to fix this problem is a method that detects this point using tracking memory pages and monitoring pages, then executes it, and then dumps the memory for creating a new file that has been unpacked. Our method has two advantages: first, there is no need for knowledge of packaging type, and second it can also be used for packers that are created in the future. Finally, in the evaluation section, we have shown that this method has a very high performance for current packers and more than 90% of them can be unpacked with it, so it can be used on an antivirus engine.