DyVSoR: Dynamic Malware Detection Based on Extracting Patterns from Value Sets of Registers

To control the exponential growth of malware files, security analysts pursue dynamic approaches thatautomatically identify and analyze malicious software samples. Obfuscation and polymorphism employedby malware make it difficult for signature-based systems to detect sophisticated malware files, the dynamicanalysis or run-time behavior provides a better technique for identifying the threat.In this paper, a dynamic approach for extracting features from binaries is proposed. Run-timebehavior of the binary files were found and recorded in a controlled environment tool developed in-house.The approach based on DyVSoR assumes that run-time behavior of each binary can be represented by thevalues of registers contents. A method to compute the similarity between two binaries based on their registersvalue sets is presented. To do, registers values are traced before and after invoked API calls in eachbinary and mapped to some vectors. To detect an unknown file, it is enough to compare it with datasetbinaries by computing the distance between registers content of this file and all binaries. This methodcould detect malicious samples with 96.1% accuracy and 4% false positive. List of execution traces anddataset can be found at: http://home.shirazu.ac.ir/~sami/malware.