An approach to rootkit detection based on virtual machine introspection

Kernel rootkits have posed serious security threats due to their stealthy manner. To hide their presence and activities, many rootkits hijack control flows by modifying control data or hooks in the kernel space function pointers, especially those dynamically allocated from heaps and memory pools. These areas of kernel memory are currently not monitored by kernel integrity checkers. On the other hand, traditional host-based detection tools are executed inside the host they are protecting, therefore, since these tools are executed within the kernel, they could be easily detected by the rootkits. To solve this problem, current rootkit detection tools deploy virtual machine introspection technique that monitors the state of running virtual machine at hypervisor level, without rootkits interposition. The goal of this thesis is to present an approach based on virtual machine introspection, to detect rootkits which hide themselves and their associated malwares in the main memory using system control flow modification. The proposed approach monitors the integrity of windows kernel function pointers that are potentially prone to malicious exploits, based entirely on virtual machine introspection. This approach is evaluated with a set of rootkits which use advanced hooking techniques and it is shown that it detects all of the stealth techniques utilized