Web Anomaly Detection by Using Access Log Usage Profile

Due to increasing in cyber-attacks, the need for web servers attack detection technique has drawn attentions today. Unfortunately, many available security solutions are inefficient in identifying web-based attacks.
The main aim of this study is to detect abnormal web navigations based on web usage profiles. In this paper, comparing scrolling behavior of a normal user with an attacker, and simultaneous use of the access control policy alarms provided in web pages crawling with high access level, leads to an attacker to be detected among ordinary users. Indeed, the proposed method in this research includes two main steps: firstly web usage profiles are extracted as web main patterns of users’ behavior. In order to cluster similar web sessions we used a system inspired by artificial immune system. In the employed method, the rate at which a particular web page is visited as well as the time a user spends on the pages, is calculated so as to estimate how interesting a specific page is in a user’s session. Therefore, the similarity in the web page is defined based on the combination of the similarity of web pages URLs and that of the users’ level of interest in visiting them. Secondly, the difference between each current user session from the main profiles is calculated. Additionally, the access control logs are derived from corresponding sessions in this stage. Regarding the noisy nature of web server logs, a method was required so that a slight change in the data would not make a noticeable change in the results validity. Hence, a fuzzy neural network has been applied to distinguish normal and abnormal scrolling behavior in second step.
Due to the lack of a standard data that contains both web pages scrolling and access control logs corresponding to it, providing such a data was required. At first, those intended logs were produced. To do so, an Apache web server was run on the platform of a Centos machine. In order to create the logs completely similar to a real server’s log, an e-commerce website was set up on Apache server. This website had about 160 different web pages to be visited by different users. At this point, a novel method is proposed to simulate the behavior of web users when they visit a website. Likewise, the abnormal data was generated by means of a large number of existing attack tools. It should also be noted that the access control policy has been used is SELinux and It has been added to Linux kernel.
As mentioned, web server access log varies greatly with changing user behaviors, the stability of the proposed method against noise should be evaluated. For this reason, the results has been investigated on noisy profiles created by making random changes on the main profiles, and only the testing phase is conducted again. Subsequently, the distance from the profiles having noise is compared with the main ones. To demonstrate the ability of this method, the results have been compared with a Support Vector Machine (SVM). The carried out evaluations show that our approach performs efficiently in identifying normal and abnormal scrolling.