Web Usage Mining: Dynamic Methodology to Preprocessing Web Logs

Author Name(s): Mahesh Manchanda, Dr. Neena Gupta
Author Email: manchandamahesh@gmail.com


Internet is a huge source of massive information for retrieving information and searching knowledge from WWW, leading to increase network traffic, access delay & server overload, which results in poor web services. With the use of Web-caching & web prefetching techniques to enhance the performance of web services where web mining techniques play an important role to decide which web object should be pre-fetched from server and stored in proxy cache memory so that the web object with high probability of request, in the next couple of days, serves as the base of the proxy cache. But for efficient web mining and to extract meaningful usage access pattern, the raw log file must be transformed into a meaningful & formatted file. This paper proposed a new dynamic preprocess technique to create a dynamic training dataset for prediction model using web mining, and Graph based substructure Pattern Mining (GSPAN) for improved preprocessing using proxy log. The proposed model would help in minimizing the cache size by 40% thus improving the overall performance.


The dependency on the web services for day to day usage has swiftly increased in the last two decades. Web services are frequently used for many purposes like shopping, banking, filling application forms, exams etc. User access to these services recorded in log files stored in three places client browser, proxy server and finally in origin web server. These different sources of logs have its own advantages and limitation but their importance to record the user access event for web usage mining is pertinent [1]. Genuine access pattern of individual client can be depicted from browser log document [2]. It is as oneto-numerous connections of customer and sites went to by that specific client. Proxy log also contained user access entries in log file. It is in many-tonumerous relationships. On the other side, Server log files having numerous-to-one relationship. Server log recorded the access pattern of several users visiting a particular web site. Server log caters only to a group of users accessing the same web site. Real motivation behind choosing the proxy server log file for creating a training dataset for our new dynamic model, since proxy server is an intermediary server, sits between the client (browser) and the origin Web server. As the proxy server resides on the same network as the user, resulting in a much faster load page, reduction in bandwidth usage, network congestion and enhance the performance for group of users. In Literature review, we notice that maximum of researcher considered server log for mining purpose but it is also evident that server log does not record the cached pages’ requests hence not considered for web mining[4]. The growth of web services and excessive use of these web services increase the size of web log file rapidly. The data which is stored in web log files will be consisting of huge amount of information with some kind of incomplete, noisy and unwanted data too. It is difficult to deal with entire data which is huge in size. So, unwanted or irrelevant data must be removed by using efficient preprocessing technique. Thus, preprocessing plays a vital role in web mining to reduce the size of a log file by removing irrelevant entries from the log files. The quality preprocessed log file decides the quality of the accurate pattern analysis required for efficient Web prefetching. The process of discovering hidden knowledge from log files and finds the interesting patterns is known as web usage mining.


Preprocessing of web log file is important and a required task before filtering web log. Removing irrelevant log entries before preprocessing is a firm base for identifying used access behavior. We have used java based pattern matching technique to remove and filter the irrelevant log entries. By applying efficient preprocessing method we can ensure the quality of pattern mining and pattern analysis. For future work we should explore Dynamic, Robust and customizable preprocessing techniques, consider all the attributes of web log file, and apply to all types of web logs to discover the quality usage pattern. The authors are working on designing a Machine learning algorithm for the entire process so that during certain intervals the process automatically cleans and updates the log files.

447 total views, 1 views today

Download PDF File

About the author: admin