Anomaly Topic and Emerging Topics Discovery Using Social Media

Author Name(s): Yogita P. Shewale, Harshal Kumar R. Khairnar
Author Email:


Anomaly detection is an approach to detect anomalies from high dimensional discrete data. Several approaches for anomaly detection have been proposed which is only capable of detecting individual anomaly. It is very time consuming and infeasible task. With proposed ATD approach group anomalies are detected. Some techniques used all features for anomaly detection which get fail. In our system, batch of text documents are given to discover anomalies therefore, topic based algorithmic approach is utilized. With group anomalies detection, emerging topic discovery by extracting links between social users is contributed in proposed system. There is large growth in social medias detecting the latest trending topic from social medias links are receiving interest , conventional methods link text mining and text-frequency because the data is not in a social network post including images, URL’S and video so focusing on the emerging topics we required the user- links on social medias on the behavior of user which they comments on social networks basis on that be can find the anomaly that not match with the regular environment so that anomaly can comes in trend when it finds some link with recent trending environment, we calculate the anomaly score from various user which are use social medias the data set of social media may be large we need to consider social posts the datasets gathered from Facebook or twitter. The post which is consider as anomaly have time span of 30 days to be an emerging trend.


In proposed anomaly detection system, patterns which exhibit abnormal behavior get grouped into clusters. Anomaly pattern do not tune normal behavior. AD has several applications in credit card fraud detection, insurance fraud detection, network intrusions. Traditional approaches are only capable of detecting an individual anomaly from given input. Therefore, Anomalous Topic Discovery (ATD) approach is proposed. It contains two phases such as training and testing phase. In training phase, Parsimonious Topic Model (PTM) is used. Rather than LDA model, PTM model is used to find accurate frequent words (salient words) which have accuracy & estimate the normal topics from test batch. In PTM, normal data is extracted and used to construct null model whereas, in anomaly detection phase, null model is used as, reference model to identify group or clusters of anomalies from test batch of documents. System works on training corpus to detect normal topics based on PTM model technique. Pattern matching and group anomaly in cluster is then carried out into testing phase. In testing, similar documents based on similar patterns are club into clusters hence unusual or anomalous topic remains into side. In this process, topic relevance score is calculated. In each step of proposed algorithm candidate anomalous cluster (S) is detected which exhibits maximum “deviance” from normal topic. Cluster significance is calculated to get d*. d* is candidate document belongs to S. Bootstrapping algorithm is utilized.


We proposed ATD approach to detect cluster of anomalies from input dataset. Traditional approaches of anomaly detection such as, MGMM and FGM can efficiently works on high density dataset. But it can only detect individual anomaly from huge data input which is infeasible task. Hence, proposed approach mainly aims to discover group anomaly. PTM model is utilised for normal topic discovery in training phase whereas, in testing phase, it is used to construct M1. Anomalies are nothing but abnormal patterns, in cluster formation relevance score is used for construct anomaly cluster’s. With proposed work system contributes emerging trend detection. With experimental set up, proposed system proves it’s efficiency in terms of accuracy.


96 total views, 2 views today

Download PDF File

About the author: admin