Using Evolutionary Clustering for Topic Detection in Microblogging Considering Social Network Information

Short texts of social media like Twitter provide a lot of information about hot topics and public opinions. For better understanding of such information, topic detection and tracking is essential. In many of the available studies in this field, the number of topics must be specified beforehand and cannot be changed during time. From this perspective, these methods are not suitable for increasing and dynamic data. In addition, non-parametric topic evolution models lack appropriate performance on short texts due to the lack of sufficient data. In this paper, we present a new evolutionary clustering algorithm, which is implicitly inspired by the distance-dependent Chinese Restaurant Process (dd-CRP). In the proposed method, to solve the data sparsity problem, social networking information along with textual similarity has been used to improve the similarity evaluation between the tweets. In addition, in the proposed method, unlike most methods in this field, the number of clusters is calculated automatically. In fact, in this method, the tweets are connected with a probability proportional to their similarity, and a collection of these connections constitutes a topic. To speed up the implementation of the algorithm, we use a cluster-based summarization method. The method is evaluated on a real data set collected over two and a half months from the Twitter social network. Evaluation is performed by clustering the texts and comparing the clusters. The results of the evaluations show that the proposed method has a better coherence compared to other methods, and can be effectively used for topic detection from social media short texts.