Design and production of Persian news data set IHU-PersianNewsDataSetJavadzade-et-al Imam Hossein Comprehensive University

Although the lack of data is one of the important challenges for research in the field of natural language processing, but this challenge is more acute in the Persian language, so finding a high-quality and comprehensive dataset in the Persian language is a difficult task. In addition to that, having some problems such as the ability to categorize and not complying with the storage standard are among the problems of the existing datasets, each of which can affect the learning rate of the model, the results, and the error rate in the experiments. For this reason, all these reasons made us seek to collect and prepare a dataset that covers all such problems and reduces the amount of error when using data in different models. In this research, we have designed and used a crawler to collect textual data. By crawling on one of the news bases, it has been able to collect data sets in five columns: title, summary, text, tag, and publication date. The textual data has been normalized with the help of one of the Persian language libraries in the Python programming language and stored in csv and xml formats and made available to fellow researchers. The tags in this dataset include 13 main tags of sports, art and media, culture, science and progress, political, foreign policy, life, family, society, education and training, international, economic and provinces. Among the tasks that can be done on this data set are text classification, text extraction, text summarization and title recognition. Also, one of the prominent features of this data set is its comprehensiveness, the amount of suitable data, the existence of useful features, having unique features, as well as storage in a standard format. This dataset is a product of the Language Processing Department of Imam Hossein Comprehensive University and can be downloaded and used through the link mentioned in the footnote of the next page and with respect to copyright.