Proposing a process to integrate and identify repetitions to improve the quality of data

Recently, information in the workplace and decision has a major role. Due to the importance of deciding, it is also necessary to ensure data quality. Data quality can be improved by data cleaning methods. In this research, we propose a process for discovering duplications and contradictory types of records, integrating and identifying duplications to improve the quality of data. Our proposed process consists of different activities. These activities are coding records, clustering by expectation maximization algorithm, making token for records, integrate coding records methods and making token for records methods, and extracting association rules by Fp-growth Algorithm. The results of the tests show that the proposed process has averaged 96% recall, 99% precision, 95% accuracy and 95% f-score. The proposed method is compared with a duplication and error detection method. The results indicate an increase of 13% for recall, 1% for accuracy and 6% for f-score in the proposed process.