Automatic Data Cleaning using Functional Dependency and Ensemble Learning

Data accuracy is one of the important aspects of data quality. According to high volume of data sources an automatic method is required. In this article an automatic method is proposed for cleaning of data with various data types. Initially, records that may contain incorrect attributes are detected using functional dependency, so that each record that inconsistent more than records for one functional dependency, probably is error. Thereafter ensemble learning is done for each attribute of data source. Ensemble learning contains 3 classifiar naïve bayes, decision tree and MLP, and voting is combination strategy. It is trained using correct records that identified in the initial steps. After training, each incorrect attribute is placed as target and predict values for it. Proposed method is able to clean data in data sources with different data types. Experiments show the true negative rate in detecting error part of the proposed algorithm is averagely 93.7% and in cleaning error part is averagely 90.6%. Also experiments show that evaluation parametrs are improved in proposed method compared with 2 similar methods.