An Improved Approach for Loading Changes into Data Warehouse

Extraction, transformation and loading (ETL) of data sources changes are an important subject in the data warehousing field. During ETL process data are extracted from operational data sources, and after the extraction, these data can be transformed and loaded into the data warehouse. Because operational data sources change over time, it is required that the process of the data warehouse refreshment be done in the maintenance phase and changed data loaded into the data warehouse. Incremental loading is a widely used method to load data in data warehouses from the respective source systems. The main problem of this method is that if the number of changes increases then it will become time consuming. Usually, these processes must be completed in a certain time window; thus, it is necessary to optimize their execution time. The main purpose of a proposed method in this paper is to optimize time of changes loading by reducing number of log records and changing possible delete and update operations into insert operation in incremental loading. According to the results of tests conducted by real and test data and the comparisons, a significant improvement has been achieved due to the proposed data loading method of this paper (53.02 on real data).