Replacing kappa index with quantitative and spatial agreement and disagreement components for the accuracy assessment of different thematic maps

In the last few decades, thematic maps and models were usually assessed using Kappa index of agreement. The index gives us the relative observed agreement among raters (identical to accuracy), but lacks any useful information to make practical decision making about the model’svalidity easier. In other words, Kappa index does not provide an explanation about classification quality or an idea about increasing theaccuracyof the predicted map. Moreover, the index does not explain the causes of disagreement.Thus, giving indices of agreement without any interpretation will not be satisfactory. Today, new complementary methods are required to show the quantitative and spatialagreement and disagreement between two maps. It is necessary to show how a modeled map can be produced with better accuracy. The present study seeks to introduce and explain concepts of agreement and disagreement components with an example. Finally, these components are introduced as a useful method for the validation of digital maps.  

An area of 410 hectares which belongs toZanjanUniversity was used to express the findings of this study. The area is located 5 km before the beginning of Zanjan-Miyaneh Road at 48.4° eastern latitude and 36.68° northern longitude. A digital soil mapin which probability distribution of different soil classes is obtained using multinomial logistic regression algorithm and a reference soil mapproduced with the conventional methods are usedto explain the concepts and investigate the spatial and quantitative agreement and disagreement indices. Validation and calculation of quantitative and spatial agreement and disagreements are performed using IDRISI software (SELVA version). To simplify the process, two maps with a grid structure (3 x 3) are introduced as a reference map and a predicted map. The reference map is used for spatial and quantitative evaluation and validation of the predicted map cells. Each map contains 9 cells and each grid cell has a membership value of either white or gray categories.  

In the validation process of two maps, most researchers seek to find answers to two important questions: 1- How much agreement is there between the cells of each mapping class group? And 2- How much agreement is there between the map used in modeling and the reference map regarding the position of the cells in each class? The present study expresses agreement between the two soil maps using an index of (M (m)) which equals 60.69%. With an average level of quantitative and spatial information about different classes of the digital soil map (DSM), the H (m) index equals46.4%. Results indicate that if the produced map is modified or rearranged (provided that the level of quantitative information remains unchanged but the amount of spatial information increases), the agreement between the maps increases dramatically and reaches 87.17%. Quantitative and spatialagreement and disagreement between the digital and traditional soil maps also equal 61% (M(m) = 61%) and 39%, respectively. The DSM accuracy can be increased to 87% (P (m) = 87%) compared to thetraditional soil map through spatial modification of cells(without changing quantitative information).  

Evaluating the accuracy and validity of digital maps are considered to be an important and sensitive stepof research projects. Therefore, introducing more accurate indices is very important. According to the results of the present study, displayingquantitative and spatialagreement and disagreement in the form of a matrix and according to the different levels of quantitative and spatial information can be a new strategy to verify modeling methods. The method presented here not only introduces and interprets sources of (quantitative and spatial)error, but also provides information on the possible ways of reducing these errors. Thus, introducing the amount of error without any scientific interpretation cannot be useful for predicted maps. Unfortunately, researchers does not concur on how to report agreement and disagreement. However, it seems thatwhen it comes to explaining errors and finding a method to reduce such errors,the components of disagreement and its related parameters are more useful than agreement component and its indices. Therefore, it is recommended tointerpretdisagreement components before other components of agreement. The advantage of this method is that complex analyses can be reported in a simple form. Finally, this assessment and validation method is expected to be used in different studies as an appropriate and alternative method.