Comparing the Estimation Power of Machine Learning Models and Statistical Models in Predicting Profit Component Changes and Selecting the Optimal Model

The aim of predicting profit changes is to create awareness for investors, financial analysts, managers, stock market officials, creditors, and other users to judge the business unit, make decisions about buying or selling stocks, or granting or denying loans and credits. The goal of this research is to evaluate the performance and compare the accuracy of machine learning models and statistical models in predicting the direction of changes in three profit components including net profit (loss), gross profit (loss), and operating profit (loss).

In this research, using the financial information of 139 manufacturing companies listed on the Tehran Stock Exchange over a 15-year period, from 2008 to 2022, and employing 25 machine learning models and 10 statistical models, the efficiency of machine learning models and statistical models in predicting the direction of changes in profit components including net profit (loss), gross profit (loss), and operating profit (loss) has been compared. In the present study, Excel software was used for data sorting, Eviews software for extracting descriptive statistics, and data mining software SPSS Modeler and Rapidminer for predicting profit changes. The performance of machine learning models was evaluated using two criteria: accuracy (predictive accuracy of the model) and AUC (area under the curve), and the performance of statistical models was evaluated only by the accuracy criterion. Finally, in order to select the model with the best performance for predicting the direction of changes in net profit (loss), gross profit (loss), and operating profit (loss), the best model among the machine learning models was chosen using the ROC curve.

After calculating the average predictive accuracy of machine learning and statistical models, it was found that the average predictive accuracy of machine learning models for dependent variables including the percentage of changes in net profit (loss), percentage of changes in gross profit (loss), and percentage of changes in operating profit (loss) ranges from 83% to 93%. It was also found that the average predictive accuracy of statistical models for all three profit components varies from 76% to 83%. After confirming the non-normality of the average accuracy of machine learning and statistical models for profit components using the Kolmogorov-Smirnov test, the non-parametric Mann-Whitney U test was used to compare the predictive accuracy of machine learning models and statistical models in predicting the direction of changes in profit components.

The results of the research hypotheses test indicate the high efficiency of machine learning models in predicting the direction of changes in net profit (loss), gross profit (loss), and operating profit (loss), compared to statistical models. The ROC curve results indicate that the decision tree model achieved a predictive accuracy of 100% in forecasting the direction of changes in net profit (loss) and 99.38% accuracy in predicting the direction of changes in gross profit (loss). Additionally, the rule-based inference model demonstrated a predictive accuracy of 86.76% for forecasting the direction of changes in operating profit (loss). These models exhibited the best performance and were selected as the optimal models.