The article proposes the use of the permutation method for assessment of the predictive ability of models based on artificial neural networks. To test this method, three models based on artificial neural networks were implemented: a multilayer perceptron, a radial basis function network, and a generalized regression neural network. For modeling, data on the spatial distribution of copper and iron in the topsoil (depth 0.05 m) on the territory of the subarctic city of Noyabrsk, Yamalo-Nenets Autonomous Okrug, Russia, were used. A total of 237 soil samples were collected. For modelling, the copper and iron concentration data were divided into two subsets: training and test. The modelled spatial datasets were compared with the observed values of the test subset. To assess the performance of the constructed models, three approaches were used: 1) calculation of correlation coefficients, error or agreement indexes, 2) graphical approach (Taylor diagram), 3) randomization assessment of the probability of obtaining a divergence between the observed and modelled datasets, assuming that both of these datasets taken from the same population. For the randomization algorithm, two statistics were used: difference in means and correlation coefficient. The permutation method proved its productivity, as it allowed to assess the significance of the divergence between the observed and predicted datasets.

Alexander P. Sergeev
Сandidate of Physical and Mathematical Sciences
Leading Researcher, Acting Head of the laboratory of the Institute of Industrial Ecology, UB RAS
20, S. Kovalevskoy str., Ekaterinburg, 620990, Russia
е-mail: sergeev@ecko.uran.ru

Anastasia S. Butorova
Research engineer of the Institute of Industrial Ecology, UB RAS
20, S. Kovalevskoy str., Ekaterinburg, 620990, Russia;
1st year Postgraduate Student of the Institute of Radio Electronics and Information Technologies – RtF of the Ural Federal University named after B.N. Yeltsin
19, Mira str., Ekaterinburg, 620002, Russia
е-mail: amoskalyova11@yandex.ru

Andrey V. Shichkin
Researcher of the Institute of Industrial Ecology, UB RAS
20, S. Kovalevskoy str., Ekaterinburg, 620990, Russia
е-mail: and@ecko.uran.ru

Alexander G. Buevich
Researcher of the Institute of Industrial Ecology, UB RAS
20, S. Kovalevskoy str., Ekaterinburg, 620990, Russia
е-mail: bag@ecko.uran.ru

Elena M. Baglaeva
Candidate of Physical and Mathematical Sciences
Senior Researcher of the Institute of Industrial Ecology, UB RAS
20, S. Kovalevskoy str., Ekaterinburg, 620990, Russia
е-mail: e.m.baglaeva@urfu.ru

Irina Ev. Subbotina
Candidate of Physical and Mathematical Sciences
Researcher of the Institute of Industrial Ecology, UB RAS
20, S. Kovalevskoy str., Ekaterinburg, 620990, Russia
е-mail: iesub@mail.ru

1. Subbotina I.E., Buevich A.G., Sergeev A.P., Shichkin A.V, Baglaeva E.M., Remezova M.S. Two-step combined algorithm for improving the accuracy of predicting methane concentration in atmospheric air based on the NARX neural network and subsequent prediction of residuals. Arctic: ecology and economy. 2020;38:59–67. DOI:10.25283/2223-4594-2020-2-59-67.
2. Ali I., Alharbi O.M.L., ALOthman Z.A., Al-Mohaimeed A.M., Alwarthan  A. Modeling of fenuron pesticide adsorption on CNTs for mechanistic insight and removal in water. Environmental Research. 2019;170:389–397. DOI:10.1016/j.envres.2018.12.066.
3. Baglaeva E.M., Sergeev A.P., Shichkin A.V., Buevich  A. G. The extraction of the training subset for the spatial distribution modelling of the heavy metals in topsoil. Catena. 2021;207:105699. DOI:10.1016/j.catena.2021.105699.
4. Berry K.J., Johnston J.E., Mielke P.W., Johnston L.A. Permutation methods. Part II. Wiley Interdisciplinary Reviews: Computational Statistics. 2018;10(3):e1429. DOI:10.1002/wics.1429.
5. Broomhead D.S., Lowe  D. Radial basis functions, multi-variable functional interpolation and adaptive networks. Malvern: Royals Signals & Radar Establishment; 1988. 34 p.
6. Fernández Jaramillo J.M., Mayerle R. Sample selection via angular distance in the space of the arguments of an artificial neural network. Computers and Geosciences. 2018;114:98–106. DOI:10.1016/j.cageo.2018.02.003.
7. Fişek M.H., Barlas Z. Permutation tests for goodness-of-fit testing of mathematical models to experimental data. Social Science Research. 2013;42(2):482–495. DOI:10.1016/j.ssresearch.2012.09.010.
8. Goovaerts P. Geostatistics in soil science: state-of-the-art and perspectives. Geoderma.1999;89:1–45. DOI:10.1016/S0016-7061(98)00078-0.
9. Graf R., Zhu S., Sivakumar B. Forecasting river water temperature time series using a wavelet–neural network hybrid modelling approach. Journal of Hydrology. 2019;578:124115. DOI:10.1016/j.jhydrol.2019.124115.
10. Hamilton S.H., Fu B., Guillaume J.H.A., Badham J., Elsawah S., Gober P., Hunt R.J., Iwanaga T., Jakeman A.J., Ames D.P., Curtis A., Hill M.C., Pierce S., Zare F. A framework for characterising and evaluating the effectiveness of environmental modelling. Environmental Modelling & Software. 2019;118:83–98. DOI:10.1016/j.envsoft.2019.04.008.
11. Hastie T., Tibshirani R., Friedman J. The Elements of Statistical Learning: Data Mining, Inference, and Prediction. New York: Springer; 2009. 745 p.
12. Lambiotte R., Rosvall M., Scholtes I. From networks to optimal higher-order models of complex systems. Nature Physics. 2019;15:313–320. DOI:10.1038/s41567-019-0459-y.
13. Liodakis S., Kyriakidis P., Gaganis P. Conditional Latin Hypercube Simulation of (Log) Gaussian Random Fields. Mathematical Geosciences. 2018;50:127–146. DOI:10.1007/s11004-017-9715-9.
14. McKay M.D., Beckman R.J., Conover W.J. Comparison of Three Methods for Selecting Values of Input Variables in the Analysis of Output from a Computer Code. Technometrics. 1979;21(2):239–245. DOI: 10.1080/00401706.1979.10489755.
15. Mielke P.W., Berry K.J. Permutation Methods. A Distance Function Approach. New York: Springer; 2001. DOI:10.1007/978-1-4757-3449-2.
16. Moré J.J. The Levenberg-Marquardt algorithm: Implementation and theory. In: Numerical Analysis. (Lecture Notes in Mathematics. Vol. 630.) Watson G.A., ed.  Berlin, Heidelberg: Springer; 1978. p. 105–116. DOI:10.1007/BFb0067700.
17. Nath A., Subbiah K. The role of pertinently diversified and balanced training as well as testing data sets in achieving the true performance of classifiers in predicting the antifreeze proteins. Neurocomputing. 2018;272:294–305. DOI:10.1016/j.neucom.2017.07.004.
18. Rosenblatt F. Principles of Neurodynamics: Perceptrons and the Theory of Brain Mechanisms. Washington: Spartan Books; 1962. 616 p.
19. Ross S.M. Simulation, bootstrap statistical methods, and permutation tests. In: Introduction to Probability and Statistics for Engineers and Scientists. Amsterdam: Academic Press, Elsevier; 2021. p. 619–647. DOI:10.1016/B978-0-12-824346-6.00024-7.
20. Sakizadeh M., Mirzaei R., Ghorbani H. Support vector machine and artificial neural network to model soil pollution: a case study in Semnan Province, Iran. Neural Computing and Applications. 2017:28:3229–3238.
21. Samadianfard S., Asadi E., Jarhan S., Kazemi H., Kheshtgar S., Kisi O., Sajjadi S., Manaf A.A. Wavelet neural networks and gene expression programming models to predict short-term soil temperature at different depths. Soil and Tillage Research. 2018;175:37–50. DOI:10.1016/j.still.2017.08.012.
22. Sergeev A.P., Buevich A.G., Baglaeva E.M., Shichkin A.V. Combining spatial autocorrelation with machine learning increases prediction accuracy of soil heavy metals. Catena. 2019;174:425–435. DOI: 10.1016/j.catena.2018.11.037.
23. Simão M.L., Videiro P.M., Silva P.B.A., Assad L.P.F., Sagrilo L.V.S. Application of Taylor diagram in the evaluation of joint environmental distributions’ performances. Marine Systems & Ocean Technology. 2020;15:151–159. DOI:10.1007/s40868-020-00081-5.
24. Specht D.F. A general regression neural network. IEEE Transactions on Neural Networks. 1991;2(6):568–576.
25. Tan A.P., Cheong C.H., Lee T., Seng K.Y., Teo C.J. Computer modelling of heat strain responses of exercising personnel in tropical climate. Computers in Biology and Medicine. 2021;134:104530. DOI:10.1016/j.compbiomed.2021.104530.
26. Taylor A.B., MacKinnon D.P. Four applications of permutation methods to testing a single-mediator model. Behavior Research Methods. 2012;44(3):806–844. DOI:10.3758/s13428-011-0181-x.
27. Taylor K. Summarizing multiple aspects of model performance in a single diagram. Journal of Geophysical Research Atmospheres. 2001;106(D7):7183–7192. DOI:10.1029/2000JD900719.
28. Ucun Ozel H., Gemici B. T., Gemici E., Ozel H.B., Cetin M., Sevik H. Application of artificial neural networks to predict the heavy metal contamination in the Bartin River. Environmental Science and Pollution Research. 2020;27:42495–42512. DOI:10.1007/s11356-020-10156-w.
29. Wang F., Gao J., Zha Y. Hyperspectral sensing of heavy metals in soil and vegetation: Feasibility and challenges. ISPRS Journal of Photogrammetry and Remote Sensing. 2018;136:73–84. DOI:10.1016/j.isprsjprs.2017.12.003.
30. Willmott C.J. On the validation of models. Physical Geography. 1981;2:184–194. DOI:10.1080/02723646.1981.10642213.
31. Willmott C.J., Robeson S.M., Matsuura K. A refined index of model performance. International Journal of Climatology. 2012;32(13):2088–2094. DOI:10.1002/joc.2419.
32. Xu W., Peng H., Zeng X., Zhou F., Tian X., Peng X. Deep belief network-based AR model for nonlinear time series forecasting. Applied Soft Computing. 2019;77:605–621. DOI:10.1016/J.ASOC.2019.02.006.
33. Ziggah Y.Y., Youjian H., Tierra A.R., Laari P. B. Coordinate Transformation between Global and Local Data Based on Artificial Neural Network with K-Fold Cross-Validation: A Case Study , Ghana. Earth Sciences Research Journal. 2019;23(1):67–77. doi:10.15446/esrj.v23n1.63860