Метод последовательных приближений для построения модели динамической полиномиальной регрессии

Anna G. Golovkina; Vladimir A. Kozynchenko; Ilia S. Klimenko

doi:10.21638/11701/spbu10.2022.404

Authors

Anna G. Golovkina St Petersburg State University, 7-9, Universitetskaya nab., St Petersburg, 199034, Russian Federation https://orcid.org/0000-0002-8906-5227
Vladimir A. Kozynchenko St Petersburg State University, 7-9, Universitetskaya nab., St Petersburg, 199034, Russian Federation https://orcid.org/0000-0003-1011-2455
Ilia S. Klimenko St Petersburg State University, 7-9, Universitetskaya nab., St Petersburg, 199034, Russian Federation https://orcid.org/0000-0002-2957-4951

DOI:

https://doi.org/10.21638/11701/spbu10.2022.404

Abstract

Predicting the behavior of a certain process in time is an important task that arises in many applied areas, and information about the system that generated this process can either be completely absent or be partially limited. The only available knowledge is the accumulated data on past states and process parameters. Such a task can be successfully solved using machine learning methods, but when it comes to modeling physical experiments or areas where the ability of a model to generalize and interpretability of predictions are important, then the most machine learning methods do not fully satisfy these requirements. The forecasting problem is solved by building a dynamic polynomial regression model, and a method for finding its coefficients is proposed, based on the connection with dynamic systems. Thus, the constructed model corresponds to a deterministic process, potentially described by differential equations, and the relationship between its parameters can be expressed in an analytical form. As an illustration of the applicability of the proposed approach to solving forecasting problems, we consider a synthetic data set generated as a numerical solution of a system of differential equations that describes the Van der Pol oscillator.

Keywords:

polynomial regression, dynamic systems, Taylor map

Downloads

Download data is not yet available.

References

Литература

Jansson M., Wahlberg B. A linear regression approach to state-space subspace system identification // Signal Processing. 1996. Vol. 2. P. 103-129. https://doi.org/10.1016/0165-1684(96)00048-5

Herceg S., Ujevic Z., Bolf A. N. Development of soft sensors for isomerization process based on support vector machine regression and dynamic polynomial models // Chemical Engineering Research and Design. 2019. Vol. 149. P. 95-103. https://doi.org/10.1016/j.cherd.2019.06.034

Dette H. Optimal designs for identifying the degree of a polynomial regression // Annals of Statistics. 1995. Vol. 23. N 4. P. 1248-1266. https://doi.org/10.1214/aos/1176324708

Kim B., Ko Ch.-Y., Wong N. Tensor network subspace identification of polynomial state space models // Automatica. 2018. Vol. 95. P. 187-196. https://doi.org/10.1016/j.automatica.2018.05.015

Blondel M., Ishihata M., Fujino A., Ueda N. Polynomial networks and factorization machines: New insights and efficient training algorithms // Proceedings of ICML 2016. New York City, 2016. P. 850-858.

Blondel M., Niculae V., Otsuka T., Ueda N. Multi-output polynomial networks and factorization machines // Proceedings of the 31st International Conference on Neural Information Processing Systems (NIPS'17). Long Beach, 2017. P. 3351-3361.

Chen T., Guestrin C. XGBoost: A Scalable tree boosting system // Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD'16). San Francisco, 2016. P. 785-794. https://doi.org/10.1145/2939672.2939785

Hornik K., Stinchcombe M., White H. Multilayer feedforward networks are universal approximators // Neural Networks. 1989. Vol. 2. P. 359-366.

Rao S., Sethuraman S., Ramamurthi V. A recurrent neural network for nonlinear time series prediction: a comparative study // IEEE 1992 Workshop on Neural Networks for Signal Processing (NNSP'92). Helsingoer, 1992. P. 531-539.

Kaheman K., Kutz J. N., Brunton S. L. SINDy-PI: a robust algorithm for parallel implicit sparse identification of nonlinear dynamics // Proceedings of the Royal Society A. 2020. Vol. 476. N 2242. Art. N 20200279.

Brunton S. L., Joshua L. P., Kutz N. Discovering governing equations from data by sparse identification of nonlinear dynamical systems // Proceedings of the National Academy of Sciences. 2016. Vol. 113. N 15. P. 3932-3937.

Andrianov S. N. Dynamical modeling of control systems for particle beams. St Petersburg: St Petersburg University Press, 2004. 368 p.

Dragt A. Lie methods for nonlinear dynamics with applications to accelerator physics. 2011. URL: inspirehep.net/record/955313/files/TOC28Nov2011.pdf (дата обращения: 10.08.2022).

Andrianov S. Symbolic computation of approximate symmetries for ordinary differential equations // Mathematics and Computers in Simulation. 2001. Vol. 57. N 3-5. P. 147-154.

Andrianov S. A matrix representation of the Lie transformation // Proceedings of the Abstracts of the International Congress on Computer Systems and Applied Mathematics. St Petersburg, 1993. Vol. 14. P. 19-23.

Ivanov A., Golovkina A., Iben U. Polynomial neural networks and taylor maps for dynamical systems simulation and learning // 24th European Conference on Artificial Intelligence, including 10th Conference on Prestigious Applications of Artificial Intelligence, PAIS 2020. Proceedings. IOS Press, 2020. P. 1230-1237. (Frontiers in Artificial Intelligence and Applications). https://doi.org/10.3233/FAIA200223

Golovkina A., Kozynchenko V., Kulabukhova N. Reconstruction and identification of dynamical systems based on taylor maps // Computational Science and its Applications - 21st International Conference, Proceedings. Pt VIII. Switzerland: Springer Nature, 2021. P. 360-369. (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)). https://doi.org/10.1007/978-3-030-87010-2_26

Golovkina A. G., Kozynchenko V. A., Kulabukhova N. V. Reconstruction of ordinary differential equations from irregularly distributed time-series data // Proceedings of the 9th International Conference ``Distributed Computing and Grid Technologies in Science and Education'' (GRID'2021). Dubna, Russia, July 5-9, 2021. Vol. 3041. P. 342-347.

Головкина А. Г., Ганаева Д. Д. Метод реконструкции нелинейных динамических систем по временным рядам // Процессы управления и устойчивость. 2022. Т. 9. № 1. С. 197-201.

Клименко И. С. Реализация метода матричных отображений для решения системы дифференциальных уравнений // Процессы управления и устойчивость. 2022. Т. 9. № 1. С. 53-57.

Cartwright M. L. Van der Pol’s equation for relaxation oscillations // Contributions to the theory of nonlinear oscillations. II. Princeton Ann. Math. Stud. Princeton: Princeton University Press, 1952. P. 3-18.

Abrevaya G., Rish I., Aravkin A. Y., Cecchi G., Kozloski J., Polosecki P., Zheng P., Dawson S. R., Rhee J., Cox D. Learning nonlinear brain dynamics: Van der Pol Meets LSTM // bioRxiv 330548. https://doi.org/10.1101/330548

LSODA. Ordinary differential equation solver for stiff or non-stiff system (September 2005). URL: http://www.nea.fr/abs/html/uscd1227.html (дата обращения: 10.08.2022).

References

Jansson M., Wahlberg B. A linear regression approach to state-space subspace system identification. Signal Processing, 1996, vol. 2, pp. 103-129. https://doi.org/10.1016/0165-1684(96)00048-5

Herceg S., Ujevic Z., Bolf A. N. Development of soft sensors for isomerization process based on support vector machine regression and dynamic polynomial models. Chemical Engineering Research and Design, 2019, vol. 149, pp. 95-103. https://doi.org/10.1016/j.cherd.2019.06.034

Dette H. Optimal designs for identifying the degree of a polynomial regression. Annals of Statistics, 1995, vol. 23, no. 4, pp. 1248-1266. https://doi.org/10.1214/aos/1176324708

Kim B., Ko Ch.-Y., Wong N. Tensor network subspace identification of polynomial state space models. Automatica, 2018, vol. 95, pp. 187-196. https://doi.org/10.1016/j.automatica.2018.05.015

Blondel M., Ishihata M., Fujino A., Ueda N. Polynomial networks and factorization machines: New insights and efficient training algorithms. Proceedings of ICML 2016. New York City, 2016, pp. 850-858.

Blondel M., Niculae V., Otsuka T., Ueda N. Multi-output polynomial networks and factorization machines. Proceedings of the 31st International Conference on Neural Information Processing Systems (NIPS'17). Long Beach, 2017, pp. 3351-3361.

Chen T., Guestrin C. XGBoost: A Scalable tree boosting system. Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD'16). San Francisco, 2016, pp. 785-794. https://doi.org/10.1145/2939672.2939785

Hornik K., Stinchcombe M., White H. Multilayer feedforward networks are universal approximators. Neural Networks, 1989, vol. 2, pp. 359-366.

Rao S., Sethuraman S., Ramamurthi V. A recurrent neural network for nonlinear time series prediction: a comparative study. IEEE 1992 Workshop on Neural Networks for Signal Processing (NNSP'92). Helsingoer, 1992, pp. 531-539.

Kaheman K., Kutz J. N., Brunton S. L. SINDy-PI: a robust algorithm for parallel implicit sparse identification of nonlinear dynamics. Proceedings of the Royal Society A., 2020, vol. 476, no. 2242, Art. no. 20200279.

Brunton S. L., Joshua L. P., Kutz N. Discovering governing equations from data by sparse identification of nonlinear dynamical systems. Proceedings of the National Academy of Sciences, 2016, vol. 113, no. 15, pp. 3932-3937.

Andrianov S. N. Dynamical modeling of control systems for particle beams. St Petersburg, St Petersburg University Press, 2004, 368 p.

Dragt A. Lie methods for nonlinear dynamics with applications to accelerator physics. 2011. Available at: inspirehep.net/record/955313/files/TOC28Nov2011.pdf (accessed: August 10, 2022).

Andrianov S. Symbolic computation of approximate symmetries for ordinary differential equations. Mathematics and Computers in Simulation, 2001, vol. 57, no. 3-5, pp. 147-154.

Andrianov S. A matrix representation of the Lie transformation. Proceedings of the Abstracts of the International Congress on Computer Systems and Applied Mathematics. St Petersburg, 1993, vol. 14, pp. 19-23.

Ivanov A., Golovkina A., Iben U. Polynomial neural networks and taylor maps for dynamical systems simulation and learning. 24th European Conference on Artificial Intelligence, including 10th Conference on Prestigious Applications of Artificial Intelligence, PAIS 2020. Proceedings. IOS Press, 2020, pp. 1230-1237. (Frontiers in Artificial Intelligence and Applications). https://doi.org/10.3233/FAIA200223

Golovkina A., Kozynchenko V., Kulabukhova N. Reconstruction and identification of dynamical systems based on taylor maps. Computational Science and its Applications - 21st International Conference. Proceedings. Pt VIII. Switzerland, Springer Nature Publ. 2021, pp. 360-369. (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)). https://doi.org/10.1007/978-3-030-87010-2_26

Golovkina A. G., Kozynchenko V. A., Kulabukhova N. V. Reconstruction of ordinary differential equations from irregularly distributed time-series data. Proceedings of the 9th International Conference ``Distributed Computing and Grid Technologies in Science and Education'' (GRID'2021). Dubna, Russia, July 5-9, 2021, vol. 3041, pp. 342-347.

Golovkina А. G., Ganaeva D. D. Metod reconstructsii nelineinykh dinamicheskih sistem po vremennym ryadam [Method of nonlinear dynamical systems reconstruction based on time series data]. Control Processes and Stability, 2022, vol. 9, no. 1, pp. 197-201. (In Russian)

Klimenko I. S. Realizatsia metoda matrichnykh otobrajenii dlya reshenia sistemy differentsialnyh uravnenii [Implementation of matrix map method for solving a system of ordinary equations]. Control Processes and Stability, 2022, vol. 9, no. 1, pp. 53-57. (In Russian)

Cartwright M. L. Van der Pol’s equation for relaxation oscillations. Contributions to the Theory of Nonlinear Oscillations. II. Princeton Ann. Math. Stud. Princeton, Princeton University Press, 1952, pp. 3-18.

Abrevaya G., Rish I., Aravkin A. Y., Cecchi G., Kozloski J., Polosecki P., Zheng P., Dawson S. P., Rhee J., Cox D. Learning nonlinear brain dynamics: Van der Pol Meets LSTM. bioRxiv 330548. https://doi.org/10.1101/330548

LSODA. Ordinary differential equation solver for stiff or non-stiff system (September 2005). Available at: http://www.nea.fr/abs/html/uscd1227.html (accessed: August 10, 2022).