A New Transformer-Based Hybrid Model for Forecasting Crude Oil Returns

Document Type : Research Article

Authors

1 Department of Computer Engineering, Amirkabir University of Technology, Tehran, Iran

2 Department of Industrial Engineering and Management Systems, Amirkabir University of Technology, Tehran, Iran

Abstract

In recent years, crude oil has been one of the most important energy sources in the world which impacts political stability and economic security in many countries. Furthermore, Crude oil price also has a huge influence on the world economic pattern due to being directly utilized in different industries in various ways. The purpose of this paper is to improve the ability of existing models in forecasting Brent crude oil returns. Hence, we propose two new deep learning-based models. The first model is based on the transformer which has been very popular in Natural Language Processing over the past few years. Moreover, different widely used deep learning-based methods of time series modeling such as SVR, MLP, GPR, and LSTM are implemented. The second model takes the outputs of all implemented methods as new features and feeds them to a Multilayer Perceptron network. The obtained results by each proposed model have been compared together concerning closeness to the real returns according to the predefined metrics. It is demonstrated that the new transformer-based model (the first model) has better results than the other four common machine learning-based methods. Consequently, the new hybrid model (the second model) provides better price forecasts among all implemented models.

Keywords

Main Subjects

References

[1] Abdollahi H, Ebrahimi SB. A new hybrid model for forecasting Brent crude oil price. Energy 2020;p. 117520.
[2] Zhang YJ, Zhang JL. Volatility forecasting of crude oil market: A new hybrid method. Journal of Forecasting 2018;37(8):781–789.
[3] Deng S, Xiang Y, Fu Z, Wang M, Wang Y. A hybrid method for crude oil price direction forecasting using multiple timeframes dynamic time wrapping and genetic algorithm. Applied Soft Computing 2019;82:105566.
[4] Zhao Y, Li J, Yu L. A deep learning ensemble approach for crude oil price forecasting. Energy Economics 2017;66:9–16.
[5] Li X, Shang W, Wang S. Text-based crude oil price forecasting: A deep learning approach. International Journal of Forecasting 2019;35(4):1548–1560.
[6] Cen Z,Wang J. Crude oil price prediction model with long short term memory deep learning based on prior knowledge data transfer. Energy 2019;169:160–171.
[7] Längkvist M, Karlsson L, Lout_ A. A review of unsupervised feature learning and deep learning for time-series modeling. Pattern Recognition Letters 2014;42:11–24.
[8] Raheem ID, Vo XV. A new approach to exchange rate forecast: The role of global _nancial cycle and time-varying parameters. International Journal of Finance & Economics 2020;.
[9] Enilov M, Fazio G, Ghoshray A. Global connectivity between commodity prices and national stock markets: A timevarying MIDAS analysis;.
[10] Gong X, Lin B. Predicting the volatility of crude oil futures: The roles of leverage e_ects and structural changes. International Journal of Finance & Economics 2020;.
[11] Cheng F, Li T, Wei Ym, Fan T. The VEC-NAR model for short-term forecasting of oil prices. Energy Economics 2019;78:656–667.
[12] Livieris IE, Pintelas E, Pintelas P. A CNN–LSTM model for gold price time-series forecasting. Neural Computing and Applications 2020;p. 1–10.
[13] Yu P, Yan X. Stock price prediction based on deep neural networks. Neural Computing and Applications 2020;32(6):1609–1628.
[14] Kodogiannis V, Lolis A. Forecasting _nancial time series using neural network and fuzzy system-based techniques. Neural computing & applications 2002;11(2):90–102.
[15] Chiroma H, Abdulkareem S, Herawan T. Evolutionary Neural Network model forWest Texas Intermediate crude oil price prediction. Applied Energy 2015;142:266–273.
[16] Hamdia KM, Zhuang X, Rabczuk T. An e_cient optimization approach for designing machine learning models based on genetic algorithm. Neural Computing and Applications 2020;p. 1–11.
[17] Yu L, Dai W, Tang L, Wu J. A hybrid grid-GA-based LSSVR learning paradigm for crude oil price forecasting. Neural computing and applications 2016;27(8):2193–2215.
[18] Fazelabdolabadi B. A hybrid Bayesian-network proposition for forecasting the crude oil price. Financial Innovation 2019;5(1):1–21.
[19] Khashman A, Nwulu NI. Intelligent prediction of crude oil price using Support Vector Machines. In: 2011 IEEE 9thInternational Symposium on Applied Machine Intelligence and Informatics (SAMI) IEEE; 2011. p. 165–169.
[20] Drucker H, Burges CJ, Kaufman L, Smola AJ, Vapnik V. Support Vector Regression machines. In: Advances in neural information processing systems; 1997. p. 155–161.
[21] Salvi H, Shah A, Mehta M, Correia S. Long Short-Term Model for Brent Oil Price Forecasting. Int J Res Appl Sci Eng Technol 2019;7:315–9.
[22] Gupta N, Nigam S. Crude Oil Price Prediction using Arti_cial Neural Network. Procedia Computer Science 2020;170:642–647.
[23] Wang J, Lei C, Guo M. Daily natural gas price forecasting by a weighted hybrid data-driven model. Journal of Petroleum Science and Engineering 2020;192:107240.
[24] Vaswani A, Shazeer N, Parmar N, Uszkoreit J, Jones L, Gomez AN, et al. Attention is all you need. In: Advances in neural information processing systems; 2017. p. 5998–6008.
[25] Wu N, Green B, Ben X, O’Banion S. Deep Transformer Models for Time Series Forecasting: The In_uenza Prevalence Case. arXiv preprint arXiv:200108317 2020;.
[26] Cohen M, Charbit M, Cor_ SL, Preda M, Nozière G. End-to-end deep metamodeling to calibrate and optimize energy loads. arXiv preprint arXiv:200612390 2020;.
[27] Lim B, Arik SO, Loe_ N, P_ster T. Temporal fusion transformers for interpretable multi-horizon time series forecasting. arXiv preprint arXiv:191209363 2019;.
[28] Li S, Jin X, Xuan Y, Zhou X, Chen W, Wang YX, et al. Enhancing the locality and breaking the memory bottleneck of transformer on time series forecasting. In: Advances in Neural Information Processing Systems; 2019. p. 5243–5253.
[29] Liu J, Lin H, Liu X, Xu B, Ren Y, Diao Y, et al. Transformer-based capsule network for stock movement prediction. In: Proceedings of the First Workshop on Financial Technology and Natural Language Processing; 2019. p. 66–73.
[30] Quintanar A, Fernández-Llorca D, Parra I, Izquierdo R, Sotelo MA. Predicting vehicles trajectories in urban scenarios with transformer networks and augmented information. IEEE Intelligent Vehicles Symposium (IV) 2021 Jul 11 (pp. 1051-1056).
[31] Pan X, Wang L, Wang Z, Huang C. Short-term wind speed forecasting based on spatial-temporal graph transformer networks. Energy. 2022 Aug 15;253:124095.
[32] Sridhar, S. and Sanagavarapu, S., 2021, July. Multi-head self-attention transformer for dogecoin price prediction. IEEE 14th International Conference on Human System Interaction (HSI), 2021, (pp. 1-6).
[33] Ramos-Pérez E, Alonso-González PJ, Núñez-Velázquez JJ. Multi-transformer: A new neural network-based architecture for forecasting S&P volatility. Mathematics. 2021 Jul 28;9(15):1794.
[34] Hochreiter S, Schmidhuber J. Long Short-Term Memory. Neural computation 1997;9(8):1735–1780.
[35] Smola AJ, Schölkopf B. A tutorial on Support Vector Regression. Statistics and computing 2004;14(3):199–222.
[36] Bishop CM. Pattern recognition and machine learning. springer; 2006.
[37] Hajizadeh E, Seifi A, Zarandi MF, Turksen IB. A hybrid modeling approach for forecasting the volatility of S&P 500 index return. Expert Systems with Applications. 2012 Jan 1;39(1):431-6.
[38] Kingma DP, Ba J. Adam: A Method for Stochastic Optimization. In: ICLR (Poster); 2015.
AUT Journal of Modeling and Simulation
Volume 54, Issue 1
June 2022
Pages 19-30

Files

Share

How to cite

Statistics