Enhancing Multi-Output Time Series Forecasting with Encoder-Decoder Networks
Downloads
Background: Multi-output Time series forecasting is a complex problem that requires handling interdependencies and interactions between variables. Traditional statistical approaches and machine learning techniques often struggle to predict such scenarios accurately. Advanced techniques and model reconstruction are necessary to improve forecasting accuracy in complex scenarios.
Objective: This study proposed an Encoder-Decoder network to address multi-output time series forecasting challenges by simultaneously predicting each output. This objective is to investigate the capabilities of the Encoder-Decoder architecture in handling multi-output time series forecasting tasks.
Methods: This proposed model utilizes a 1-Dimensional Convolution Neural Network with Bidirectional Long Short-Term Memory, specifically in the encoder part. The encoder extracts time series features, incorporating a residual connection to produce a context representation used by the decoder. The decoder employs multiple unidirectional LSTM modules and Linear transformation layers to generate the outputs each time step. Each module is responsible for specific output and shares information and context along the outputs and steps.
Results: The result demonstrates that the proposed model achieves lower error rates, as measured by MSE, RMSE, and MAE loss metrics, for all outputs and forecasting horizons. Notably, the 6-hour horizon achieves the highest accuracy across all outputs. Furthermore, the proposed model exhibits robustness in single-output forecast and transfer learning, showing adaptability to different tasks and datasets.
Conclusion: The experiment findings highlight the successful multi-output forecasting capabilities of the proposed model in time series data, with consistently low error rates (MSE, RMSE, MAE). Surprisingly, the model also performs well in single-output forecasts, demonstrating its versatility. Therefore, the proposed model effectively various time series forecasting tasks, showing promise for practical applications.
Keywords: Bidirectional Long Short-Term Memory, Convolutional Neural Network, Encoder-Decoder Networks, Multi-output forecasting, Multi-step forecasting, Time-series forecasting
Q.Q. He, P.C.I. Pang, and Y.W.Si, "Transfer Learning for Financial Time Series Forecasting,” in PRICAI 2019: Trends in Artificial Intelligence, pp. 24–36, 2019.
A. Mahmoud and A. Mohammed, "A Survey on Deep Learning for Time-Series Forecasting,” in Machine Learning and Big Data Analytics Paradigms: Analysis, Applications and Challenges, A. E. Hassanien and A. Darwish, Eds. Cham: Springer International Publishing, pp. 365-392, 2021, doi: 10.1007/978-3-030-59338-4_19.
S. Sengupta et al., "A review of deep learning with special emphasis on architectures, applications and recent trends,” Knowledge-Based Syst., vol. 194, 105596, 2020, doi: 10.1016/j.knosys.2020.105596.
H. Wang, Y. Zhao, and S. Tan, "Short-Term load forecasting of power system based on time convolutional network,” 2019 8th Int. Symp. Next Gener. Electron. ISNE 2019, pp. 1-3, 2019, doi: 10.1109/ISNE.2019.8896684.
K. Bandara, C. Bergmeir, and S. Smyl, "Forecasting across time series databases using recurrent neural networks on groups of similar series: A clustering approach,” Expert Syst. Appl., vol. 140, 112896, 2020, doi: 10.1016/j.eswa.2019.112896.
G. Zerveas, S. Jayaraman, D. Patel, A. Bhamidipaty, and C. Eickhoff, "A Transformer-based Framework for Multivariate Time Series Representation Learning,” Proc. ACM SIGKDD Int. Conf. Knowl. Discov. Data Min., pp. 2114–2124, 2021, doi: 10.1145/3447548.3467401.
S. Ranathunga, E.S. A. Lee, M. Prifti Skenduli, R. Shekhar, M. Alam, and R. Kaur, "Neural Machine translation for low-resource languages: A survey,” ACM Comput. Surv., vol. 55, no. 11, pp. 1-37, 2023, doi: 10.1145/3567592.
Y. Wang et al., "Lednet: A lightweight encoder-decoder network for real-time semantic segmentation,” in 2019 IEEE International Conference on Image Processing (ICIP), pp. 1860–1864, 2019, doi: 10.1109/ICIP.2019.8803154.
S. Wu and Y. Wang, "Attention-based Encoder-Decoder Recurrent Neural Networks for HTTP Payload Anomaly Detection,” in 2021 IEEE Intl Conf on Parallel & Distributed Processing with Applications, Big Data & Cloud Computing, Sustainable Computing & Communications, Social Computing & Networking (ISPA/BDCloud/SocialCom/SustainCom), pp. 1452–1459, 2021, doi: 10.1109/ISPA-BDCloud-SocialCom-SustainCom52081.2021.00196.
N. P. Winkler, H. Matsukura, P. P. Neumann, E. Schaffernicht, H. Ishida, and A. J. Lilienthal, "Super-Resolution for Gas Distribution Mapping: Convolutional Encoder-Decoder Network,” in 2022 IEEE International Symposium on Olfaction and Electronic Nose (ISOEN), pp. 1-3, 2022, doi: 10.1109/ISOEN54820.2022.9789555.
Y. Sarker, M.N. Islam Mondal, S.R. Fahim, S. Shahriar, S.K. Sarker, and S.K. Das, "A Novel Diagnosis System Using Regularized Encoder-Decoder Based Generative Probabilistic Network for Parkinson's Disease,” in IEEE Region 10 Symposium (TENSYMP), pp. 1444-1447, 2020, doi: 10.1109/TENSYMP50017.2020.9230792.
S. Du, T. Li, Y. Yang, and S.J. Horng, "Multivariate time series forecasting via attention-based encoder–decoder framework,” Neurocomputing, vol. 388, pp. 269–279, 2020, doi: 10.1016/j.neucom.2019.12.118.
R. Laubscher, "Time-series forecasting of coal-fired power plant reheater metal temperatures using encoder-decoder recurrent neural networks,” Energy, vol. 189, 116187, 2019, doi: 10.1016/j.energy.2019.116187.
H. Zhou et al., "Informer: Beyond Efficient Transformer for Long Sequence Time-Series Forecasting,” 35th AAAI Conf. Artif. Intell. AAAI 2021, vol. 12B, pp. 11106-11115, 2021, doi: 10.1609/aaai.v35i12.17325.
X.B. Jin, W.Z. Zheng, J.L. Kong, X.Y. Wang, Y.T. Bai, T.L. Su, and S. Lin, , "Deep-learning forecasting method for electric power load via attention-based encoder-decoder with bayesian optimization,” Energies, vol. 14, no. 6, 1596, 2021, doi: 10.3390/en14061596.
P. Lyu, N. Chen, S. Mao, and M. Li, "LSTM based encoder-decoder for short-term predictions of gas concentration using multi-sensor fusion,” Process Saf. Environ. Prot., vol. 137, pp. 93-105, 2020, doi: 10.1016/j.psep.2020.02.021.
D. Xu, Y. Shi, I. W. Tsang, Y.S. Ong, C. Gong, and X. Shen, "Survey on Multi-Output Learning,” IEEE Trans. Neural Networks Learn. Syst., vol. 31, no. 7, pp. 2409-2429, 2020, doi: 10.1109/TNNLS.2019.2945133.
K.K.R. Samal, K.S. Babu, and S.K. Das, "Time Series Forecasting of Air Pollution using Deep Neural Network with Multi-output Learning,” in IEEE 18th India Council International Conference (INDICON), pp. 1-5, 2021, doi: 10.1109/INDICON52576.2021.9691669.
K.K.R. Samal, A.K. Panda, K.S. Babu, and S.K. Das, "Multi-output TCN autoencoder for long-term pollution forecasting for multiple sites,” Urban Clim., vol. 39, 100943, 2021, doi: https://doi.org/10.1016/j.uclim.2021.100943.
Y. Zhou, F.J. Chang, L.C. Chang, I.F. Kao, and Y.S. Wang, "Explore a deep learning multi-output neural network for regional multi-step-ahead air quality forecasts,” J. Clean. Prod., vol. 209, pp. 134-145, 2019, doi: https://doi.org/10.1016/j.jclepro.2018.10.243.
J.S. Chou, D.N. Truong, and T.L. Le, "Interval Forecasting of Financial Time Series by Accelerated Particle Swarm-Optimized Multi-Output Machine Learning System,” IEEE Access, vol. 8, no. 2008, pp. 14798-14808, 2020, doi: 10.1109/ACCESS.2020.2965598.
R. Talavera-Llames, R. Pérez-Chacón, A. Troncoso, and F. Martínez-Álvarez, "MV-kWNN: A novel multivariate and multi-output weighted nearest neighbours algorithm for big data time series forecasting,” Neurocomputing, vol. 353, pp. 56–73, 2019, doi: 10.1016/j.neucom.2018.07.092.
P. Jiang, Y. Nie, J. Wang, and X. Huang, "Multivariable short-term electricity price forecasting using artificial intelligence and multi-input multi-output scheme,” Energy Econ., vol. 117, 106471, 2023, doi: https://doi.org/10.1016/j.eneco.2022.106471.
X. Zhan, S. Zhang, W.Y. Szeto, and X. Chen, "Multi-step-ahead traffic speed forecasting using multi-output gradient boosting regression tree,” J. Intell. Transp. Syst. Technol. Planning, Oper., vol. 24, no. 2, pp. 125-141, 2020, doi: 10.1080/15472450.2019.1582950.
Z. Al Sadeque and F.M. Bui, "A deep learning approach to predict weather data using cascaded LSTM network,” Can. Conf. Electr. Comput. Eng., pp. 1-5, 2020, doi: 10.1109/CCECE47787.2020.9255716.
N. Azizi, M. Yaghoubirad, M. Farajollahi, and A. Ahmadi, "Deep learning based long-term global solar irradiance and temperature forecasting using time series with multi-step multivariate output,” Renew. Energy, vol. 206, pp. 135-147, 2023, doi: https://doi.org/10.1016/j.renene.2023.01.102.
Z. Qu et al., "Temperature forecasting of grain in storage: A multi-output and spatiotemporal approach based on deep learning,” Comput. Electron. Agric., vol. 208, 107785, 2023, doi: https://doi.org/10.1016/j.compag.2023.107785.
Z. Zhang, H. Tang, H. Qin, B. Luo, C. Zhou, and H. Zhou, "Multi-step ahead probabilistic forecasting of multiple hydrological variables for multiple stations,” J. Hydrol., vol. 617, 129094, 2023, doi: https://doi.org/10.1016/j.jhydrol.2023.129094.
P. Lu et al., "A novel spatio-temporal wind power forecasting framework based on multi-output support vector machine and optimization strategy,” J. Clean. Prod., vol. 254, 119993, 2020, doi: 10.1016/j.jclepro.2020.119993.
X. He, S. Shi, X. Geng, and L. Xu, "Dynamic Co-Attention Networks for multi-horizon forecasting in multivariate time series,” Futur. Gener. Comput. Syst., vol. 135, pp. 72-84, 2022, doi: https://doi.org/10.1016/j.future.2022.04.029.
Y. Wang, T. Li, W. Lu, and Q. Cao, "Attention-inspired RNN Encoder-Decoder for Sensory Time Series Forecasting,” Procedia Comput. Sci., vol. 209, pp. 112-121, 2022, doi: 10.1016/j.procs.2022.10.104.
I. Lloret, J. A. Troyano, F. Enríquez, and J. J. González-de-la-Rosa, "Two deep learning approaches to forecasting disaggregated freight flows: convolutional and encoder–decoder recurrent,” Soft Comput., vol. 25, no. 12, pp. 7769-7784, 2021, doi: 10.1007/s00500-021-05678-5.
G. Shi and C. Wang, "Multivariate Multi-Step Agrometeorological Forecast Model for Rapid Spray,” IEEE Access, vol. 9, pp. 159271-159282, 2021, doi: 10.1109/ACCESS.2021.3131649.
H. Yin, Z. Ou, S. Huang, and A. Meng, "A cascaded deep learning wind power prediction approach based on a two-layer of mode decomposition,” Energy, vol. 189, 116316, 2019, doi: 10.1016/j.energy.2019.116316.
R. Wan, S. Mei, J. Wang, M. Liu, and F. Yang, "Multivariate temporal convolutional network: A deep neural networks approach for multivariate time series forecasting,” Electron., vol. 8, no. 8, 876, 2019, doi: 10.3390/electronics8080876.
S. Khullar and N. Singh, "Water quality assessment of a river using deep learning Bi-LSTM methodology: forecasting and validation,” Environ. Sci. Pollut. Res., vol. 29, no. 9, pp. 12875-12889, 2022, doi: 10.1007/s11356-021-13875-w.
S. Chen, "Beijing Multi-Site Air-Quality Data.” 2019.
R. Yan, J. Liao, J. Yang, W. Sun, M. Nong, and F. Li, "Multi-hour and multi-site air quality index forecasting in Beijing using CNN, LSTM, CNN-LSTM, and spatiotemporal clustering,” Expert Syst. Appl., vol. 169, 114513, 2021, doi: https://doi.org/10.1016/j.eswa.2020.114513.
Z. Zhang, Y. Zeng, and K. Yan, "A hybrid deep learning technology for PM2.5 air quality forecasting,” Environ. Sci. Pollut. Res., vol. 28, no. 29, pp. 39409-39422, 2021, doi: 10.1007/s11356-021-12657-8.
M. Niu, Y. Zhang, and Z. Ren, "Deep Learning-Based PM2.5 Long Time-Series Prediction by Fusing Multisource Data-A Case Study of Beijing,” Atmosphere (Basel)., vol. 14, no. 2, 340, 2023, doi: 10.3390/atmos14020340.
X. Cheng, W. Zhang, A. Wenzel, and J. Chen, "Stacked ResNet-LSTM and CORAL model for multi-site air quality prediction,” Neural Comput. Appl., vol. 34, no. 16, pp. 13849-13866, 2022, doi: 10.1007/s00521-022-07175-8.
R. Rakholia, Q. Le, B. Quoc Ho, K. Vu, and R. Simon Carbajo, "Multi-output machine learning model for regional air pollution forecasting in Ho Chi Minh City, Vietnam,” Environ. Int., vol. 173, 107848, 2023, doi: 10.1016/j.envint.2023.107848.
M. Benhaddi and J. Ouarzazi, "Multivariate Time Series Forecasting with Dilated Residual Convolutional Neural Networks for Urban Air Quality Prediction,” Arab. J. Sci. Eng., vol. 46, no. 4, pp. 3423-3442, 2021, doi: 10.1007/s13369-020-05109-x.
E. J. Prasetyo and K. D. Hartomo, "Multi-industry stock forecasting using GRU-LSTM deep transfer learning method,” vol. 15, no. 2, pp. 30-43, 2023.
C. Erden, "Genetic algorithm-based hyperparameter optimization of deep learning models for PM2.5 time-series prediction,” Int. J. Environ. Sci. Technol., vol. 20, no. 3, pp. 2959-2982, 2023, doi: 10.1007/s13762-023-04763-6.
K. Liu et al., "Time series prediction of the chemical components of PM2.5 based on a deep learning model,” Chemosphere, vol. 342, 140153, 2023, doi: https://doi.org/10.1016/j.chemosphere.2023.140153.
Copyright (c) 2023 The Authors. Published by Universitas Airlangga.
This work is licensed under a Creative Commons Attribution 4.0 International License.
Authors who publish with this journal agree to the following terms:
All accepted papers will be published under a Creative Commons Attribution 4.0 International (CC BY 4.0) License. Authors retain copyright and grant the journal right of first publication. CC-BY Licenced means lets others to Share (copy and redistribute the material in any medium or format) and Adapt (remix, transform, and build upon the material for any purpose, even commercially).
