Temporal Consistency Transformer for Long Sequence Modeling
Main Article Content
Abstract
Long-range dependency modeling is still difficult in sequence learning because useful information may weaken as the sequence becomes longer. This problem affects both language modeling and time-series prediction. This study proposes a sequence modeling method that combines Transformer learning, temporal consistency constraints, and a memory module. The temporal consistency term reduces sharp changes between adjacent hidden states. The memory module keeps useful historical information for later sequence steps. Experiments were conducted on WikiText-103 and a traffic forecasting dataset. The results showed that the proposed method reduced perplexity by about 11.5% in language modeling and reduced MAE by about 9.3% in traffic prediction. When the sequence length was greater than 512, overall performance improved by about 14%. These results show that temporal smoothing and memory updating can reduce information loss and improve long-sequence modeling. The method provides a practical approach for language modeling, traffic prediction, financial sequence analysis, and sensor monitoring.
Article Details
This work is licensed under a Creative Commons Attribution 4.0 International License.
All articles published in the Journal of the Artificial Intelligence Engineers Consortium (JAIEC) are licensed under the Creative Commons Attribution 4.0 International License (CC BY 4.0).
This license permits unrestricted use, distribution, reproduction, and adaptation in any medium, provided the original author(s) and source are properly credited.
Authors retain copyright of their work while granting the journal the right of first publication.
References
Chen, X., Xiao, H., Zeng, Z., Zhang, S., & Xiao, R. (2025). Fine-Grained Multivariate Time Series Anomaly Detection via Causal Inference. Knowledge-Based Systems, 114765.
Bianchi, G., Fanelli, C., Freddi, F., Giuliani, F., & La Placa, A. (2025). Systematic review railway infrastructure monitoring: From classic techniques to predictive maintenance. Advances in Mechanical Engineering, 17(1), 16878132241285631.
Gao, X., & Lan, Y. (2025, December). Data-Driven Identification Method for Low-Voltage Distribution Station-User Relationships Based on Voltage Sequence Correlation. In 2025 7th International Academic Exchange Conference on Science and Technology Innovation (IAECST) (pp. 508-512). IEEE.
Tufail, R., Tassinari, P., & Torreggiani, D. (2025). Deep learning applications for crop mapping using multi-temporal Sentinel-2 data and red-edge vegetation indices: integrating convolutional and recurrent neural networks. Remote Sensing, 17(18), 3207.
Yan, B. (2026). A Comparative Study: Greedy versus GAN-Based Compressed Sensing for Fast MRI Reconstruction. International Journal of Advance in Applied Science Research, 5(3), 36-42.
Kamatala, S., Jonnalagadda, A. K., & Naayini, P. (2025). Transformers beyond nlp: Expanding horizons in machine learning. Yes it was accepted by IRE Journals, 8(7).
Liu, S., & Yim, J. (2025). Research on Generative AI Creation Systems Based on Visual Language Modeling: Human-Machine Collaboration and Cognitive Feedback Mechanisms. Available at SSRN 6139770.
Ahmad Jan, M., Adil, M., Brik, B., Harous, S., & Abbas, S. (2025). Making sense of big data in intelligent transportation systems: Current trends, challenges and future directions. ACM Computing Surveys, 57(8), 1-43.
Gao, G., Gao, R., Gao, R., & Zhou, H. (2026). Engineering Analysis and Quantitative Research on the Platform-Based Evolution of Enterprise Communication Systems.
Vanaja, C. B., & Prakasam, P. (2025). Convolutional block attention gate-based Unet framework for microaneurysm segmentation using retinal fundus images. BMC medical imaging, 25(1), 83.
Wang, Y., Yin, X., Chen, J., & Wang, Y. (2026). Evidence-Based Study on Low-Burden Digital Phenotyping for Precision Screening of Oral Anti-Obesity Drug Efficacy.
Taha, M. A. (2025). Logarithmic memory networks (lmns): Efficient long-range sequence modeling for resource-constrained environments. arXiv preprint arXiv:2501.07905.
Jiao, Y., Wang, A., Zhao, B., & Shi, T. (2026). Quantitative Study on the Construction and Application Effectiveness of Graffiti Wall Painting Teaching Models in Public Space Contexts.
Nawrot, P., Li, R., Huang, R., Ruder, S., Marchisio, K., & Ponti, E. M. (2025). The sparse frontier: Sparse attention trade-offs in transformer llms. arXiv preprint arXiv:2504.17768.
Liu, S., Liu, X., & Feng, H. (2025, November). Research on AI-Driven Visual Design and Immersive Interactive Experiences Based on Multimodal Cognition and User. In Proceedings of the 2025 International Conference on Digital Society and Intelligent Computing (pp. 734-740).
Verdone, A., Scardapane, S., Araneo, R., & Panella, M. (2025). Graph State-Space Models for Spatio-Temporal Renewable Energy Forecasting. IEEE Transactions on Industry Applications.
Xu, S., Ma, Q., Liu, H., & Yue, L. (2026). Continuous Reorganization and Performance Preservation of Agent Memory Structure Under Distributed Change Environments.
Kumar, A., Clune, J., Lehman, J., & Stanley, K. O. (2025). Questioning representational optimism in deep learning: The fractured entangled representation hypothesis. arXiv preprint arXiv:2505.11581.
Chen, F., Li, S., Liang, H., Xu, P., & Yue, L. (2025). Optimization Study of Thermal Management of Domestic SiC Power Semiconductor Based on Improved Genetic Algorithm.
Madhyastha, P., & Adamcova, D. (2026). Working Memory Constraints Scaffold Learning in Transformers under Data Scarcity. arXiv preprint arXiv:2604.20789.
Liang, R., Feifan, F. N. U., Liang, Y., & Ye, Z. (2025). Emotion-Aware Interface Adaptation in Mobile Applications Based on Color Psychology and Multimodal User State Recognition. Frontiers in Artificial Intelligence Research, 2(1), 51-57.
Omar, M., Yakub, F., Wijaya, A. A., Mohammad, A. F., Maslan, M. N., Dhamanti, I., & Zhongchao, S. (2025). Evaluation of encoder-only transformer for multi-step traffic flow prediction. IEEE Access.
Zhu, W., Yao, Y., & Yang, J. (2025). Optimizing Financial Risk Control for Multinational Projects: A Joint Framework Based on CVaR-Robust Optimization and Panel Quantile Regression.
Saleh, Z. Z., Abbod, M. F., & Nilavalan, R. (2025). Intelligent resource allocation via hybrid reinforcement learning in 5G network slicing. IEEE Access, 13, 47440-47458.
Feng, H. (2024, October). Design of Intelligent Charging System for Portable Electronic Devices Based on Internet of Things (IoT). In 2024 5th International Conference on Big Data, Artificial Intelligence and Internet of Things Engineering (ICBAIE) (pp. 568-571). IEEE.