Investigating the Impact of Cross-lingual Acoustic-Phonetic Similarities on Multilingual Speech Emotion Recognition
DOI:
https://doi.org/10.53799/xsdr1t51Abstract
Speech emotion recognition (SER) is a crucial aspect of effective human-computer interaction applications. However, due to diverse phonetic structures and patterns, it is difficult to accurately recognize emotions in speech across different languages. In order to enhance the performance of emotion recognition systems in diverse linguistic circumstances, this study explores how cross-linguistic acoustic-phonetic similarities impact multilingual speech emotion recognition. The study involves a comprehensive analysis of emotion-labeled speech data for Bangla, Hindi, Odia, English, and German languages, with a focus on identifying acousticphonetic features that exhibit similarity across languages. We have carried out cross-lingual and multilingual experiments for the five languages. SVM, Random forest, MLP, XGBoost, and CNN classifiers were used for the experiments. A randomized search method with five-fold cross-validation was implemented for hyperparameter tuning, and the machine learning models were evaluated using the parameters obtained. Based on the experimental results, we have found that, machine learning models trained with the languages of a family are capable of recognizing emotions of other languages of the same family better than those of a different one.
References
[1] Al-onazi, B.B., Nauman, M.A., Jahangir, R., Malik, M.M., Alkhammash, E.H., Elshewey, A.M., 2022. Transformer-based multilingual speech emotion recognition using data augmentation and feature fusion. Applied Sciences 12, 9188.
[2] Burkhardt, F., Paeschke, A., Rolfes, M., Sendlmeier, W.F., Weiss, B., et al., 2005. A database of german emotional speech., in: Interspeech, pp. 1517–1520.
[3] Ekman, P., 1992. Are there basic emotions?
[4] Goel, S., Beigi, H., 2020. Cross lingual cross corpus speech emotion recognition. arXiv preprint arXiv:2003.07996 .
[5] Heracleous, P., Mohammad, Y., Yoneyama, A., 2020. Integrating language and emotion features for multilingual speech emotion recognition, in: International Conference on Human-Computer Interaction, Springer. pp. 187–196.
[6] Heracleous, P., Yoneyama, A., 2019. A comprehensive study on bilingual and multilingual speech emotion recognition using a twopass classification scheme. PloS one 14, e0220386.
[7] Hozjan, V., Kačič, Z., 2003. Context-independent multilingual emotion recognition from speech signals. International journal of speech technology 6, 311–320.
[8] Kamaruddin, N., Wahab, A., Quek, C., 2012. Cultural dependency analysis for understanding speech emotion. Expert Systems with Applications 39, 5115–5133.
[9] Koolagudi, S.G., Reddy, R., Yadav, J., Rao, K.S., 2011. Iitkgp-sehsc: Hindi speech corpus for emotion analysis, in: 2011 International conference on devices and communications (ICDeCom), IEEE. pp. 1–5.
[10] Lee, S.w., 2019. The generalization effect for multilingual speech emotion recognition across heterogeneous languages, in: ICASSP 2019-2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), IEEE. pp. 5881–5885.
[11] Li, X., Akagi, M., 2019. Improving multilingual speech emotion recognition by combining acoustic features in a three-layer model. Speech Communication 110, 1–12.
[12] Livingstone, S.R., Russo, F.A., 2018. The ryerson audio-visual database of emotional speech and song (ravdess): A dynamic, multimodal set of facial and vocal expressions in north american english. PloS one 13, e0196391.
[13] Maji, B., Swain, M., 2022. Advanced fusion-based speech emotion recognition system using a dual-attention mechanism with conv-caps and bi-gru features. Electronics 11, 1328.
[14] Monisha, S.T.A., Sultana, S., 2025. A deep learning approach toward analyzing the cross-lingual acoustic-phonetic similarities in multilingual speech emotion recognition. Journal of Electrical and Computer Engineering 2025, 4748790.
[15] Scotti, V., Galati, F., Sbattella, L., Tedesco, R., 2021. Combining deep and unsupervised features for multilingual speech emotion recognition, in: International Conference on Pattern Recognition, Springer. pp. 114–128.
[16] Sultana, S., Iqbal, M.Z., Selim, M.R., Rashid, M.M., Rahman, M.S., 2021a. Bangla speech emotion recognition and cross-lingual study using deep cnn and blstm networks. IEEE Access 10, 564–578.
[17] Sultana, S., Rahman, M.S., 2023. Acoustic feature analysis and optimization for bangla speech emotion recognition. Acoustical Science and Technology 44, 157–166.
[18] Sultana, S., Rahman, M.S., Selim, M.R., Iqbal, M.Z., 2021b. Sust bangla emotional speech corpus (subesco): An audio-only emotional speech corpus for bangla. Plos one 16, e0250173.
[19] Swain, M., Sahoo, S., Routray, A., Kabisatpathy, P., Kundu, J.N., 2015. Study of feature combination using hmm and svm for multilingual odiya speech emotion recognition. International Journal of Speech Technology 18, 387–393.
[20] Wang, C., Ren, Y., Zhang, N., Cui, F., Luo, S., 2022. Speech emotion recognition based on multi-feature and multi-lingual fusion. Multimedia Tools and Applications 81, 4897–4907.
[21] Williamson, J.D., 1978. Speech analyzer for analyzing pitch or frequency perturbations in individual speech pattern to determine the emotional state of the person. US Patent 4,093,821.
[22] Yadav, A., Vishwakarma, D.K., 2020. A multilingual framework of cnn and bi-lstm for emotion classification, in: 2020 11th International Conference on Computing, Communication and Networking Technologies (ICCCNT), IEEE. pp. 1–6.
[23] Yu, Y., 2022. Multilingual speech emotion research based on data mining. Advances in Multimedia 2022.
[24] Zehra, W., Javed, A.R., Jalil, Z., Khan, H.U., Gadekallu, T.R., 2021. Cross corpus multi-lingual speech emotion recognition using ensemble learning. Complex & Intelligent Systems 7, 1845–1854.
Published
Issue
Section
License
Copyright (c) 2026 AIUB Journal of Science and Engineering
This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License.
AJSE contents are under the terms of the Creative Commons Attribution License. This permits anyone to copy, distribute, transmit and adapt the work non-commercially provided the original work and source is appropriately cited.
