Penerapan Transfer Learning MobileNetV2 untuk Sistem Pengenalan Gerakan Tangan Interaktif

Marcellino Andelta Pinem; Fathony Mursyid; Eldika Rubiana; Abraham Imanuel Sinaga; Haikal Ryan Saputra

Authors

Marcellino Andelta Pinem, Universitas Bina Sarana Informatika, Indonesia
Fathony Mursyid, Universitas Bina Sarana Informatika, Indonesia
Eldika Rubiana, Universitas Bina Sarana Indonesia, Indonesia
Abraham Imanuel Sinaga, Universitas Bina Sarana Informatika, Indonesia
Haikal Ryan Saputra, Universitas Bina Sarana Informatika, Indonesia

Keywords:

Hand Gesture Recognition, Transfer Learning, MobileNetV2, Deep Learning, HaGRID Dataset, Two-phase Fine-Tuning

Abstract

Pengenalan gerakan tangan merupakan komponen penting dalam interaksi manusia-komputer yang memerlukan akurasi tinggi untuk aplikasi praktis. Penelitian ini menerapkan transfer learning MobileNetV2 dengan strategi two-phase fine-tuning untuk meningkatkan akurasi pengenalan tujuh gerakan tangan pada dataset HaGRID. Dataset terdiri dari 175.000 gambar yang terbagi menjadi 140.000 data latih, 17.500 data validasi, dan 17.500 data uji. Metode two-phase meliputi Phase 1 dengan frozen base layers menghasilkan akurasi 75,83%, dan Phase 2 dengan fine-tuning selective layers meningkatkan akurasi menjadi 98,88% pada data validasi dan 98,86% pada data uji. Peningkatan signifikan sebesar 23,05% berhasil dicapai hanya dalam 10 epochs total dengan durasi training 6,5 jam. Model berhasil mengeliminasi seluruh confusion pairs yang sebelumnya mencapai 18,64% pada Phase 1 menjadi 0% confusion di Phase 2. Kontribusi utama penelitian ini adalah demonstrasi strategi two-phase fine-tuning yang efisien untuk model lightweight dengan akurasi setara arsitektur kompleks, memberikan solusi praktis untuk implementasi sistem pengenalan gerakan real-time pada perangkat mobile dan embedded system tanpa mengorbankan performa.

Downloads

Download data is not yet available.

References

K. Aurangzeb, K. Javeed, M. Alhussein, I. Rida, S. I. Haider, and A. Parashar, “Deep Learning Approach for Hand Gesture Recognition: Applications in Deaf Communication and Healthcare,” Computers, Materials & Continua, vol. 78, no. 1, pp. 127–144, 2024, doi: 10.32604/cmc.2023.042886.

A. Mujahid et al., “Real-Time Hand Gesture Recognition Based on Deep Learning YOLOv3 Model,” Applied Sciences, vol. 11, no. 9, p. 4164, May 2021, doi: 10.3390/app11094164.

M. A. Haq, L. N. Q. Huy, M. Ridlwan, and I. Naila, “Leveraging Self-Attention Mechanism for Deep Learning in Hand-Gesture Recognition System,” E3S Web of Conferences, vol. 500, p. 01009, Mar. 2024, doi: 10.1051/e3sconf/202450001009.

M. Rahim, A. S. M. Miah, H. Akash, J. Shin, M. Hossain, and M. Hossain, An Advanced Deep Learning Based Three-Stream Hybrid Model for Dynamic Hand Gesture Recognition. 2024. doi: 10.48550/arXiv.2408.08035.

Yaseen, O.-J. Kwon, J. Kim, S. Jamil, J. Lee, and F. Ullah, “Next-Gen Dynamic Hand Gesture Recognition: MediaPipe, Inception-v3 and LSTM-Based Enhanced Deep Learning Model,” Electronics (Basel), vol. 13, no. 16, p. 3233, Aug. 2024, doi: 10.3390/electronics13163233.

N. Zerrouki et al., “Deep Learning for Hand Gesture Recognition in Virtual Museum Using Wearable Vision Sensors,” IEEE Sens J, vol. 24, no. 6, pp. 8857–8869, Mar. 2024, doi: 10.1109/JSEN.2024.3354784.

Md. A. A. Faisal, F. F. Abir, M. U. Ahmed, and M. A. R. Ahad, “Exploiting domain transformation and deep learning for hand gesture recognition using a low-cost dataglove,” Sci Rep, vol. 12, no. 1, p. 21446, Dec. 2022, doi: 10.1038/s41598-022-25108-2.

Y. Gulzar, “Fruit Image Classification Model Based on MobileNetV2 with Deep Transfer Learning Technique,” Sustainability, vol. 15, no. 3, p. 1906, Jan. 2023, doi: 10.3390/su15031906.

R. K. Banoth and B. V. R. Murthy, “Soil Image Classification Using Transfer Learning Approach: MobileNetV2 with CNN,” SN Comput Sci, vol. 5, no. 1, p. 199, Jan. 2024, doi: 10.1007/s42979-023-02500-x.

T. Barman and S. Susan, “Multi-Label Remote Sensing Image Classification using MobileNetV2,” in 2024 15th International Conference on Computing Communication and Networking Technologies (ICCCNT), IEEE, Jun. 2024, pp. 1–4. doi: 10.1109/ICCCNT61001.2024.10725506.

Q. Xiang, X. Wang, R. Li, G. Zhang, J. Lai, and Q. Hu, “Fruit Image Classification Based on MobileNetV2 with Transfer Learning Technique,” in Proceedings of the 3rd International Conference on Computer Science and Application Engineering, New York, NY, USA: ACM, Oct. 2019, pp. 1–7. doi: 10.1145/3331453.3361658.

K. Alexander, K. Karina, N. Alexander, K. Roman, and M. Andrei, “HaGRID – HAnd Gesture Recognition Image Dataset,” in 2024 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), IEEE, Jan. 2024, pp. 4560–4569. doi: 10.1109/WACV57701.2024.00451.

A. Nuzhdin, A. Nagaev, A. Sautin, A. Kapitanov, and K. Kvanchiani, “HaGRIDv2: 1M Images for Static and Dynamic Hand Gesture Recognition,” 2025. doi: 10.24132/CSRN.2025-1.

T. ValizadehAslani et al., “Two-stage fine-tuning with ChatGPT data augmentation for learning class-imbalanced data,” Neurocomputing, vol. 592, p. 127801, Aug. 2024, doi: 10.1016/j.neucom.2024.127801.

T. ValizadehAslani et al., “Two-Stage Fine-Tuning: A Novel Strategy for Learning Class-Imbalanced Data,” Jul. 2022.