INTUITIVE GESTURE-DRIVEN AUGMENTED REALITY FOR USER INTERACTION IN LOW-COST AR SYSTEMS
DOI:
https://doi.org/10.33003/fjs-2025-0910-3908Keywords:
Augmented Reality, Gesture Recognition, Human-computer Interaction, Low-Cost AR, MediaPipe, OpenCVAbstract
Augmented Reality (AR) offers promising applications in education, healthcare, and industry, but its adoption in low-resource settings is hindered by expensive hardware and complex interaction methods. This study presents a lightweight, gesture-driven AR interface designed to enable intuitive user interaction on low-cost devices. Using standard webcams and open-source tools, MediaPipe for hand landmark detection and OpenCV for visual overlays, a rule-based system was developed to recognize three core gestures (open palm, fist, point) and trigger real-time AR overlays. The system was optimized for latency and frame rate, achieving 81.8% accuracy in gesture recognition during user testing with 20 participants. Real-time performance averaged 44.8 ms latency and 24.6 FPS, demonstrating responsive feedback on entry-level hardware. Despite limitations such as variable lighting and lack of depth sensing, the system proved intuitive and effective for immersive interaction. This work contributes a scalable AR interaction model that enhances accessibility in educational and low-cost environments.
References
Azuma, R. T. (1997). A survey of augmented reality. Presence: Teleoperators and Virtual Environments, 6(4), 355–385. https://doi.org/10.1162/pres.1997.6.4.355
Billinghurst, M., Clark, A., and Lee, G. (2015). A survey of augmented reality. Foundations and Trends® in Human–Computer Interaction, 8(2–3), 73–272. https://doi.org/10.1561/1100000049
Cao, J., Lam, K.-Y., Lee, L.-H., Liu, X., Hui, P., and Su, X. (2022). Mobile augmented reality: User interfaces, frameworks, and intelligence. ACM Computing Surveys, 55(9). https://doi.org/10.1145/3557999
Carmigniani, J., Furht, B., Anisetti, M., Ceravolo, P., Damiani, E., and Ivkovic, M. (2011). Augmented reality technologies, systems and applications. Multimedia Tools and Applications, 51(1), 341–377. https://doi.org/10.1007/s11042-010-0660-6
Ghazwani, Y., and Smith, S. (2020). Interaction in augmented reality. In Proceedings of the 2020 4th International Conference on Virtual and Augmented Reality Simulations*. https://doi.org/10.1145/3385378.3385384
Guna, J., Jakus, G., Pogačnik, M., Tomažič, S., and Sodnik, J. (2014). An analysis of the precision and reliability of the Leap Motion sensor and its suitability for static and dynamic tracking. Sensors, 14(2), 3702–3720. https://doi.org/10.3390/s140203702
Lee, K., Lee, J., and Lee, J. Y. (2018). Comparative study on augmented reality approaches for interactive learning. Interactive Learning Environments, 26(4), 516–529. https://doi.org/10.1080/10494820.2017.1337038
Lugaresi, C., Tang, J., Nash, H., McClanahan, C., Uboweja, E., Hays, M., Zhang, F., ... and Grundmann, M. (2019). MediaPipe: A framework for building perception pipelines. arXiv Preprint arXiv:1906.08172. https://arxiv.org/abs/1906.08172
Majdoub Bhiri, N., Ameur, S., Alouani, I., Mahjoub, M. A., and Ben Khalifa, A. (2023). Hand gesture recognition with focus on leap motion: An overview, real-world challenges, and future directions. Expert Systems with Applications, 226, 120125. https://doi.org/10.1016/j.eswa.2023.120125
Pavlovic, V. I., Sharma, R., and Huang, T. S. (1997). Visual interpretation of hand gestures for human-computer interaction: A review. IEEE Transactions on Pattern Analysis and Machine Intelligence, 19(7), 677–695. https://doi.org/10.1109/34.598226
Rautaray, S. S., and Agrawal, A. (2015). Vision based hand gesture recognition for human computer interaction: A survey. Artificial Intelligence Review, 43(1), 1–54. https://doi.org/10.1007/s10462-012-9356-9
Sanna, A., Lamberti, F., Paravati, G., and Manuri, F. (2013). A Kinect-based natural interface for quadrotor control. Entertainment Computing, 4(3), 179–186. https://doi.org/10.1016/j.entcom.2013.01.001
Shen, J., Dudley, J., Mo, G., and Kristensson, P. O. (2022). Gesture spotter: A rapid prototyping tool for key gesture spotting in virtual and augmented reality applications. IEEE Transactions on Visualization and Computer Graphics, 28(11), 3618–3628. https://doi.org/10.1109/tvcg.2022.3203004
Syed, T. A., and et al. (2023). In-depth review of augmented reality: Tracking technologies, development tools, AR displays, collaborative AR, and security concerns. Sensors, 23(1), 146. https://www.mdpi.com/1424-8220/23/1/146
Tati, S., Sahu, N. K., and Rath, S. K. (2019). A robust hand gesture recognition system using MediaPipe framework. International Journal of Interactive Multimedia and Artificial Intelligence, 5(7), 52–59. https://doi.org/10.9781/ijimai.2019.07.003
Torres, W., and et al. (2024). A framework for real-time gestural recognition and augmented reality for industrial applications. Sensors, 24(8), 2407. https://doi.org/10.3390/s24082407
Vertucci, R., D’Onofrio, S., Ricciardi, S., and De Nino, M. (2023). History of augmented reality. In Springer Handbooks (pp. 35–50). https://doi.org/10.1007/978-3-030-67822-7_2
Wang, X., Kim, M. J., Love, P. E., and Kang, S. C. (2013). Augmented reality in built environment: Classification and implications for future research. Automation in Construction, 32, 1–13. https://doi.org/10.1016/j.autcon.2012.11.021
Zhang, X., and Zhu, S. (2019). Real-time hand gesture recognition based on deep features and support vector machines. IEEE Access, 7, 103138–103145. https://doi.org/10.1109/ACCESS.2019.2931064
Zhou, Z., and Deng, Z. (2017). Rule-based hand gesture recognition for human–computer interaction. Multimedia Tools and Applications, 76(19), 20067–20083. https://doi.org/10.1007/s11042-016-4028-5
Downloads
Published
Issue
Section
Categories
License
Copyright (c) 2025 Bilkisu Larai Muhammad-Bello, Awoke Victor Ndubuisi, Muhammad Aliyu Suleiman, Ibrahim Anka Salihu
This work is licensed under a Creative Commons Attribution 4.0 International License.
