Assistive mobile application for visually impaired based on real time object recognition using machine learning with voice feedback

Abstract

Visual impairment significantly restricts individuals’ autonomy, mobility, and capacity to interact with their surroundings. According to the World Health Organization, more than 2.2 billion people live with vision impairment, many of whom face daily challenges in navigating unfamiliar environments, avoiding obstacles, and identifying essential objects [1]. Recent advancements in artificial intelligence (AI) and computer vision offer promising opportunities to develop assistive technologies capable of addressing these challenges through real-time environmental understanding. This project presents the development of a mobile application designed to assist visually impaired individuals through real-time object detection combined with audio feedback. Three different approaches were explored. The first two approaches used custom deep- learning models based on YOLOv5, trained on datasets of varying size and complexity. While these models demonstrated good accuracy, deployment on mobile devices proved challenging due to computational constraints and limited class coverage. The final approach leveraged the Google Cloud Vision API to off-load inference to the cloud, enabling broad object recognition without the need for local training or heavy computation. The resulting system captures an image through the Android camera, processes it via a FastAPI backend, and returns a concise voice description of detected objects. User interviews guided the design of the interface, ensuring accessibility and simplicity for individuals with different levels of visual impairment. The results demonstrate that cloud-based computer vision significantly improves recognition breadth, reliability, and real-time performance. However, trade-offs include dependence on network connectivity and third-party services. The final prototype shows strong potential as a low-cost, scalable assistive solution. Future work will explore hybrid systems combining offline detection for essential objects with cloud-based recognition for complex scenes.