Mixture reality-based assistive system for visually impaired people

Individuals with visual impairments often face challenges in their daily lives, particularly in terms of independent mobility. To address this issue, we present a mixed reality-based assistive system for visually impaired individuals, which comprises a Microsoft Hololens2 device and a website and utilizes a simultaneous localization and mapping (SLAM) algorithm to capture various large indoor scenes in real-time. This system incorporates remote multi-person assistance technology and navigation technology to aid visually impaired individuals. To evaluate the effectiveness of our system, we conducted an experiment in which several participants completed a large indoor scene maintenance task. Our experimental results demonstrate that the system is robust and can be utilized in a wide range of indoor environments. Additionally, the system enhances environmental perception and enables visually impaired individuals to navigate independently, thus facilitating successful task completion.     

A computer vision-based perception system for visually impaired

In this paper, we introduce a novel computer vision-based perception system, dedicated to the autonomous navigation of visually impaired people. A first feature concerns the real-time detection and recognition of obstacles and moving objects present in potentially cluttered urban scenes. To this purpose, a motion-based, real-time object detection and classification method is proposed. The method requires no a priori information about the obstacle type, size, position or location. In order to enhance the navigation/positioning capabilities offered by traditional GPS-based approaches, which are often unreliably in urban environments, a building/landmark recognition approach is also proposed. Finally, for the specific case of indoor applications, the system has the possibility to learn a set of user-defined objects of interest. Here, multi-object identification and tracking is applied in order to guide the user to localize such objects of interest. The feedback is presented to user by audio warnings/alerts/indications. Bone conduction headphones are employed in order to allow visually impaired to hear the systems warnings without obstructing the sounds from the environment. At the hardware level, the system is totally integrated on an android smartphone which makes it easy to wear, non-invasive and low-cost.

     

Interactive mixed reality white cane simulation for the training of the blind and the visually impaired

This paper presents a mixed reality tool developed for the training of the visually impaired based on haptic and auditory feedback. The proposed approach focuses on the development of a highly interactive and extensible Haptic Mixed Reality training system that allows visually impaired to navigate into real size Virtual Reality environments. The system is based on the use of the CyberGrasp™ haptic device. An efficient collision detection algorithm based on superquadrics is also integrated into the system so as to allow real time collision detection in complex environments. A set of evaluation tests is designed in order to identify the importance of haptic, auditory and multimodal feedback and to compare the MR cane against the existing Virtual Reality cane simulation system.     

Context-aware obstacle detection for navigation by visually impaired

This paper presents a context-aware smartphone-based based visual obstacle detection approach to aid visually impaired people in navigating indoor environments. The approach is based on processing two consecutive frames (images), computing optical flow, and tracking certain points to detect obstacles. The frame rate of the video stream is determined using a context-aware data fusion technique for the sensors on smartphones. Through an efficient and novel algorithm, a point dataset on each consecutive frames is designed and evaluated to check whether the points belong to an obstacle. In addition to determining the points based on the texture in each frame, our algorithm also considers the heading of user movement to find critical areas on the image plane. We validated the algorithm through experiments by comparing it against two comparable algorithms. The experiments were conducted in different indoor settings and the results based on precision, recall, accuracy, and f-measure were compared and analyzed. The results show that, in comparison to the other two widely used algorithms for this process, our algorithm is more precise. We also considered time-to-contact parameter for clustering the points and presented the improvement of the performance of clustering by using this parameter.     

Guide robot intelligent navigation in urban environments

This paper presents an intelligent robotic system to guide visually impaired people in urban environments. The robot is equipped with two laser range finders, global positioning system (GPS), camera, and compass sensors. All the sensors data are processed by a single laptop computer. We have implemented different navigation algorithms enabling the robot to move autonomously in different urban environments. In pedestrian walkways, we utilize the distance to the edge (left, right, or both) to determine the robot steering command. In difference from pedestrian walkways, in open squares where there is no edge information, artificial neural networks map the GPS and compass sensor data to robot steering command guiding the visually impaired to the goal location. The neural controller is designed such as to be employed even in environments different from those in which they have been evolved. Another important advantage is that a single neural network controls the robot to reach multiple goal locations inside the open square. The proposed algorithms are verified experimentally in a navigation task inside the University of Toyama Campus, where the robot moves from the initial to goal location.     

Toward a computer vision-based wayfinding aid for blind persons to access unfamiliar indoor environments

Independent travel is a well-known challenge for blind and visually impaired persons. In this paper, we propose a proof-of-concept computer vision-based wayfinding aid for blind people to independently access unfamiliar indoor environments. In order to find different rooms (e.g. an office, a laboratory, or a bathroom) and other building amenities (e.g. an exit or an elevator), we incorporate object detection with text recognition. First, we develop a robust and efficient algorithm to detect doors, elevators, and cabinets based on their general geometric shape, by combining edges and corners. The algorithm is general enough to handle large intra-class variations of objects with different appearances among different indoor environments, as well as small inter-class differences between different objects such as doors and door-like cabinets. Next, to distinguish intra-class objects (e.g. an office door from a bathroom door), we extract and recognize text information associated with the detected objects. For text recognition, we first extract text regions from signs with multiple colors and possibly complex backgrounds, and then apply character localization and topological analysis to filter out background interference. The extracted text is recognized using off-the-shelf optical character recognition software products. The object type, orientation, location, and text information are presented to the blind traveler as speech.     

Seeing Eye Phone: a smart phone-based indoor localization and guidance system for the visually impaired

In order to help the visually impaired as they navigate unfamiliar environment such as public buildings, this paper presents a novel smart phone, vision-based indoor localization, and guidance system, called Seeing Eye Phone. This system requires a smart phone from the user and a server. The smart phone captures and transmits images of the user facing forward to the server. The server processes the phone images to detect and describe 2D features by SURF and then matches them to the 2D features of the stored map images that include their corresponding 3D information of the building. After features are matched, Direct Linear Transform runs on a subset of correspondences to find a rough initial pose estimate and the Levenberg–Marquardt algorithm further refines the pose estimate to find a more optimal solution. With the estimated pose and the camera’s intrinsic parameters, the location and orientation of the user are calculated using 3D location correspondence data stored for features of each image. Positional information is then transmitted back to the smart phone and communicated to the user via text-to-speech. This indoor guiding system uses efficient algorithms such as SURF, homographs, multi-view geometry, and 3D to 2D reprojection to solve a very unique problem that will benefit the visually impaired. The experimental results demonstrate the feasibility of using a simple machine vision system design to accomplish a complex task and the potential of building a commercial product based on this design.     

A Multiagent Approach to Qualitative Landmark-Based Navigation

In this paper we present a multiagent system for landmark-based navigation in unknown environments. We propose a bidding mechanism to coordinate the actions requested by the different agents. The navigation system has been tested on a real robot on indoor unstructured environments.     

Design and development of an indoor navigation system using denoising autoencoder based convolutional neural network for visually impaired people

A challenging area of research is the development of a navigation system for visually impaired people in an indoor environment such as a railway station, commercial complex, educational institution, and airport. Identifying the current location of the users can be a difficult task for those with visual impairments. The entire selection of the navigation path depends upon the current location of the user. This work presents a detailed analysis of the recent user positioning techniques and methodologies on the indoor navigation system based on the parameters, such as techniques, cost, the feasibility of implementation, and limitations. This paper presents a denoising auto encoder based on the convolutional neural network (DAECNN) to identify the present location of the users. The proposed approach uses the de-noising autoencoder to reconstruct the noisy image and the convolution neural network (CNN) to classify the users' current position. The proposed method is compared with the existing deep learning approaches such as deep autoencoder, sparse autoencoder, CNN, multilayer perceptron, radial basis function neural network, and the performances are analyzed. The experimental findings indicate that the DAECNN methodology works better than the existing classification approaches.

     

An Open Road Evaluation of a Self-Driving Vehicle Human–Machine Interface Designed for Visually Impaired Users

ABSTRACT

Fully autonomous or “self-driving” vehicles are an emerging technology that may hold tremendous mobility potential for individuals who are visually impaired who have been previously disadvantaged by an inability to operate conventional motor vehicles. Prior studies however, have suggested that these consumers have significant concerns regarding the accessibility of this technology and their ability to effectively interact with it. We present the results of a quasi-naturalistic study, conducted on public roads with 20 visually impaired users, designed to test a self-driving vehicle human–machine interface. This prototype system, ATLAS, was designed in participatory workshops in collaboration with visually impaired persons with the intent of satisfying the experiential needs of blind and low vision users. Our results show that following interaction with the prototype, participants expressed an increased trust in self-driving vehicle technology, an increased belief in its likely usability, an increased desire to purchase it and a reduced fear of operational failures. These findings suggest that interaction with even a simulated self-driving vehicle may be sufficient to ameliorate feelings of distrust regarding the technology and that existing technologies, properly combined, are promising solutions in addressing the experiential needs of visually impaired persons in similar contexts.     

Ontology-assisted automatic precise information extractor for visually impaired inhabitants

As the internet grows rapidly, millions of web pages are being added on a daily basis. The extraction of precise information is becoming more and more difficult as the volume of data on the internet increases. Several search engines and information fetching tools are available on the internet, all of which claim to provide the best crawling facilities. For the most part, these search engines are keyword based. This poses a problem for visually impaired people who want to get the full use from online resources available to other users. Visually impaired users require special aid to get?along with any given computer system. Interface and content management are no exception, and special tools are required to facilitate the extraction of relevant information from the internet for visually impaired users. The HOIEV (Heavyweight Ontology Based Information Extraction for Visually impaired User) architecture provides a mechanism for highly precise information extraction using heavyweight ontology and built-in vocal command system for visually impaired internet users. Our prototype intelligent system not only integrates and communicates among different tools, such as voice command parsers, domain ontology extractors and short message engines, but also introduces an autonomous mechanism of information extraction (IE) using heavyweight ontology. In this research we designed domain specific heavyweight ontology using OWL 2 (Web Ontology Language 2) and for axiom writing we used PAL (Protégé Axiom Language). We introduced a novel autonomous mechanism for IE by developing prototype software. A series of experiments were designed for the testing and analysis of the performance of heavyweight ontology in general, and our information extraction prototype specifically.     

Face detection and recognition in an unconstrained environment for mobile visual assistive system

We present a visual assistive system that features mobile face detection and recognition in an unconstrained environment from a mobile source using convolutional neural networks. The goal of the system is to effectively detect individuals that approach facing towards the person equipped with the system. We find that face detection and recognition becomes a very difficult task due to the movement of the user which causes camera shakes resulting in motion blur and noise in the input for the visual assistive system. Due to the shortage of related datasets, we create a dataset of videos captured from a mobile source that features motion blur and noise from camera shakes. This makes the application a very challenging aspect of face detection and recognition in unconstrained environments. The performance of the convolutional neural network is further compared with a cascade classifier. The results show promising performance in daylight and artificial lighting conditions while the challenges lie for moonlight conditions with the need for reduction of false positives in order to develop a robust system. We also provide a framework for implementation of the system with smartphones and wearable devices for video input and auditory notification from the system to guide the visually impaired.     

BETWEEN LOCATION AWARENESS AND AWARE LOCATIONS: WHERE TO PUT THE INTELLIGENCE

This article presents a review on vision-aided systems and proposes an approach for visual rehabilitation using stereo vision technology. The proposed system utilizes stereo vision, image processing methodology, and a sonification procedure to support blind mobilization. The developed system includes wearable computer, stereo cameras as vision sensor, and stereo earphones, all molded in a helmet. The image of the scene in front of the visually handicapped is captured by the vision sensors. The captured images are processed to enhance the important features in the scene in front for mobilization assistance. The image processing is designed as a model of human vision by identifying the obstacles and their depth information. The processed image is mapped onto musical stereo sound for the blind's understanding of the scene in front. The developed method has been tested in the indoor and outdoor environments and the proposed image processing methodology is found to be effective for object identification.     

Visualizing for the Non‐Visual: Enabling the Visually Impaired to Use Visualization

The majority of visualizations on the web are still stored as raster images, making them inaccessible to visually impaired users. We propose a deep‐neural‐network‐based approach that automatically recognizes key elements in a visualization, including a visualization type, graphical elements, labels, legends, and most importantly, the original data conveyed in the visualization. We leverage such extracted information to provide visually impaired people with the reading of the extracted information. Based on interviews with visually impaired users, we built a Google Chrome extension designed to work with screen reader software to automatically decode charts on a webpage using our pipeline. We compared the performance of the back‐end algorithm with existing methods and evaluated the utility using qualitative feedback from visually impaired users.     

Sea of images

Our sea of images approach provides new methods for acquiring, analyzing, representing, and rendering photorealistic models of complex indoor environments. We present our image-based rendering walk-through system based on the sea of images approach. We describe the system and give results for its implementation in three environments of different sizes and types.     

Feature extraction method for a robot map using neural networks

Many map-building algorithms using ultrasonic sensors have been developed for mobile robot applications. In indoor environments, the ultrasonic sensor system gives some uncertain data. To compensate for this effect, a new feature extraction method using neural networks is proposed. A new, effective representation of the target is defined, and the reflection wave data patterns are learnt using neural networks. As a consequence, the targets are classified as planes, corners, or edges, which all frequently occur in indoor environments. We constructed our own robot system for the experiments which were carried out to show the performance. This work was presented in part at the 7th International Symposium on Artificial Life and Robotics, Oita, Japan, January 16–18, 2002     

Artificial intelligence for visually impaired

The eyes are an essential tool for human observation and perception of the world, helping people to perform their tasks. Visual impairment causes many inconveniences in the lives of visually impaired people. Therefore, it is necessary to focus on the needs of the visually impaired community. Researchers work from different angles to help visually impaired people live normal lives. The advent of the digital age has profoundly changed the lives of the visually impaired community, making life more convenient. Deep learning, as a promising technology, is also expected to improve the lives of visually impaired people. It is increasingly being used in the diagnosis of eye diseases and the development of visual aids. The earlier accurate diagnosis of the eye disease by the doctor, the sooner the patient can receive the appropriate treatment and the better chances of a cure. This paper summarises recent research on the development of artificial intelligence-based eye disease diagnosis and visual aids. The research is divided according to the purpose of the study into deep learning methods applied in diagnosing eye diseases and smart devices to help visually impaired people in their daily lives. Finally, a summary is given of the directions in which artificial intelligence may be able to assist the visually impaired in the future. In addition, this overview provides some knowledge about deep learning for beginners. We hope this paper will inspire future work on the subjects..     

Reconstructing the World’s Museums

Virtual exploration tools for large indoor environments (e.g. museums) have so far been limited to either blueprint-style 2D maps that lack photo-realistic views of scenes, or ground-level image-to-image transitions, which are immersive but ill-suited for navigation. On the other hand, photorealistic aerial maps would be a useful navigational guide for large indoor environments, but it is impossible to directly acquire photographs covering a large indoor environment from aerial viewpoints. This paper presents a 3D reconstruction and visualization system for automatically producing clean and well-regularized texture-mapped 3D models for large indoor scenes, from ground-level photographs and 3D laser points. The key component is a new algorithm called “inverse constructive solid geometry (CSG)” for reconstructing a scene with a CSG representation consisting of volumetric primitives, which imposes powerful regularization constraints. We also propose several novel techniques to adjust the 3D model to make it suitable for rendering the 3D maps from aerial viewpoints. The visualization system enables users to easily browse a large-scale indoor environment from a bird’s-eye view, locate specific room interiors, fly into a place of interest, view immersive ground-level panorama views, and zoom out again, all with seamless 3D transitions. We demonstrate our system on various museums, including the Metropolitan Museum of Art in New York City—one of the largest art galleries in the world.     

A wearable system for mobility improvement of visually impaired people

Degradation of the visual system can lead to a dramatic reduction of mobility by limiting a person to his sense of touch and hearing. This paper presents the development of an obstacle detection system for visually impaired people. While moving in his environment the user is alerted to close obstacles in range. The system we propose detects an obstacle surrounding the user by using a multi-sonar system and sending appropriate vibrotactile feedback. The system aims at increasing the mobility of visually impaired people by offering new sensing abilities.     

Accessibility barriers for users of screen readers in the Moodle learning content management system

In recent decades, the use of the Internet has spread rapidly into diverse social spheres including that of education. Currently, most educational centers make use of e-learning environments created through authoring tool applications like learning content management systems (LCMSs). However, most of these applications currently present accessibility barriers that make the creation of accessible e-learning environments difficult for teachers and administrators. In this paper, the accessibility of the Moodle authoring tool, one of the most frequently used LCMSs worldwide, is evaluated. More specifically, the evaluation is carried out from the perspective of two visually impaired users accessing content through screen readers, as well as a heuristic evaluation considering the World Wide Web Consortium’s Authoring Tool Accessibility Guidelines. The evaluation results demonstrate that Moodle presents barriers for screen reader users, limiting their ability to access the tool. One example of accessibility problems for visually impaired users is the frequent inability to publish learning contents without assistance. In light of these results, the paper offers recommendations that can be followed to reduce or eliminate these accessibility barriers.