Multimodal Image Perception System for Visually Impaired Users
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current assistive technologies for visually impaired individuals are inadequate in conveying the richness and complexity of visual data, limiting their participation in scientific research and experimentation, as they rely on single-modality human-computer interfaces that cannot effectively substitute visual information with tactile or auditory feedback.
Innovation Solution
A real-time multimodal image perception system that uses a combination of auditory, haptics, and vibrotactile feedback, employing a Bayesian network to characterize images through primary and peripheral features, allowing visually impaired users to interpret data via multiple sensory modalities.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of information
If single-modality sensory substitution technologies are used, then the system complexity is reduced, but the information richness and measurement precision are insufficient
Solution Approach 1:
The patent combines multiple sensory modalities (auditory, tactile, and proprioceptive feedback) into a single integrated system. The auditory-vision sensory substitution device integrates audio output capabilities with navigation controls, while tactile feedback is incorporated through haptic actuators in the headphones and controller, creating a unified multimodal interface that preserves comprehensive visual information without requiring separate complex systems for each modality.
Solution Approach 2:
The sensory substitution device serves multiple functions simultaneously: it provides spatial localization through auditory cues, object recognition through audio patterns, navigation guidance through proprioceptive feedback, and environmental mapping through integrated feedback loops. This multi-functionality allows a single device to replace multiple specialized tools, reducing overall system complexity while maintaining information richness.
2Measurement precision
If high-resolution tactile display arrays are used, then the measurement precision of visual information is improved, but the ease of operation and device complexity are reduced
Solution Approach 1:
The patent replaces complex mechanical tactile display arrays with an auditory-based sensory substitution system. Instead of using mechanical actuators to create high-resolution tactile patterns on the user's skin, the system uses audio signals to convey spatial and object information. This substitution maintains measurement precision through auditory spatial resolution while dramatically improving ease of operation, as audio feedback requires no complex mechanical components and can be processed naturally by the human auditory system.
3Measurement precision
If auditory-vision sensory substitution is used, then the measurement precision for localization and object recognition is improved, but the loss of time for training and memorization increases
Solution Approach 1:
The system incorporates real-time feedback loops that provide immediate auditory and tactile responses to user navigation actions and environmental features. This feedback mechanism allows users to learn spatial relationships and object characteristics through direct experience rather than memorization, significantly reducing training time. The proprioceptive feedback from haptic actuators provides continuous guidance that reinforces learning without requiring extensive preparatory training.
Solution Approach 2:
The system pre-processes visual information into auditory cues that directly map to spatial and object characteristics, eliminating the need for users to learn complex audio-visual mappings. By preparing the sensory substitution in advance and presenting information in naturally interpretable formats, the system reduces the time required for users to adapt and achieve accurate localization and object recognition.
4Loss of information
If multiple sensory modalities are integrated, then the information richness and measurement precision are improved, but the device complexity and manufacturing precision increase
Solution Approach 1:
The patent segments the multimodal sensory substitution system into modular functional components: an auditory processing module, a tactile feedback module, a navigation control module, and an integration layer. Each module can be developed, tested, and manufactured independently using standard audio and haptic technologies, then assembled through well-defined interfaces. This segmentation maintains information richness while significantly improving ease of manufacture compared to attempting to build a monolithic multimodal system.
Data Source
AI summary
A real-time multimodal image perception system to transform the standard lab blood smear image for persons with BVI to perceive, employing a combination of auditory, haptic, and vibrotactile feedbacks. These sensory feedbacks are used to convey visual information in appropriate perceptual channels, thus creating a palette of multimodal, sensorial information. A Bayesian network is provided to characterize images through two groups of features of interest: primary and peripheral features. A method is provided for optimal matching between primary features and sensory modalities.