Control method for blind guidance device, and related product

Abstract

Provided in the present application are a control method for a blind guidance device, and a related product. The method comprises: after determining a navigation route for a user from a current position to a destination, acquiring a visual image and environmental detection data of a reference region corresponding to the current position; fusing first feature data extracted from the visual image and second feature data extracted from the environmental detection data, so as to obtain a reference feature data set of target objects in the reference region; on the basis of the reference feature data set, determining relative position parameters between all the target objects and the current position; and when there is an obstacle in the traveling direction of the user and the distance between the obstacle and the current position is less than or equal to a preset threshold value, updating the navigation route on the basis of the relative position parameters corresponding to the obstacle, and generating a corresponding navigation control signal. By means of the implementation of the solution of the present application, accurate and comprehensive road condition analysis can be realized, thereby providing safe and efficient obstacle avoidance guidance for a user.

Description

A control method for guide devices and related products Technical Field

[0001] This application relates to the field of guide assist technology, and in particular to a control method for guide devices and related products. Background Technology

[0002] Visually impaired individuals, as a vulnerable group in society, lack a crucial means of perceiving the outside world, facing numerous challenges in their work and daily lives. Among relevant technologies, the mainstream methods of guidance for the visually impaired involve installing tactile paving and training guide dogs. However, the functionality of existing tactile paving is very limited; users cannot accurately determine their surroundings solely based on it. Furthermore, existing tactile paving is typically tactile and requires special paving stones, making construction cumbersome and repairs difficult. In practice, the utilization rate of existing tactile paving is also very low. Meanwhile, the number of guide dogs is limited, training costs are high, and universality is lacking. The efficiency and safety of travel for visually impaired individuals need improvement. Technical issues

[0003] This application provides a method for controlling guide devices and related products, which can at least solve the problem of inconvenience for visually impaired patients in traveling in related technologies. Technical solutions

[0004] The first aspect of this application provides a method for controlling a guide device, including:

[0005] After determining the navigation route for the user to reach their destination from their current location, visual images and environmental detection data of a reference area corresponding to the current location are acquired; the extent of the reference area is determined based on the current location and the navigation route.

[0006] The first feature data extracted from the visual image and the second feature data extracted from the environmental detection data are fused to obtain a reference feature dataset of the target object in the reference region; wherein the target object includes obstacles and non-obstacles.

[0007] Based on the reference feature dataset, determine the relative position parameters between all target objects and the current position;

[0008] When the preset navigation control conditions are met based on all relative position parameters, the navigation route is updated according to the relative position parameters corresponding to the obstacles, and a corresponding navigation control signal is generated. The navigation control conditions include the presence of obstacles in the user's direction of travel and the distance between the obstacles and the current position being less than or equal to a preset threshold. The navigation control signal is used to control the output of navigation information by the guide device. The navigation information is used to guide the user to avoid obstacles.

[0009] A second aspect of this application provides a control device for a guide device for the visually impaired, comprising:

[0010] The acquisition module is used to acquire visual images and environmental detection data of a reference area corresponding to the current location after determining the navigation route for the user to reach the destination from the current location; wherein the range of the reference area is determined based on the current location and the navigation route;

[0011] The fusion module is used to fuse the first feature data extracted from the visual image and the second feature data extracted from the environmental detection data to obtain a reference feature dataset of the target object in the reference region; wherein the target object includes obstacles and non-obstacles;

[0012] The determination module is used to determine the relative position parameters between all target objects and the current position based on the reference feature dataset;

[0013] The generation module is used to update the navigation route according to the relative position parameters corresponding to the obstacles and generate corresponding navigation control signals when it is determined that the preset navigation control conditions are met based on all relative position parameters. The navigation control conditions include the presence of obstacles in the user's direction of travel and the distance between the obstacles and the current position being less than or equal to a preset threshold. The navigation control signals are used to control the guide device to output navigation information. The navigation information is used to guide the user to avoid obstacles.

[0014] A third aspect of this application provides a guide device for the visually impaired, comprising: a guide device, a memory, and a controller. The guide device is equipped with an audio output component, an image acquisition unit, and a lidar detector, all of which are communicatively connected to the controller. The audio output component is used to output navigation information based on navigation control signals sent by the controller. The image acquisition unit is used to acquire visual images, and the lidar detector is used to acquire environmental detection data. The controller is used to execute a computer program stored in the memory. When the controller executes the computer program, it implements the steps of the guide device control method provided in the first aspect of this application.

[0015] The fourth aspect of this application provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a controller, it implements the steps of the guide device control method provided in the first aspect of this application. Beneficial effects

[0016] As can be seen from the above, according to the guide device control method and related products provided in this application, after determining the navigation route for the user to reach the destination from the current location, visual images and environmental detection data of the reference area corresponding to the current location are acquired; wherein, the range of the reference area is determined based on the current location and the navigation route; the first feature data extracted from the visual image and the second feature data extracted from the environmental detection data are fused to obtain a reference feature dataset of target objects in the reference area; wherein, the target objects include obstacles and non-obstacles; based on the reference feature dataset, the relative position parameters between all target objects and the current location are determined; when it is determined based on all relative position parameters that the preset navigation control conditions are met, the navigation route is updated according to the relative position parameters corresponding to the obstacles, and a corresponding navigation control signal is generated; wherein, the navigation control conditions include the presence of obstacles in the user's direction of travel, and the distance between the obstacles and the current location is less than or equal to a preset threshold; the navigation control signal is used to control the guide device to output navigation information; the navigation information is used to guide the user to avoid obstacles. By implementing the solution in this application, multiple sensors are used to obtain road condition information of the reference area corresponding to the user's current location. In other words, the visual images and environmental detection data collected by multiple sensors are fused and processed to accurately locate the position information of obstacles, improve the accuracy, robustness and adaptability of obstacle detection, and significantly enhance the user's walking safety and convenience in complex environments. Attached Figure Description

[0017] Figure 1 is a basic flowchart of a guide device control method provided in the first embodiment of this application;

[0018] Figure 2 is a schematic diagram of the structure of the first-view guide device for the visually impaired provided in the first embodiment of this application;

[0019] Figure 3 is a structural schematic diagram of the second-view guide device for the visually impaired provided in the first embodiment of this application;

[0020] Figure 4 is a structural schematic diagram of a tactile guide provided in the first embodiment of this application;

[0021] Figure 5 is a detailed flowchart of a guide device control method provided in the second embodiment of this application;

[0022] Figure 6 is a schematic diagram of the program modules of the guide device control device provided in the third embodiment of this application;

[0023] Figure 7 is a schematic diagram of the structure of the guide device provided in the fourth embodiment of this application. Embodiments of the present invention

[0024] To make the inventive objectives, features, and advantages of this application more apparent and understandable, the technical solutions in the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0025] In the description of the embodiments of this application, it should be understood that the terms "length", "width", "upper", "lower", "front", "rear", "left", "right", "vertical", "horizontal", "top", "bottom", "inner", "outer", etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the drawings. They are only for the convenience of describing the embodiments of this application and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limiting the present invention.

[0026] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Therefore, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of the embodiments of this application, "multiple" means two or more, unless otherwise explicitly specified.

[0027] In the embodiments of this application, unless otherwise explicitly specified and limited, the terms "installation," "connection," "linking," "fixing," etc., should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral part; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; they can refer to the internal communication of two components or the interaction between two components. For those skilled in the art, the specific meaning of the above terms in the embodiments of this application can be understood according to the specific circumstances.

[0028] To address the problem of mobility difficulties for visually impaired patients in related technologies, the first embodiment of this application provides a guide device control method. Figure 1 is a basic flowchart of the guide device control method provided in this embodiment, which includes the following steps:

[0029] Step 101: After determining the navigation route for the user to reach the destination from the current location, obtain visual images and environmental detection data of the reference area corresponding to the current location.

[0030] Specifically, the reference area is determined based on the current location and navigation route. As shown in Figures 2 and 3, in this embodiment, the guide device can be configured with a small and lightweight guide device 100, which can integrate an image acquisition unit 1, a lidar detector 2, and an ultrasonic detector 3 to acquire road condition information. The guide device 100 is also equipped with a wearing component (not shown in the figure), which allows users to install it on their helmet, guide cane, shoulder, chest, waist, or other locations according to their personal habits and needs. The image acquisition unit 1 may include a night camera 11 and a daytime camera 12. In addition, the guide device 100 may also be equipped with a keyboard 4, a display 5, a wireless charging induction component 6, etc., wherein the keyboard 4 can be used to set parameters and debug functions, and the display screen can be used to display relevant information to facilitate function debugging. The intelligent guide device 100 in this embodiment can be powered by a rechargeable battery to ensure long-term use. Meanwhile, it can be equipped with a low battery reminder function to promptly remind users to charge when the battery is low; considering the convenience of users, the guide device 100 is also equipped with a wireless charging induction component 6, which supports wireless charging technology and can be used for wireless charging so that users can charge it anytime when they are out.

[0031] In some specific implementations, the image acquisition device 1 can be a 180-degree wide-angle camera. The visual images captured by the 180-degree wide-angle camera can be used to analyze road conditions, such as obstacles, puddles, and steps. Image processing algorithms can then be used to identify and analyze the visual images to determine the optimal walking path. The lidar detector 2 and the ultrasonic detector 3 can be used to measure the distance and position of obstacles in front and around in real time, providing accurate spatial information for navigation and ensuring the comprehensiveness, accuracy, and real-time nature of data acquisition. In one specific implementation, the aforementioned environmental detection data can refer to the three-dimensional point cloud data of the lidar detector 2. That is, the lidar detector 2 can acquire three-dimensional point cloud data of the user's surrounding environment by emitting a laser beam and receiving reflected signals. The three-dimensional point cloud data includes information such as the distance between the target object and the user, the shape of the target object, and its position.

[0032] It is understood that in other implementations, the image acquisition device 1 may be a visible light camera, an infrared camera, a depth camera, etc., and there is no limitation here; different sensors have their own advantages in different regions and weather conditions; for example, in low light or nighttime environments, infrared cameras can capture the thermal radiation of objects to help identify the outline of objects; depth cameras can provide distance information of objects, which helps to determine the position and distance of obstacles.

[0033] It should be noted that in some embodiments, the navigation route determined by the guide device 100 can be pre-recorded or determined by loading map navigation. That is, the guide device 100 can pre-record designated routes that the user frequently travels, such as commuting routes, routes to parks, routes to supermarkets, etc. When the user needs to go to places they don't often go or remote places, they can load map navigation and update the navigation route according to real-time traffic information. In this embodiment, the guide device 100 can not rely entirely on maps, replacing the blind person's eyes to observe road conditions, and can also identify road obstacles, road signals, and signs, allowing the user to walk easily and safely on the road.

[0034] Step 102: The first feature data extracted from the visual image and the second feature data extracted from the environmental detection data are fused to obtain a reference feature dataset of the target object in the reference region.

[0035] Specifically, the target object includes both obstacles and non-obstacles, and can be either static or dynamic. Before data fusion, the aforementioned visual images can be preprocessed with denoising, enhancement (such as brightness and contrast adjustment), and edge detection to obtain optimized two-dimensional image data (i.e., the aforementioned first feature data), thereby improving the accuracy of subsequent recognition. The environmental detection data acquired by the lidar detector 2 can be filtered, downsampled, registered (when using multiple lidar detectors to acquire detection data), and segmented to extract useful second feature data. Through the fusion of the first and second feature data, a more comprehensive and real-time perception and intelligent analysis of the surrounding environment can be achieved.

[0036] Step 103: Based on the reference feature dataset, determine the relative position parameters between all target objects and the current position.

[0037] Specifically, trained deep learning models (such as convolutional neural networks) can be used to detect target objects such as pedestrians, vehicles, and road obstacles. Geometric feature analysis combined with machine learning methods can also be used to extract target objects from point clouds.

[0038] Step 104: When it is determined that the preset navigation control conditions are met based on all relative position parameters, the navigation route is updated according to the relative position parameters corresponding to the obstacles, and the corresponding navigation control signals are generated.

[0039] Specifically, navigation control conditions include the presence of obstacles in the user's direction of travel, and the distance between the obstacle and the current position being less than or equal to a preset threshold; the navigation control signal is used to control the output of navigation information by the guide device; the navigation information is used to guide the user to avoid obstacles. As shown in Figure 2, the guide device 100 in this embodiment can also be configured with an audio output component (including a speaker 7). In addition, for users with hearing impairments or who do not wish to disturb others, a tactile guide 200 as shown in Figure 4 can also be configured. The tactile guide 200 can be equipped with a vibrating straightener 8 and a Braille display component 9. Correspondingly, the navigation control signal in this embodiment can refer to the control signal of the audio output component, the control signal of the vibrating straightener 8, or the control signal of the Braille display component 9. The audio output component can clearly and accurately provide the user with voice prompts regarding the road conditions ahead, the location of obstacles, and suggested walking directions, ensuring that the user can obtain navigation information in a timely manner. The vibrating straightener 8 can generate rolling vibrations in different directions, such as forward, backward, left and right turns, and turning, to achieve tactile navigation, allowing the user to perceive the guiding direction through touch. The Braille display component 9 can convert text navigation information into Braille format, allowing blind or visually impaired individuals to read it by touch and receive directional guidance. In practical applications, users can choose the desired signal output method according to their actual needs; there are no restrictions.

[0040] In some embodiments of this example, when determining the navigation route, road condition information can be determined based on a preset path search algorithm to find the optimal walking path (i.e., the navigation route). Specifically, the nodes and edges in the graph structure are first determined; nodes can represent positions, and edges can represent walking directions and distances. Based on the preset path search algorithm, the estimated distance from each node to the target location and the actual distance traveled are calculated, thereby determining the optimal navigation route. Optionally, the above path search algorithm can be the A* algorithm, Dijkstra's algorithm, etc.

[0041] In some embodiments of this example, the parameters of the path search algorithm can be adjusted according to different terrain and weather conditions. For example, in mountainous areas, due to the complex terrain, the safety weight of the path can be increased to avoid selecting overly steep or dangerous paths. On rainy days, due to slippery roads, the stability weight of the path can be increased to avoid selecting paths that are easy to slip on.

[0042] In some specific implementations, route planning can be optimized by incorporating user feedback. For example, if a user reports that a particular route is too congested or unsafe, the parameters of the route search algorithm can be adjusted to avoid selecting that route. Furthermore, personalized route planning services can be provided based on user preferences, such as prioritizing the shortest, fastest, or most scenic routes.

[0043] In some implementations of this embodiment, when determining a specific walking path, a machine learning algorithm suitable for road condition recognition and path planning can be selected. For example, a convolutional neural network (CNN) can be used for feature data extraction and object detection from visual images; that is, a large amount of labeled image data can be used to train the CNN model. By adjusting the network structure and hyperparameters, the accuracy and robustness of the CNN can be improved. During training, data augmentation techniques, such as random rotation, flipping, and scaling, can be used to increase data diversity and prevent overfitting.

[0044] In some implementations of this embodiment, reinforcement learning algorithms can also be used for specific walking path planning. Specifically, a suitable state space, action space, and reward function can be designed based on a preset reinforcement learning algorithm. The state space can include information such as the current position, image features of the surrounding environment, and the target position. The action space can include walking directions such as forward, backward, left turn, and right turn. The reward function can be designed based on factors such as the length of the walking path, safety, and proximity to the target. The agent learns in a simulated environment and continuously adjusts its strategy to find the optimal walking path.

[0045] In some embodiments of this example, before the steps of obtaining visual images and environmental detection data of the reference area corresponding to the current location, the method further includes: after determining the user's destination, determining the user's item carrying information, and obtaining all possible routes for the user to reach the destination from the current location and the road condition information of each possible route; wherein, the road condition information includes terrain information, population density and weather conditions; based on the item carrying information and road condition information, determining the navigation route for the user to reach the destination from the current location from all possible routes.

[0046] Specifically, weighting coefficients can be set for each factor related to road conditions and for each item carried. By combining actual road conditions and the user's item load information, the priority of each available route can be determined. Then, the optimal navigation route can be selected by comparing the priorities of each available route. For example, when a user is carrying luggage, routes with shorter walking times and fewer or no steps can be prioritized.

[0047] In some embodiments of this example, when preprocessing the aforementioned visual images, different algorithm models can be applied to preprocess visual images captured under different terrain and weather conditions. For example, in hot regions, strong direct sunlight may cause images to be overly bright; an adaptive brightness adjustment algorithm can be used to reduce the overall brightness of the image while enhancing contrast, making objects clearer. In cold regions, snow cover may affect the recognition of roads and objects; a snow removal algorithm can be used to remove the interference of snow by analyzing the color and texture features in the image. On rainy days, images may be blurred by raindrops; a rain removal algorithm can be applied to remove the influence of raindrops on the image by detecting the shape and trajectory of raindrops. On foggy days, the clarity of the image will be reduced; a dehazing algorithm, such as a dark channel prior-based dehazing method, can be used to restore the contrast and details of the image.

[0048] In some embodiments of this example, before fusing the first feature data extracted from the visual image and the second feature data extracted from the environmental detection data to obtain the reference feature dataset of the target object in the reference region, potential target objects (such as pedestrians, vehicles, obstacles, etc.) can be detected and identified from the visual image to obtain the first feature data. Specifically, a deep learning model (such as YOLO, Faster R-CNN, etc.) can be used for target detection. When training the target detection model, an image dataset containing different terrain and weather conditions is used to improve the model's generalization ability. The geometric characteristics (such as position, velocity, etc.) of each detected target are extracted from the environmental detection data to obtain the second feature data.

[0049] Furthermore, in some specific implementations, when extracting the first feature data and detecting targets, feature extraction methods adapted to different terrain and weather conditions can be designed. For example, in urban areas with numerous buildings and traffic signs, geometric features such as edges and corners, as well as features such as color and texture, can be extracted to identify targets such as buildings, traffic signs, and vehicles. In mountainous areas with complex terrain, features such as mountain outlines and rivers can be extracted to determine the direction of roads and the location of obstacles. The parameters for feature extraction can be adjusted under different weather conditions; for example, in rainy weather, the color and texture of objects may change due to rain reflection, so the methods for extracting color and texture features can be adjusted.

[0050] In some embodiments of this example, the step of fusing the first feature data extracted from the visual image and the second feature data extracted from the environmental detection data to obtain a reference feature dataset of the target object in the reference region includes: aligning the first feature data extracted from the visual image and the second feature data extracted from the environmental detection data based on a preset timestamp; after data alignment, associating the first feature data with the geometric features of the target object determined based on the second feature data to obtain association information; wherein, the target object is an object in the reference region; and fusing the first feature data and the second feature data based on the association information to obtain a reference feature dataset of the target object.

[0051] Specifically, in this embodiment, when associating the geometric features of the first feature data and the second feature data, for each target object detected by the lidar detector, the spatial distance between it and the target object detected in the visual image is calculated. Euclidean distance or other metrics can be used to determine the degree of matching between the two (a threshold is set). If an object matches the position of an object in the visual image within a certain spatial error range (the degree of matching is greater than the set threshold), the two are associated to obtain the corresponding association information. Furthermore, in some implementations, for complex scenes with multiple target objects, a joint probability data association method can be used to comprehensively consider the relationships between the detected target objects, achieving more accurate information association.

[0052] Furthermore, in some embodiments of this example, the step of fusing the first feature data and the second feature data based on the association information to obtain a reference feature dataset of the target object includes: inputting the first feature data into a trained neural network model to obtain a reference image with the target object identifier; and mapping the second feature data onto the reference image based on the association information to generate a reference feature dataset of the target object.

[0053] Specifically, in this embodiment, weighted averaging, feature-based fusion, particle filtering, and other methods can be used to fuse the associated visual images and environmental detection data to obtain a reference feature dataset of the target object; wherein, the reference feature dataset includes the final position, category, and other relevant information of each target object in space.

[0054] In other embodiments of this example, the first feature data and the second feature data can be concatenated or weighted at the same level to obtain fused feature data. The fused feature data is then input into the trained deep learning model, which can output the required reference feature dataset. Specifically, when building the deep learning model, the output layer architecture, such as target classification and detection box regression, can be set according to actual needs. Road condition image data and detection data under different terrain and weather conditions are collected extensively and accurately labeled. The labeling content can include the location and category of target objects such as roads, obstacles, and traffic signs, ensuring that the dataset covers various terrain features (such as urban, rural, and mountainous areas) and weather conditions (such as sunny, rainy, snowy, and foggy days). The dataset is divided into training, validation, and test sets for model training, adjustment, and evaluation. During training, multiple iterations of training can be performed on training data containing visual image features and LiDAR detection data features. The loss value and accuracy during training are monitored, and methods such as early stopping and cross-validation can be used to prevent overfitting. The model performance is evaluated in the validation set.

[0055] In some implementations of this example, deep learning models can be trained using a combination of transfer learning and incremental learning methods, enabling the model to quickly adapt to new geographical and weather conditions. Transfer learning allows a model trained under one terrain and weather condition to be applied to other similar conditions and fine-tuned to improve its adaptability. Incremental learning allows the model to learn on new data, continuously updating and optimizing its parameters without retraining the entire model. In practical applications, the model can be periodically evaluated and updated, adjusted and optimized based on feedback from actual navigation and newly collected data. For example, if the model's performance degrades under certain terrain and weather conditions, data under those conditions can be collected again for retraining to improve the model's accuracy.

[0056] In some embodiments of this example, after the step of fusing the first feature data extracted from the visual image and the second feature data extracted from the environmental detection data to obtain the reference feature dataset of the target objects in the reference area, the method further includes: establishing a three-dimensional motion model of each target object based on the environmental detection data; determining the movement trajectory of each target object within a preset time period based on the three-dimensional motion model; adding the positional features of the navigation route in the reference area to the three-dimensional motion model to determine whether there is an intersection or overlap between each movement trajectory and the navigation route within the preset time period; if the determination result is yes, updating the navigation route based on each movement trajectory and generating corresponding navigation control signals.

[0057] Specifically, the target object in this embodiment can be either a dynamic or static object. During feature extraction and target detection, a Kalman filter can be used to estimate the target object's position and velocity to help handle noise and uncertainty, and generate a corresponding three-dimensional motion model. This three-dimensional motion model can be used to predict the target object's future position. Therefore, the three-dimensional motion model and navigation route can be combined to determine whether the user's journey will be obstructed by obstacles within a preset time period. If obstruction exists, the navigation route is updated in real time, and corresponding navigation control signals are generated to guide the user to their destination smoothly. The guide device in this embodiment can control various sensors to continuously acquire new road condition information during actual walking and adjust the walking path in real time. If new obstacles or changes in road conditions are detected, the path is replanned in a timely manner to ensure safety and efficiency.

[0058] In some embodiments of this example, the control method further includes: when the rescue function of the guide device is triggered, acquiring the user's distress information; wherein the distress information includes at least one of the user's vital signs information and the surrounding environmental safety information of the current location; determining the target rescue platform based on the distress information, and sending a rescue request to the target rescue platform; wherein the rescue request carries the distress information and the location information of the guide device.

[0059] Specifically, the guide device in this embodiment can be configured with an SOS alarm function component. An SOS one-button alarm button 7 is equipped on the guide device 100. In an emergency, pressing the alarm button 7 immediately triggers the alarm process of the rescue function. The intelligent guide device 100 can automatically acquire the user's current location information and send it to the corresponding SOS system through its built-in GPS positioning system. The guide device 100 can also be linked with its camera and audio system to capture on-site images and sounds, sending the data to the SOS system together to provide more comprehensive information support for rescue. Based on the received location data and on-site information, it can quickly analyze and determine the target rescue platform, and then notify the target rescue platform to carry out timely and accurate rescue operations. In this embodiment, the target rescue platform can be public security, hospitals, fire departments, etc., without limitation. It should be noted that in some implementations, the target rescue platform can be determined by a preset evaluation model in the SOS system based on the received location data and on-site information, or by SOS system personnel based on the received location data and on-site information, without limitation.

[0060] In practical applications, the wearing method and sensor configuration of the Smart Guide Device 100 can be adjusted according to user needs and scenario characteristics. The algorithm can be optimized to adapt to navigation requirements under different geographical and weather conditions, ensuring the practicality and reliability of the Guide Device 100. Simultaneously, a companion mobile app or remote controller can be developed to allow users to view historical walking records, adjust navigation settings, receive emergency notifications, and perform other related operations. Furthermore, user-customizable functions can be considered, such as setting emergency contacts and adjusting alarm volume and vibration intensity, to meet the personalized needs of different users.

[0061] Based on the technical solutions of the embodiments of this application described above, a guide system for the visually impaired can be implemented by combining multi-sensor fusion technology, intelligent algorithms, multiple navigation prompt systems, emergency rescue functions (including SOS alarm button 7, location data transmission, camera and voice system linkage, etc.) and a supporting APP (or remote controller). This highly integrated intelligent technology and human-centered design significantly improves the safety and convenience of users walking in complex environments. It represents a significant innovative achievement in modern technology assisting people with disabilities, with broad application prospects and social value. It fully considers the characteristics of different regions and weather conditions, combining multi-sensor data fusion, specific image preprocessing, feature extraction and target detection adapted to different conditions, and machine learning algorithms to optimize the guidance function of the guide device, thereby improving the accuracy, robustness, and adaptability of navigation route planning.

[0062] The method shown in Figure 5 is a refined guide device control method provided in the second embodiment of this application. This guide device control method includes:

[0063] Step 501: After determining the navigation route for the user to reach the destination from the current location, acquire visual images and environmental detection data of the reference area corresponding to the current location.

[0064] Specifically, in this embodiment, the range of the reference area is determined based on the current location and navigation route.

[0065] Step 502: Based on the preset timestamp, align the first feature data extracted from the visual image with the second feature data extracted from the environmental detection data.

[0066] Step 503: After data alignment, associate the first feature data with the geometric features of the target object determined based on the second feature data to obtain association information.

[0067] Specifically, the target object is the object in the reference area.

[0068] Step 504: Based on the association information, the first feature data and the second feature data are fused to obtain the reference feature dataset of the target object.

[0069] Step 505: Based on the reference feature dataset, determine the relative position parameters between all target objects and the current position.

[0070] Step 506: When it is determined, based on all relative position parameters, that there is an obstacle in the user's direction of travel, and the distance between the obstacle and the current position is less than or equal to a preset threshold, the navigation route is updated according to the relative position parameters corresponding to the obstacle, and a corresponding navigation control signal is generated.

[0071] Specifically, navigation control signals are used to control the output of navigation information by the guide device for the visually impaired, and the navigation information is used to guide the user to avoid obstacles.

[0072] By implementing this embodiment, road condition analysis is performed by combining data collected from multiple sensors (visible light cameras, infrared cameras, depth cameras, etc.). Data fusion algorithms are used to fuse images from different sensors, which helps to improve the accuracy and robustness of road condition recognition. This enables comprehensive and real-time perception and intelligent analysis of the surrounding environment, greatly improving the user's walking safety and autonomy.

[0073] It should be understood that the sequence number of each step in this embodiment does not imply the order in which the steps are executed. The execution order of each step should be determined by its function and internal logic, and should not constitute a unique limitation on the implementation process of this application embodiment.

[0074] Figure 6 illustrates a guide device control apparatus according to a third embodiment of this application. This guide device control apparatus can be applied to the aforementioned guide device control method. As shown in Figure 7, the guide device control apparatus mainly includes:

[0075] The acquisition module 601 is used to acquire visual images and environmental detection data of a reference area corresponding to the current location after determining the navigation route for the user to reach the destination from the current location; wherein the range of the reference area is determined based on the current location and the navigation route;

[0076] The fusion module 602 is used to fuse the first feature data extracted from the visual image and the second feature data extracted from the environmental detection data to obtain a reference feature dataset of the target object in the reference region; wherein the target object includes obstacles and non-obstacles;

[0077] The determination module 603 is used to determine the relative position parameters between all target objects and the current position based on the reference feature dataset;

[0078] The generation module 604 is used to update the navigation route according to the relative position parameters corresponding to the obstacles and generate corresponding navigation control signals when it is determined that the preset navigation control conditions are met based on all relative position parameters. The navigation control conditions include the presence of obstacles in the user's direction of travel and the distance between the obstacles and the current position being less than or equal to a preset threshold. The navigation control signals are used to control the guide device to output navigation information. The navigation information is used to guide the user to avoid obstacles.

[0079] In some embodiments of this example, a navigation module is also included. The navigation module is specifically used to: determine the user's item carrying information after determining the user's destination, and obtain all possible routes for the user to reach the destination from the current location and the road condition information of each possible route; wherein, the road condition information includes terrain information, population density and weather conditions; and determine the navigation route for the user to reach the destination from the current location from all possible routes based on the item carrying information and the road condition information.

[0080] In some embodiments of this example, the fusion module is specifically used to: align first feature data extracted from a visual image and second feature data extracted from environmental detection data based on a preset timestamp; after data alignment, associate the first feature data with the geometric features of the target object determined based on the second feature data to obtain association information; wherein, the target object is an object in the reference area; and based on the association information, fuse the first feature data and the second feature data to obtain a reference feature dataset of the target object.

[0081] Furthermore, in some embodiments of this example, when the fusion module performs the function of fusing the first feature data and the second feature data based on the association information to obtain a reference feature dataset of the target object, it is specifically used to: input the first feature data into the trained neural network model to obtain a reference image with the target object identifier; and map the second feature data onto the reference image based on the association information to generate a reference feature dataset of the target object.

[0082] In some embodiments of this example, the navigation module is further specifically used for: establishing a three-dimensional motion model of each target object based on environmental detection data; determining the movement trajectory of each target object within a preset time period based on the three-dimensional motion model; adding the positional features of the navigation route in the reference area to the three-dimensional motion model to determine whether there is an intersection or overlap between each movement trajectory and the navigation route within the preset time period; if the determination result is yes, updating the navigation route based on each movement trajectory.

[0083] In some embodiments of this example, a rescue module is also included. The rescue module is specifically used to: obtain the user's distress information when the rescue function of the guide device is triggered; wherein the distress information includes at least one of the user's vital signs information and the surrounding environmental safety information of the current location; determine the target rescue platform based on the distress information, and send a rescue request to the target rescue platform; wherein the rescue request carries the distress information and the location information of the guide device.

[0084] According to the guide device control device provided in this embodiment, after determining the navigation route for the user to reach the destination from the current location, visual images and environmental detection data corresponding to the current location are acquired; wherein, the range of the reference area is determined based on the current location and the navigation route; the first feature data extracted from the visual image and the second feature data extracted from the environmental detection data are fused to obtain a reference feature dataset of target objects in the reference area; wherein, the target objects include obstacles and non-obstacles; based on the reference feature dataset, the relative position parameters between all target objects and the current location are determined; when it is determined based on all relative position parameters that a preset navigation control condition is met, the navigation route is updated according to the relative position parameters corresponding to the obstacles, and a corresponding navigation control signal is generated; wherein, the navigation control condition includes the presence of obstacles in the user's direction of travel, and the distance between the obstacles and the current location is less than or equal to a preset threshold; the navigation control signal is used to control the guide device to output navigation information; the navigation information is used to guide the user to avoid obstacles. By implementing the solution in this application, multiple sensors are used to obtain road condition information of the reference area corresponding to the user's current location. In other words, the visual images and environmental detection data collected by multiple sensors are fused and processed to accurately locate the position information of obstacles, improve the accuracy, robustness and adaptability of obstacle detection, and significantly enhance the user's walking safety and convenience in complex environments.

[0085] Figure 7 illustrates a guide device for the visually impaired according to a fourth embodiment of this application. This guide device can be used to implement the guide device control method described in the foregoing embodiments, and mainly includes:

[0086] The system comprises a memory 701, a controller 702, a computer program 703 stored in the memory 701 and executable on the controller 702, and a guide device 704. The memory 701 and the controller 702 are communicatively connected. The guide device 704 is equipped with an audio output component, an image acquisition unit, and a lidar detector, all communicatively connected to the controller. The audio output component outputs navigation information based on navigation control signals sent by the controller. The image acquisition unit acquires visual images, and the lidar detector obtains environmental detection data. When the controller 702 executes the computer program 703, it implements the method described in Embodiment 1 or 2. The number of controllers can be one or more.

[0087] In this embodiment, a tactile guide is also configured with a communication connection to the controller. The tactile guide includes a vibration generator and / or a Braille display component for outputting navigation information based on navigation control signals sent by the controller. It is understood that in some other embodiments, the guide device and the tactile guide may also be integrated into a single device, which is not a limitation here.

[0088] The memory 701 can be a high-speed random access memory (RAM) or a non-volatile memory, such as a disk drive. The memory 701 is used to store executable program code, and the controller 702 is coupled to the memory 701.

[0089] Furthermore, this application embodiment also provides a computer-readable storage medium, which may be disposed in the above-mentioned guide device for the visually impaired, and the computer-readable storage medium may be the memory in the embodiment shown in FIG7 above.

[0090] The computer-readable storage medium stores a computer program that, when executed by the controller, implements the guide device control method described in the foregoing embodiments. Furthermore, the computer-readable storage medium can also be a USB flash drive, portable hard drive, read-only memory (ROM), RAM, magnetic disk, or optical disk, or any other medium capable of storing program code.

[0091] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of modules is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple modules or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or modules may be electrical, mechanical, or other forms.

[0092] The modules described as separate components may or may not be physically separate. Similarly, the components shown as modules may or may not be physical modules; they may be located in one place or distributed across multiple network modules. Some or all of the modules can be selected to achieve the purpose of this embodiment, depending on actual needs.

[0093] Furthermore, the functional modules in the various embodiments of this application can be integrated into one processing module, or each module can exist physically separately, or two or more modules can be integrated into one module. The integrated modules described above can be implemented in hardware or as software functional modules.

[0094] If the integrated module is implemented as a software functional module and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a readable storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned readable storage medium includes various media capable of storing program code, such as USB flash drives, external hard drives, ROM, RAM, magnetic disks, or optical disks.

[0095] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that this application is not limited to the described order of actions, as some steps may be performed in other orders or simultaneously according to this application. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are preferred embodiments, and the actions and modules involved are not necessarily essential to this application.

[0096] In the above embodiments, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0097] The above is a description of the control method and related products for guide devices for the visually impaired provided in this application. For those skilled in the art, based on the ideas of the embodiments of this application, there will be changes in the specific implementation methods and application scope. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A control method for a guide device for the visually impaired, characterized in that, include: After determining the navigation route for the user to reach their destination from their current location, visual images and environmental detection data of a reference area corresponding to the current location are acquired; wherein, the extent of the reference area is determined based on the current location and the navigation route; The first feature data extracted from the visual image and the second feature data extracted from the environmental detection data are fused to obtain a reference feature dataset of the target object in the reference region; wherein, the target object includes obstacles and non-obstacles; Based on the reference feature dataset, determine the relative position parameters between all the target objects and the current position; When it is determined that the preset navigation control conditions are met based on all the relative position parameters, the navigation route is updated according to the relative position parameters corresponding to the obstacle, and a corresponding navigation control signal is generated; wherein, the navigation control conditions include the presence of the obstacle in the user's direction of travel, and the distance between the obstacle and the current position is less than or equal to a preset threshold; the navigation control signal is used to control the guide device to output navigation information; the navigation information is used to guide the user to avoid the obstacle.

2. The guide dog control method according to claim 1, wherein Before acquiring the visual image and environmental detection data of the reference area corresponding to the current location, the method further includes: After determining the user's destination, the system determines the user's belongings information and obtains all possible routes for the user to reach the destination from the current location, as well as the road condition information for each possible route; wherein, the road condition information includes terrain information, population density, and weather conditions; Based on the item carrying information and the road condition information, a navigation route for the user to reach their destination from their current location is determined from all the available routes.

3. The method of claim 1, wherein, The process of fusing the first feature data extracted from the visual image and the second feature data extracted from the environmental detection data to obtain a reference feature dataset of the target object in the reference region includes: Based on a preset timestamp, the first feature data extracted from the visual image and the second feature data extracted from the environmental detection data are aligned; After data alignment, the first feature data is associated with the geometric features of the target object determined based on the second feature data to obtain association information; wherein, the target object is an object in the reference region; Based on the association information, the first feature data and the second feature data are fused to obtain a reference feature dataset of the target object.

4. The method according to claim 3, wherein The step of fusing the first feature data and the second feature data based on the association information to obtain a reference feature dataset of the target object includes: The first feature data is input into the trained neural network model to obtain a reference image with the target object identifier; Based on the association information, the second feature data is mapped onto the reference image to generate a reference feature dataset of the target object.

5. The method of claim 1, wherein, After fusing the first feature data extracted from the visual image and the second feature data extracted from the environmental detection data to obtain a reference feature dataset of the target object in the reference region, the method further includes: Based on the environmental detection data, a three-dimensional motion model of each target object is established; Based on the three-dimensional motion model, the movement trajectory of each target object within a preset time period is determined; The positional features of the navigation route in the reference area are added to the three-dimensional motion model, and it is determined whether there is any intersection or overlap between each movement trajectory and the navigation route within a preset time period; If the determination result is yes, the navigation route is updated based on each of the movement trajectories, and corresponding navigation control signals are generated.

6. The guide dog control method according to any one of claims 1 to 5, characterized in that, Also includes: When the rescue function of the guide device is triggered, the user's distress information is obtained; wherein, the distress information includes at least one of the user's vital signs information and the safety information of the surrounding environment of the current location; Based on the distress message, a target rescue platform is identified, and a rescue request is sent to the target rescue platform; wherein, the rescue request carries the distress message and the location information of the guide device.

7. A method of controlling a guide dog apparatus, characterized by, include: The acquisition module is used to acquire visual images and environmental detection data of a reference area corresponding to the current location after determining the navigation route for the user to reach the destination from the current location; wherein the range of the reference area is determined based on the current location and the navigation route; The fusion module is used to fuse first feature data extracted from the visual image and second feature data extracted from the environmental detection data to obtain a reference feature dataset of the target object in the reference region; wherein the target object includes obstacles and non-obstacles; The determination module is used to determine the relative position parameters between all the target objects and the current position based on the reference feature dataset; A generation module is used to update the navigation route according to the relative position parameters corresponding to the obstacle and generate a corresponding navigation control signal when it is determined that a preset navigation control condition is met based on all the relative position parameters; wherein, the navigation control condition includes the presence of the obstacle in the user's direction of travel, and the distance between the obstacle and the current position is less than or equal to a preset threshold; the navigation control signal is used to control the guide device to output navigation information; the navigation information is used to guide the user to avoid the obstacle.

8. A guide device for the visually impaired, characterized in that, Includes a guide device, memory, and controller, among which: The guide device is equipped with an audio output component, an image acquisition unit, and a lidar detector, all of which are communicatively connected to the controller. The audio output component is used to output navigation information based on navigation control signals sent by the controller, the image acquisition unit is used to acquire visual images, and the lidar detector is used to acquire environmental detection data. The controller is used to execute computer programs stored in the memory; When the controller executes the computer program, it implements the steps in the method of any one of claims 1 to 6.

9. The guide dog device according to claim 8, characterized by It also includes a tactile guide that is communicatively connected to the controller, the tactile guide including a vibratory straightener and / or a Braille display component for outputting navigation information based on navigation control signals sent by the controller.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the controller, it implements the steps of the method according to any one of claims 1 to 6.

Images