Control method for self-moving cleaning device, and device

Abstract

A control method for a self-moving cleaning device, and a device. The method comprises: during execution of a cleaning task by a self-moving cleaning device, identifying an obstacle and determining the obstacle as a wire bundle obstacle, and then acquiring a wire bundle obstacle image (S201); and determining position coordinates of a wire bundle in the image, and avoiding the wire bundle obstacle on the basis of the position coordinates; wherein determining position coordinates of a wire bundle in the image comprises: determining a wire bundle adjacent area in the wire bundle obstacle image, determining a connecting line between position coordinates of each pixel point in the wire bundle adjacent area and position coordinates of the wire bundle obstacle, and determining the position coordinates of the wire bundle obstacle on the basis of the connecting line; the wire bundle adjacent area at least comprises an envelope area of the wire bundle obstacle; and the envelope area is a closed area created around the boundary of a wire bundle obstacle area (S202).

Description

Self-Moving Cleaning Equipment Control Method and Equipment

[0001] This application claims priority to Chinese Patent Application No. 202411648475.1, filed on November 18, 2024, entitled "Control Method and Apparatus for Self-Moving Cleaning Equipment", the entire contents of which are incorporated herein by reference. Technical Field

[0002] This application relates to the field of intelligent robots, and more particularly to a method and apparatus for controlling a self-moving cleaning device. Background Technology

[0003] In the fields of automation and robotics, especially in the application of self-moving robots such as robotic vacuum cleaners, identifying and avoiding obstacles is one of the important tasks to ensure the safe operation of the equipment. However, wire harness obstacles, such as power cords, network cables, and charging cables, are irregular in shape and may be distributed in different directions and positions. If they are not accurately identified, they may get caught in the drive wheels of the robotic vacuum cleaner during the cleaning process, affecting the operation of the robotic vacuum cleaner.

[0004] In existing technologies, self-moving robots typically use box detection algorithms to identify wire harness obstacles. Box detection algorithms use rectangular boxes to represent the position and extent of objects. However, wire harnesses are usually long, curved, and may be randomly distributed in space. Using rectangular boxes to represent the position of wire harnesses results in low positioning accuracy.

[0005] However, due to the complex shape of the wire harness, different annotators may give different annotations to the same wire harness. This inconsistency affects the detection results of the algorithm, which in turn leads to a decrease in the accuracy of the self-moving robot when detecting the position of the wire harness. Summary of the Invention

[0006] This application provides a self-moving cleaning device control method and device, which can effectively identify the position of the wire harness obstacle by considering the connection between each pixel point in the vicinity of the wire harness and the wire harness obstacle, and accurately locating the wire harness obstacle based on the connection.

[0007] In a first aspect, this application provides a control method for a self-moving cleaning device, applied to a self-moving cleaning device, the method comprising:

[0008] During the cleaning process of the self-moving cleaning equipment, after identifying the obstacle and determining that the obstacle is a wire harness obstacle, an image of the wire harness obstacle is acquired;

[0009] Determine the position coordinates of the wire harness in the obstacle image, and avoid the wire harness obstacle based on the position coordinates;

[0010] Determining the position coordinates of the wire harness in the obstacle image includes:

[0011] The image of the wire harness obstacle is determined to include the wire harness neighboring region, and a line is drawn connecting the position coordinates of each pixel in the wire harness neighboring region to the position coordinates of the wire harness obstacle. The position coordinates of the wire harness obstacle are determined based on the line. The wire harness neighboring region includes at least the envelope region of the wire harness obstacle. The envelope region is a closed region created around the boundary of the wire harness obstacle region.

[0012] In this way, during the cleaning process of the self-moving cleaning equipment, by analyzing the connection between the pixel points and the wire harness obstacles in detail, the position of the wire harness obstacles can be accurately determined. After the position coordinates are determined, the path can be planned more effectively to avoid potential collisions or interference. This reduces the physical contact between the equipment and the wire harness, lowers the risk of tangling, damaging the wire harness or the equipment itself, improves safety, and can complete the cleaning task more efficiently.

[0013] Optionally, determining the position coordinates of the harness obstacle based on the connection includes:

[0014] The position coordinates of the wire harness obstacle are determined based on the distance between the connecting lines; where the distance between each connecting line is the minimum distance between the position coordinates of the corresponding pixel in the adjacent region of the wire harness and the position coordinates of the wire harness obstacle.

[0015] In this way, by taking into account the distance from each pixel in the vicinity of the wire harness to the wire harness obstacle, and by calculating and comparing the minimum distance, the wire harness obstacle can be accurately located, and the position of the wire harness obstacle can be effectively identified. Especially in complex environments, the wire harness obstacle may have an irregular shape, and the distance calculation method based on the connection can adapt to these complexities and provide more reliable positioning results.

[0016] Optionally, determining the region adjacent to the wire harness in the wire harness obstacle image includes:

[0017] Obtain a predefined neighborhood window and calculate the neighborhood features of each pixel in the wire harness obstacle image within the predefined neighborhood window;

[0018] Based on neighborhood features, threshold segmentation is performed on pixels in the wire bundle obstacle image to obtain the wire bundle neighboring region and non-wire bundle neighboring region.

[0019] The self-moving cleaning equipment does not treat areas not adjacent to the wire harness.

[0020] In this way, by identifying and ignoring non-wire harness neighboring areas, the processing speed can be improved. Furthermore, by determining the wire harness neighboring and non-wire harness neighboring areas through the aforementioned neighborhood segmentation method, the wire harness neighboring areas that need to be focused on can be identified more accurately, thereby improving the subsequent recognition accuracy of wire harness obstacles.

[0021] Optionally, determining the position coordinates of the wire harness in the wire harness obstacle image includes:

[0022] The image of the wire harness obstacle is input into the trained wire harness detection model to predict the position coordinates of the wire harness obstacle in the image.

[0023] In this way, using the trained wire harness detection model to predict the position coordinates of wire harness obstacles can accurately identify wire harness obstacles in complex backgrounds, adapt to various environmental changes, and have strong robustness. Furthermore, the wire harness detection model can also adapt to new environments or identify various types of wire harness obstacles through transfer learning or further training, demonstrating good scalability.

[0024] Optionally, the wire harness detection model includes a classification model and a regression model; the wire harness obstacle image is input into the trained wire harness detection model to predict the position coordinates of the wire harness obstacle in the image, including:

[0025] Input the image of the wire harness obstacle into the trained classification model to obtain the wire harness neighborhood in the image of the wire harness obstacle;

[0026] By inputting the area near the wire harness into the trained regression model, the position coordinates of the wire harness obstacle in the area near the wire harness are obtained.

[0027] In this way, by decomposing the model prediction into two steps, classification and regression, this application can optimize the performance of each model and improve the overall detection accuracy. The classification model first narrows down the area that needs to be accurately located, reduces the computational burden of the regression model, and improves the processing efficiency. Then, the trained regression model can be used to perform accurate regression localization, fit the irregular shape of the wire bundle obstacle to identify and accurately locate the wire bundle, and then output the specific location of the wire bundle.

[0028] Optionally, the wire harness detection model is a neural network model based on deep learning.

[0029] It should be noted that deep learning-based neural network models perform well in image recognition and classification tasks, can handle complex image features, provide high-precision detection results, and can effectively identify target objects in complex and variable backgrounds. They are suitable for wire harness detection in various environments and conditions. In particular, the model has strong robustness to noise, occlusion and lighting changes, and can adapt to different image quality and environmental conditions.

[0030] In this way, the deep learning-based neural network model can identify the wire bundle and accurately locate its specific position.

[0031] Optionally, the training process for the harness detection model includes:

[0032] Obtain the training dataset; the training dataset includes multiple training samples, each training sample includes a wire harness obstacle image and the position coordinates of the wire harness obstacle in the wire harness obstacle image;

[0033] The training dataset is input into the wire harness detection model for training, resulting in a trained wire harness detection model.

[0034] The training process includes: determining the neighborhood of each obstacle in the bundle image, determining the minimum distance between the position coordinates of each pixel in the neighborhood and the position coordinates of the obstacle, and using the line corresponding to the minimum distance, the position coordinates of the pixel corresponding to each line, and the position coordinates of the obstacle to perform regression training.

[0035] In this way, by calculating the minimum distance and performing regression training, the model can accurately locate the position of the wire harness obstacle, improving detection accuracy. Furthermore, through precise regression training, the model can reduce errors in position prediction and improve overall reliability. In particular, the wire harness detection model can automatically learn complex features in the image, reducing reliance on manual intervention.

[0036] Optionally, the wire harness detection model includes a classification model and a regression model; the training dataset is input into the wire harness detection model for training to obtain a trained wire harness detection model, including:

[0037] For each wire harness obstacle image, obtain the wire harness neighboring region in the labeled wire harness obstacle image;

[0038] Multiple images of wire harness obstacles and the adjacent regions of each wire harness obstacle image are input into the classification model for training to obtain a trained classification model.

[0039] The position coordinates of multiple wire harness neighborhoods and wire harness obstacles in the wire harness neighborhoods are input into the regression model for training, and a trained regression model is obtained.

[0040] The trained classification model and the trained regression model are combined to obtain the trained wire harness detection model.

[0041] In this way, by combining classification and regression models, possible neighboring areas of the wire harness can be identified, and the location of the wire harness can be accurately located, improving the overall detection accuracy. Furthermore, the independent training and optimization of the classification and regression models make the overall design more modular, facilitating maintenance and expansion. Specifically, the classification model first narrows down the area that needs to be processed accurately, effectively filtering out irrelevant areas, reducing the computational burden on the regression model, and improving processing efficiency. In turn, the regression model accurately locates the wire harness obstacle, improving the positioning accuracy.

[0042] Optionally, the training process for the regression model includes:

[0043] Obtain the position coordinates of each pixel in the vicinity of the wire harness;

[0044] For each pixel, using the pixel's position coordinates and the position coordinates of the wire bundle obstacle, calculate the multiple distances from the pixel to the multiple pixels corresponding to the wire bundle obstacle.

[0045] Determine the minimum distance among multiple distances, and then determine the connection between the pixels corresponding to the minimum distance;

[0046] Regression training is performed using the lines connecting the pixels corresponding to the minimum distance, the position coordinates of the pixels corresponding to each line, and the position coordinates of the obstacles in the line bundle.

[0047] In this way, by utilizing the minimum distance between pixels, the model can accurately capture the position of the wire harness obstacle, improving positioning accuracy. This minimum distance is determined through precise distance calculation and connection, which allows the model to reduce errors in position prediction and improve overall reliability.

[0048] Optionally, avoiding harness obstacles based on location coordinates includes at least one of the following cases:

[0049] The cleaning route is replanned based on the location coordinates of the wire harness obstacle, and the self-moving cleaning equipment performs the cleaning task according to the cleaning route to avoid the wire harness obstacle;

[0050] The cleaning no-go zone is determined based on the location coordinates of the wire harness obstacle. After the self-moving cleaning equipment cleans around the cleaning no-go zone, it continues to perform the cleaning task according to the original planned path.

[0051] The location coordinates of the wiring harness obstacle are sent to the user terminal for display, and the obstacle is avoided in response to the user's control operation.

[0052] In this way, once the self-moving cleaning equipment determines the location coordinates of the wire harness obstacle, it can avoid the obstacle in various ways to adapt to different environmental changes and user needs. This improves application flexibility and reduces the risk of collision between the equipment and the wire harness obstacle, thereby protecting the equipment and other items in the environment.

[0053] Secondly, this application provides a self-moving cleaning device control method, applied to a user terminal, the method comprising:

[0054] Receive the position coordinates of the wire harness in the image of the wire harness obstacle determined during the cleaning task sent by the mobile cleaning equipment, and visualize the position coordinates;

[0055] Determining the position coordinates of the wire harness in the obstacle image includes:

[0056] After identifying the obstacle and determining that it is a wire harness obstacle, an image of the wire harness obstacle is acquired;

[0057] The image of the wire harness obstacle is determined to include the wire harness neighboring region, and a line is drawn connecting the position coordinates of each pixel in the wire harness neighboring region to the position coordinates of the wire harness obstacle. The position coordinates of the wire harness obstacle are determined based on the line. The wire harness neighboring region includes at least the envelope region of the wire harness obstacle. The envelope region is a closed region created around the boundary of the wire harness obstacle region.

[0058] Understandably, displaying the location of wiring harness obstacles on the user terminal allows users to take preventative measures to avoid collisions between the self-propelled cleaning device and the wiring harness obstacles, thereby protecting the self-propelled cleaning device and the surrounding environment.

[0059] In this way, by providing real-time visualizations and interactive functions, the opportunities for users to participate in the operation of the device are increased, improving the user experience and satisfaction. Furthermore, users can dynamically adjust the cleaning strategy of the device based on the real-time visualizations, more accurately control the behavior of the device, reduce the possibility of misoperation, and thus reduce the risk of damage to the self-propelled cleaning device.

[0060] Thirdly, this application provides a self-moving cleaning device, which includes: a processor and a memory communicatively connected to the processor;

[0061] The memory stores the instructions that the computer executes;

[0062] The processor executes computer-executable instructions stored in memory to implement the method as described in any of the first aspects.

[0063] Fourthly, this application provides a control device for a self-moving cleaning equipment, applied to a self-moving cleaning equipment, the device comprising:

[0064] The identification module is used to identify obstacles and, after determining that the obstacle is a wire harness obstacle, acquire an image of the wire harness obstacle during the cleaning task performed by the self-moving cleaning device.

[0065] The determination module is used to determine the position coordinates of the wire harness in the wire harness obstacle image and avoid the wire harness obstacle based on the position coordinates;

[0066] Determining the position coordinates of the wire harness in the obstacle image includes:

[0067] The image of the wire harness obstacle is determined to include the wire harness neighboring region, and a line is drawn connecting the position coordinates of each pixel in the wire harness neighboring region to the position coordinates of the wire harness obstacle. The position coordinates of the wire harness obstacle are determined based on the line. The wire harness neighboring region includes at least the envelope region of the wire harness obstacle. The envelope region is a closed region created around the boundary of the wire harness obstacle region.

[0068] Fifthly, this application provides a self-moving cleaning equipment control device, applied to a user terminal, the device comprising:

[0069] The receiving module is used to receive the position coordinates of the wire harness in the image of the wire harness obstacle determined during the execution of the cleaning task sent by the mobile cleaning device, and to visualize the position coordinates.

[0070] Determining the position coordinates of the wire harness in the obstacle image includes:

[0071] After identifying the obstacle and determining that it is a wire harness obstacle, an image of the wire harness obstacle is acquired;

[0072] The image of the wire harness obstacle is determined to include the wire harness neighboring region, and a line is drawn connecting the position coordinates of each pixel in the wire harness neighboring region to the position coordinates of the wire harness obstacle. The position coordinates of the wire harness obstacle are determined based on the line. The wire harness neighboring region includes at least the envelope region of the wire harness obstacle. The envelope region is a closed region created around the boundary of the wire harness obstacle region.

[0073] In a sixth aspect, this application provides a computer-readable storage medium storing computer-executable instructions that, when executed by a processor, are used to implement the method as described in any of the first aspects.

[0074] In a seventh aspect, this application provides a computer program product, including a computer program that, when executed by a processor, implements the method as described in any of the first aspects.

[0075] In summary, this application provides a control method and apparatus for a self-moving cleaning device. During the cleaning process, after identifying a wire harness obstacle, an image of the obstacle is acquired to identify the adjacent region of the wire harness in the image. This adjacent region at least includes the envelope region of the obstacle, which is a closed region created around the boundary of the obstacle. Further, a line connecting the position coordinates of each pixel in the adjacent region of the wire harness to the position coordinates of the obstacle is found. This line intersects the obstacle at a common point. Intersection points can better fit the non-fixed shape of the wire harness. Furthermore, based on these connections, the position coordinates of wire harness obstacles can be accurately determined. Then, using the determined wire harness position coordinates, the self-moving cleaning equipment adjusts its path planning to avoid wire harness obstacles, ensuring the smooth progress of the cleaning task. In this way, accurate image analysis and position calculation through connections improve the accuracy of wire harness obstacle location identification. Moreover, by accurately identifying and avoiding wire harness obstacles, the failures and damages caused by wire harness entanglement or pulling of the self-moving cleaning equipment are reduced, improving the safety and reliability of the equipment. Attached Figure Description

[0076] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0077] Figure 1 is a schematic diagram of an application scenario provided by an embodiment of this application;

[0078] Figure 2 is a flowchart illustrating a self-moving cleaning equipment control method provided in an embodiment of this application;

[0079] Figure 3 is a schematic diagram of a scenario in which a user terminal interacts with a self-moving cleaning device according to an embodiment of this application;

[0080] Figure 4 is a schematic diagram of the structure of a self-moving cleaning equipment control device provided in an embodiment of this application;

[0081] Figure 5 is a structural schematic diagram of a self-moving cleaning device provided in an embodiment of this application.

[0082] The accompanying drawings illustrate specific embodiments of this application, which will be described in more detail below. These drawings and descriptions are not intended to limit the scope of the concept in any way, but rather to illustrate the concept of this application to those skilled in the art through reference to particular embodiments. Detailed Implementation

[0083] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.

[0084] To facilitate a clear description of the technical solutions in the embodiments of this application, the terms "first" and "second" are used in the embodiments of this application to distinguish identical or similar items with essentially the same function and purpose. For example, "first device" and "second device" are merely used to distinguish different devices and do not limit their order of execution. Those skilled in the art will understand that the terms "first" and "second" do not limit the quantity or execution order, and that "first" and "second" do not necessarily imply that they are different.

[0085] It should be noted that, in this application, the terms "exemplary" or "for example" are used to indicate that something is being described as an example, illustration, or illustration. Any embodiment or design described as "exemplary" or "for example" in this application should not be construed as being more preferred or advantageous than other embodiments or design solutions. Specifically, the use of terms such as "exemplary" or "for example" is intended to present the relevant concepts in a concrete manner.

[0086] In this application, "at least one" means one or more, and "more than one" means two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can mean: A alone, A and B simultaneously, or B alone, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one of a, b, or c can mean: a, b, c, ab, ac, bc, or abc, where a, b, and c can be single or multiple.

[0087] Self-moving cleaning equipment is a type of self-moving robot specifically designed for cleaning tasks, such as robotic vacuum cleaners and automatic floor scrubbers. It is used to sweep surfaces, vacuum, and mop to keep the environment clean.

[0088] In one possible implementation, the self-moving cleaning device can use a box detection algorithm to identify and avoid obstacles in the wiring harness during the cleaning process.

[0089] However, box detection algorithms use rectangles to represent the position and extent of objects. Therefore, using rectangles cannot accurately represent the irregular shape of wire harness obstacles, making it difficult to locate the true position of wire harness obstacles and causing misjudgments when self-moving cleaning devices avoid obstacles. Moreover, during the annotation stage, how to define the boundaries of wire harness obstacles may be ambiguous, especially when wire harnesses are intersecting or partially occluded. Therefore, box detection algorithms are usually designed for objects with clear boundaries and shapes, and their adaptability to wire harnesses without fixed shapes is poor.

[0090] Understandably, if the location of wiring harness obstructions is not accurately identified during the cleaning process of the self-propelled cleaning equipment, the obstructions may get caught in the drive wheels of the self-propelled cleaning equipment, causing the equipment to stop working or be damaged; it may also cause items on tables and chairs to fall off due to the self-propelled cleaning equipment pulling on the wiring harness obstructions, resulting in user losses and affecting the user experience.

[0091] In another possible implementation, the self-moving cleaning device can identify objects and terrain in the environment through semantic segmentation and perform obstacle avoidance, navigation, path planning, etc.; the semantic segmentation is a deep learning algorithm that associates labels or categories with each pixel of an image to identify a set of pixels that constitutes a distinguishable category, in which each pixel is labeled as a specific category.

[0092] Labeling pixel categories is a time-consuming and expensive process, especially when high precision is required. Thus, while semantic segmentation can identify the shape and location of wire harness obstacles during the cleaning process of self-moving cleaning equipment, it requires processing a large amount of pixel-level data, resulting in high computational complexity. This can lead to latency, especially in real-time applications, and consequently affect the obstacle avoidance performance of self-moving cleaning equipment.

[0093] To address the aforementioned problems, this application provides a control method for a self-moving cleaning device. During the cleaning process, after identifying a wire harness obstacle, an image of the obstacle is acquired to identify the adjacent region of the wire harness in the image. This adjacent region at least includes the envelope region of the obstacle, which is a closed region created around the boundary of the obstacle. Further, lines connecting the position coordinates of each pixel in the adjacent region to the position coordinates of the obstacle are found. These lines intersect with the obstacle at common points, which can better fit the non-fixed shape of the wire harness. Based on these lines, the position coordinates of the obstacle are accurately determined. Using these determined position coordinates, the self-moving cleaning device adjusts its path planning to avoid the obstacle, ensuring the smooth progress of the cleaning task. Thus, precise image analysis and position calculation through these lines improve the accuracy of wire harness obstacle location identification. Furthermore, by accurately identifying and avoiding obstacles, the failures and damage caused by wire harness entanglement or pulling are reduced, improving the safety and reliability of the device.

[0094] For example, Figure 1 is a schematic diagram of an application scenario provided by an embodiment of this application. As shown in Figure 1, the application scenario is applied to a scenario in which a self-moving cleaning device performs a cleaning task. Taking a sweeping robot as an example, the application scenario includes: a sweeping robot 100 and a wire harness obstacle 101. The sweeping robot 100 has a built-in Neural Processing Unit (NPU) chip, which is used to execute the method of the embodiment of this application.

[0095] During the cleaning process of the robot vacuum cleaner 100 in the living room, if it detects the wire harness obstacle 101, it can acquire an image of the wire harness obstacle 101. The image can be an RGB image captured by an RGB (Red, Green, Blue) camera. This application embodiment does not specifically limit the type of image or the capturing device.

[0096] Furthermore, the RGB image is transmitted to the NPU chip of the robotic vacuum cleaner 100 so that the NPU chip can process the RGB image, determine the position coordinates of the wire harness obstacle 101 in the image, and then avoid the wire harness obstacle 101 based on the position coordinates and continue to perform the cleaning task.

[0097] Wherein, part of the wire harness in the wire harness obstacle 101 is located on the surface to be cleaned, and the rest of the wire harness is suspended in the air, the image of the wire harness obstacle 101 captured by the sweeping robot 100 includes at least the image of the part of the wire harness located on the surface to be cleaned.

[0098] Optionally, if the image of the wire harness obstacle 101 captured by the robot vacuum cleaner 100 is an image of a portion of the wire harness on the surface to be cleaned, the remaining portion of the wire harness above the robot vacuum cleaner 100 does not need to be captured, because the remaining portion of the wire harness may not affect the robot vacuum cleaner 100 from performing the cleaning task. In this way, the processing requirements for image data can be reduced and the processing efficiency can be improved.

[0099] Optionally, if the image of the wire harness obstacle 101 captured by the robot vacuum cleaner 100 is an image corresponding to all wire harnesses within the scanning field of view of the robot vacuum cleaner 100, and the image may include some wire harnesses higher than the robot vacuum cleaner 100, then the NPU chip can use a projection algorithm to project the wire harnesses in the image, and then determine the position coordinates of the wire harness obstacle 101 in the image after projection processing.

[0100] It should be noted that the self-moving cleaning equipment control method can also be applied to shopping malls, schools, and offices. This application does not limit the specific application scenarios of the self-moving cleaning equipment control method; the above are merely illustrative examples.

[0101] For example, Figure 2 is a flowchart illustrating a self-moving cleaning device control method provided in an embodiment of this application. As shown in Figure 2, the self-moving cleaning device control method is applied to a self-moving cleaning device. The execution entity of the self-moving cleaning device control method can be the NPU chip mentioned above, or it can be a processor or controller within the self-moving cleaning device. This embodiment of the application does not specifically limit this. The method includes the following steps:

[0102] S201. During the cleaning process of the self-moving cleaning equipment, after identifying the obstacle and determining that the obstacle is a wire harness obstacle, an image of the wire harness obstacle is acquired.

[0103] In this application, during the cleaning process of the self-moving cleaning equipment, sensors such as cameras, lidar, infrared sensors, and ultrasonic sensors can be used to scan the surrounding environment to determine whether there are obstacles in the surrounding environment. The embodiments of this application do not specifically limit the type of sensor for identifying obstacles.

[0104] Once the existence of an obstacle is determined, the type of obstacle can be identified based on computer vision technology, such as artificial intelligence (AI) algorithms, image processing algorithms, etc. This application embodiment does not specifically limit the method for identifying the type of obstacle.

[0105] Furthermore, after determining that the obstacle is a wire harness obstacle, an image of the wire harness obstacle can be obtained for further analysis and processing.

[0106] It should be noted that the embodiments of this application do not specifically limit the content of the image of the wire harness obstacle acquired by the self-moving cleaning device. It can be determined based on the functional settings of the self-moving cleaning device, but the image of the wire harness obstacle shall at least include the image of the wire harness located on the surface to be cleaned.

[0107] S202. Determine the position coordinates of the wire harness in the wire harness obstacle image, and avoid the wire harness obstacle based on the position coordinates;

[0108] Determining the position coordinates of the wire harness in the obstacle image includes:

[0109] The image of the wire harness obstacle is determined to include the wire harness neighboring region, and a line is drawn connecting the position coordinates of each pixel in the wire harness neighboring region to the position coordinates of the wire harness obstacle. The position coordinates of the wire harness obstacle are determined based on the line. The wire harness neighboring region includes at least the envelope region of the wire harness obstacle. The envelope region is a closed region created around the boundary of the wire harness obstacle region.

[0110] In this embodiment, the envelope region is a closed convex polygon that contains all points in the point set corresponding to the harness obstacle area, and has the smallest area among all possible convex regions that may contain these point sets. The envelope region is a closed region created around the boundary of the harness obstacle, which can ensure that all possible harness obstacles are within the scope of consideration.

[0111] In this step, an image containing the wire harness obstacle is acquired, and the wire harness proximity region of the obstacle in the image is determined. The wire harness proximity region includes at least the envelope region of the wire harness obstacle. Further, within the wire harness proximity region, the position coordinates of each pixel are identified, which are usually represented by the pixel position of the image. Further, the line connecting the position coordinates of each pixel and the position coordinates of the wire harness obstacle is determined, and then a predefined algorithm is used to determine the position coordinates of the wire harness obstacle based on the line.

[0112] The predefined algorithm can be the minimum value method, Euclidean distance algorithm, Dijkstra's algorithm, etc., and this application does not specifically limit it.

[0113] Optionally, determining the position coordinates of the harness obstacle based on the connection includes:

[0114] The position coordinates of the wire harness obstacle are determined based on the distance between the connecting lines; where the distance between each connecting line is the minimum distance between the position coordinates of the corresponding pixel in the adjacent region of the wire harness and the position coordinates of the wire harness obstacle.

[0115] For example, for each pixel, the line connecting it to the wire harness obstacle can be calculated. By analyzing the distances of these lines, it can be determined which pixel(s) have the smallest line distance to the wire harness obstacle. These pixels with the smallest distance are used to help determine the precise location coordinates of the obstacle.

[0116] In this way, by taking into account the distance from each pixel in the vicinity of the wire harness to the wire harness obstacle, and by calculating and comparing the minimum distance, the wire harness obstacle can be accurately located, and the position of the wire harness obstacle can be effectively identified. Especially in complex environments, the wire harness obstacle may have an irregular shape, and the distance calculation method based on the connection can adapt to these complexities and provide more reliable positioning results.

[0117] In this application, after determining the position coordinates of the wire harness obstacle, a path can be planned or the operation adjusted based on the position coordinates of the wire harness obstacle to avoid the obstacle; wherein, the adjustment operation may involve adjusting the movement path of the device to ensure that it does not collide with the wire harness obstacle. The embodiments of this application do not specifically limit the obstacle avoidance strategy adopted.

[0118] Therefore, during the cleaning process of self-moving cleaning equipment, by analyzing the connection between pixels and wire harness obstacles in detail, the position of the wire harness obstacles can be accurately determined. After the position coordinates are determined, the path can be planned more effectively to avoid potential collisions or interference. This reduces the physical contact between the equipment and the wire harness, lowers the risk of tangling, damaging the wire harness or the equipment itself, improves safety, and can complete the cleaning task more efficiently.

[0119] Understandably, since the above process can be executed automatically by algorithms, the need for human intervention is reduced, and the above method relies on data calculation rather than vision or other imprecise methods, thereby improving the accuracy and consistency of positioning.

[0120] Optionally, determining the region adjacent to the wire harness in the wire harness obstacle image includes:

[0121] Obtain a predefined neighborhood window and calculate the neighborhood features of each pixel in the wire harness obstacle image within the predefined neighborhood window;

[0122] Based on neighborhood features, threshold segmentation is performed on pixels in the wire bundle obstacle image to obtain the wire bundle neighboring region and non-wire bundle neighboring region.

[0123] The self-moving cleaning equipment does not treat areas not adjacent to the wire harness.

[0124] It should be noted that the area adjacent to the wire harness and the area not adjacent to the wire harness together constitute the wire harness obstacle image, and these two areas do not overlap; there are no wire harness obstacles in the non-wire harness area, so the non-wire harness area is not processed, thereby optimizing processing efficiency.

[0125] In this application, the image of the wire harness obstacle is segmented in a manner different from conventional segmentation methods. Since the wire harness is not a convex polygon and cannot be fitted with a polygon, the NPU chip uses a neighborhood segmentation method to obtain the non-wire harness region and the wire harness neighboring region. Among them, since the wire harness is special and has no regular shape, it is not possible to obtain an accurate region in one step. Therefore, the wire harness neighboring region is the approximate region of the wire harness obstacle obtained through neighborhood segmentation.

[0126] In this step, a predefined neighborhood window size is selected to analyze the local features of each pixel in the image. The predefined neighborhood window can be a rectangular or square region of a fixed size, and this embodiment does not specifically limit it.

[0127] Furthermore, for each pixel in the image, its neighborhood features within a predefined neighborhood window are calculated, which may include information such as color, brightness, and texture. By analyzing these neighborhood features, it is possible to identify which pixels belong to the wire harness obstacle.

[0128] Furthermore, based on the calculated neighborhood features, the pixels in the image are segmented by thresholding, that is, by selecting an appropriate threshold, the pixels are divided into the line bundle neighbor region and the non-line bundle neighbor region.

[0129] The threshold can be selected based on experience, statistical analysis or machine learning methods to ensure accurate segmentation. This application does not specify the threshold size.

[0130] In this way, by identifying and ignoring non-wire harness neighboring areas, the processing speed can be improved. Furthermore, by determining the wire harness neighboring and non-wire harness neighboring areas through the aforementioned neighborhood segmentation method, the wire harness neighboring areas that need to be focused on can be identified more accurately, thereby improving the subsequent recognition accuracy of wire harness obstacles.

[0131] Optionally, semantic segmentation can also be used to segment the wire harness neighboring region and the non-wire harness neighboring region, performing pixel-by-pixel segmentation. The boundaries of the two segmented sub-image regions are clear, namely the wire harness neighboring region and the non-wire harness neighboring region. This application embodiment does not specifically limit the method for segmenting the wire harness neighboring region and the non-wire harness neighboring region.

[0132] Optionally, determining the position coordinates of the wire harness in the wire harness obstacle image includes:

[0133] The image of the wire harness obstacle is input into the trained wire harness detection model to predict the position coordinates of the wire harness obstacle in the image.

[0134] In this embodiment of the application, using a trained wire harness detection model to determine the position coordinates of the wire harness in the wire harness obstacle image is a machine learning-based method. This embodiment of the application does not limit the specific model type corresponding to the wire harness detection model. For example, the wire harness detection model can be a convolutional neural network, a recurrent neural network, an autoencoder, a generative adversarial network, a variational autoencoder, etc.

[0135] For example, the image of the wire harness obstacle to be processed is input into a trained wire harness detection model. The model extracts image features through its neural network layer. Furthermore, the model analyzes the image features and predicts the position coordinates of the wire harness obstacle.

[0136] Therefore, using a trained wire harness detection model to predict the position coordinates of wire harness obstacles can accurately identify wire harness obstacles in complex backgrounds, adapt to various environmental changes, and have strong robustness. Furthermore, the aforementioned wire harness detection model can also adapt to new environments or identify various types of wire harness obstacles through transfer learning or further training, demonstrating good scalability.

[0137] Optionally, the wire harness detection model is a neural network model based on deep learning.

[0138] For example, as shown in Figure 1, the NPU chip uses a pre-trained neural network model to process RGB images to identify and locate the precise position of the wire bundle; that is, the neural network model is pre-trained by learning from the annotation results of wire bundles in a large number of images.

[0139] It should be noted that deep learning-based neural network models perform well in image recognition and classification tasks, can handle complex image features, provide high-precision detection results, and can effectively identify target objects in complex and variable backgrounds. They are suitable for wire harness detection in various environments and conditions. In particular, the model has strong robustness to noise, occlusion and lighting changes, and can adapt to different image quality and environmental conditions.

[0140] In this way, the deep learning-based neural network model can identify the wire bundle and accurately locate its specific position.

[0141] The deep learning-based neural network model includes a regression process to output the position coordinates of the wire harness obstacle. Therefore, the inaccuracy of the wire harness boundary caused by neighborhood segmentation can be offset by accurate regression localization to obtain the accurate position of the wire harness.

[0142] Optionally, the wire harness detection model includes a classification model and a regression model; the wire harness obstacle image is input into the trained wire harness detection model to predict the position coordinates of the wire harness obstacle in the image, including:

[0143] Input the image of the wire harness obstacle into the trained classification model to obtain the wire harness neighborhood in the image of the wire harness obstacle;

[0144] By inputting the area near the wire harness into the trained regression model, the position coordinates of the wire harness obstacle in the area near the wire harness are obtained.

[0145] It should be noted that regression is a predictive action to form a value. Unlike existing box detection methods, which use regression to form a detection rectangle, the regression model in this application regresses the neighborhood of the wire bundle, that is, the coordinate values ​​of the pixel points corresponding to the wire bundle obstacles.

[0146] For example, an image of a wire harness obstacle is input into a trained classification model, and the model outputs an image containing the area adjacent to the wire harness; this classification model has been trained with a large amount of labeled data and can identify areas in the image containing the wire harness.

[0147] Furthermore, the neighborhood of the wire bundle identified by the classification model is input into the trained regression model, which outputs the precise location coordinates of the wire bundle obstacle. This regression model can predict the precise location coordinates of the wire bundle by learning the positional features of the wire bundle in the image.

[0148] For example, the prediction process of this regression model includes: for each pixel in the vicinity of the wire harness, predicting the position of the nearest wire harness. For instance, for pixel A in the vicinity of the wire harness, draw a perpendicular line to the wire harness, that is, find the minimum distance between pixel A and the line connecting the wire harness obstacle area, and then obtain the position of pixel B on the wire harness obstacle. Then, the position B is obtained by regression through pixel A. Similarly, the above operation is performed for each pixel (A1, A2, ..., An) in the vicinity of the wire harness, so that the positions of all pixels (B1, B2, ..., Bn) on the wire harness obstacle can be obtained by regression. B1, B2, ..., Bn are the pixels located on the wire harness obstacle, and the regression model accurately predicts the position of the entire wire harness obstacle.

[0149] Understandably, the regression model identified the obstacles in the wire harness and accurately determined their specific locations, thus solving the problem that the wire harness in the vicinity of the wire harness obtained from the neighborhood segmentation could not be accurately located.

[0150] In this way, by decomposing the model prediction into two steps, classification and regression, this application can optimize the performance of each model and improve the overall detection accuracy. The classification model first narrows down the area that needs to be accurately located, reduces the computational burden of the regression model, and improves the processing efficiency. Then, the trained regression model can be used to perform accurate regression localization, fit the irregular shape of the wire bundle obstacle to identify and accurately locate the wire bundle, and then output the specific location of the wire bundle.

[0151] It should be noted that the classification model and the regression model are combined to obtain the wire harness detection model. The classification model and the regression model are decoupled. Since the classification model itself has the risk of boundary instability, the classification model is only used to identify the vicinity of the wire harness and does not accurately locate its position. Instead, the regression model is used to accurately locate the wire harness obstacle.

[0152] Optionally, the training process for the harness detection model includes:

[0153] Obtain the training dataset; the training dataset includes multiple training samples, each training sample includes a wire harness obstacle image and the position coordinates of the wire harness obstacle in the wire harness obstacle image;

[0154] The training dataset is input into the wire harness detection model for training, resulting in a trained wire harness detection model.

[0155] The training process includes: determining the neighborhood of each obstacle in the bundle image, determining the minimum distance between the position coordinates of each pixel in the neighborhood and the position coordinates of the obstacle, and using the line corresponding to the minimum distance, the position coordinates of the pixel corresponding to each line, and the position coordinates of the obstacle to perform regression training.

[0156] For example, during the training of the wire harness detection model, a large image dataset containing wire harness obstacles can be collected. This image dataset includes multiple training samples, each of which includes the image itself and the location coordinates of the wire harness obstacle.

[0157] Furthermore, the training dataset is input into the wire harness detection model for training. This model can be a deep learning model, such as a convolutional neural network model, which is not specifically limited in this embodiment. During the training process, the model first identifies the wire harness neighborhood in each image, and then, for each wire harness neighborhood, calculates the distance between the position coordinates of each pixel and the position coordinates of the wire harness obstacle, and determines the connection with the smallest distance among these connections to accurately locate the wire harness obstacle.

[0158] Furthermore, regression training is performed using the lines corresponding to the minimum distances, the position coordinates of the pixels corresponding to each line, and the position coordinates of the obstacles in the duct bundle. This allows the model to learn these relationships and predict the precise location of the obstacles in the duct bundle in new images.

[0159] In this way, by calculating the minimum distance and performing regression training, the model can accurately locate the position of the wire harness obstacle, improving detection accuracy. Furthermore, through precise regression training, the model can reduce errors in position prediction and improve overall reliability. In particular, the wire harness detection model can automatically learn complex features in the image, reducing reliance on manual intervention.

[0160] Optionally, the wire harness detection model includes a classification model and a regression model; the training dataset is input into the wire harness detection model for training to obtain a trained wire harness detection model, including:

[0161] For each wire harness obstacle image, obtain the wire harness neighboring region in the labeled wire harness obstacle image;

[0162] Multiple images of wire harness obstacles and the adjacent regions of each wire harness obstacle image are input into the classification model for training to obtain a trained classification model.

[0163] The position coordinates of multiple wire harness neighborhoods and wire harness obstacles in the wire harness neighborhoods are input into the regression model for training, and a trained regression model is obtained.

[0164] The trained classification model and the trained regression model are combined to obtain the trained wire harness detection model.

[0165] In this embodiment of the application, the area adjacent to the wire harness is labeled for each wire harness obstacle image. This can be done manually to ensure that the area adjacent to the wire harness in each image is identified. It should be noted that this embodiment of the application does not specifically limit the method of labeling the area adjacent to the wire harness in the wire harness obstacle image.

[0166] For example, multiple images of wire harness obstacles and their corresponding adjacent regions are input into a classification model. The classification model learns the features in the images to identify which regions may contain wire harness obstacles, thereby obtaining a well-trained classification model that can effectively identify adjacent regions of wire harnesses.

[0167] The identified multiple adjacent regions of the wire harness and their corresponding wire harness position coordinates are input into the regression model. The regression model learns the relationship between the pixels in these regions and the wire harness position coordinates in order to accurately predict the position of the wire harness. This results in a well-trained regression model that can accurately locate the position of the wire harness obstacle in the identified adjacent regions of the wire harness.

[0168] Furthermore, the trained classification and regression models are combined to form a complete wire harness detection model, which can identify the area near the wire harness and then accurately locate the position of wire harness obstacles within that area.

[0169] In this way, by combining classification and regression models, possible neighboring areas of the wire harness can be identified, and the location of the wire harness can be accurately located, improving the overall detection accuracy. Furthermore, the independent training and optimization of the classification and regression models make the overall design more modular, facilitating maintenance and expansion. Specifically, the classification model first narrows down the area that needs to be processed accurately, effectively filtering out irrelevant areas, reducing the computational burden on the regression model, and improving processing efficiency. In turn, the regression model accurately locates the wire harness obstacle, improving the positioning accuracy.

[0170] Optionally, the training process for the regression model includes:

[0171] Obtain the position coordinates of each pixel in the vicinity of the wire harness;

[0172] For each pixel, using the pixel's position coordinates and the position coordinates of the wire bundle obstacle, calculate the multiple distances from the pixel to the multiple pixels corresponding to the wire bundle obstacle.

[0173] Determine the minimum distance among multiple distances, and then determine the connection between the pixels corresponding to the minimum distance;

[0174] Regression training is performed using the lines connecting the pixels corresponding to the minimum distance, the position coordinates of the pixels corresponding to each line, and the position coordinates of the obstacles in the line bundle.

[0175] In this embodiment, the position coordinates of each pixel are extracted from the area adjacent to the wire harness to calculate the distance between the wire harness and obstacles.

[0176] For example, for each pixel, multiple distances between its position coordinates and the wire harness obstacle are calculated. Further, among the calculated multiple distances, the minimum distance is found. This minimum distance represents the closest contact point between the pixel and the wire harness obstacle. Then, the line connecting the pixel and the pixel corresponding to the minimum distance is determined. This line is used to define the direct relationship between the pixel and the wire harness obstacle.

[0177] Furthermore, using these connections, the position coordinates of the pixels corresponding to each connection, and the position coordinates of the bundle obstacles, regression training is performed so that the regression model can learn these relationships and predict the precise location of the bundle obstacles in new data.

[0178] In this way, by utilizing the minimum distance between pixels, the model can accurately capture the position of the wire harness obstacle, improving positioning accuracy. This minimum distance is determined through precise distance calculation and connection, which allows the model to reduce errors in position prediction and improve overall reliability.

[0179] Understandably, the methods described above can help the model understand spatial relationships and geometric features in an image, enabling it to better adapt to different images and environments.

[0180] Optionally, avoiding harness obstacles based on location coordinates includes at least one of the following cases:

[0181] The cleaning route is replanned based on the location coordinates of the wire harness obstacle, and the self-moving cleaning equipment performs the cleaning task according to the cleaning route to avoid the wire harness obstacle;

[0182] The cleaning no-go zone is determined based on the location coordinates of the wire harness obstacle. After the self-moving cleaning equipment cleans around the cleaning no-go zone, it continues to perform the cleaning task according to the original planned path.

[0183] The location coordinates of the wiring harness obstacle are sent to the user terminal for display, and the obstacle is avoided in response to the user's control operation.

[0184] In some embodiments, after the self-moving cleaning device determines the location coordinates of the wire harness obstacle, a path planning algorithm, such as the A* algorithm or the Dijkstra algorithm, can be used to recalculate the cleaning route to ensure that the wire harness obstacle is avoided. Thus, the self-moving cleaning device can perform the task according to the newly planned cleaning route, thereby effectively avoiding the obstacle.

[0185] In other embodiments, after the self-moving cleaning device determines the location coordinates of the wiring harness obstacle, it determines a cleaning no-go zone that the self-moving cleaning device should not enter based on these location coordinates. Then, the self-moving cleaning device cleans around the cleaning no-go zone and continues to perform the task according to the originally planned path after completion, so as to ensure that the device can operate efficiently in complex environments.

[0186] Understandably, replanning routes or designating no-go zones can ensure that self-propelled cleaning equipment completes its cleaning tasks in the shortest possible time, thus improving overall efficiency.

[0187] In some other embodiments, after the self-moving cleaning device determines the location coordinates of the wire harness obstacle, it can send the location coordinates of the wire harness obstacle to a user terminal device, such as a smartphone or computer, so that the user can view the location of the wire harness obstacle on the terminal and instruct the device to avoid the wire harness obstacle through control operations; wherein, the self-moving cleaning device can adjust its path according to the user's instructions to avoid the wire harness obstacle.

[0188] Understandably, by sending the location of obstacles in the wiring harness to the user terminal, users can participate in the control of the device, increasing the flexibility and customization of operation. This provision of real-time information and interactive functions can enhance the user's sense of control and satisfaction with the device.

[0189] In this way, once the self-moving cleaning equipment determines the location coordinates of the wire harness obstacle, it can avoid the obstacle in various ways to adapt to different environmental changes and user needs. This improves application flexibility and reduces the risk of collision between the equipment and the wire harness obstacle, thereby protecting the equipment and other items in the environment.

[0190] Optionally, this application also provides a self-moving cleaning device control method, applied to a user terminal, the method comprising:

[0191] Receive the position coordinates of the wire harness in the image of the wire harness obstacle determined during the cleaning task sent by the mobile cleaning equipment, and visualize the position coordinates;

[0192] Determining the position coordinates of the wire harness in the obstacle image includes:

[0193] After identifying the obstacle and determining that it is a wire harness obstacle, an image of the wire harness obstacle is acquired;

[0194] The image of the wire harness obstacle is determined to include the wire harness neighboring region, and a line is drawn connecting the position coordinates of each pixel in the wire harness neighboring region to the position coordinates of the wire harness obstacle. The position coordinates of the wire harness obstacle are determined based on the line. The wire harness neighboring region includes at least the envelope region of the wire harness obstacle. The envelope region is a closed region created around the boundary of the wire harness obstacle region.

[0195] In this application, by visually displaying the location coordinates of wiring harness obstacles on the user terminal, the user can intuitively understand the environment around the device and potential wiring harness obstacles, so as to better understand the working status of the device and the environment.

[0196] The user terminal can also be referred to as terminal equipment, user equipment (UE), mobile station (MS), mobile terminal, terminal, etc. In practical applications, user terminals include, for example, desktop computers, laptops, personal digital assistants (PDAs), smartphones, tablets, in-vehicle devices, wearable devices (such as smartwatches and smart bracelets), and smart home devices (such as smart display devices), etc. This application does not specifically limit the types of devices mentioned.

[0197] Optionally, users can view the location of wiring harness obstacles detected by the self-propelled cleaning device in real time, improving the transparency of device operation and giving users a clearer understanding of the device's behavior and decision-making process. In turn, users can make informed decisions based on the visualized location of wiring harness obstacles, such as adjusting the cleaning route of the self-propelled cleaning device or setting new cleaning no-go zones to improve cleaning efficiency and safety.

[0198] In this way, by displaying the location of the wiring harness obstacle on the user terminal, the user can take preventive measures to avoid collisions between the self-moving cleaning device and the wiring harness obstacle, thereby protecting the self-moving cleaning device and the surrounding environment.

[0199] It should be noted that the process of the self-moving cleaning device determining the position coordinates of the wire harness obstacle can be referred to the description in the above embodiments, and will not be repeated here.

[0200] For example, Figure 3 is a schematic diagram of a scenario of interaction between a user terminal and a self-moving cleaning device provided in an embodiment of this application. As shown in Figure 3, the scenario includes a sweeping robot 100 and a user terminal 200. After the sweeping robot 100 determines the position coordinates of the wire harness obstacle, it can send the position coordinates of the wire harness obstacle to the user terminal 200 so that the user terminal 200 can display the position of the wire harness obstacle. In the figure, there are many black dots, which are the position coordinates of the wire harness in the determined wire harness obstacle image. Optionally, in the visualization display, the position coordinates of the wire harness can be connected to generate the approximate shape of the wire harness obstacle. This embodiment of the application does not specifically limit this process.

[0201] In this way, by providing real-time visualizations and interactive functions, the opportunities for users to participate in the operation of the device are increased, improving the user experience and satisfaction. Furthermore, users can dynamically adjust the cleaning strategy of the device based on the real-time visualizations, more accurately control the behavior of the device, reduce the possibility of misoperation, and thus reduce the risk of damage to the self-propelled cleaning device.

[0202] In the foregoing embodiments, the self-moving cleaning device control method provided in this application has been described. To achieve the functions of the methods provided in the embodiments of this application, the self-moving cleaning device, as the executing entity, may include hardware structures and / or software modules, implementing the above functions in the form of hardware structures, software modules, or a combination of hardware structures and software modules. Whether a particular function is executed in the form of hardware structures, software modules, or a combination of hardware structures and software modules depends on the specific application and design constraints of the technical solution.

[0203] For example, Figure 4 is a schematic diagram of a self-moving cleaning equipment control device provided in an embodiment of this application, applied to a self-moving cleaning equipment. As shown in Figure 4, the device 400 includes:

[0204] The identification module 401 is used to identify obstacles and, after determining that the obstacle is a wire harness obstacle, acquire an image of the wire harness obstacle during the cleaning task performed by the self-moving cleaning device.

[0205] The determining module 402 is used to determine the position coordinates of the wire harness in the wire harness obstacle image and avoid the wire harness obstacle based on the position coordinates;

[0206] Specifically, the determining module 402 is used for:

[0207] The image of the wire harness obstacle is determined to include the wire harness neighboring region, and a line is drawn connecting the position coordinates of each pixel in the wire harness neighboring region to the position coordinates of the wire harness obstacle. The position coordinates of the wire harness obstacle are determined based on the line. The wire harness neighboring region includes at least the envelope region of the wire harness obstacle. The envelope region is a closed region created around the boundary of the wire harness obstacle region.

[0208] Optionally, the determining module 402 includes a first determining unit, which is used for:

[0209] The position coordinates of the wire harness obstacle are determined based on the distance between the connecting lines; where the distance between each connecting line is the minimum distance between the position coordinates of the corresponding pixel in the adjacent region of the wire harness and the position coordinates of the wire harness obstacle.

[0210] Optionally, the determining module 402 includes an obstacle avoidance unit, which is used for:

[0211] The cleaning route is replanned based on the location coordinates of the wire harness obstacle, and the self-moving cleaning equipment performs the cleaning task according to the cleaning route to avoid the wire harness obstacle;

[0212] The cleaning no-go zone is determined based on the location coordinates of the wire harness obstacle. After the self-moving cleaning equipment cleans around the cleaning no-go zone, it continues to perform the cleaning task according to the original planned path.

[0213] The location coordinates of the wiring harness obstacle are sent to the user terminal for display, and the obstacle is avoided in response to the user's control operation.

[0214] Optionally, the determining module 402 includes a second determining unit, which is used for:

[0215] Obtain a predefined neighborhood window and calculate the neighborhood features of each pixel in the wire harness obstacle image within the predefined neighborhood window;

[0216] Based on neighborhood features, threshold segmentation is performed on pixels in the wire bundle obstacle image to obtain the wire bundle neighboring region and non-wire bundle neighboring region.

[0217] The self-moving cleaning equipment does not treat areas not adjacent to the wire harness.

[0218] Optionally, the first determining unit is specifically used for:

[0219] The image of the wire harness obstacle is input into the trained wire harness detection model to predict the position coordinates of the wire harness obstacle in the image.

[0220] Optionally, the harness detection model includes a classification model and a regression model; the prediction unit is specifically used for:

[0221] Input the image of the wire harness obstacle into the trained classification model to obtain the wire harness neighborhood in the image of the wire harness obstacle;

[0222] By inputting the area near the wire harness into the trained regression model, the position coordinates of the wire harness obstacle in the area near the wire harness are obtained.

[0223] Optionally, the wire harness detection model is a neural network model based on deep learning.

[0224] Optionally, the device 400 also includes a training module, which is used for:

[0225] Obtain the training dataset; the training dataset includes multiple training samples, each training sample includes a wire harness obstacle image and the position coordinates of the wire harness obstacle in the wire harness obstacle image;

[0226] The training dataset is input into the wire harness detection model for training, resulting in a trained wire harness detection model.

[0227] The training process includes: determining the neighborhood of each obstacle in the bundle image, determining the minimum distance between the position coordinates of each pixel in the neighborhood and the position coordinates of the obstacle, and using the line corresponding to the minimum distance, the position coordinates of the pixel corresponding to each line, and the position coordinates of the obstacle to perform regression training.

[0228] Optionally, the harness detection model includes a classification model and a regression model; the training module is specifically used for:

[0229] For each wire harness obstacle image, obtain the wire harness neighboring region in the labeled wire harness obstacle image;

[0230] Multiple images of wire harness obstacles and the adjacent regions of each wire harness obstacle image are input into the classification model for training to obtain a trained classification model.

[0231] The position coordinates of multiple wire harness neighborhoods and wire harness obstacles in the wire harness neighborhoods are input into the regression model for training, and a trained regression model is obtained.

[0232] The trained classification model and the trained regression model are combined to obtain the trained wire harness detection model.

[0233] Optionally, the training module includes a training submodule for training the regression model. Specifically, this training submodule is used for:

[0234] Obtain the position coordinates of each pixel in the vicinity of the wire harness;

[0235] For each pixel, using the pixel's position coordinates and the position coordinates of the wire bundle obstacle, calculate the multiple distances from the pixel to the multiple pixels corresponding to the wire bundle obstacle.

[0236] Determine the minimum distance among multiple distances, and then determine the connection between the pixels corresponding to the minimum distance;

[0237] Regression training is performed using the lines connecting the pixels corresponding to the minimum distance, the position coordinates of the pixels corresponding to each line, and the position coordinates of the obstacles in the line bundle.

[0238] It should be noted that the specific implementation principle and effect of the above-mentioned self-moving cleaning equipment control device 400 can be found in the relevant description and effect of the above embodiments, and will not be elaborated further here.

[0239] Optionally, this application also provides a self-propelled cleaning equipment control device, which is applied to a user terminal. The device includes a receiving module for:

[0240] Receive the position coordinates of the wire harness in the image of the wire harness obstacle determined during the cleaning task sent by the mobile cleaning equipment, and visualize the position coordinates;

[0241] Determining the position coordinates of the wire harness in the obstacle image includes:

[0242] After identifying the obstacle and determining that it is a wire harness obstacle, an image of the wire harness obstacle is acquired;

[0243] The image of the wire harness obstacle is determined to include the wire harness neighboring region, and a line is drawn connecting the position coordinates of each pixel in the wire harness neighboring region to the position coordinates of the wire harness obstacle. The position coordinates of the wire harness obstacle are determined based on the line. The wire harness neighboring region includes at least the envelope region of the wire harness obstacle. The envelope region is a closed region created around the boundary of the wire harness obstacle region.

[0244] It is understood that the specific implementation principle and effect of the above-mentioned self-moving cleaning equipment control device can be found in the relevant descriptions and effects of the above-mentioned related embodiments, and will not be elaborated further here.

[0245] This application also provides a self-moving cleaning device. Figure 5 is a structural schematic diagram of a self-moving cleaning device provided in this application. As shown in Figure 5, the device may include: a processor 501 and a memory 502 communicatively connected to the processor; the memory 502 stores a computer program; the processor 501 executes the computer program stored in the memory 502, causing the processor 501 to perform the method described in any of the above embodiments.

[0246] The memory 502 and the processor 501 can be connected via bus 503.

[0247] This application also provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, are used to implement the methods described in any of the foregoing embodiments of this application.

[0248] This application also provides a chip for executing instructions, which is used to perform the methods described in any of the foregoing embodiments of this application as executed by the self-moving cleaning device.

[0249] This application also provides a computer program product, which includes a computer program that, when executed by a processor, can implement the methods described in any of the foregoing embodiments of this application as performed by the self-moving cleaning device.

[0250] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of modules is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple modules or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or modules may be electrical, mechanical, or other forms.

[0251] The modules described as separate components may or may not be physically separate. The components shown as modules may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to implement the solution of this embodiment according to actual needs.

[0252] Furthermore, the functional modules in the various embodiments of this application can be integrated into one processing unit, or each module can exist physically separately, or two or more modules can be integrated into one unit. The unit composed of the above modules can be implemented in hardware or in the form of hardware plus software functional units.

[0253] The integrated modules implemented as software functional modules described above can be stored in a computer-readable storage medium. These software functional modules, stored in a storage medium, include several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) or processor to execute some steps of the methods described in the various embodiments of this application.

[0254] It should be understood that the aforementioned processor can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), etc. A general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in the application can be directly manifested as being executed by a hardware processor, or executed by a combination of hardware and software modules within the processor.

[0255] The memory may include high-speed random access memory (RAM) or non-volatile memory (NVM), such as at least one disk storage device, and may also be a USB flash drive, external hard drive, read-only memory, disk or optical disc, etc.

[0256] The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of illustration, the buses shown in the accompanying drawings are not limited to a single bus or a single type of bus.

[0257] The aforementioned storage media can be implemented from any type of volatile or non-volatile storage device or a combination thereof, such as Static Random-Access Memory (SRAM), Electrically Erasable Programmable Read-Only Memory (EEPROM), Erasable Programmable Read-Only Memory (EPROM), Programmable Read-Only Memory (PROM), Read-Only Memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk. The storage media can be any available medium accessible to general-purpose or special-purpose computers.

[0258] An exemplary storage medium is coupled to a processor, enabling the processor to read information from and write information to the storage medium. Alternatively, the storage medium can be an integral part of the processor. Both the processor and the storage medium can reside in an Application Specific Integrated Circuit (ASIC). Alternatively, the processor and storage medium can exist as discrete components within a self-cleaning device or a main control device.

[0259] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that this application is not limited to the described order of actions, as some steps may be performed in other orders or simultaneously according to this application. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are all optional embodiments, and the actions and modules involved are not necessarily essential to this application.

[0260] It should be further noted that although the steps in the flowchart are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowchart may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the sub-steps or stages of other steps.

[0261] In the above embodiments, the descriptions of each embodiment have their own emphasis. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments. The technical features of the above embodiments can be combined arbitrarily. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as the combination of these technical features does not contradict each other, it should be considered within the scope of this specification.

[0262] Other embodiments of this application will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of this application that follow the general principles of this application and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this application are indicated by the claims.

[0263] The above description is merely a specific implementation of the embodiments of this application, but the protection scope of the embodiments of this application is not limited thereto. Any changes or substitutions within the technical scope disclosed in the embodiments of this application should be covered within the protection scope of the embodiments of this application. Therefore, the protection scope of the embodiments of this application should be determined by the protection scope of the claims.

Claims

1. A self-moving cleaning device control method, characterized by, Applied to self-propelled cleaning devices, the method includes: During the cleaning process of the self-moving cleaning device, after identifying the obstacle and determining that the obstacle is a wire harness obstacle, an image of the wire harness obstacle is acquired; Determine the position coordinates of the wire harness in the wire harness obstacle image, and avoid the wire harness obstacle based on the position coordinates; The step of determining the position coordinates of the wire bundle in the obstacle image includes: The wire harness neighboring region of the wire harness obstacle image is determined, and a line is drawn connecting the position coordinates of each pixel in the wire harness neighboring region and the position coordinates of the wire harness obstacle. The position coordinates of the wire harness obstacle are determined based on the line. The wire harness neighboring region includes at least the envelope region of the wire harness obstacle. The envelope region is a closed region created around the boundary of the wire harness obstacle region.

2. The method of claim 1, wherein, Determining the position coordinates of the obstacle based on the connection line includes: The position coordinates of the wire harness obstacle are determined based on the distance of the connecting lines; wherein, the distance of each connecting line is the minimum distance between the position coordinates of the corresponding pixel in the vicinity of the wire harness and the line connecting the position coordinates of the wire harness obstacle.

3. The method according to claim 1 or 2, characterized in that, Determining the region adjacent to the wire harness in the wire harness obstacle image includes: Obtain a predefined neighborhood window and calculate the neighborhood features of each pixel in the wire harness obstacle image within the predefined neighborhood window; Based on the neighborhood features, the pixels in the wire harness obstacle image are segmented by thresholding to obtain the wire harness neighboring region and the non-wire harness neighboring region. The self-moving cleaning device does not process the area not adjacent to the wire harness.

4. The method according to any one of claims 1-3, characterized in that, Determining the position coordinates of the wire harness in the obstacle image includes: The image of the wire harness obstacle is input into the trained wire harness detection model to predict the position coordinates of the wire harness obstacle in the image.

5. The method of claim 4, wherein, The wire harness detection model includes a classification model and a regression model; the step of inputting the wire harness obstacle image into the trained wire harness detection model to predict the position coordinates of the wire harness obstacle in the wire harness obstacle image includes: The image of the wire harness obstacle is input into a trained classification model to obtain the wire harness neighborhood in the image of the wire harness obstacle. By inputting the area adjacent to the wire harness into the trained regression model, the position coordinates of the wire harness obstacles in the area adjacent to the wire harness are obtained.

6. The method according to claim 4 or 5, characterized in that, The wire harness detection model is a neural network model based on deep learning.

7. The method according to any one of claims 4-6, characterized in that, The training process of the wire harness detection model includes: Obtain a training dataset; the training dataset includes multiple training samples, each training sample including a wire harness obstacle image and the position coordinates of the wire harness obstacle in the wire harness obstacle image; The training dataset is input into the wire harness detection model for training to obtain a trained wire harness detection model. The training process includes: determining the neighborhood of each wire bundle obstacle image, determining the minimum distance between the position coordinates of each pixel in the neighborhood of the wire bundle and the line connecting the position coordinates of the wire bundle obstacle, and performing regression training using the line corresponding to the minimum distance, the position coordinates of the pixel corresponding to each line, and the position coordinates of the wire bundle obstacle.

8. The method of claim 7, wherein, The wire harness detection model includes a classification model and a regression model; the step of inputting the training dataset into the wire harness detection model for training to obtain a trained wire harness detection model includes: For each wire harness obstacle image, obtain the labeled wire harness adjacent region in the wire harness obstacle image; Multiple images of the wire harness obstacles and the adjacent regions of the wire harnesses corresponding to each image are input into the classification model for training to obtain a trained classification model. The position coordinates of multiple adjacent regions of the wire harness and the wire harness obstacles in the adjacent regions of the wire harness are input into the regression model for training to obtain a trained regression model. The trained classification model and the trained regression model are combined to obtain the trained wire harness detection model.

9. The method of claim 8, wherein, The training process of the regression model includes: Obtain the position coordinates of each pixel in the vicinity of the wire harness; For each pixel, using the position coordinates of the pixel and the position coordinates of the wire harness obstacle, calculate the multiple distances from the pixel to the multiple pixels corresponding to the wire harness obstacle. Determine the minimum distance among the plurality of distances, and determine the line connecting the pixels corresponding to the minimum distance; Regression training is performed using the lines connecting the pixels corresponding to the minimum distance, the position coordinates of the pixels corresponding to each line, and the position coordinates of the obstacles in the wire bundle.

10. The method according to any one of claims 1-9, characterized in that, The avoidance of the wire harness obstacle based on the position coordinates includes at least one of the following situations: The cleaning route is replanned based on the position coordinates of the wire harness obstacle, and the self-moving cleaning device performs the cleaning task according to the cleaning route to avoid the wire harness obstacle; Based on the position coordinates of the wire harness obstacle, a cleaning no-go zone is determined. After the self-moving cleaning device cleans around the cleaning no-go zone, it continues to perform the cleaning task according to the originally planned path. The location coordinates of the wire harness obstacle are sent to the user terminal for display, and the obstacle is avoided in response to the user's control operation. 11.A method for controlling a self-moving cleaning device, the method comprising: Applied to a user terminal, the method includes: Receive the position coordinates of the wire harness in the wire harness obstacle image determined during the execution of the cleaning task sent by the self-moving cleaning device, and visualize the position coordinates; The step of determining the position coordinates of the wire bundle in the obstacle image includes: After identifying the obstacle and determining that it is a wire harness obstacle, an image of the wire harness obstacle is acquired; The wire harness neighboring region of the wire harness obstacle image is determined, and a line is drawn connecting the position coordinates of each pixel in the wire harness neighboring region and the position coordinates of the wire harness obstacle. The position coordinates of the wire harness obstacle are determined based on the line. The wire harness neighboring region includes at least the envelope region of the wire harness obstacle. The envelope region is a closed region created around the boundary of the wire harness obstacle region.

12. A self-moving cleaning device, characterized in that, The self-moving cleaning device includes: a processor, and a memory communicatively connected to the processor; The memory stores computer-executed instructions; The processor executes computer execution instructions stored in the memory to implement the method as described in any one of claims 1-10.

13. A self-moving cleaning device control apparatus, characterized by, Applied to self-propelled cleaning equipment, the device includes: The identification module is used to identify obstacles and, after determining that the obstacle is a wire harness obstacle, acquire an image of the wire harness obstacle during the cleaning task performed by the self-moving cleaning device. The determination module is used to determine the position coordinates of the wire harness in the wire harness obstacle image, and avoid the wire harness obstacle based on the position coordinates; The step of determining the position coordinates of the wire bundle in the obstacle image includes: The wire harness neighboring region of the wire harness obstacle image is determined, and a line is drawn connecting the position coordinates of each pixel in the wire harness neighboring region and the position coordinates of the wire harness obstacle. The position coordinates of the wire harness obstacle are determined based on the line. The wire harness neighboring region includes at least the envelope region of the wire harness obstacle. The envelope region is a closed region created around the boundary of the wire harness obstacle region. 14.A self-moving cleaning device control device, characterized by, The device, applied to a user terminal, includes: The receiving module is used to receive the position coordinates of the wire harness in the wire harness obstacle image determined during the execution of the cleaning task sent by the self-moving cleaning device, and to visualize the position coordinates; The step of determining the position coordinates of the wire bundle in the obstacle image includes: After identifying the obstacle and determining that it is a wire harness obstacle, an image of the wire harness obstacle is acquired; The wire harness neighboring region of the wire harness obstacle image is determined, and a line is drawn connecting the position coordinates of each pixel in the wire harness neighboring region and the position coordinates of the wire harness obstacle. The position coordinates of the wire harness obstacle are determined based on the line. The wire harness neighboring region includes at least the envelope region of the wire harness obstacle. The envelope region is a closed region created around the boundary of the wire harness obstacle region.

15. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions, which, when executed by a processor, are used to implement the method as described in any one of claims 1-11.

16. A computer program product, characterised in that, Includes a computer program that, when executed by a processor, implements the method as described in any one of claims 1-11.

Images