Component hoisting pose control method and apparatus, device, medium, and product

Abstract

A component hoisting pose control method and apparatus, a device, a medium, and a product, the method comprising: acquiring an optimal motion pose of a component to be tracked during a hoisting process from an initial position to a mounting position (S101); during the hoisting process of the component, using a pre-arranged total station and prism to acquire a current motion position of the component, and also using a pre-arranged video acquisition device to acquire video stream data of the component (S102); on the basis of the video stream data of the component, determining a current spatial orientation of the component at the current motion position (S103); and, on the basis of an optimal motion position, an optimal spatial orientation, the current motion position, and the current spatial orientation, providing an adjustment suggestion for the component (S104).

Description

Methods, devices, equipment, media, and products for controlling the lifting posture of components.

[0001] This application claims priority to Chinese Patent Application No. 202510370969.6, filed with the Chinese Patent Office on March 27, 2025, the entire contents of which are incorporated herein by reference. Technical Field

[0002] This application relates to the field of building construction technology, such as a method, device, equipment, medium, and product for controlling the hoisting posture of components. Background Technology

[0003] In hoisting operations during building construction, real-time and visualized monitoring of the movement and spatial attitude of components—from the initial lifting to their final placement—is crucial for ensuring the accuracy and safety of the operation. Related technologies typically involve using multiple total stations to observe the component's movement and spatial attitude. However, this method is unsuitable for scenarios with limited space and poor visibility. For example, due to the location of tower cranes / cranes, the hoisting operator, on-site supervisor, and project management personnel may not be able to obtain real-time information on the moving component's position and spatial attitude. Summary of the Invention

[0004] This application provides a method, device, equipment, medium, and product for controlling the hoisting posture of components, enabling the monitoring and control of component posture even in scenarios with limited space and poor visibility.

[0005] Firstly, this embodiment provides a method for controlling the hoisting posture of a component, the method comprising:

[0006] The optimal motion pose of the component to be tracked is obtained during the process of hoisting from the initial position to the installation position. The optimal motion pose includes the optimal motion position and the optimal spatial pose at the optimal motion position.

[0007] During the hoisting process of the component to be tracked, the current movement position of the component to be tracked is collected by a total station and prism pre-arranged in the relative coordinate system of the engineering environment, while video stream data of the component to be tracked is collected by a pre-arranged video acquisition device.

[0008] Based on the video stream data of the component to be tracked, determine the current spatial pose of the component at the current motion position;

[0009] Based on the optimal motion position, the optimal spatial pose, the current motion position, and the current spatial pose, adjustment suggestions are made for the component to be tracked.

[0010] Secondly, this embodiment provides a control device for the hoisting posture of a component, the device comprising:

[0011] The optimal pose determination module is configured to obtain the optimal motion pose of the component to be tracked during the process of hoisting from the initial position to the installation position. The optimal motion pose includes the optimal motion position and the optimal spatial pose at the optimal motion position.

[0012] The information acquisition module is configured to acquire the current movement position of the component to be tracked by a total station and prism pre-arranged in the relative coordinate system of the engineering environment during the hoisting process of the component to be tracked, while simultaneously acquiring video stream data of the component to be tracked by a pre-arranged video acquisition device.

[0013] The current attitude determination module is configured to determine the current spatial attitude of the component to be tracked at the current motion position based on the video stream data of the component to be tracked.

[0014] The component adjustment module is configured to provide adjustment suggestions for the component to be tracked based on the optimal motion position, the optimal spatial attitude, the current motion position, and the current spatial attitude.

[0015] Thirdly, this embodiment provides an electronic device, including:

[0016] At least one processor; and

[0017] A memory communicatively connected to the at least one processor; wherein,

[0018] The memory stores a computer program that can be executed by the at least one processor, which enables the at least one processor to perform the component hoisting posture control method described in any embodiment of this application.

[0019] Fourthly, this embodiment provides a computer-readable storage medium storing computer instructions, which are used to cause a processor to execute the component hoisting posture control method described in any embodiment of this application.

[0020] Fifthly, embodiments of this application also provide a computer program product, the computer program product including a computer program, which, when executed by a processor, implements the component hoisting posture control method as described in any embodiment of this application. Attached Figure Description

[0021] Figure 1 is a flowchart illustrating a method for controlling the hoisting posture of a component according to Embodiment 1 of this application;

[0022] Figure 2 is a flowchart illustrating another method for controlling the hoisting posture of a component according to Embodiment 2 of this application;

[0023] Figure 3 is an example diagram of the human-machine interface in the execution of a component hoisting posture control method provided in Embodiment 2 of this application;

[0024] Figure 4 is a structural schematic diagram of a component hoisting posture control device provided in Embodiment 3 of this application;

[0025] Figure 5 is a schematic diagram of the structure of an electronic device provided in Embodiment 4 of this application. Detailed Implementation

[0026] The terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion, for example, including processes, methods, systems, products, or devices that, in addition to comprising the series of steps or units shown in the embodiments of this application, may also include processes, methods, systems, products, or devices that do not explicitly list such series of steps or units, or other steps or units inherent to such processes, methods, systems, products, or devices.

[0027] Example 1

[0028] Figure 1 is a flowchart illustrating a component hoisting posture control method provided in Embodiment 1 of this application. This method is applicable to situations where the component hoisting posture is monitored and controlled during the component hoisting process. This method can be executed by a component hoisting posture control device, which can be implemented in hardware and / or software and is generally integrated into electronic equipment.

[0029] As shown in Figure 1, the component hoisting posture control method provided in this embodiment may include the following steps:

[0030] S101. Obtain the optimal motion pose of the component to be tracked during the process of hoisting it from the initial position to the installation position.

[0031] In this embodiment, the optimal motion pose includes the optimal motion position and the optimal spatial pose at the optimal motion position. The component to be tracked can be understood as a component to be hoisted and installed on the engineering site. The initial position can be understood as the position of the component before hoisting, and the installation position can be understood as the position where the component to be tracked is to be installed. To ensure that the component to be tracked can be safely and accurately hoisted from the initial position to the installation position, the optimal movement path of the component to be tracked during the entire hoisting process can be simulated in advance. The optimal movement path can be regarded as composed of multiple optimal motion positions in sequence, and the optimal spatial pose at each motion position can also be simulated. The optimal motion position represents which positions the component to be tracked should be hoisted to during the hoisting process. For example, the optimal motion position can be represented by three-dimensional coordinates. The optimal spatial pose represents the spatial pose that the component to be tracked should maintain at each optimal motion position during the hoisting process. For example, the optimal spatial pose can be represented by the rotation angles of the three axes in the relative coordinate system of the engineering environment. The optimal motion position and the optimal spatial pose at the optimal motion position constitute the optimal motion pose.

[0032] For example, modeling and motion simulation software can be used to simulate the optimal motion pose of the component to be tracked during the process of hoisting it from its initial position to its installation position. Using pre-set modeling and motion simulation software, the optimal motion positions and simulated spatial poses of the component to be tracked during the process of hoisting it from its initial position to its installation position are simulated. Scene rendering is performed on each simulated spatial pose to obtain keyframe images corresponding to the component to be tracked at each optimal motion position. Based on each keyframe image and a pre-trained component recognition model, the optimal spatial pose of the component to be tracked at each optimal motion position is obtained.

[0033] S101 can be considered a preliminary step performed before the hoisting of the component to be tracked. This step determines the optimal motion path of the component from its initial position to its installation position, as well as the spatial attitude corresponding to each position along the motion path. This provides a basis for subsequent adjustments to the attitude of the component during the hoisting process. When determining the optimal spatial attitude of the component during the entire hoisting process, not only safety factors such as avoiding collisions with obstacles are considered, but also factors such as ensuring the convenience and cost-effectiveness of the hoisting process are taken into account.

[0034] In this embodiment, before the hoisting operation begins, all equipment is set up. The system can input the component number of the component to be tracked or scan the QR code on the component via a terminal. The system will then automatically import the AI ​​training data of the component, i.e., the optimal motion pose of the component during the hoisting process from its initial position to its installation position. For example, the terminal can be augmented reality (AR) glasses, a tablet computer, a mobile phone, etc.

[0035] S102. During the hoisting process of the component to be tracked, the current movement position of the component to be tracked is collected by a total station and prism pre-arranged in the relative coordinate system of the engineering environment, while video stream data of the component to be tracked is collected by a pre-arranged video acquisition device.

[0036] The relative coordinate system of the engineering environment described in this application is a coordinate system established with fixed reference objects (such as building axis benchmark points, tower crane foundation center points, and permanent measurement control points) preset at the engineering site as the origin and the horizontal and vertical directions as coordinate axes. Its origin and coordinate axis directions remain fixed throughout the hoisting operation, providing a unified benchmark for total station measurement and video image analysis.

[0037] During the hoisting operation of the component to be tracked, it is necessary to have real-time and concrete control over the component's position and status during the initial lifting, movement, and final placement. To meet this requirement, a total station (or automated surveying robot) with automatic tracking measurement capabilities and a video acquisition device can be set up in the relative coordinate system of the engineering environment, and their spatial coordinate positions in the relative coordinate system of the engineering environment can be confirmed. A prism is installed on the component to be tracked, ensuring unobstructed line of sight between the total station with automatic tracking measurement capabilities and the prism. For example, a high-definition camera can be used for the video acquisition device, and a 360° prism can be used. The positions of the video acquisition device and the total station can be set according to the actual situation, but they must maintain line of sight with the component to be tracked. Based on the above setup, two data sources are obtained through the total station with automatic tracking measurement capabilities and the video acquisition device.

[0038] The total station must have automatic tracking measurement capabilities, or an automated measuring robot can be used. A prism is mounted on the component to be tracked. By tracking the prism on the component, the total station can obtain the real-time 3D coordinates of the component during hoisting, and this real-time 3D coordinate is used as the current position. Understandably, once the 3D coordinates of the prism mounting point are known, the overall 3D coordinates of the component can be calculated based on its external dimensions and spatial orientation. For example, assuming the total station acquires the coordinates of the lower left corner of the component, the overall spatial coordinates of the component can be determined based on its external dimensions and spatial orientation. Video acquisition equipment can collect real-time video stream data of the hoisting process. By processing this real-time video stream data, the component can be identified and located, and its real-time spatial orientation can be determined.

[0039] Based on the above description, it can be seen that the component to be tracked has two data sources during the hoisting process: the real-time current position of the component can be provided by a total station and a prism, and real-time video stream data of the component can be provided by a video acquisition device. This data can be sent to the executing entity via a communication module.

[0040] S103. Based on the video stream data of the component to be tracked, determine the current spatial pose of the component at the current motion position.

[0041] In this embodiment, video stream data of the component to be tracked during the hoisting process is acquired in real time by a video acquisition device, and the video stream data is processed. The video stream data consists of multiple frames of images. The current frame image corresponding to the component to be tracked at the current motion position is extracted from the video stream data. The component to be tracked is identified and marked in the current frame image, thereby using the spatial pose of the component to be tracked as the current spatial pose at the current motion position. Since the component to be tracked is hoisted from the initial position to the installation position, the current motion position of the component to be tracked is continuously updated during the hoisting process, and each current motion position has a corresponding current spatial pose.

[0042] For example, in this embodiment, a pre-trained component recognition model can be used to identify and label the components to be tracked in the video stream data. The component recognition model can be understood as a model used to identify and label components from an image. After extracting the current frame image at the current motion position from the video stream data, the current frame image and the original component image of the component to be tracked can be used as input data and input into the pre-trained component recognition model, thereby labeling the component to be tracked in the current frame image and determining the spatial pose of the labeled component to be tracked as the current spatial pose of the component to be tracked at the current position.

[0043] S104. Based on the optimal motion position, optimal spatial attitude, current motion position, and current spatial attitude, propose adjustment suggestions for the component to be tracked.

[0044] In this embodiment, to achieve pose control of the component to be tracked, adjustments can be made to both its position and orientation. For example, the current motion position is compared with the optimal motion position. If the difference between the two does not exceed a set error range, no movement operation suggestion is needed for the component to be tracked; if the difference exceeds the set error range, a movement operation suggestion is needed until the difference is less than the set error range. Similarly, the current spatial orientation is compared with the optimal spatial orientation. If the difference between the two does not exceed a set error range, no rotation operation suggestion is needed for the component to be tracked; if the difference exceeds the set error range, a rotation operation suggestion is needed until the difference is less than the set error range.

[0045] The process of hoisting the component to be tracked from its initial position to its installation position is a continuous process. Therefore, during the entire hoisting process, the current motion position and current spatial attitude are continuously acquired. The current motion position is compared with the corresponding optimal motion position, and the current spatial attitude is compared with the corresponding optimal spatial attitude to determine whether the position and spatial attitude of the component to be tracked need to be adjusted.

[0046] Unlike related technologies that rely on multiple total stations to observe component posture, which is unsuitable for scenarios with limited space and poor visibility, the embodiments described in this application utilize a single total station and a video acquisition device. The total station captures the real-time position of the component, while the video acquisition device captures video stream data. Processing this video stream data allows for the identification of the component's real-time spatial posture. Based on this, the component's position and spatial posture can be observed in real time, and adjustment suggestions can be made based on the optimal motion posture. This achieves real-time monitoring of the component's position and spatial posture during hoisting and provides motion suggestions, overcoming the limitations of component posture monitoring and control in scenarios with limited space and poor visibility.

[0047] Example 2

[0048] Figure 2 is a flowchart illustrating another component hoisting posture control method provided in Embodiment 2 of this application. This embodiment is an adjustment based on the above embodiment. In this embodiment, the following adjustments are made: "obtaining the optimal motion posture of the component to be tracked during the hoisting process from the initial position to the installation position", "determining the current spatial posture of the component to be tracked at the current motion position based on the video stream data of the component to be tracked", and "providing adjustment suggestions for the component to be tracked based on the optimal motion position, the optimal spatial posture, the current motion position, and the current spatial posture".

[0049] As shown in Figure 2, this embodiment 2 provides a method for controlling the hoisting posture of a component, including the following steps:

[0050] S201. Using preset modeling and motion simulation software, based on the relative position of the video acquisition device and the component to be tracked, simulate the optimal motion positions and simulated spatial postures of the component to be tracked during the process of hoisting from the initial position to the installation position.

[0051] In this embodiment, within the relative coordinate system of the engineering environment, the video acquisition device is set up at a preset fixed position. Based on the relative position of the video acquisition device and the component to be tracked, this is equivalent to setting an effect point during the modeling and motion simulation software simulation. The image that might correspond to the hoisting process of the component to be tracked is simulated from the perspective of this effect point, thereby improving the image recognition accuracy. For example, the modeling and motion simulation software is Building Information Modeling (BIM) software.

[0052] For example, based on the relative position of the video acquisition device and the component to be tracked, the path motion state of the component to be tracked from the initial position to the installation position is simulated in the modeling and motion simulation software. That is, the optimal motion path of the component to be tracked from the initial position to the installation position is simulated, which determines the optimal motion positions to which the component to be tracked moves during the hoisting process, and determines the simulated spatial attitude of the component to be tracked at each optimal motion position.

[0053] S202. Render each simulated spatial posture based on the real environment scene to obtain the keyframe image corresponding to the component to be tracked at each optimal motion position.

[0054] In this embodiment, the modeling and motion simulation software has a realistic scene rendering function. By rendering the simulated spatial posture corresponding to each optimal motion position in a realistic environment, images of the component to be tracked at each optimal motion position throughout the entire hoisting process can be obtained, and these images are recorded as keyframe images corresponding to the component to be tracked. For example, a large number of keyframe images are extracted through realistic scene rendering by BIM software.

[0055] S203. Based on each keyframe image and the pre-trained component recognition model, obtain the optimal spatial pose of the component to be tracked at each optimal motion position.

[0056] In this embodiment, the component recognition model can be understood as a model used to identify and label components from an image. The component recognition model can be obtained by training an initial neural network model based on a training sample set. After obtaining keyframe images rendered by modeling and motion simulation software, a pre-trained component recognition model can be used to identify and label the components to be tracked in each keyframe image. For example, the keyframe images and the original component images of the components to be tracked can be input into the component recognition model to segment and label the components to be tracked from the keyframe images, and determine the spatial pose of the components to be tracked as the optimal spatial pose of the components.

[0057] As some implementation methods, based on each keyframe image and a pre-trained component recognition model, the optimal spatial pose of the component to be tracked at each optimal motion position is obtained, including:

[0058] a1) For each optimal motion position, the keyframe image of the component to be tracked at the optimal motion position and the original component image of the component to be tracked are used as input data and input into the pre-trained component recognition model to mark the component to be tracked in the keyframe image.

[0059] In this embodiment, the original image of the component to be tracked refers to the actual image of the component, such as an image of the component manufactured at the factory. For example, for each optimal motion position, the keyframe image corresponding to the component at the optimal motion position and the original component image are used together as input data and input into the component recognition model. The keyframe image output by the component recognition model can then annotate the component to be tracked. The component recognition model includes a segmentation algorithm that can achieve pixel-level segmentation and extraction of component edges.

[0060] b1) Determine the spatial pose of the component to be tracked marked in the keyframe image as the optimal spatial pose of the component to be tracked at the optimal motion position.

[0061] For example, spatial pose can be represented by rotation angles of the three coordinate axes in a relative coordinate system with respect to the engineering environment. For example, after marking the component to be tracked in the keyframe image, the rotation angles of the component to be tracked relative to the three coordinate axes in the relative coordinate system with respect to the engineering environment can be determined as the optimal spatial pose of the component to be tracked at the current motion position.

[0062] The above scheme describes the steps for determining the optimal spatial pose of the component to be tracked based on the component recognition model, providing a basis for subsequent spatial pose adjustment of the component to be tracked.

[0063] S204. During the hoisting process of the component to be tracked, the current movement position of the component to be tracked is collected by a total station and prism pre-arranged in the relative coordinate system of the engineering environment, while video stream data of the component to be tracked is collected by a pre-arranged video acquisition device.

[0064] S205. Extract the current frame image corresponding to the current motion position of the component to be tracked from the video stream data of the component to be tracked.

[0065] In this embodiment, the video stream data of the component to be tracked consists of multiple frames of images in sequence. The image corresponding to the component to be tracked at the current motion position is extracted from the video stream data and recorded as the current frame image.

[0066] S206. Input the current frame image and the original component image of the component to be tracked as input data into the pre-trained component recognition model, and mark the component to be tracked in the current frame image.

[0067] For example, for each current motion position, the current frame image corresponding to the component to be tracked at the current motion position and the original component image are used together as input data and input into the component recognition model. The current frame image output by the component recognition model can then annotate the component to be tracked. The component recognition model includes a segmentation algorithm that can achieve pixel-level segmentation and extraction of component edges. For example, the annotation method can be to identify and highlight or select the component to be tracked during hoisting in the video stream data.

[0068] S207. Determine the spatial pose of the component to be tracked marked in the current frame image as the current spatial pose of the component to be tracked at the current position.

[0069] For example, spatial pose can be represented by rotation angles of three coordinate axes in a relative coordinate system with respect to the engineering environment. For example, after marking the component to be tracked in the current frame image, the rotation angles of the component to be tracked relative to the three coordinate axes in the relative coordinate system with respect to the engineering environment can be determined as the current spatial pose of the component to be tracked at the current motion position.

[0070] S208. Compare whether the difference between the current motion position and the optimal motion position is within the first set error range. If yes, do not make a movement operation suggestion for the tracking component. If no, make a movement operation suggestion for the tracking component.

[0071] After simulating the optimal movement path and optimal spatial posture using modeling and motion simulation software, these are compared in real time with the movement position and spatial posture of the component to be tracked, allowing for adjustments to the component. In this embodiment, the first set error range can be set according to actual conditions. The absolute value of the difference between the current movement position and the optimal movement position is taken. If the absolute value of this difference is within the first set error range, the current movement position of the component to be tracked is considered to conform to the planned optimal movement path, and no movement operation suggestion is needed. If the absolute value of this difference exceeds the first set error range, the current movement position of the component to be tracked is considered to not conform to the planned optimal movement path, and a movement operation suggestion is needed to ensure that the absolute value of the difference between the movement position and the optimal movement position of the component to be tracked is within the first set error range. For example, assuming the current movement position is (x, y, z) and the optimal movement position is (x, y, z1), and |z - z1| > the first set error range, the component to be tracked needs to be moved in the z-axis direction so that the coordinate difference after the movement is less than the first set error range.

[0072] S209. Compare whether the difference between the current spatial attitude and the optimal spatial attitude is within the second set error range. If yes, do not suggest a rotation operation for the component to be tracked. If no, suggest a rotation operation for the component to be tracked.

[0073] In this embodiment, the second preset error range can be set according to actual conditions. The absolute value of the difference between the current spatial attitude and the optimal spatial attitude is taken. If the absolute value of this difference is within the second preset error range, the current spatial attitude of the tracked component is considered to conform to the planned optimal spatial attitude, and no rotation operation suggestion is needed for the tracked component. If the absolute value of this difference exceeds the second preset error range, the current spatial attitude of the tracked component is considered to not conform to the planned optimal spatial attitude, and a rotation operation suggestion is needed for the tracked component so that the absolute value of the difference between the spatial attitude of the tracked component and the optimal spatial attitude is within the second preset error range. For example, suppose a frame of image shows the tracked component as a rectangle, but the image is labeled as a parallelogram, indicating that the component needs to be rotated by a certain angle to reach the correct hoisting position.

[0074] The above scheme describes the steps for obtaining the optimal motion pose, determining the current spatial attitude of the component to be tracked, and adjusting the component. Modeling and motion simulation software is used to model and render the pose of the component during the hoisting process, obtaining the optimal motion pose. Then, a total station is used to collect the real-time motion position of the component, and video stream data is acquired using video acquisition equipment to obtain the real-time spatial attitude. The optimal motion pose is compared with the real-time position and attitude of the component to achieve adjustments. This scheme enables real-time control of the component's position and spatial attitude during hoisting and provides correction suggestions, improving the accuracy and safety of the hoisting process.

[0075] As an optional embodiment of this application, based on the above embodiments, the training steps of the component recognition model in this optional embodiment include:

[0076] a2) Collect a first set number of first images from the completion of component processing to the preparation of hoisting, and a second set number of second images obtained by modeling and motion simulation software based on the simulation rendering of the component in a real environment scene. Record the first images and the second images as sample images.

[0077] In this embodiment, the first and second preset quantities can be set according to actual conditions. The first image can be understood as an image of the collected components during the historical hoisting process. The second image can be understood as an image obtained by simulating and rendering the components based on modeling and motion simulation software. Both the first and second images can be used as sample images for training the component recognition model. It is understood that the training steps of the component recognition model are pre-executed steps, and at the engineering site, the trained component recognition model can be directly used to identify components.

[0078] b2) Label each sample image to obtain a sample training set. The sample training set includes at least one sample training pair. Each sample training pair includes a sample image and the corresponding sample image with labeled components.

[0079] In this embodiment, component annotations are performed on each sample image, and components are identified and highlighted or selected by bounding boxes. A sample image and its corresponding annotated sample image constitute a sample training pair, and a large number of sample training pairs constitute a sample training set.

[0080] c2) Train the initial neural network model based on the sample training set, and use the trained initial neural network model as the component recognition model.

[0081] For example, the initial neural network model can employ a lightweight YOLOv8-Seg algorithm. The initial neural network model is trained using a sample training set. This involves inputting sample images into the initial neural network model to obtain images of component edge segmentation, comparing these images with the sample images labeled with the components, calculating a loss function, and adjusting the parameters of the initial neural network model accordingly. This process is repeated until the model can recognize the components to be tracked through extensive training, until the loss function value meets a set condition. The trained initial neural network model is then used as the component recognition model.

[0082] In this embodiment, based on the component number, the AI ​​training results data of the component, including model weights, evaluation charts, log files and data statistics, can be independently archived, retrieved and managed.

[0083] The above scheme describes the training steps of the component recognition model, which provides a foundation for the subsequent recognition and annotation of the components to be tracked.

[0084] As an optional embodiment of this application, based on the above embodiments, the method further includes:

[0085] a3) Collect safety monitoring information of the components to be tracked based on safety monitoring equipment.

[0086] In this embodiment, an IoT-based safety monitoring device is added. This device can be installed on the component to be tracked or on a building at a predetermined distance from the component during hoisting. The predetermined distance can be set according to actual conditions. The installed safety monitoring device collects relevant safety monitoring information about the component to be tracked. For example, the device could be an anemometer or an ultrasonic impact detector. The anemometer collects the wind speed around the component to be tracked or nearby buildings and records this wind speed as safety monitoring information. The ultrasonic impact detector collects the distance between the component to be tracked and the building and records this distance as safety monitoring information.

[0087] b3) Issue a safety warning in response to safety monitoring information exceeding a set safety information threshold.

[0088] In this embodiment, a safety information threshold is set. When the safety monitoring information exceeds the set threshold, a safety warning is issued because a risk is considered to exist during the hoisting of the component to be tracked. The safety information threshold can be set according to actual conditions. For example, if the safety monitoring information is wind speed, hoisting is advised to stop if the current wind speed exceeds the set safe wind speed threshold. Similarly, if the safety monitoring information is the current distance between the component to be hoisted and surrounding buildings, hoisting is advised to stop if the current distance is less than the set safe distance threshold, indicating a risk of collision between the component and the building. A safety warning is also issued if the component is less than 50 centimeters from the building and a collision is possible. For example, the safety warning prompts may include at least one of text prompts, sound prompts, and indicator light prompts.

[0089] c3) No safety warning will be issued if the safety monitoring information is less than or equal to the set safety information threshold.

[0090] In this embodiment, if the safety monitoring information is less than or equal to a set safety information threshold, it is considered that there is no risk during the hoisting of the component to be tracked, and no safety warning is issued. If the safety monitoring information is wind speed, then if the current wind speed is less than a set safe wind speed threshold, no safety warning is issued. If the safety monitoring information is the current distance between the component to be hoisted and surrounding buildings, then if the current distance is less than or equal to a set safe distance threshold, there is no risk of collision between the component and the buildings, and no safety warning is issued.

[0091] The aforementioned technology adds a safety warning function during the component hoisting process, enabling safety alerts when there are safety risks during component hoisting, thereby improving the safety of the entire hoisting process.

[0092] As an optional embodiment of this application, based on the above embodiments, the method further includes:

[0093] The component-related information of the component to be tracked is presented on the human-computer interaction interface. The component-related information includes at least one of the following: component parameter information, current frame image of the component to be tracked, current motion position, installation position, current position deviation, and safety warning information. The current position deviation is obtained by subtracting the installation position from the current motion position.

[0094] In this embodiment, during the hoisting process of the component to be tracked, an enhanced display method can be used to display the component-related information of the component in real time on the human-machine interface. For example, the component-related information can be presented on the human-machine interface of the electronic device integrated into this execution entity, such as on the display terminal configured for operators (e.g., tower crane operators) and managers. The display terminal can be various visualization terminals such as AR glasses, tablets, and mobile phones, or it can be an enhanced display system integrating artificial intelligence installed on a high-performance host in the on-site command room (e.g., in the project department's computer room).

[0095] As described above, the engineering project utilizes a digital management system for project management. By inputting the component number, information such as the manufacturer, materials used, and welding precautions can be obtained. Since the component number is pre-entered, the implementing entity can remotely read and obtain relevant parameters of the component to be tracked, such as material and installation information, through the project's digital management platform. This information is recorded as component parameter information and displayed in real-time on the augmented reality display system. Furthermore, the component identification model annotates the current frame image captured in real-time. This allows the current frame image of the annotated component to be tracked to be displayed on the human-computer interaction interface, providing the user with the visual effect of locking the component to be tracked in the captured video stream data using methods such as box selection. In actual working conditions, multiple components may be being hoisted. By inputting the component number, which corresponds to the component diagram, it can be identified which component to observe. Locking the component to be tracked allows staff to focus on it, facilitating better guidance. For example, a green box can be used to mark and lock the component to be tracked, while also displaying its relevant coordinates. If the component presented in the human-computer interaction interface is not the component to be observed, a prompt will be displayed indicating that there is no component to be observed in the current image.

[0096] In this embodiment, the total station collects the three-dimensional coordinates of the component to be tracked in real time and transmits them to the execution entity in real time via the communication module. The execution entity can then use the received three-dimensional coordinates as the current movement position of the component to be tracked and display it on the human-computer interaction interface. Simultaneously, based on the component number of the component to be tracked, the endpoint coordinates of the component to be tracked, i.e., the installation position, can be read from the modeling and motion simulation software and displayed on the human-computer interaction interface. The current position deviation can also be obtained by subtracting the installation position from the current movement position and displayed on the human-computer interaction interface. For example, high-definition video streams installed at fixed points and data from real-time tracking and measurement by an intelligent total station equipped with an automatic measurement robot are fused in the system, and dynamic highlighted outlines and measurement data are rendered in real time on various display terminals such as AR glasses.

[0097] In this embodiment, safety warning information can also be presented on the human-computer interaction interface. For example, if the current wind speed is too high, a warning message such as "Wind speed too high, hoisting recommended" can be generated; if the current distance is too close, a warning message such as "Distance too close, avoid collision" can be generated. Based on these safety warning messages, safety reminders can be displayed on the human-computer interaction interface. If there is no safety warning, this part of the content on the human-computer interaction interface will be empty.

[0098] Based on the above description, the video stream data acquired by the video acquisition equipment is displayed on the human-machine interface not only as the video stream data but also as locked and labeled components to be tracked. Furthermore, component parameter information, real-time movement position, final installation position, current distance from the endpoint, and safety warning messages are added. This enhanced display information improves the visual effect, enhances the user's visual experience, and increases the readability of the human-machine interface, allowing users to more intuitively obtain relevant component information. This enhanced display information is then fed back to operations and management personnel to assist on-site implementation until the hoisting is completed.

[0099] For example, to illustrate the layout of the human-computer interaction interface, a real-world scenario is used as an example. Figure 3 is an example diagram of the human-computer interaction interface in the execution of a component hoisting posture control method provided in Embodiment 2 of this application. As shown in Figure 3, the content presented on the human-computer interaction interface 1 includes component parameter information 11, such as component number and material; the current frame image 12 of the component to be tracked is marked and locked; the current movement position 13 is represented as (X1, Y1, Z1), where X1, Y1, and Z1 represent the current coordinates of the component to be tracked on the three coordinate axes in the relative coordinate system of the engineering environment; the installation position 15 is represented as (X2, Y2, Z2), where X2, Y2, and Z2 represent the final coordinates of the component to be tracked on the three coordinate axes in the relative coordinate system of the engineering environment; the current position deviation 14 is represented as (ΔX, ΔY, ΔZ), where ΔX, ΔY, and ΔZ represent the coordinate deviations on the three coordinate axes; and safety warning information 16 is also presented on the human-computer interaction interface.

[0100] Example 3

[0101] Figure 4 is a structural schematic diagram of a component hoisting posture control device provided in Embodiment 3 of this application. This device is applicable to situations where the hoisting posture of a component is monitored and controlled during the hoisting process. The component hoisting posture control device can be implemented in hardware and / or software and is generally integrated into an electronic device. As shown in Figure 4, the device includes: an optimal posture determination module 31, an information acquisition module 32, a current posture determination module 33, and a component adjustment module 34.

[0102] The optimal pose determination module 31 is configured to obtain the optimal motion pose of the component to be tracked during the process of hoisting from the initial position to the installation position. The optimal motion pose includes the optimal motion position and the optimal spatial pose at the optimal motion position.

[0103] The information acquisition module 32 is configured to acquire the current movement position of the component to be tracked by a total station and prism pre-arranged in the relative coordinate system of the engineering environment during the hoisting process of the component to be tracked, while simultaneously acquiring video stream data of the component to be tracked by a pre-arranged video acquisition device.

[0104] The current attitude determination module 33 is configured to determine the current spatial attitude of the component to be tracked at the current motion position based on the video stream data of the component to be tracked.

[0105] The component adjustment module 34 is configured to provide adjustment suggestions for the component to be tracked based on the optimal motion position, optimal spatial attitude, current motion position, and current spatial attitude.

[0106] Unlike related technologies that rely on multiple total stations to observe component posture, which is unsuitable for scenarios with limited space and poor visibility, the embodiments described in this application utilize a single total station and a video acquisition device. The total station captures the real-time position of the component, while the video acquisition device captures video stream data. Processing this video stream data allows for the identification of the component's real-time spatial posture. Based on this, the component's position and spatial posture can be observed in real time, and adjustment suggestions can be made based on the optimal motion posture. This achieves real-time monitoring of the component's position and spatial posture during hoisting and provides motion suggestions, overcoming the limitations of component posture monitoring and control in scenarios with limited space and poor visibility.

[0107] Optionally, the optimal pose determination module 31 includes:

[0108] The attitude simulation unit is configured to use preset modeling and motion simulation software to simulate the optimal motion positions and simulated spatial attitudes of the component being tracked during the process of hoisting the component from its initial position to its installation position, based on the relative position of the video acquisition device and the component to be tracked.

[0109] The scene rendering unit is set to render each simulated spatial pose based on the real environment scene to obtain the keyframe image corresponding to the component to be tracked at each optimal motion position.

[0110] The optimal pose determination unit is configured to obtain the optimal spatial pose of the component to be tracked at each optimal motion position based on each keyframe image and a pre-trained component recognition model.

[0111] Optionally, the optimal attitude determination unit is set as follows:

[0112] For each optimal motion position, the keyframe image of the component to be tracked at the optimal motion position and the original component image of the component to be tracked are used as input data and input into the pre-trained component recognition model to mark the component to be tracked in the keyframe image.

[0113] The spatial pose of the component to be tracked, marked in the keyframe image, is determined as the optimal spatial pose of the component to be tracked at the optimal motion position.

[0114] Optionally, the current attitude determination module 33 is configured as follows:

[0115] Extract the current frame image corresponding to the current motion position of the component to be tracked from the video stream data of the component to be tracked;

[0116] The current frame image and the original component image of the component to be tracked are used as input data and fed into the pre-trained component recognition model to mark the component to be tracked in the current frame image.

[0117] The spatial pose of the component to be tracked marked in the current frame image is determined as the current spatial pose of the component to be tracked at the current position.

[0118] Optionally, the component adjustment module 34 is configured as follows:

[0119] Compare whether the difference between the current motion position and the optimal motion position is within a first set error range; if the difference between the current motion position and the optimal motion position is within the first set error range, do not suggest a movement operation for the component to be tracked; if the difference between the current motion position and the optimal motion position is not within the first set error range, suggest a movement operation for the component to be tracked.

[0120] If the difference between the current spatial attitude and the optimal spatial attitude is within the second set error range, and the difference is within the second set error range, no rotation operation suggestion is made for the component to be tracked, and if the difference is not within the second set error range, a rotation operation suggestion is made for the component to be tracked.

[0121] Optionally, the device further includes a model training module configured to train the component recognition model in the following manner:

[0122] Collect a first set number of first images from the completion of component processing to the preparation for hoisting, and a second set number of second images obtained by modeling and motion simulation software based on the simulation rendering of the component in a real environment scene. Record the first images and the second images as sample images.

[0123] Each sample image is labeled with components to obtain a sample training set. The sample training set includes at least one sample training pair. Each sample training pair includes a sample image and the corresponding sample image after the components are labeled.

[0124] The initial neural network model is trained based on the sample training set, and the trained initial neural network model is used as the component recognition model.

[0125] Optionally, the device also includes a safety warning module, configured as follows:

[0126] Based on the safety monitoring information collected by the safety monitoring equipment, the safety monitoring equipment is installed on the component to be tracked or on a building at a set distance from the component to be tracked during the hoisting process;

[0127] A safety warning is issued in response to safety monitoring information exceeding a set safety information threshold;

[0128] No safety warning will be issued if the safety monitoring information is less than or equal to the set safety information threshold.

[0129] Optionally, the device also includes an information display module, configured as follows:

[0130] The component-related information of the component to be tracked is presented on the human-computer interaction interface. The component-related information includes at least one of the following: component parameter information, current frame image of the component to be tracked, current motion position, installation position, current position deviation, and safety warning information. The current position deviation is obtained by subtracting the installation position from the current motion position.

[0131] The component hoisting posture control device provided in this application embodiment can execute the component hoisting posture control method provided in any embodiment of this application, and has the corresponding functional modules and beneficial effects of the execution method.

[0132] Example 4

[0133] Figure 5 is a schematic diagram of the structure of an electronic device provided in Embodiment 4 of this application. The electronic device can represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices (such as helmets, glasses, watches, etc.), and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples.

[0134] As shown in Figure 5, the electronic device 40 includes at least one processor 41 and a memory, such as a read-only memory (ROM) 42 or a random access memory (RAM) 43, communicatively connected to the at least one processor 41. The memory stores computer programs executable by the at least one processor. The processor 41 can perform various appropriate actions and processes based on the computer program stored in the ROM 42 or loaded from storage unit 48 into the RAM 43. The RAM 43 can also store various programs and data required for the operation of the electronic device 40. The processor 41, ROM 42, and RAM 43 are interconnected via a bus 44. An input / output (I / O) interface 45 is also connected to the bus 44.

[0135] Multiple components in electronic device 40 are connected to I / O interface 45, including: input unit 46, such as keyboard, mouse, etc.; output unit 47, such as various types of monitors, speakers, etc.; storage unit 48, such as disk, optical disk, etc.; and communication unit 49, such as network card, modem, wireless transceiver, etc. Communication unit 49 allows electronic device 40 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0136] Processor 41 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of processor 41 include a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, digital signal processors (DSPs), and any suitable processor, controller, microcontroller, etc. Processor 41 performs the various methods and processes described above, such as methods for controlling the posture of component hoisting.

[0137] In some embodiments, the method for controlling the lifting posture of a component can be implemented as a computer program tangibly contained in a computer-readable storage medium, such as storage unit 48. In some embodiments, part or all of the computer program can be loaded and / or installed on electronic device 40 via ROM 42 and / or communication unit 49. When the computer program is loaded into RAM 43 and executed by processor 41, one or more steps of the method for controlling the lifting posture of the component described above can be performed. Alternatively, in other embodiments, processor 41 can be configured to perform the method for controlling the lifting posture of the component by any other suitable means (e.g., by means of firmware).

[0138] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard parts (ASSPs), systems on chips (SoCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0139] Computer programs used to implement the methods of this application may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, such that when executed by the processor, the computer programs cause the functions / operations specified in the flowcharts and / or block diagrams to be performed. The computer programs may be executed entirely on a machine, partially on a machine, or as a standalone software package, partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0140] In the context of this application, a computer-readable storage medium can be a tangible medium that may contain or store a computer program for use by or in conjunction with an instruction execution system, apparatus, or device. A computer-readable storage medium may include electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. Examples of machine-readable storage media may include electrical connections based on one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM), flash memory, optical fiber, compact disc-read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0141] To provide interaction with a user, the systems and techniques described herein can be implemented on an electronic device having: a display device (e.g., a cathode ray tube (CRT) or liquid crystal display (LCD) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the electronic device. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0142] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as data servers), or computing systems that include middleware components (e.g., application servers), or computing systems that include frontend components (e.g., user computers with graphical user interfaces or web browsers through which users can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication (e.g., communication networks) of any form or medium. Examples of communication networks include local area networks (LANs), wide area networks (WANs), blockchain networks, and the Internet.

[0143] A computing system can include clients and servers. Clients and servers are generally located far apart and typically interact through communication networks. The client-server relationship is created by computer programs running on the respective computers and having a client-server relationship with each other. The server can be a cloud server, also known as a cloud computing server or cloud host, which is a hosting product within the cloud computing service system. It addresses the shortcomings of traditional physical hosts and Virtual Private Server (VPS) services, such as high management difficulty and weak business scalability.

[0144] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the component hoisting posture control method provided in any embodiment of this application.

[0145] In implementing a computer program product, computer program code for performing the operations of this disclosure can be written in one or more programming languages ​​or a combination thereof. These programming languages ​​may include object-oriented programming languages ​​such as Java, Smalltalk, and C++, as well as conventional procedural programming languages ​​such as C or similar languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or it may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0146] It should be understood that the various processes shown above can be used to rearrange, add, or delete steps. For example, the multiple steps described in this application can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution of this application can be achieved, and this is not limited herein.

Claims

1. A method for controlling the hoisting posture of a component, comprising: The optimal motion pose of the component to be tracked is obtained during the process of hoisting from the initial position to the installation position. The optimal motion pose includes the optimal motion position and the optimal spatial pose at the optimal motion position. During the hoisting process of the component to be tracked, the current movement position of the component to be tracked is collected by a total station and prism pre-arranged in the relative coordinate system of the engineering environment, while video stream data of the component to be tracked is collected by a pre-arranged video acquisition device. Based on the video stream data of the component to be tracked, determine the current spatial pose of the component at the current motion position; Based on the optimal motion position, the optimal spatial pose, the current motion position, and the current spatial pose, adjustment suggestions are made for the component to be tracked.

2. The method according to claim 1, wherein, The process of obtaining the optimal motion pose of the component to be tracked during its hoisting from its initial position to its installation position includes: Using pre-set modeling and motion simulation software, based on the relative position of the video acquisition device and the component to be tracked, the optimal motion positions of the component to be tracked during the process of hoisting from the initial position to the installation position and the simulated spatial posture at each of the optimal motion positions are simulated. Render the simulated spatial poses based on the real environment scene to obtain keyframe images corresponding to the tracked components at the optimal motion positions; Based on the keyframe images and the pre-trained component recognition model, the optimal spatial pose of the component to be tracked at each optimal motion position is obtained.

3. The method according to claim 2, wherein, The step of obtaining the optimal spatial pose of the tracked component at each of the optimal motion positions based on each of the keyframe images and the pre-trained component recognition model includes: For each optimal motion position, the keyframe image corresponding to the component to be tracked at the optimal motion position and the original component image of the component to be tracked are used as input data and input into the pre-trained component recognition model, and the component to be tracked is marked in the keyframe image. The spatial pose of the component to be tracked, marked in the keyframe image, is determined as the optimal spatial pose of the component to be tracked at the optimal motion position.

4. The method according to claim 1, wherein, Determining the current spatial pose of the component to be tracked at the current motion position based on the video stream data of the component to be tracked includes: Extract the current frame image corresponding to the current motion position of the component being tracked from the video stream data of the component being tracked; The current frame image and the original component image of the component to be tracked are used as input data and input into a pre-trained component recognition model to mark the component to be tracked in the current frame image. The spatial pose of the component to be tracked marked in the current frame image is determined as the current spatial pose of the component to be tracked at the current position.

5. The method according to claim 1, wherein, The step of proposing adjustment suggestions for the tracked component based on the optimal motion position, the optimal spatial pose, the current motion position, and the current spatial pose includes: Compare whether the difference between the current motion position and the optimal motion position is within a first set error range; if the difference between the current motion position and the optimal motion position is within the first set error range, do not suggest a movement operation for the tracked component; if the difference between the current motion position and the optimal motion position is not within the first set error range, suggest a movement operation for the tracked component. The difference between the current spatial attitude and the optimal spatial attitude is compared to see if it falls within a second set error range. If the difference between the current spatial attitude and the optimal spatial attitude falls within the second set error range, no rotation operation suggestion is made for the tracked component. If the difference between the current spatial attitude and the optimal spatial attitude does not fall within the second set error range, a rotation operation suggestion is made for the tracked component.

6. The method according to claim 2, wherein, The component recognition model is trained in the following way: Collect a first set number of first images from the completion of component processing to the preparation for hoisting, and a second set number of second images obtained by the modeling and motion simulation software based on the simulation rendering of the component in a real environment scene. Record the first images and the second images as sample images. Each of the sample images is labeled with components to obtain a sample training set. The sample training set includes at least one sample training pair. Each sample training pair includes a sample image and a corresponding sample image with the labeled components. The initial neural network model is trained based on the sample training set, and the trained initial neural network model is used as the component recognition model.

7. The method according to claim 1, further comprising: Based on the safety monitoring information collected by the safety monitoring equipment, the safety monitoring equipment is installed on the component to be tracked or on a building at a set distance from the component to be tracked during the hoisting process; A security warning is issued in response to the security monitoring information exceeding a set security information threshold; If the security monitoring information is less than or equal to a set security information threshold, no security warning will be issued.

8. The method according to claim 1, further comprising: The component-related information of the component to be tracked is presented on the human-computer interaction interface. The component-related information includes at least one of the following: component parameter information, current frame image with the component to be tracked marked, current motion position, installation position, current position deviation, and safety warning information. The current position deviation is obtained by subtracting the installation position from the current motion position.

9. A control device for the hoisting position of a component, comprising: The optimal pose determination module is configured to obtain the optimal motion pose of the component to be tracked during the process of hoisting from the initial position to the installation position. The optimal motion pose includes the optimal motion position and the optimal spatial pose at the optimal motion position. The information acquisition module is configured to acquire the current movement position of the component to be tracked by a total station and prism pre-arranged in the relative coordinate system of the engineering environment during the hoisting process of the component to be tracked, while simultaneously acquiring video stream data of the component to be tracked by a pre-arranged video acquisition device. The current attitude determination module is configured to determine the current spatial attitude of the component to be tracked at the current motion position based on the video stream data of the component to be tracked. The component adjustment module is configured to provide adjustment suggestions for the component to be tracked based on the optimal motion position, the optimal spatial attitude, the current motion position, and the current spatial attitude.

10. An electronic device, comprising: At least one processor; as well as A memory communicatively connected to the at least one processor; wherein, The memory stores a computer program that can be executed by the at least one processor, the computer program being executed by the at least one processor to enable the at least one processor to perform the component hoisting posture control method as described in any one of claims 1-8.

11. A computer-readable storage medium storing computer instructions for causing a processor to execute a method for controlling the lifting posture of a component as described in any one of claims 1-8.

12. A computer program product comprising a computer program that, when executed by a processor, implements the component hoisting posture control method as described in any one of claims 1-8.

Images