Video display method and device and automated guided vehicle

Abstract

A video display method and device and an automated guided vehicle, relating to the technical field of mobile robots. The method comprises: displaying an overlay layer on a display screen of an automatic guided vehicle, wherein the overlay layer comprises a user layer located at a top layer and a video layer located at a bottom layer, a preset first position of the user layer is provided with a transparent area of a preset size, and a preset second position of the user layer is provided with at least one first control corresponding to at least one image acquisition device; in response to an operation for any first control, acquiring a video filmed by the image acquisition device corresponding to the first control for which the operation is performed; scaling the acquired video to a preset size to obtain a target video; and displaying the target video at a first position of the video layer, and displaying, by means of the display screen, a calculation result calculated by the automated guided vehicle on the basis of the acquired video. Embodiments of the present application can improve the efficiency of resolving faults at an automated guided vehicle operation site.

Description

A video display method, device and automated guided vehicle

[0001] This application claims priority to Chinese Patent Application No. 202510350312.3, filed on March 24, 2025, entitled "A Video Display Method, Apparatus and Automated Guided Vehicle", the entire contents of which are incorporated herein by reference. Technical Field

[0002] This application relates to the field of mobile robot technology, and in particular to a video display method, device, and automated guided vehicle. Background Technology

[0003] AGVs (Automated Guided Vehicles) are industrial vehicles that load goods automatically or manually, travel automatically along a set route or tow a cargo trolley to a designated location, and then load and unload goods automatically or manually. AGVs are typically equipped with multiple image acquisition devices for auxiliary positioning and recognition, such as binocular cameras and barcode readers. Binocular cameras are used for object recognition, visual navigation, and obstacle avoidance, while barcode readers are used to identify and read information such as barcodes and QR codes to achieve cargo management, AGV positioning, and navigation.

[0004] During the actual operation of automated guided vehicles, various anomalies or malfunctions often occur, such as failure to recognize barcodes, low accuracy of carrying cargo, low positioning accuracy, and abnormality when encountering obstacles with binocular cameras. In order to analyze the above anomalies or malfunctions, the relevant technologies usually use a host computer to obtain and analyze the image information collected by the binocular camera and barcode reader to determine the solution to the problem.

[0005] However, this method relies on a host computer, resulting in low efficiency in resolving malfunctions at the automated guided vehicle (AGV) operation site. Summary of the Invention

[0006] The purpose of this application is to provide a video display method, device, and automated guided vehicle (AGV) to improve the efficiency of troubleshooting at the AGV's operational site. The specific technical solution is as follows:

[0007] A first aspect of this application provides a video display method applied to an automated guided vehicle, the method comprising:

[0008] An overlay layer is displayed on the screen of the automated guided vehicle. The overlay layer includes a user layer at the top and a video layer at the bottom. A transparent area of ​​a preset size is set at a preset first position of the user layer, and at least one first control is set at a preset second position of the user layer. The first control corresponds to at least one image acquisition device of the automated guided vehicle.

[0009] In response to an operation on any of the first controls, acquire the video captured by the image acquisition device corresponding to the first control targeted by the operation;

[0010] The acquired video is scaled to the preset size to obtain the target video;

[0011] The target video is displayed at the first position of the video layer, and the calculation result obtained by the automated guided vehicle based on the acquired video is displayed on the screen.

[0012] In one possible implementation, the image acquisition device of the automated guided vehicle includes an upper barcode reader camera;

[0013] The step of displaying the calculation results obtained by the automated guided vehicle based on the acquired video on the display screen includes:

[0014] If the first control targeted by the operation is the first control corresponding to the upper barcode reader camera, then the display screen shows the position of the cargo code relative to the automated guided vehicle and the barcode reading result of the cargo code captured by the upper barcode reader camera.

[0015] In one possible implementation, the image acquisition device of the automated guided vehicle includes a lower barcode scanner;

[0016] The step of displaying the calculation results obtained by the automated guided vehicle based on the acquired video on the display screen includes:

[0017] If the first control targeted by the operation is the first control corresponding to the lower barcode reader camera, and the navigation method of the automated guided vehicle is QR code navigation, then the display screen will show the pose of the ground code captured by the lower barcode reader relative to the automated guided vehicle, the barcode reading result, and the pose of the automated guided vehicle in its own operating map.

[0018] In one possible implementation, the image acquisition device of the automated guided vehicle includes a lower barcode scanner;

[0019] The step of displaying the calculation results obtained by the automated guided vehicle based on the acquired video on the display screen includes:

[0020] If the first control targeted by the operation is the first control corresponding to the lower barcode reader camera, and the navigation mode of the automated guided vehicle is texture navigation, then the display screen shows the position and pose of the automated guided vehicle in its own running map, calculated based on the ground texture captured by the lower barcode reader camera.

[0021] In one possible implementation, the image acquisition device of the automated guided vehicle includes a depth camera;

[0022] The step of displaying the target video at the first position of the video layer and displaying the calculation results obtained by the automated guided vehicle based on the acquired video on the display screen includes:

[0023] A second control is displayed in the user layer;

[0024] In response to an operation on the second control, the target video is displayed at the first position of the video layer, or a calculation result calculated based on the image captured by the depth camera is displayed at the first position of the video layer.

[0025] In one possible implementation, the depth camera is a binocular camera, and the second control includes a left-eye control, a right-eye control, and a color image control;

[0026] The step of displaying the target video at the first position of the video layer in response to an operation on the second control includes:

[0027] In response to an operation on the left eye control, a video captured by the left eye of the binocular camera at a preset size is displayed at the first position of the video layer;

[0028] In response to an operation on the right eye control, a video captured by the right eye of the binocular camera at a preset size is displayed at the first position of the video layer;

[0029] In response to an operation on the color image control, a color image video of the preset size captured by the binocular camera is displayed at the first position of the video layer.

[0030] In one possible implementation, the second control includes an obstacle map control and a depth map control;

[0031] The step of displaying the target video at the first position of the video layer in response to an operation on the second control includes:

[0032] In response to an operation on the obstacle map control, an obstacle map calculated based on the video captured by the depth camera is displayed at the first position of the video layer. The obstacle map is marked with detected obstacles located within a preset obstacle encounter range of the AGV.

[0033] In response to an operation on the depth map control, a depth map calculated based on the video captured by the depth camera is displayed at the first position of the video layer, the depth map marking all detected obstacles;

[0034] In one possible implementation, the method further includes:

[0035] In response to an operation on the second control, the display screen shows marking information indicating whether the automated guided vehicle has encountered an obstacle.

[0036] A second aspect of this application provides an automated guided vehicle, the automated guided vehicle including a processor, a display screen and at least one image acquisition device;

[0037] The image acquisition device is used to capture video and send it to the processor;

[0038] The processor is configured to, in response to an operation on any of the first controls, acquire a video captured by an image acquisition device corresponding to the first control targeted by the operation; scale the acquired video to the preset size to obtain a target video; and send the target video and a calculation result calculated based on the acquired video to the display screen.

[0039] The display screen is used to display an overlay layer, wherein the overlay layer includes a user layer at the top and a video layer at the bottom. A transparent area of ​​a preset size is provided at a preset first position of the user layer, and at least one first control is provided at a preset second position of the user layer. The first control corresponds to at least one image acquisition device of the automated guided vehicle. The target video is displayed at the first position of the video layer. The calculation result is displayed.

[0040] A third aspect of this application provides a video display device for use in an automated guided vehicle, the device comprising:

[0041] A layer display module is used to display an overlay layer on the display screen of the automated guided vehicle. The overlay layer includes a user layer at the top layer and a video layer at the bottom layer. A transparent area of ​​a preset size is provided at a preset first position of the user layer, and at least one first control is provided at a preset second position of the user layer. The first control corresponds to at least one image acquisition device of the automated guided vehicle.

[0042] The video acquisition module is used to acquire the video captured by the image acquisition device corresponding to the first control targeted by the operation in response to any operation of the first control.

[0043] The video scaling module is used to scale the acquired video to the preset size to obtain the target video;

[0044] The video display module is used to display the target video at the first position of the video layer, and to display the calculation results obtained by the automated guided vehicle based on the acquired video through the display screen.

[0045] In one possible implementation, the image acquisition device of the automated guided vehicle includes an upper barcode reader camera;

[0046] The step of displaying the calculation results obtained by the automated guided vehicle based on the acquired video on the display screen includes:

[0047] If the first control targeted by the operation is the first control corresponding to the upper barcode reader camera, then the display screen shows the position of the cargo code relative to the automated guided vehicle and the barcode reading result of the cargo code captured by the upper barcode reader camera.

[0048] In one possible implementation, the image acquisition device of the automated guided vehicle includes a lower barcode scanner;

[0049] The video display module is specifically used for:

[0050] If the first control targeted by the operation is the first control corresponding to the lower barcode reader camera, and the navigation method of the automated guided vehicle is QR code navigation, then the display screen will show the pose of the ground code captured by the lower barcode reader relative to the automated guided vehicle, the barcode reading result, and the pose of the automated guided vehicle in its own operating map.

[0051] In one possible implementation, the image acquisition device of the automated guided vehicle includes a lower barcode scanner;

[0052] The video display module is specifically used for:

[0053] If the first control targeted by the operation is the first control corresponding to the lower barcode reader camera, and the navigation mode of the automated guided vehicle is texture navigation, then the display screen shows the position and pose of the automated guided vehicle in its own running map, calculated based on the ground texture captured by the lower barcode reader camera.

[0054] In one possible implementation, the image acquisition device of the automated guided vehicle includes a depth camera;

[0055] The video display module is specifically used for:

[0056] A second control is displayed in the user layer;

[0057] In response to an operation on the second control, the target video is displayed at the first position of the video layer, or a calculation result calculated based on the image captured by the depth camera is displayed at the first position of the video layer.

[0058] In one possible implementation, the depth camera is a binocular camera, and the second control includes a left-eye control, a right-eye control, and a color image control;

[0059] The step of displaying the target video at the first position of the video layer in response to an operation on the second control includes:

[0060] In response to an operation on the left eye control, a video captured by the left eye of the binocular camera at a preset size is displayed at the first position of the video layer;

[0061] In response to an operation on the right eye control, a video captured by the right eye of the binocular camera at a preset size is displayed at the first position of the video layer;

[0062] In response to an operation on the color image control, a color image video of the preset size captured by the binocular camera is displayed at the first position of the video layer.

[0063] In one possible implementation, the second control includes an obstacle map control and a depth map control;

[0064] The step of displaying the target video at the first position of the video layer in response to an operation on the second control includes:

[0065] In response to an operation on the obstacle map control, an obstacle map calculated based on the video captured by the depth camera is displayed at the first position of the video layer. The obstacle map is marked with detected obstacles located within a preset obstacle encounter range of the automated guided vehicle.

[0066] In response to an operation on the depth map control, a depth map calculated based on the video captured by the depth camera is displayed at the first position of the video layer, the depth map indicating all detected obstacles.

[0067] In one possible implementation, the device further includes:

[0068] The information display module is used to display, in response to an operation on the second control, annotation information indicating whether the automated guided vehicle has encountered an obstacle via the display screen.

[0069] A fourth aspect of this application also provides an electronic device, comprising:

[0070] Memory, used to store computer programs;

[0071] The processor, when executing a program stored in memory, implements any of the video display methods described above.

[0072] A fifth aspect of this application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements any of the video display methods described above.

[0073] This application also provides a computer program product containing instructions that, when run on a computer, cause the computer to execute any of the video display methods described above.

[0074] Beneficial effects of the embodiments in this application:

[0075] This application provides a video display method, apparatus, and automated guided vehicle (AGV). The AAV displays an overlay layer on its screen. This overlay layer includes a top-level user layer and a bottom-level video layer. A preset first position of the user layer has a transparent area of ​​a preset size, and a preset second position of the user layer has at least one first control. Each first control corresponds to at least one image acquisition device of the AAV. In response to an operation on any first control, the AAV acquires video captured by the at least one image acquisition device corresponding to that control, scales the acquired video to a preset size to obtain a target video, and finally displays the target video at the first position of the video layer. The display screen also shows the result calculated by the AAV based on the acquired video. The method in this application embodiment allows users to browse the target video displayed at the first position of the video layer through the transparent area at the first position of the user layer and at least one first control corresponding to at least one image acquisition device. Furthermore, the display screen shows the calculation results obtained by the automated guided vehicle (AGV) based on the acquired video. Compared to the prior art, which primarily involves connecting a host computer to the AGV to acquire the video captured by the AGV's image acquisition device and then analyzing the cause of the fault based on the video displayed on the host computer, this application allows users to view the calculation results obtained by the AGV based on the video captured by each image acquisition device while simultaneously viewing the video on the AGV's display screen. No host computer intervention is required; the cause of the abnormality or fault can be analyzed directly based on the video and calculation results displayed on the AGV's display screen, improving the efficiency of troubleshooting at the AGV's work site.

[0076] Of course, implementing any product or method of this application does not necessarily require achieving all of the advantages described above at the same time. Attached Figure Description

[0077] The accompanying drawings, which are provided to further illustrate this application and form part of this application, illustrate exemplary embodiments of this application and are used to explain this application, but do not constitute an undue limitation of this application.

[0078] Figure 1 is an example of the image information displayed by the barcode reader camera in the host computer.

[0079] Figure 2 is an example of image information displayed in a host computer using a stereo camera.

[0080] Figure 3 is a schematic diagram of the video display method provided in an embodiment of this application;

[0081] Figure 4 is a schematic diagram of the overlay layer provided in an embodiment of this application;

[0082] Figure 5a is a first interface display example diagram of the AGV display screen provided in the embodiment of this application;

[0083] Figure 5b is a second interface display example diagram of the AGV display screen provided in the embodiment of this application;

[0084] Figure 5c is a third interface display example of the AGV display screen provided in the embodiment of this application;

[0085] Figure 5d is a fourth interface display example of the AGV display screen provided in the embodiment of this application;

[0086] Figure 5e is a fifth interface display example diagram of the AGV display screen provided in the embodiments of this application;

[0087] Figure 6 is an example diagram showing the recognition result of the barcode reader camera provided in the embodiment of this application;

[0088] Figure 7 is a first example of the display of the recognition result of the code reader provided in the embodiment of this application;

[0089] Figure 8 is a second example of the display of the recognition result of the code reader provided in the embodiment of this application;

[0090] Figure 9a is a first example of a display of depth camera recognition results provided in an embodiment of this application;

[0091] Figure 9b is a second example of the depth camera recognition results provided in an embodiment of this application;

[0092] Figure 9c is a third example of the depth camera recognition results provided in the embodiments of this application;

[0093] Figure 9d is a fourth example of the depth camera recognition results provided in the embodiments of this application;

[0094] Figure 9e is a fifth example of the depth camera recognition results provided in the embodiments of this application;

[0095] Figure 9f is a sixth example of the depth camera recognition results provided in the embodiments of this application;

[0096] Figure 10 is a basic framework diagram of the display software of the display screen of the automated guided vehicle provided in the embodiment of this application;

[0097] Figure 11 is a schematic diagram of the structure of the video display device provided in an embodiment of this application;

[0098] Figure 12 is a schematic diagram of the structure of the electronic device provided in the embodiment of this application. Detailed Implementation

[0099] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided with reference to the accompanying drawings and embodiments. Obviously, the described embodiments are merely some embodiments of this application, and not all embodiments. All other embodiments obtained by those skilled in the art based on the embodiments in this application are within the scope of protection of this application.

[0100] First, the technical terms used in the embodiments of this application will be explained:

[0101] Depth camera: A device capable of capturing distance information between an object and the camera, including structured light depth cameras and binocular cameras. Structured light depth cameras use light emitted from the camera to receive and process the time difference or phase difference of the returning light, thereby calculating distance information and generating a depth image. Binocular cameras are imaging devices that mimic the principle of human binocular vision. They consist of two cameras that acquire two images of the object from different positions. The three-dimensional geometric information of the object is then obtained by calculating the positional deviation between corresponding points in these two images. The two cameras of a binocular camera are referred to as the left and right eyes. Binocular cameras are typically used on AGVs for visual navigation or obstacle avoidance.

[0102] Barcode reader: Used to identify textures or QR codes for texture positioning (texture navigation) or QR code positioning (QR code navigation). Typically, two barcode readers are installed on an AGV: one for identifying ground codes or textures, and the other for identifying cargo codes.

[0103] During AGV cargo handling operations, the differences between the images captured by the left and right eyes of the binocular cameras installed on the AGV are typically used to calculate the three-dimensional geometric information of objects in the images, thereby achieving navigation, positioning, obstacle detection, and obstacle avoidance. One barcode reader camera on the AGV can obtain location information by scanning preset QR codes or barcodes, thus guiding the AGV along a predetermined route; simultaneously, another barcode reader camera can scan preset barcodes or QR codes on the goods to obtain information such as type, quantity, and destination. However, in actual AGV operation, malfunctions may occur such as barcode recognition failure, low cargo carrying accuracy, low positioning accuracy, and binocular obstacle detection anomalies.

[0104] After the aforementioned malfunctions occur, manual intervention is required for analysis and troubleshooting. Current technology typically involves connecting a host computer to the AGV to view image information captured by the various cameras installed on the AGV. Furthermore, analysis on the host computer usually involves extracting a single frame from the video for display and analysis. For example, if a code reading error occurs at the AGV's operating site, current technology involves connecting a host computer to the AGV, extracting a single frame from the video captured by the AGV's code reading camera, and then viewing that single frame on the host computer.

[0105] Although the host computer can select the camera type, such as a barcode reader camera or a binocular camera, when a barcode reader camera is selected, the host computer can only display the code captured by the barcode reader camera, such as a QR code image. Users need to analyze the QR code reading results; however, in existing technologies, the host computer cannot display information such as the QR code value. For example, Figure 1 shows an example of the host computer displaying image information from a barcode reader camera. Suppose a barcode recognition error occurs at the AGV operation site; the host computer only displays the QR code image captured by the barcode reader camera and cannot display information such as the QR code value or the QR code's pose relative to the AGV. Users cannot determine whether the barcode recognition error is caused by the QR code's pose relative to the AGV.

[0106] When the camera type is selected as a stereo camera, Figure 2 shows an example of the image information displayed on the host computer. The host computer only displays the image currently captured by the stereo camera and cannot show obstacle information or other information determined based on that image. However, these issues all affect the efficiency of personnel in resolving AGV operation site malfunctions.

[0107] To improve the efficiency of troubleshooting AGV operation site faults, a first aspect of this application provides a video display method applied to AGVs. Figure 3 shows a schematic diagram of the first type of video display method provided by this application. The method includes the following steps:

[0108] Step S10: Display the overlay layer on the AGV's display screen;

[0109] The overlay layer includes a user layer at the top and a video layer at the bottom. The user layer has a transparent area of ​​a preset size at a preset first position and at least one first control at a preset second position. The first control corresponds to at least one image acquisition device of the automated guided vehicle.

[0110] Step S20: In response to an operation on any first control, acquire the video captured by the image acquisition device corresponding to the first control targeted by the operation;

[0111] Step S30: Scale the acquired video to a preset size to obtain the target video;

[0112] Step S40: Display the target video at the first position of the video layer, and display the calculation results obtained by the AGV based on the acquired video on the display screen.

[0113] The method of this application embodiment includes a user layer at the top and a video layer at the bottom of the overlay layer. The user layer has a preset first position with a transparent area of ​​a preset size, and at least one first control corresponding to at least one image acquisition device is provided at the preset first position. Therefore, the user can select the image acquisition device through the first control and browse the target video displayed at the first position of the video layer through the transparent area at the first position of the user layer in the overlay layer. Furthermore, the display screen will also show the calculation results obtained by the AGV based on the acquired video. Compared to the prior art, which mainly involves connecting a host computer to the AGV to obtain the video captured by the AGV's image acquisition device, and then combining the video displayed by the host computer to analyze the cause of the fault, in this application, the user can view the calculation results obtained by the AGV based on the video captured by each image acquisition device while viewing the video on the AGV's display screen. No host computer intervention is required; the cause of the abnormality or fault can be analyzed directly based on the video and calculation results displayed on the AGV's display screen, improving the efficiency of troubleshooting at the AGV's work site.

[0114] In addition, it also makes troubleshooting easier and faster when maintaining AGVs later on, reducing maintenance difficulty and workload.

[0115] Taking the aforementioned issue of code recognition errors occurring at the AGV operation site as an example, assuming the AGV in this application is equipped with a code reader camera and a binocular camera, an overlay layer with a user layer at the top and a video layer at the bottom will be displayed on the AGV's screen. A transparent area of ​​a preset size will be set at a preset position on the user layer. When personnel operate the first control on the AGV's screen (hereinafter referred to as the screen) for the code reader camera, the AGV will first acquire the video captured by the code reader camera, scale it to a preset size (e.g., 400×200), and then send it to the preset position of the video layer in the overlay layer for display. Furthermore, the AGV will also display the result of the code reading by the code reader camera (e.g., the code value) on the screen. This allows personnel to view the video and QR code value captured by the code reader camera on the screen, analyze the cause of the code reading error by combining the video and QR code, and take appropriate action.

[0116] The following is a detailed explanation of steps S10 to S40:

[0117] In step S10 above, the user layer refers to the layer used to construct user interface elements when designing the display software for the screen, and the video layer refers to a layer obtained by embedding a video stream in a separate layer. It is understood that the user layer and the video layer have the same size, and are the same size as the display screen.

[0118] The overlay layer is obtained by pre-overlaying the user layer and the video layer. Figure 4 shows a schematic diagram of the overlay layer provided in this embodiment. The user layer is located at the top layer, and the video layer is located at the bottom layer. A transparent area of ​​a preset size is set on the user layer. Since this transparent area is transparent, after the user layer and the video layer are overlaid to obtain the overlay layer, and a video stream is embedded on the video layer, the user can view the video of the video layer through the transparent area on the video layer. Therefore, the transparent area set on the user layer is actually a video preview area for viewing the video. The shape of the transparent area is usually a regular shape, such as a rectangle or a circle.

[0119] In order to enable interactive functions of the user interface through the user layer, such as allowing users to view videos captured by different image acquisition devices through the user interface, a first control is set in the preset second position of the user layer.

[0120] The first preset position is used to represent a region, and the second preset position can represent one region or multiple regions. It is understood that the regions represented by the first preset position and the regions represented by the second preset position do not overlap.

[0121] The areas represented by the preset first position and the preset second position are preset sizes, and the sizes of the areas represented by the preset first position and the preset second position are different. To facilitate user viewing of the video, the size of the area represented by the preset first position is larger than the size of the area represented by the preset second position.

[0122] The preset first position and preset second position can be represented by multiple coordinate points, or by coordinate points and a radius. For example, if A(10,10), B(10,60), C(60,10), and D(60,60) represent the preset first position, then the area represented by the preset first position is a rectangle with a size of 50×50, and a transparent area with a preset size of 50×50 is generated on the user layer. In another example, if E(20,20) and r=10 represent the preset first position, then the area represented by the preset first position is a circle with its center at (20,20) and a radius of 10, and a transparent area with a preset size of r=10 is generated on the user layer with coordinates (20,20) as the center. If P(1,1), B(1,6), C(6,1), and D(6,6) represent the preset second position, then the area represented by the preset second position is a rectangle with a size of 5×5, and a first control is set within this rectangular area.

[0123] It is understood that the transparent area on the user layer can be obtained by processing with existing image processing software, or by setting the transparency of a preset first position on the user layer, or by using other image processing methods. This application embodiment does not limit this.

[0124] Image acquisition devices are installed on AGVs for navigation, obstacle avoidance, cargo identification and positioning, etc. Depending on their function, image acquisition devices are classified into different types, such as depth cameras, upper barcode readers, lower barcode readers, etc. The image acquisition devices in the embodiments of this application include, but are not limited to, depth cameras and barcode readers.

[0125] When a preset second position represents a region, this region is equipped with a first control. This first control can correspond to multiple image acquisition devices or to a single image acquisition device. The image acquisition devices corresponding to the first control can be all the image acquisition devices on the AGV, or it can be one or some of the image acquisition devices on the AGV.

[0126] For example, assuming the AGV has 5 image acquisition devices, 5 first controls can be set in the user layer, each corresponding to one image acquisition device; alternatively, 1 first control can be set in the user layer, corresponding to all 5 image acquisition devices; or 4 first controls can be set in the user layer, with 2 image acquisition devices corresponding to the same first control, and the remaining 3 image acquisition devices each corresponding to a first control. If the user only needs to view images captured by 3 of the 5 image acquisition devices, 3 first controls can be set in the user layer, each corresponding to one of the 3 image acquisition devices; alternatively, 1 first control can be set in the user layer, corresponding to all 3 image acquisition devices; or 2 first controls can be set in the user layer, with 2 image acquisition devices corresponding to the same first control, and the remaining image acquisition device corresponding to another first control.

[0127] To facilitate users in accurately selecting image acquisition devices, each first control corresponds to one image acquisition device. Users can view the videos acquired by different image acquisition devices in the video preview area by selecting the first control corresponding to different image acquisition devices.

[0128] In this embodiment, the first control can be fully displayed on the screen. For example, as shown in Figure 5a, which is a first interface display example diagram of the AGV screen provided in this application embodiment, the first controls corresponding to the AGV image acquisition device are all displayed on the screen. The display interface has a video preview area 10, and the screen is also provided with different upper barcode reader control 201, lower barcode reader control 202 and binocular camera control 203 respectively for the upper barcode reader camera, lower barcode reader camera and depth camera.

[0129] The first control can also be displayed on the screen through a hidden menu, such as a drop-down list. For example, Figure 5b shows a second interface example of the AGV display screen provided in this application embodiment. The display screen has a video preview area 10 and controls 20 for selecting different image acquisition devices. Users can expand the first controls corresponding to different image acquisition devices through controls 20. As shown in Figure 5c, after clicking control 20, the user will expand the upper barcode reader control 201, lower barcode reader control 202, and depth camera control 203, respectively. It can be understood that the camera name displayed in control 20 indicates the name of the currently selected camera. For example, in Figure 5b, control 20 displaying "Depth Camera 1" indicates that the currently selected camera is depth camera 1.

[0130] If the user selects the upper barcode reader camera control 201 on the display screen, the video captured by the upper barcode reader camera will be displayed in the video preview display area 10; if the user selects the lower barcode reader camera control 202, the video captured by the lower barcode reader camera will be displayed in the video preview display area 10; if the user clicks the depth camera control 203, the video captured by the depth camera will be displayed in the video preview area 10.

[0131] The depth camera can be a stereo camera, a structured light depth camera, or a depth camera using other technologies; this application does not limit this.

[0132] To detect obstacles in all directions, AGVs are typically equipped with different depth cameras for different directions. When the first control is displayed as a drop-down list on the interface, clicking control 20 expands the first controls corresponding to all image acquisition devices of the AGV. As shown in Figure 5d, assuming the AGV's image acquisition devices include an upper barcode reader camera, a lower barcode reader camera, depth camera 1, depth camera 2, and depth camera 3, clicking control 20 will display the upper barcode reader control 201, lower barcode reader control 202, depth camera 1 control 203, depth camera 2 control 204, and depth camera 3 control 205, respectively.

[0133] To avoid excessive obstruction of the video image by the controls corresponding to the image acquisition device, in another possible implementation, controls corresponding to the same type of image acquisition device can be displayed through a secondary menu. For example, as shown in Figure 5e, assume the AGV's image acquisition device includes an upper barcode reader camera, a lower barcode reader camera, and five depth cameras. After the user clicks control 20, the upper barcode reader camera control 201 corresponding to the upper barcode reader camera, the lower barcode reader camera control 202 corresponding to the lower barcode reader camera, and a total depth camera control 203 corresponding to all depth cameras are first displayed. After the user clicks the total depth camera control 203, the controls for depth cameras 1, 2, 3, 4, and 5 are expanded to include depth camera 1 controls 2031, 2032, 2033, 2034, and 2035, respectively.

[0134] Understandably, if there are many controls corresponding to the AGV's image acquisition devices, and they cannot all be displayed on the same page, a scroll bar can be used to facilitate users viewing all the AGV's image acquisition devices. In one possible implementation, after the user selects the control corresponding to the image acquisition device, only the number corresponding to the selected image acquisition device is displayed in the area where control 20 is located.

[0135] Since the size of the videos captured by each image acquisition device may differ from the size of the video preview area in the overlay layer, if the video captured by the image acquisition device is directly displayed in the first position of the video layer, the video preview area of ​​the overlay layer may only display a portion of the video or the video may be too small to be clearly seen, resulting in the user being unable to obtain useful information from the video preview area. Therefore, after obtaining the video captured by the image acquisition device corresponding to the first control targeted by the user's operation in step S20, step S30 requires adaptively scaling the acquired video proportionally according to the size of the video preview area to ensure that the acquired video will not suffer from image distortion due to scaling when displayed on the AGV's screen.

[0136] For example, suppose that in step S20 above, the original video captured by the image acquisition device is obtained, and the size of the original video is 2000×1000. However, the preset size of the transparent area set on the user layer in step S10 above is 500×500, that is, the size of the video preview area is 500×500. Therefore, the size of the original video can be reduced from 2000×1000 to 1000×500 or 500×250. As another example, suppose that in step S20 above, the original video captured by the image acquisition device is obtained, and the size of the original video is 500×500. However, the preset size of the transparent area set on the user layer in step S10 above is 2000×1000, that is, the size of the video preview area is 2000×1000. Therefore, the size of the original video needs to be enlarged from 500×500 to 1000×1000.

[0137] In one possible embodiment, the format of the video captured by the image acquisition device may be different from the format supported by the AGV, or the video captured by the image acquisition device may have a large memory size, resulting in a long loading time. In this case, when scaling the acquired video, operations such as rotation, compression, and format conversion can also be performed on the video so that the target video can be played on the AGV's display screen, improving the user experience.

[0138] In step S40, displaying the target video at the first position of the video layer means displaying the target video at the first position of the overlay video layer, not at the first position of the original video layer. This means that the transparent area of ​​the user layer is directly above the first position of the video layer. After displaying the target video at the first position of the video layer, the transparent area of ​​the user layer allows the target video to be fully displayed in the video preview area of ​​the overlay layer.

[0139] It is understood that the AGV in this embodiment is equipped with various image acquisition devices. The AGV calculates its pose, ground code value, cargo code value, relative deviation between the AGV and the ground code, angular position deviation, and obstacle position by combining the videos captured by these devices. This enables AGV navigation, obstacle avoidance, cargo identification, and positioning. To facilitate user analysis of anomalies or malfunctions, when the user views the videos captured by the image acquisition devices on the AGV's display screen, the calculation results obtained by the AGV based on the acquired videos are also displayed on the screen.

[0140] For example, if the first control in step S20 is a depth camera control, then the video obtained in step S30 is the video captured by the depth camera. The AGV can then calculate the following results based on its travel route and the video captured by the depth camera: the distance of each object in the video, the object category, and whether there are obstacles on the travel route. If the first control in step S20 is an upper code reader control, then the video obtained in step S30 is the video captured by the upper code reader camera. The AGV can then calculate the following results based on the video captured by the upper code reader camera: the pose of the AGV, the relative deviation between the AGV and the ground code, etc.

[0141] As can be seen, the categories of calculation results differ depending on the type of image device. The video display methods provided in this application will be described below according to the different types of image devices:

[0142] In one possible implementation, the image acquisition device for the AGV is an upper barcode reader camera. The upper barcode reader camera is used to read the barcode. If the first control in the aforementioned step S20 is the first control corresponding to the upper barcode reader camera, then the calculation result displayed on the screen is the pose of the barcode relative to the AGV captured by the upper barcode reader camera and the barcode reading result.

[0143] The reading result of the code is the code value of the code. The position of the code relative to the AGV can be represented by the deviation information of the code relative to the AGV, the turntable angle information, the turntable height information, etc.

[0144] For example, Figure 6 shows an example of the display of the recognition result of the upper barcode reader provided in this application embodiment. Since the upper barcode reader is selected, it can be seen that the first control in the above step S20 is the upper barcode reader control. The video preview area 10 displays a video containing a QR code image for representing the goods code captured by the upper barcode reader. The AGV calculates the code value of the goods code, the angular position deviation of the goods code relative to the AGV, the turntable angle (the angle of rotation of the AGV's turntable relative to the AGV body), and the turntable height (the lifting height of the AGV's turntable relative to the vehicle body) based on the video. The display screen shows that the angular position deviation is 3°, the turntable angle is 90°, the turntable height is 300mm, and the barcode reading result is 10090. The barcode reading result is the code value obtained by recognizing the QR code in the video captured by the upper barcode reader.

[0145] Using the method of this application embodiment, users can not only view the video captured by the barcode reader camera on the AGV's display screen, but also view the position and orientation of the barcode relative to the AGV and the barcode reading result captured by the barcode reader camera. In this way, users can analyze the cause of abnormalities without checking the host computer, which improves the efficiency of troubleshooting at the AGV operation site.

[0146] In one possible implementation, the image acquisition device for the AGV is a bottom-mounted barcode camera. The bottom-mounted barcode camera is used for navigation, and can employ either QR code navigation or texture navigation.

[0147] When navigation is performed using QR code navigation, if the first control in the aforementioned step S20 is the first control corresponding to the lower code reader camera, then the calculation results displayed on the screen are the pose of the AGV relative to the ground code captured by the lower code reader camera, the code reading result of the ground code, and the pose of the AGV.

[0148] For example, Figure 7 shows a first example of the display of the recognition result of the bottom-reading camera provided in this application embodiment. Since the bottom-reading control is selected, it can be seen that the first control in the above step S20 is the bottom-reading control. The video preview area 10 displays a video containing a QR code image for representing the ground code, captured by the bottom-reading camera. The AGV calculates the code value of the ground code, the AGV's pose (i.e., the vehicle pose) and the deviation of the AGV relative to the ground code (i.e., the relative deviation) based on the video. The vehicle pose is displayed on the display screen as (12555, 21005), the relative deviation as (5, 4, 0), and the code reading result of the ground code as 0122502100.

[0149] Using the method of this application embodiment, when an AGV using QR code navigation malfunctions or fails, the user can not only view the video captured by the QR code reader camera on the AGV's display screen, but also view the position of the ground code relative to the AGV captured by the QR code reader camera, the reading result of the ground code, and the position of the AGV in its own running map. In this way, the user can analyze the cause of the malfunction without having to check the host computer, thus improving the efficiency of troubleshooting at the AGV operation site.

[0150] When using texture navigation, if the first control in step S20 is the first control corresponding to the lower barcode reader camera, the calculation result displayed on the screen is the pose of the AGV in its own running map.

[0151] For example, Figure 8 shows a second example of the display of the recognition result of the bottom-reading camera provided in the embodiment of this application. Since the bottom-reading control is selected, it can be seen that the first control in step S20 above is the bottom-reading control. The video containing real-time texture information captured by the bottom-reading camera is displayed in the video preview area 10. The AGV calculates the AGV pose (i.e., the car pose) based on the video and displays the car pose on the display screen: (12555, 21005, 0).

[0152] Using the method of this application embodiment, when an AGV using texture navigation malfunctions or fails, the user can not only view the video captured by the barcode reader camera on the AGV's display screen, but also view the AGV's pose. This allows the user to determine whether there is an abnormality in the AGV's pose by comparing the actual pose of the AGV on-site with the pose of the AGV on the display screen, without needing to connect to a host computer, thus improving the efficiency of troubleshooting at the AGV's work site.

[0153] In one possible implementation, the image acquisition device for the AGV is a depth camera. The depth camera is used for obstacle recognition, obstacle avoidance, and localization. The following explanation uses a binocular camera as an example. It is understood that the left and right eyes of the binocular camera can acquire images separately, typically RGB images. Based on the positional deviations between corresponding points in the RGB images acquired by the left and right eyes of the binocular camera, three-dimensional coordinate information can be calculated, and the three-dimensional coordinate information of each point can be represented by a depth map.

[0154] To facilitate users in selecting and viewing either the RGB image or the depth image captured by the stereo camera, if the first control in step S20 is the first control corresponding to the stereo camera, a second control is displayed on the user layer. When the user clicks the second control, the RGB image captured by the stereo camera or the depth image calculated based on the video captured by the stereo camera is displayed.

[0155] Based on the RGB images captured by the left and right eyes of the binocular camera, a stereo image, i.e., a depth image, can be calculated. Based on the depth image, a corresponding obstacle image can be obtained, which is labeled with the obstacle and its information. Therefore, in one possible implementation, the second control includes a left-eye control, a right-eye control, a color image control, an obstacle image control, and a depth image control. For example, Figure 9a shows a first example of displaying the depth camera recognition results provided in this application embodiment. After the user selects the depth camera using control 20, the display interface shows second controls for different types of images, such as obstacle image control 21, depth image control 22, color image control 23, left-eye control 24, and right-eye control 25. The user can click on different controls to display the corresponding images in the video preview area 10 of the display screen. The display screen can also show the depth camera's recognition results; for example, if no obstacle is encountered, it will display "Obstacle Status: Binocular Normal".

[0156] It is understandable that when the first control is a depth control, the images captured by the depth camera include the images captured by the left eye and the right eye of the binocular camera. In the above step S30, the images captured by the left eye and the right eye are scaled simultaneously.

[0157] When a user wants to view the video captured by the left eye of the stereo camera, upon selecting the left eye control 24, the AGV responds by displaying the image captured by the left eye of the stereo camera at a preset size in the first position of the video layer. For example, Figure 9b shows a second example of the display of depth camera recognition results provided in this embodiment. Since the left eye control is selected, it can be seen that the first control in step S20 above is the depth control, and the second control mentioned above is the left eye control. The video captured by the left eye of the stereo camera is displayed in the video preview area 10.

[0158] When a user wants to view the video captured by the right eye of the stereo camera, the user selects the right eye control. The AGV responds to this selection by displaying the video captured by the right eye of the stereo camera at a preset size in the first position of the video layer. For example, Figure 9c shows a third example of the depth camera recognition result display provided in this embodiment. Since the right eye control is selected, it can be seen that the first control in step S20 above is the depth control, and the second control mentioned above is the right eye control. The video captured by the right eye of the stereo camera is displayed in the video preview area 10.

[0159] When a user wants to view the stereo image captured by the binocular camera, the user selects the color image control. In response to this selection, the AGV displays the stereo video (i.e., the color image video) calculated based on the RGB images captured by the left and right eyes of the binocular camera in the first position of the video layer. For example, Figure 9d shows a fourth example of the depth camera recognition result display provided in this embodiment. The color image control is selected, indicating that the first control in step S20 is the depth control, and the second control mentioned above is the color image control. The color image video calculated based on the RGB images captured by the left and right eyes of the binocular camera is displayed in the video preview area 10.

[0160] Using the method of this application embodiment, users can not only view the stereoscopic images captured by the binocular camera on the AGV's display screen, but also view the videos captured by the left and right eyes of the left eye camera respectively, thus improving the user experience.

[0161] In one possible implementation, after obtaining the two-dimensional and three-dimensional images captured by the binocular camera, an algorithm is used to calculate the depth map and the position information of each object in the depth map, thereby determining whether there is an obstacle.

[0162] The depth map displays the distance information of all detected surrounding objects relative to the AGV. This distance information is typically expressed in grayscale values; for example, pixels with higher grayscale values ​​represent objects farther from the AGV, and pixels with lower grayscale values ​​represent objects closer to the AGV. However, because grayscale images are not visually significant when pixel values ​​change slightly, they cannot intuitively display the distance information of each object. Therefore, to facilitate users' intuitive understanding of the distance information of each object, in one possible implementation, obstacles at different distances in the depth map can be represented by different colors. Obstacles at distances between a preset first distance and a preset second distance from the AGV are represented by a first color, obstacles at distances between a preset second distance and a preset third distance from the AGV are represented by a second color, and obstacles at distances between a preset third distance and a preset fourth distance from the AGV are represented by a third color. It is understood that the preset distances and colors are set by the user according to their needs, and this application embodiment does not limit this.

[0163] For example, suppose a user pre-sets the depth map to indicate obstacles 1-2 meters away from the AGV in red, obstacles 2-3 meters away in yellow, obstacles 3-4 meters away in blue, and objects 4-5 meters away in green. The user can view the depth map to understand the distance information of each object and thus determine whether obstacles exist.

[0164] In one possible implementation, the distance between each obstacle and the AGV in the depth map can also be labeled in numerical form.

[0165] Although the depth map shows the distance information of all detected obstacles, some obstacles will not affect the AGV's movement. For example, the depth image may include the distance information of the ceiling, but the ceiling will not affect the AGV's movement.

[0166] As can be seen, users cannot intuitively understand which obstacles might affect the AGV's movement when viewing the depth map. Therefore, to facilitate users' intuitive understanding of obstacle information, they can also view the obstacle map to see if obstacles exist. The obstacle map only marks detected obstacles within the AGV's preset obstacle encounter range. This preset obstacle encounter range is set by the user based on practical experience, and this application does not limit it.

[0167] Therefore, the second control mentioned above includes an obstacle map control and a depth map control. Users can click the obstacle map control to view an obstacle map obtained from the video captured by the stereo camera. The obstacle map marks the detected obstacles within the AGV's preset obstacle encounter range. Clicking the depth map control displays a depth map obtained from the video captured by the stereo camera. This depth map marks all detected obstacles.

[0168] When a user wants to view the obstacle map, they select the obstacle map control. In response, the AGV displays the obstacle map obtained from the video captured by the binocular camera in the first position of the video layer. For example, Figure 9e shows a fifth example of the depth camera recognition result display provided in this embodiment. The obstacle map control is selected, and it can be seen that the first control in step S20 above is a depth control, and the second control mentioned above is an obstacle map control. The obstacle map obtained from the video captured by the binocular camera is displayed in the video preview area 10. The obstacle map marks the detected obstacles within the AGV's preset obstacle encounter range. Obstacles at different distances in the obstacle map can also be marked with different colors. It is understood that only obstacles within the AGV's preset obstacle encounter range are marked in the obstacle map.

[0169] In addition, to help users identify the nearest obstacle to the AGV, in one possible implementation, the nearest obstacle to the AGV can be marked with a dashed box on the obstacle map.

[0170] When a user wants to view the depth image, they select the depth map control. In response to this selection, the AGV displays a depth map calculated from the video captured by the stereo camera in the first position of the video layer. All detected obstacles are marked on the depth map. For example, Figure 9f shows a sixth example of the depth camera recognition result display provided in this embodiment. The depth map control is selected, indicating that the first control in step S20 is the depth control, and the second control mentioned above is the depth map control. The depth map obtained from the video captured by the stereo camera is displayed in the video preview area 10.

[0171] To enhance the user experience, in one possible implementation, when the user selects the second control, a label indicating whether the AGV has encountered an obstacle will also be displayed on the screen.

[0172] For example, as shown in Figure 9f, when the user selects the depth map control, the distance to the obstacle is marked as 1088mm in the depth map displayed in the video preview area, and the obstacle is marked with a dashed box. Simultaneously, the display shows "Obstacle encountered: Yes" to indicate that the AGV has encountered an obstacle. It is understood that when the user selects the left view control, right view control, color map control, or obstacle map control, the display will also show marking information indicating whether the AGV has encountered an obstacle.

[0173] The method described in this application process obtains an obstacle map by processing the depth image, thereby improving the visual effect of the depth map. This makes it easier for users to intuitively locate obstacles in the video by combining the depth map and the obstacle map, thus improving the efficiency of anomaly or fault analysis.

[0174] It is understood that the images displayed in the video preview area 10 of Figures 6 to 9f above are merely examples. In practice, the videos displayed in the video preview area 10 are actual images captured by various image acquisition devices. The videos captured by the binocular camera displayed on the AGV's screen can be black and white or color, depending on the configuration of the binocular camera and user requirements. This application does not limit this aspect.

[0175] In one possible implementation, the user can change the size and position of the video preview area of ​​the overlay layer by changing the position and size of the transparent area in the user layer.

[0176] When the depth camera is a structured light depth camera, since structured light cameras do not have left and right view controls, the second control includes a color image control, a depth map control, and an obstacle map control. In this case, the AGV's display screen interface is the same as Figures 5a to 5e above. The example images showing the recognition results from the upper and lower barcode readers are the same as Figures 6 and 7 above. When the user selects the depth camera control, the page displayed is similar to Figure 9a, the only difference being that when the depth camera is a structured light depth camera, the page displayed after the user selects the depth camera control does not have left and right view controls. Other example images showing the depth camera recognition results are the same as Figures 9d to 9f above. It can be understood that when the user selects the color image control, the display screen shows the RGB image captured by the structured light depth camera, i.e., the color image video mentioned above.

[0177] To implement the video display method provided in this application embodiment, the basic framework diagram of the display software of the AGV screen provided in this application embodiment is shown in Figure 10. First, the main process of the application side draws the user layer through the graphical user interface. After the media library receives the image information from the image acquisition device, it performs rotation, scaling, compression and format conversion to obtain the video stream of the target video. Then, the video stream of the target video is sent to the layer module. The layer module embeds the target video into a new layer to obtain the video layer. After the graphics hardware abstraction layer overlays the user layer and the video layer, it is displayed through the operating system driver and multimedia interface.

[0178] The operating system driver can be a DMR (Direct Rendering Manager) driver or an FB (Frame buffer) driver; this application embodiment does not limit this.

[0179] Corresponding to the first aspect mentioned above, the second aspect of this application provides an AGV, which includes a processor, a display screen, and at least one image acquisition device;

[0180] The image acquisition device is used to capture video and send it to the processor;

[0181] The processor is configured to, in response to an operation on any of the first controls, acquire a video captured by an image acquisition device corresponding to the first control targeted by the operation; scale the acquired video to the preset size to obtain a target video; and send the target video and a calculation result calculated based on the acquired video to the display screen.

[0182] The display screen is used to display an overlay layer, wherein the overlay layer includes a user layer at the top and a video layer at the bottom. A transparent area of ​​a preset size is provided at a preset first position of the user layer, and at least one first control is provided at a preset second position of the user layer. The first control corresponds to at least one image acquisition device of the automated guided vehicle. The target video is displayed at the first position of the video layer. The calculation result is displayed.

[0183] In this embodiment, the overlay layer includes a user layer at the top and a video layer at the bottom. The user layer has a preset first position with a transparent area of ​​a preset size, and at least one first control corresponding to at least one image acquisition device is located at the preset first position. Therefore, the user can browse the target video displayed at the first position of the video layer through the transparent area at the first position of the user layer in the overlay layer. Furthermore, the display screen will also show the calculation results obtained by the AGV based on the acquired video. Compared to the prior art, which primarily involves connecting a host computer to the AGV to obtain the video captured by the AGV's image acquisition device, and then combining the video displayed by the host computer to analyze the cause of the fault, in this application, the user can view the calculation results obtained by the AGV based on the video captured by each image acquisition device while simultaneously viewing the video on the AGV's display screen. No host computer intervention is required; the cause of the abnormality or fault can be analyzed directly based on the video and calculation results displayed on the AGV's display screen, improving the efficiency of troubleshooting at the AGV's work site.

[0184] Corresponding to the first aspect mentioned above, the third aspect of this application provides a video display device applied to an AGV, as shown in FIG11, the device comprising:

[0185] The layer display module 1101 is used to display an overlay layer on the display screen of the AGV. The overlay layer includes a user layer at the top layer and a video layer at the bottom layer. A transparent area of ​​a preset size is provided at a preset first position of the user layer, and at least one first control is provided at a preset second position of the user layer. The first control corresponds to at least one image acquisition device of the automated guided vehicle.

[0186] The video acquisition module 1102 is used to acquire the video captured by the image acquisition device corresponding to the first control targeted by the operation in response to any operation of the first control.

[0187] The video scaling module 1103 is used to scale the acquired video to the preset size to obtain the target video;

[0188] The video display module 1104 is used to display the target video at the first position of the video layer and to display the calculation result obtained by the AGV based on the acquired video through the display screen.

[0189] In this embodiment, the overlay layer includes a user layer at the top and a video layer at the bottom. The user layer has a preset first position with a transparent area of ​​a preset size, and at least one first control corresponding to at least one image acquisition device is located at the preset first position. Therefore, the user can browse the target video displayed at the first position of the video layer through the transparent area at the first position of the user layer in the overlay layer. Furthermore, the display screen will also show the calculation results obtained by the AGV based on the acquired video. Compared to the prior art, which primarily involves connecting a host computer to the AGV to obtain the video captured by the AGV's image acquisition device, and then combining the video displayed by the host computer to analyze the cause of the fault, in this application, the user can view the calculation results obtained by the AGV based on the video captured by each image acquisition device while simultaneously viewing the video on the AGV's display screen. No host computer intervention is required; the cause of the abnormality or fault can be analyzed directly based on the video and calculation results displayed on the AGV's display screen, improving the efficiency of troubleshooting at the AGV's work site.

[0190] In one possible implementation, the image acquisition device of the automated guided vehicle includes an upper barcode reader camera;

[0191] The step of displaying the calculation results obtained by the automated guided vehicle based on the acquired video on the display screen includes:

[0192] If the first control targeted by the operation is the first control corresponding to the upper barcode reader camera, then the display screen shows the position of the cargo code relative to the automated guided vehicle and the barcode reading result of the cargo code captured by the upper barcode reader camera.

[0193] In one possible implementation, the image acquisition device of the automated guided vehicle includes a lower barcode scanner;

[0194] The video display module is specifically used for:

[0195] If the first control targeted by the operation is the first control corresponding to the lower barcode reader camera, and the navigation method of the automated guided vehicle is QR code navigation, then the display screen will show the pose of the ground code captured by the lower barcode reader relative to the automated guided vehicle, the barcode reading result, and the pose of the automated guided vehicle in its own operating map.

[0196] In one possible implementation, the image acquisition device of the automated guided vehicle includes a lower barcode scanner;

[0197] The video display module is specifically used for:

[0198] If the first control targeted by the operation is the first control corresponding to the lower barcode reader camera, and the navigation mode of the automated guided vehicle is texture navigation, then the display screen shows the position and pose of the automated guided vehicle in its own running map, calculated based on the ground texture captured by the lower barcode reader camera.

[0199] In one possible implementation, the image acquisition device of the automated guided vehicle includes a depth camera;

[0200] The video display module is specifically used for:

[0201] A second control is displayed in the user layer;

[0202] In response to an operation on the second control, the target video is displayed at the first position of the video layer, or a calculation result calculated based on the image captured by the depth camera is displayed at the first position of the video layer.

[0203] In one possible implementation, the depth camera is a binocular camera, and the second control includes a left-eye control, a right-eye control, and a color image control;

[0204] The step of displaying the target video at the first position of the video layer in response to an operation on the second control includes:

[0205] In response to an operation on the left eye control, a video captured by the left eye of the binocular camera at a preset size is displayed at the first position of the video layer;

[0206] In response to an operation on the right eye control, a video captured by the right eye of the binocular camera at a preset size is displayed at the first position of the video layer;

[0207] In response to an operation on the color image control, a color image video of the preset size captured by the binocular camera is displayed at the first position of the video layer.

[0208] In one possible implementation, the second control includes an obstacle map control and a depth map control;

[0209] The step of displaying the target video at the first position of the video layer in response to an operation on the second control includes:

[0210] In response to an operation on the obstacle map control, an obstacle map calculated based on the video captured by the depth camera is displayed at the first position of the video layer. The obstacle map is marked with detected obstacles located within a preset obstacle encounter range of the automated guided vehicle.

[0211] In response to an operation on the depth map control, a depth map calculated based on the video captured by the depth camera is displayed at the first position of the video layer, the depth map indicating all detected obstacles.

[0212] In one possible implementation, the device further includes:

[0213] The information display module is used to display, in response to an operation on the second control, annotation information indicating whether the automated guided vehicle has encountered an obstacle via the display screen.

[0214] A fourth aspect of this application also provides an electronic device, as shown in FIG12, comprising:

[0215] Memory 1201 is used to store computer programs;

[0216] The processor 1202 is used to execute the program stored in the memory to implement any of the video display methods described above.

[0217] The memory may include random access memory (RAM) or non-volatile memory (NVM), such as at least one disk storage device. Optionally, the memory may also be at least one storage device located remotely from the aforementioned processor.

[0218] The processors mentioned above can be general-purpose processors, including central processing units (CPUs), network processors (NPs), etc.; they can also be digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0219] In another embodiment provided in this application, a computer-readable storage medium is also provided, which stores a computer program that, when executed by a processor, implements the steps of any of the above-described video display methods.

[0220] In another embodiment provided in this application, a computer program product containing instructions is also provided, which, when run on a computer, causes the computer to perform any of the video display methods described above.

[0221] In the above embodiments, implementation can be achieved entirely or partially through software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented entirely or partially in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a solid-state drive (SSD), etc.

[0222] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0223] The various embodiments in this specification are described in a related manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the embodiments for automated guided vehicles and devices are basically similar to the method embodiments, so the descriptions are relatively simple; relevant parts can be referred to the descriptions of the method embodiments.

[0224] The above description is only a preferred embodiment of this application and is not intended to limit this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of protection of this application.

Claims

1. A video display method, characterized in that, Applied to automated guided vehicles, the method includes: An overlay layer is displayed on the screen of the automated guided vehicle. The overlay layer includes a user layer at the top and a video layer at the bottom. A transparent area of ​​a preset size is set at a preset first position of the user layer, and at least one first control is set at a preset second position of the user layer. The first control corresponds to at least one image acquisition device of the automated guided vehicle. In response to an operation on any of the first controls, acquire the video captured by the image acquisition device corresponding to the first control targeted by the operation; The acquired video is scaled to the preset size to obtain the target video; The target video is displayed at the first position of the video layer, and the calculation result obtained by the automated guided vehicle based on the acquired video is displayed on the screen.

2. The method according to claim 1, characterized in that, The image acquisition device of the automated guided vehicle includes an upper barcode reader camera; The step of displaying the calculation results obtained by the automated guided vehicle based on the acquired video on the display screen includes: If the first control targeted by the operation is the first control corresponding to the upper barcode reader camera, then the display screen shows the position of the cargo code relative to the automated guided vehicle and the barcode reading result of the cargo code captured by the upper barcode reader camera.

3. The method according to claim 1, characterized in that, The image acquisition device of the automated guided vehicle includes a lower barcode reader camera; The step of displaying the calculation results obtained by the automated guided vehicle based on the acquired video on the display screen includes: If the first control targeted by the operation is the first control corresponding to the lower barcode reader camera, and the navigation method of the automated guided vehicle is QR code navigation, then the display screen will show the pose of the ground code captured by the lower barcode reader relative to the automated guided vehicle, the barcode reading result, and the pose of the automated guided vehicle in its own operating map.

4. The method according to claim 1, characterized in that, The image acquisition device of the automated guided vehicle includes a lower barcode reader camera; The step of displaying the calculation results obtained by the automated guided vehicle based on the acquired video on the display screen includes: If the first control targeted by the operation is the first control corresponding to the lower barcode reader camera, and the navigation mode of the automated guided vehicle is texture navigation, then the display screen shows the position and pose of the automated guided vehicle in its own running map, calculated based on the ground texture captured by the lower barcode reader camera.

5. The method according to claim 1, characterized in that, The image acquisition device of the automated guided vehicle includes a depth camera; The step of displaying the target video at the first position of the video layer and displaying the calculation results obtained by the automated guided vehicle based on the acquired video on the display screen includes: A second control is displayed in the user layer; In response to an operation on the second control, the target video is displayed at the first position of the video layer, or a calculation result calculated based on the image captured by the depth camera is displayed at the first position of the video layer.

6. The method according to claim 5, characterized in that, The depth camera is a binocular camera, and the second control includes a left eye control, a right eye control, and a color image control; The step of displaying the target video at the first position of the video layer in response to an operation on the second control includes: In response to an operation on the left eye control, a video captured by the left eye of the binocular camera at a preset size is displayed at the first position of the video layer; In response to an operation on the right eye control, a video captured by the right eye of the binocular camera at a preset size is displayed at the first position of the video layer; In response to an operation on the color image control, a color image video of the preset size captured by the binocular camera is displayed at the first position of the video layer.

7. The method according to claim 5, characterized in that, The second control includes an obstacle map control and a depth map control; The step of displaying the target video at the first position of the video layer in response to an operation on the second control includes: In response to an operation on the obstacle map control, an obstacle map calculated based on the video captured by the depth camera is displayed at the first position of the video layer. The obstacle map is marked with detected obstacles located within a preset obstacle encounter range of the automated guided vehicle. In response to an operation on the depth map control, a depth map calculated based on the video captured by the depth camera is displayed at the first position of the video layer, the depth map indicating all detected obstacles.

8. The method according to claim 6 or 7, characterized in that, The method further includes: In response to an operation on the second control, the display screen shows marking information indicating whether the automated guided vehicle has encountered an obstacle.

9. An automated guided vehicle, characterized in that, The automated guided vehicle includes a processor, a display screen, and at least one image acquisition device; The image acquisition device is used to capture video and send it to the processor; The processor is configured to, in response to an operation on any first control, acquire a video captured by an image acquisition device corresponding to the first control targeted by the operation; and scale the acquired video to a preset size to obtain a target video; The target video and the calculation results calculated based on the acquired video are sent to the display screen; The display screen is used to display an overlay layer, wherein the overlay layer includes a user layer at the top and a video layer at the bottom. A transparent area of ​​a preset size is provided at a preset first position of the user layer, and at least one first control is provided at a preset second position of the user layer. The first control corresponds to at least one image acquisition device of the automated guided vehicle. The target video is displayed at the first position of the video layer. The calculation result is displayed.

10. A video display device, characterized in that, The device, used in automated guided vehicles, includes: A layer display module is used to display an overlay layer on the display screen of the automated guided vehicle. The overlay layer includes a user layer at the top layer and a video layer at the bottom layer. A transparent area of ​​a preset size is provided at a preset first position of the user layer, and at least one first control is provided at a preset second position of the user layer. The first control corresponds to at least one image acquisition device of the automated guided vehicle. The video acquisition module is used to acquire the video captured by the image acquisition device corresponding to the first control targeted by the operation in response to any operation of the first control. The video scaling module is used to scale the acquired video to the preset size to obtain the target video; The video display module is used to display the target video at the first position of the video layer, and to display the calculation results obtained by the automated guided vehicle based on the acquired video through the display screen.

11. The apparatus according to claim 10, characterized in that, The image acquisition device of the automated guided vehicle includes an upper barcode reader camera; The step of displaying the calculation results obtained by the automated guided vehicle based on the acquired video on the display screen includes: If the first control targeted by the operation is the first control corresponding to the upper barcode reader camera, then the position of the barcode relative to the automated guided vehicle and the barcode reading result of the barcode captured by the upper barcode reader camera are displayed on the display screen. and / or The image acquisition device of the automated guided vehicle includes a lower barcode reader camera; The video display module is specifically used for: If the first control targeted by the operation is the first control corresponding to the lower barcode reader camera, and the navigation method of the automated guided vehicle is QR code navigation, then the display screen will show the pose of the ground code captured by the lower barcode reader relative to the automated guided vehicle, the barcode reading result, and the pose of the automated guided vehicle in its own operating map. and / or The image acquisition device of the automated guided vehicle includes a lower barcode reader camera; The video display module is specifically used for: If the first control targeted by the operation is the first control corresponding to the lower barcode reader camera, and the navigation mode of the automated guided vehicle is texture navigation, then the position and pose of the automated guided vehicle in its own running map, calculated based on the ground texture captured by the lower barcode reader camera, is displayed on the display screen. and / or The image acquisition device of the automated guided vehicle includes a depth camera; The video display module is specifically used for: A second control is displayed in the user layer; In response to an operation on the second control, the target video is displayed at the first position of the video layer, or a calculation result calculated based on the image captured by the depth camera is displayed at the first position of the video layer; and / or The depth camera is a binocular camera, and the second control includes a left eye control, a right eye control, and a color image control; The step of displaying the target video at the first position of the video layer in response to an operation on the second control includes: In response to an operation on the left eye control, a video captured by the left eye of the binocular camera at a preset size is displayed at the first position of the video layer; In response to an operation on the right eye control, a video captured by the right eye of the binocular camera at a preset size is displayed at the first position of the video layer; In response to an operation on the color image control, a color image video of the preset size captured by the binocular camera is displayed at the first position of the video layer; and / or The second control includes an obstacle map control and a depth map control; The step of displaying the target video at the first position of the video layer in response to an operation on the second control includes: In response to an operation on the obstacle map control, an obstacle map calculated based on the video captured by the depth camera is displayed at the first position of the video layer. The obstacle map is marked with detected obstacles located within a preset obstacle encounter range of the automated guided vehicle. In response to an operation on the depth map control, a depth map calculated based on the video captured by the depth camera is displayed at the first position of the video layer, the depth map marking all detected obstacles; and / or The device further includes: The information display module is used to display, in response to an operation on the second control, annotation information indicating whether the automated guided vehicle has encountered an obstacle via the display screen.

12. An electronic device, characterized in that, include: Memory, used to store computer programs; A processor, when executing a program stored in memory, implements the method described in any one of claims 1-8.

13. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the method described in any one of claims 1-8.

Images