Monitoring method, device, product, medium and system

Abstract

Provided in the present application are a monitoring method, a device, a product, a medium and a system. The method comprises: receiving a panoramic image sent by a vehicle, the panoramic image being composited on the basis of images acquired by a plurality of image acquisition devices of the vehicle; displaying a projection image of the panoramic image in a first region of a terminal interface, the projection image comprising a plurality of portions; in response to a first viewing operation input by a user in a first portion, sending a first viewing request to the vehicle, the first portion belonging to the plurality of portions; receiving a first viewing response returned by the vehicle on the basis of the first viewing request, the first viewing response comprising an image acquired by a first image acquisition device corresponding to the first portion, and the first image acquisition device belonging to the plurality of image acquisition devices; and displaying on the terminal interface the image acquired by the first image acquisition device.

Description

Monitoring methods, equipment, products, media and systems

[0001] This application claims priority to Chinese Patent Application No. 202411629302.5, filed on November 14, 2024, with the invention entitled "Monitoring Method, Device, Product, Medium and System", the entire contents of which are incorporated herein by reference. Technical Field

[0002] This application relates to the field of image processing, and more particularly to a monitoring method, device, product, medium, and system. Background Technology

[0003] Vehicles are prone to personal injury or property damage during operation due to negligence or accidents. Therefore, it's often necessary to install multiple image acquisition devices on vehicles to collect information about their various positions and transmit it to a terminal device. Users can then observe the vehicle's position on the terminal device's interface. Furthermore, the terminal interface features multiple buttons, each corresponding to a different image acquisition device. When a button is clicked, the terminal interface displays the image captured by that specific device for closer inspection. However, this method lacks a clear visual connection between the buttons and the image acquisition devices, making it easy for users to select the wrong button. Summary of the Invention

[0004] This application provides a monitoring method, device, product, medium, and system that can intuitively display the relationship between various parts of the projected image of a panoramic image and various image acquisition devices, making it convenient for users to select the image acquisition devices that require detailed display.

[0005] Firstly, a monitoring method is provided. A terminal device receives a panoramic image sent by a vehicle. The panoramic image is synthesized from images captured by multiple image acquisition devices on the vehicle. The terminal device can be one or more of a mobile terminal, central control unit, tablet, laptop, and desktop computer. The vehicle can be a car, electric vehicle, ship, speedboat, etc. The terminal device displays a projected image of the panoramic image in a first area of ​​its interface. The projected image can be an image projected from one angle, for example, an image projected from top to bottom. The projected image includes multiple parts, and there is a one-to-one correspondence between these multiple parts and the multiple image acquisition devices. For example, when the image acquisition devices include a front image acquisition device, a rear image acquisition device, a left front image acquisition device, a right front image acquisition device, a left rear image acquisition device, and a right rear image acquisition device, the projected image also includes a front projection part, a rear projection part, a left front projection part, a right front projection part, a left rear projection part, and a right rear projection part. Therefore, the front projection portion corresponds to the front image acquisition device, the rear image acquisition device corresponds to the rear image acquisition device, the left front projection portion corresponds to the left front image acquisition device, the right front image acquisition device corresponds to the right front image acquisition device, the left rear projection portion corresponds to the left rear image acquisition device, and the right rear image acquisition device corresponds to the right rear image acquisition device. The terminal device, in response to a user's first viewing operation input in the first part, sends a first viewing request to the vehicle. The first part belongs to the plurality of parts. The first viewing request is used to request the vehicle to display the image acquired by the first image acquisition device corresponding to the first part. The terminal device receives a first viewing response returned by the vehicle based on the first viewing request. The first viewing response includes the image acquired by the first image acquisition device corresponding to the first part, and the first image acquisition device belongs to the plurality of image acquisition devices. The terminal device displays the image acquired by the first image acquisition device on the terminal interface. The terminal device can display the image acquired by the first image acquisition device in a first area or in a second area. The second area and the first area are different areas on the terminal interface.

[0006] In the above solution, the projected image of the panoramic view can be displayed on the terminal interface. The projected image consists of multiple parts, and there is a one-to-one correspondence between these parts and multiple image acquisition devices. This makes the relationship between the projected images of the multiple parts and the multiple image acquisition devices very direct, allowing the user to select one of the image acquisition devices for display based on this intuitive presentation. For example, when a user sees an obstacle in a part of the projected image, they can directly click on that part of the projected image to select and display the image acquired by the corresponding image acquisition device for closer observation.

[0007] In some possible designs, after the terminal interface displays the image captured by the first image acquisition device, the method further includes: the terminal device sending a second viewing request to the vehicle in response to a second viewing operation input by the user in a second part. The second part belongs to the plurality of parts. The terminal device receives a second viewing response returned by the vehicle based on the second viewing request. The second viewing response includes an image captured by a second image acquisition device corresponding to the second part, and the second image acquisition device belongs to the plurality of image acquisition devices. The terminal device replaces the image captured by the first image acquisition device with the image captured by the second image acquisition device.

[0008] In the above solution, users can switch the displayed image to any image captured by any image acquisition device as needed.

[0009] In some possible designs, the method further includes: displaying the projected image in full-screen mode on the terminal interface in response to a user's full-screen operation input on the terminal interface. For example, in response to a user's click operation on a panoramic area of ​​the first region, the projected image is displayed in full-screen mode on the terminal interface. Alternatively, in response to a user's zoom-in operation on the first region, the projected image is displayed in full-screen mode on the terminal interface.

[0010] In the above solution, the projected image can be quickly switched to full-screen mode for easier observation by the user.

[0011] In some possible designs, the projected image includes obstacle and / or risk warning icons, which are located in a second portion of the projected image and are displayed in a highlighted manner. Optionally, the risk warning icons may also indicate one or more of the types of risks and the distance between the risks and the vehicle.

[0012] In the above scheme, a risk warning icon can also be displayed in a portion of the projected image to prompt the user to select the image captured by the corresponding image acquisition device for careful observation.

[0013] Secondly, a terminal device is provided, including:

[0014] A receiving module is used to receive panoramic images sent by a vehicle, wherein the panoramic images are synthesized from images captured by multiple image acquisition devices of the vehicle;

[0015] The display module is used to display a projected image of the panoramic image in a first area of ​​the terminal interface, wherein the projected image includes multiple parts, and there is a one-to-one correspondence between the multiple parts and the multiple image acquisition devices;

[0016] A sending module is configured to send a first viewing request to the vehicle in response to a first viewing operation input by a user in the first part, wherein the first part belongs to the plurality of parts;

[0017] The receiving module is further configured to receive a first viewing response returned by the vehicle based on the first viewing request, wherein the first viewing response includes an image captured by a first image acquisition device corresponding to the first part, and the first image acquisition device belongs to the plurality of image acquisition devices;

[0018] The image acquired by the first image acquisition device is displayed on the terminal interface.

[0019] In some possible designs, the sending module is used to send a second viewing request to the vehicle in response to a second viewing operation input by the user in the second part, wherein the second part belongs to the plurality of parts;

[0020] The receiving module is further configured to receive a second viewing response returned by the vehicle based on the second viewing request, wherein the second viewing response includes an image captured by a second image acquisition device corresponding to the second part, and the second image acquisition device belongs to the plurality of image acquisition devices;

[0021] The display module is further configured to replace the image acquired by the first image acquisition device with the image acquired by the second image acquisition device.

[0022] In some possible designs, the display module is also used to respond to a full-screen operation input by the user on the terminal interface, and to display the projected image in full-screen mode on the terminal interface.

[0023] In some possible designs, the display module is also used to display the projected image in full-screen mode on the terminal interface in response to a user's click operation in the panoramic area of ​​the first region.

[0024] In some possible designs, the display module is also used to display the projected image in full-screen mode on the terminal interface in response to a user's zoom-in operation in the first area.

[0025] In some possible designs, the projected image includes obstacles and / or risk warning icons, which are located in a second part of the projected image and are displayed in a highlighted manner.

[0026] In some possible designs, the risk warning icon is also used to indicate one or more of the types of risks and the distance between the risk and the vehicle.

[0027] Thirdly, a monitoring system is provided, including: terminal equipment and vehicles.

[0028] The vehicle is used to send panoramic images to the terminal, wherein the panoramic images are synthesized from images captured by multiple image acquisition devices of the vehicle;

[0029] The terminal device is used to display a projected image of the panoramic image in a first area of ​​the terminal interface, wherein the projected image includes multiple parts, and there is a one-to-one correspondence between the multiple parts and the multiple image acquisition devices;

[0030] The terminal device is used to send a first viewing request to the vehicle in response to a first viewing operation input by the user in the first part, wherein the first part belongs to the plurality of parts;

[0031] The vehicle is used to send a first viewing response based on the first viewing request, wherein the first viewing response includes an image captured by a first image acquisition device corresponding to the first part, and the first image acquisition device belongs to the plurality of image acquisition devices;

[0032] The terminal device is used to display the image acquired by the first image acquisition device on the terminal interface.

[0033] Fourthly, a terminal device is provided, the terminal device including a processor and a memory; the processor is configured to execute instructions stored in the memory to cause the terminal device to perform the operation steps of the method as described in any of the first aspects.

[0034] Fifthly, a computer program product containing instructions is provided, which, when executed by a cluster of terminal devices, causes the cluster of terminal devices to perform the operational steps of the method described in any of the first aspects.

[0035] In a sixth aspect, a computer-readable storage medium is provided, including program instructions that, when executed by a terminal device, cause the computing device to perform operational steps as described in any of the first aspects. Attached Figure Description

[0036] Figure 1 is a schematic diagram of the structure of a monitoring system provided in this application;

[0037] Figure 2 is a flowchart illustrating a monitoring system method provided in this application;

[0038] Figure 3 is a schematic diagram of an image acquisition device set up in various directions on a vehicle, as provided in this application;

[0039] Figure 4 is a structural schematic diagram of a first region provided in this application;

[0040] Figure 5 is a structural schematic diagram of a terminal device provided in this application;

[0041] Figure 6 is an exemplary structural block diagram of a vehicle provided in this application;

[0042] Figure 7 is an exemplary structural block diagram of a smartphone provided in this application. Detailed Implementation

[0043] Referring to Figure 1, Figure 1 is a schematic diagram of the structure of a monitoring system provided in this application. As shown in Figure 1, the monitoring system of this application includes: a vehicle and terminal equipment.

[0044] Vehicle 110 refers to a device or equipment used to transport people, goods or perform specific tasks and capable of moving in environments such as land, water or air.

[0045] In a land-based environment, transportation vehicles can include various vehicles, such as cars, trucks, buses, public buses, motorcycles, tricycles, etc. In a water-based environment, transportation vehicles can include ships, yachts, speedboats, hovercraft, etc. In an air-based environment, transportation vehicles can include passenger planes, cargo planes, helicopters, airships, etc. It is understood that the above-mentioned transportation vehicles are merely specific examples; in addition to existing transportation vehicles, all new types of transportation vehicles that emerge with technological advancements should be included.

[0046] Multiple image acquisition devices 111 are typically installed on vehicles to monitor the surrounding environment, including other vehicles, pedestrians, and obstacles, and to issue timely warnings to the driver to avoid collisions. The image acquisition devices 111 can be one or more of the following: visible light cameras (ordinary cameras), infrared cameras, millimeter-wave radar, lidar, ultrasonic sensors, and satellite positioning systems. Here, the multiple image acquisition devices 111 installed on the vehicle can all be of the same type, or they can be of multiple types. For example, a vehicle can have only multiple visible light cameras, only multiple ultrasonic sensors, or multiple visible light cameras along with one or more of the following: infrared cameras, millimeter-wave radar, lidar, and ultrasonic sensors. Visible light cameras receive visible light through a lens and focus it onto an image sensor, converting the light signal into an electrical signal to obtain a visible light image. Visible light cameras provide high resolution and rich detail; however, they are affected by lighting conditions, have limited detection range and distance, and are susceptible to interference from lighting, weather, and ambient light. Infrared cameras detect infrared light emitted or reflected by objects to form infrared images. They are commonly used at night or in low-light conditions. However, their detection range and distance are generally shorter than those of visible light, and their resolution is relatively low. They have high accuracy in detecting temperature differences but are sensitive to temperature changes and heat source interference. Millimeter-wave radar emits millimeter waves and receives reflected waves, measuring the round-trip time and frequency changes to determine the position and velocity of objects. It can detect long distances and penetrate fog and smoke, offering high accuracy in distance and velocity measurements. However, its angular resolution is relatively low, and it may be affected by interference from other electromagnetic signals. LiDAR emits laser beams and measures the time and angle of reflected light to generate high-precision 3D point cloud images. LiDAR has high detection range and accuracy, with high angular and distance resolution, and can accurately depict the shape of objects. However, its detection range is relatively narrow, and it has high environmental requirements. Ultrasonic sensors emit ultrasonic waves and receive echoes, calculating distance based on the sound wave propagation time. They have good accuracy over short distances and are suitable for close-range detection. However, they may be affected by interference in environments with many obstacles and noise.

[0047] Different types of image acquisition devices 111 have different advantages and limitations. For example, visible light cameras can provide rich details and color information under good lighting conditions, but may perform poorly in bad weather or poor lighting conditions; infrared cameras can detect the thermal radiation of objects at night or in low light; millimeter-wave radar can accurately measure the speed and distance of objects, unaffected by weather and lighting conditions; lidar can provide high-precision three-dimensional spatial information; and ultrasonic sensors are more accurate in short-range detection. Therefore, when multiple types of image acquisition devices are set up, they can complement each other, reduce misjudgments and omissions caused by a single image acquisition device, and improve the accuracy and reliability of environmental perception.

[0048] In the example above, the vehicle is a car and the number of image acquisition devices 111 is 6. In practical applications, the vehicle can be a tricycle, yacht, airplane, etc., and the number of image acquisition devices can be 2, 3, 4, 5, 7, 8 or even more.

[0049] Terminal device 120 can be an electronic device used to receive, process, and transmit information, and has a display function. Terminal devices can include fixed terminal devices, such as desktop computers, smart TVs, etc.; mobile terminal devices, such as laptops, smartphones, tablets, etc.; wearable terminal devices, such as smartwatches, smart glasses, and 3D helmets, etc.; and in-vehicle devices, such as in-vehicle navigation systems, intelligent in-vehicle systems, etc., one or more of these. The number of terminal devices 120 can be one; for example, a terminal device can be simply an in-vehicle device, a driver's smartphone, a driver's tablet, etc. The number of terminal devices 120 can also be multiple. In this case, the terminal devices can be of the same type or different types. For example, terminal devices can include a driver's smartphone and a passenger's smartphone, or a terminal device can include an intelligent in-vehicle system and a driver's smartphone, etc.

[0050] It is understood that the above monitoring system only exemplifies terminal devices and vehicles. In practical applications, the monitoring system can include more devices, such as servers, relay devices, or signal enhancement devices. The server stores images captured by the image acquisition devices in the vehicle; the relay device forwards signals from the terminal device to the vehicle and vice versa; the signal enhancement device amplifies and forwards signals from the terminal device to the vehicle and vice versa. Furthermore, the above monitoring system only illustrates one terminal device corresponding to one vehicle. In practical applications, one terminal device can correspond to multiple vehicles, and one vehicle can correspond to one terminal device; no specific limitation is made here.

[0051] Based on the monitoring system shown in Figure 1, this application also proposes a monitoring method. As shown in Figure 2, the monitoring method of this application includes the following steps:

[0052] S101: The vehicle acquires multiple images through multiple image acquisition devices installed on the vehicle.

[0053] In order to cover the 360-degree environment around the vehicle, ensure no blind spots, and promptly detect potential dangers from all directions, image acquisition devices can be set up in different locations.

[0054] In a specific embodiment, as shown in Figure 3, image acquisition devices can be installed at the front, rear, upper left, upper right, lower left, and lower right of the vehicle. The front image acquisition device primarily acquires images of the environment in front of the vehicle; the rear image acquisition device primarily acquires images of the environment behind the vehicle; the upper left image acquisition device primarily acquires images of the environment to the upper left of the vehicle; the upper right image acquisition device primarily acquires images of the environment to the upper right of the vehicle; the lower left image acquisition device primarily acquires images of the environment to the lower left of the vehicle; and the lower right image acquisition device primarily acquires images of the environment to the upper right of the vehicle. The above example illustrates the use of front, rear, upper left, upper right, lower left, and lower right as the locations for the image acquisition devices. In practical applications, the locations of the image acquisition devices can include more or fewer locations, or even other locations.

[0055] S102: The vehicle combines multiple images to create a panoramic image.

[0056] Panoramic images are a type of image that provides a wide field of view and a comprehensive view of the scene. Vehicles use image processing algorithms to stitch together multiple images captured by various image acquisition devices, creating a single image with a horizontal field of view approaching or reaching 360 degrees and a relatively large vertical field of view. When viewing a panoramic image, users can drag, slide, or rotate it to observe the scene from all directions as if they were actually there, gaining a richer and more complete visual experience.

[0057] The process of synthesizing a panoramic image from multiple images using a transportation vehicle involves: First, the vehicle uses a feature point detection algorithm to determine the feature points of each image. Then, a matching algorithm is used to match these feature points, identifying corresponding feature point pairs in different images. Next, based on these feature point pairs, geometric transformations between the images are calculated, including rotation, translation, and scaling, thus achieving preliminary image registration to obtain the panoramic image. At the stitching point of the registered panoramic images, an image fusion algorithm is applied to ensure a natural transition and reduce noticeable stitching artifacts. Furthermore, to ensure a more consistent overall visual effect, exposure compensation may be applied using an exposure algorithm to adjust the brightness and contrast of different parts of the panoramic image. The feature point detection algorithms may include one or more of the following: Harris corner detection algorithm, features from accelerated segment test (FAST) algorithm, scale-invariant feature transform (SIFT) algorithm, speeded up robust features (SURF) algorithm, oriented fast feature point detection and rotated binary robust independent basic feature (ORB) algorithm, good features to track (GFTT) algorithm, etc. The matching algorithms may include one or more of the following: brute force matching algorithm, fast library for approximate nearest neighbors (FLANN) algorithm, random sample consensus (RANSAC) algorithm, ratio test algorithm. Registration algorithms can include one or more of the following: SIFT registration, SURF registration, ORB registration, mutual information registration, normalized cross correlation (NCC) registration, rigid transformation registration, affine transformation registration, and projective transformation registration. Image fusion algorithms can include one or more of the following: weighted average fusion algorithm, multi-band fusion algorithm, etc.Exposure algorithms include one or more of the following: average metering algorithm, center-weighted metering algorithm, spot metering algorithm, multi-zone metering algorithm, histogram-based metering algorithm, intelligent scene recognition metering algorithm, etc.

[0058] S103: The vehicle sends a panoramic image to the terminal device. Correspondingly, the terminal device receives the panoramic image sent by the vehicle.

[0059] S104: The terminal device displays a projected image of the panoramic image in the first area of ​​the terminal interface.

[0060] A panoramic image projection image refers to an image that displays a panoramic image at a certain projection angle and in a specific projection method.

[0061] The projection angle can be the angle from the top of the vehicle to the bottom, the angle from the bottom of the vehicle to the top, the angle from directly in front of the vehicle to directly behind, the angle from the upper left to the lower right, the angle from the upper right to the lower left, the angle from the lower left to the upper right, the angle from the lower right to the upper left, and so on. The projection angles mentioned above are merely specific examples; in practical applications, other angles are also possible, and no specific limitations are made here.

[0062] The projection method can be any of the following: equirectangular projection, cube projection, Mercator projection, fisheye projection, sinusoidal projection, etc. The projection methods mentioned above are merely specific examples; in practical applications, other projection methods can also be used, and no specific limitations are made here.

[0063] In one specific embodiment, the panoramic image can be projected using an equidistant cylindrical projection method from the top to the bottom of the vehicle to obtain a projected panoramic image. Specifically, the coordinate transformation relationship between pixels in the panoramic image and pixels in the projected image can be determined based on the angle from the top to the bottom of the vehicle using equidistant cylindrical projection. Then, for each pixel position in the projected image, its corresponding position in the panoramic image is found through coordinate transformation. Since the transformed position in the panoramic image may not be an integer pixel coordinate, sampling and interpolation methods are needed to obtain the pixel value of the pixel in the panoramic image. Finally, the projected image is generated pixel by pixel according to the calculated pixel values. The interpolation method can include bilinear interpolation, bicubic interpolation, etc.

[0064] The first area can be the entire terminal interface or a portion of it. The terminal interface is the medium through which applications or operating systems interact and exchange information with the user; it converts the internal form of information into a form that the user can understand. The application's user interface is written in source code using specific computer languages ​​such as Java or Extensible Markup Language (XML). This source code is parsed and rendered on the terminal device, ultimately presenting user-recognizable content such as images, text, and buttons. Controls, also known as widgets, are the basic elements of the user interface. Typical controls include toolbars, menu bars, text boxes, buttons, scroll bars, images, and text. The attributes and content of controls in the interface are defined using tags or nodes, such as in XML. <textview> 、 <imgview> 、 <videoview>Nodes define the controls contained in the interface. A node corresponds to a control or property in the interface, and after parsing and rendering, the node is presented as the content visible to the user. In addition, many applications, such as hybrid applications, often contain web pages within their interfaces. A web page, also known as a webpage, can be understood as a special control embedded in the application interface. Web pages are source code written in a specific computer language, such as Hypertext Markup Language (HTML), Cascading Style Sheets (CSS), JavaScript (JS), etc. The webpage source code can be loaded and displayed as user-readable content by a browser or a webpage display component with browser-like functionality. The specific content contained in a webpage is also defined through tags or nodes in the webpage source code; for example, HTML uses tags or nodes to define the content. 、 、 <video> 、 <canvas>This is used to define the elements and attributes of a webpage. The most common form of terminal interface is the graphical user interface (GUI), which refers to a user interface related to computer operation displayed graphically. It can be an icon, window, control, or other interface element displayed on the screen of an electronic device. Controls can include visual interface elements such as icons, buttons, menus, tabs, text boxes, dialog boxes, status bars, navigation bars, and widgets.

[0065] The first area is used to display the projected image. In a specific embodiment, the first area includes an area for displaying the projected image and a panoramic area. When the panoramic image is projected from the top to the bottom of the vehicle using an equidistant cylindrical projection, the panoramic area can display the vehicle icon projected from the top to the bottom of the vehicle. Users can determine the projection angle of the projected image and the orientation of the vehicle, etc., through the image in the panoramic area. As shown in Figure 4, in the car icon displayed in the panoramic area, the front of the car is facing down and the rear is facing up. The projected image can also be displayed in the first area with the front facing down and the rear facing up.

[0066] The projected image comprises multiple parts, and these parts do not overlap. Optionally, in addition to the area used to display the projected image, the first region also includes a panoramic region, and the panoramic region and the area of ​​the projected image do not overlap. In a specific embodiment, as shown in Figure 4, the projected image may include six parts, for example, part A, part B, part C, part D, part E, and part F. Taking the lower left corner of the first region as the origin of the coordinate axis (0, 0), the lower edge of the first region coincides with the positive half-axis of the x-axis, the left edge of the first region coincides with the positive half-axis of the y-axis, the maximum x-coordinate of the first region on the x-axis is x0, the maximum y-coordinate of the first region on the y-axis is y0, the center of the panoramic region coincides with the center of the first region, and the width of the panoramic region is w, and the height is h (in the figure, the panoramic region displays a car icon that can rotate 360 ​​degrees).

[0067] The coordinates of part A satisfy the following conditions: y / x > y0 / x0; y / x - x0 < y0 / -x0; y > y0 / 2 - h / 2

[0068] Where y is the vertical coordinate of part A and x is the horizontal coordinate of part A.

[0069] The coordinate values ​​of part B satisfy the following conditions: y / x < y0 / x0; y / x - x0 > y0 / - x0; y < y0 / 2 - h / 2

[0070] Where y is the vertical coordinate value of part B, and x is the horizontal coordinate value of part B.

[0071] The coordinate values ​​of part C satisfy the following conditions: y / x > y0 / x0; y / x - x0 > y0 / -x0; y > y0 / 2; x < x0 / 2 - w / 2

[0072] Where y is the vertical coordinate of part C and x is the horizontal coordinate of part C.

[0073] The coordinate values ​​of part D satisfy the following conditions: y / x < y0 / x0; y / x - x0 < y0 / -x0; y > y0 / 2; x > x0 / 2 + w / 2

[0074] Where y is the vertical coordinate value of part D, and x is the horizontal coordinate value of part D.

[0075] The coordinate values ​​of part E satisfy the following conditions: y / x > y0 / x0; y / x - x0 > y0 / -x0; y < y0 / 2; x < x0 / 2 - w / 2

[0076] Where y is the vertical coordinate value of part E, and x is the horizontal coordinate value of part E.

[0077] The coordinate values ​​of part F satisfy the following conditions: y / x < y0 / x0; y / x - x0 < y0 / -x0; y < y0 / 2; x > x0 / 2 + w / 2

[0078] Where y is the vertical coordinate value of part F, and x is the horizontal coordinate value of part F.

[0079] The coordinates of the panoramic area satisfy the following conditions: x≥x0 / 2-w / 2; x≤x0 / 2+w / 2; y≥y0 / 2-h / 2; y≤y0 / 2+h / 2;

[0080] Where y is the vertical coordinate of the panoramic area and x is the horizontal coordinate of the panoramic area.

[0081] It is understood that the settings for the area displaying the projected image and the panoramic area in the first region described above are merely a specific example. In practical applications, there may be other ways to set the area displaying the projected image and the panoramic area, which are not specifically limited here.

[0082] As shown in Figure 4, since the car icon displayed in the panoramic area has the front of the car facing down and the rear facing up, for ease of correspondence, part A can be assigned to the image acquisition device directly in front, part B to the image acquisition device directly behind, part C to the image acquisition device at the upper left, part D to the image acquisition device at the upper right, part E to the image acquisition device at the lower left, and part F to the image acquisition device at the lower right. It should be understood that the above correspondence between parts and image acquisition devices is merely a specific example. In practical applications, the correspondence between multiple parts and multiple image acquisition devices can be different. For example, when the car icon displayed in the panoramic area has the front of the car facing up and the rear facing down, the correspondence between multiple parts and multiple image acquisition devices would be exactly the opposite. This is not specifically limited here.

[0083] The number of parts in a projected image can be set according to the number of image acquisition devices. Since there are six image acquisition devices in the example above, the number of parts in the projected image is also set to six. In practical applications, the number of parts in the projected image can increase or decrease accordingly as the number of image acquisition devices increases or decreases. Furthermore, in the example above, the number of image acquisition devices and the number of parts in the projected image are equal. In practical applications, the number of image acquisition devices can be greater than the number of parts in the projected image. For example, when the image acquisition devices include six visible light cameras and one infrared camera, and the infrared camera is used to enhance the image acquired by the visible light cameras, the number of parts in the projected image can be set to six.

[0084] S105: The terminal device receives the first viewing operation input by the user in the first part.

[0085] The first part comprises multiple parts of the projected image. Continuing with the example shown in Figure 4, since the multiple parts include parts A, B, C, D, E, and F, the first part can be any one or more of parts A, B, C, D, E, and F. For example, the first part can be part A, or it can be both part C and part D.

[0086] The first viewing operation can be performed at any location within the first part. For example, when a user inputs with a mouse, it could be a left-click, right-click, left-double-click, right-double-click, or swipe operation at any location within the first part. When a user inputs with a finger, it could be a single-click, double-click, zoom-in, or knuckle-tapping operation at any location within the first part. When the first part is a portion of the projected image, the first viewing operation is any of the above operations input by the user within that portion. For example, when the first part is part A, the first viewing operation is a swipe operation input by the user within part A. When there are multiple first parts, the first viewing operation is a swipe operation input by the user simultaneously within each of the projected image regions displayed. For example, when the first parts are parts C and D, the first viewing operation is a single-click operation input by the user simultaneously within parts C and part D.

[0087] S106: The terminal device sends a first viewing request to the vehicle based on the first viewing operation. Accordingly, the vehicle receives the first viewing request sent by the terminal device based on the first viewing operation.

[0088] The terminal device sends a first viewing request to the vehicle based on a first viewing operation. The first viewing request is used to request the vehicle to view the image captured by the image acquisition device corresponding to the first part where the first viewing operation is located. Specifically,

[0089] When a user inputs a first operation in part A of the first area, the terminal device sends a first viewing request to the vehicle based on the first viewing operation. The first viewing request is used to request the vehicle to view the image captured by the image acquisition device directly in front of part A.

[0090] When a user inputs a first operation in part B of the first area, the terminal device sends a first viewing request to the vehicle based on the first viewing operation. The first viewing request is used to request the vehicle to view the image captured by the image acquisition device directly behind part B.

[0091] When the user inputs the first operation in section C of the first area, the terminal device sends a first viewing request to the vehicle based on the first viewing operation. The first viewing request is used to request the vehicle to view the image captured by the image acquisition device in the upper left corner corresponding to section C.

[0092] When a user inputs a first operation in part D of the first area, the terminal device sends a first viewing request to the vehicle based on the first viewing operation. The first viewing request is used to request the vehicle to view the image captured by the image acquisition device in the upper right corner corresponding to part D.

[0093] When a user inputs a first operation in part E of the first area, the terminal device sends a first viewing request to the vehicle based on the first viewing operation. The first viewing request is used to request the vehicle to view the image captured by the image acquisition device in the lower left corner of part E.

[0094] When a user inputs a first operation in section F of the first area, the terminal device sends a first viewing request to the vehicle based on the first viewing operation. The first viewing request is used to request the vehicle to view the image captured by the image acquisition device in the lower right corner corresponding to section F.

[0095] In the examples above, the first part is a projection image. In practical applications, when there are multiple first parts, the first viewing request is used to request the vehicle to view the images captured by the image acquisition devices corresponding to these multiple parts. For example, when the user simultaneously inputs the first viewing operation in parts C and D, the first viewing request is used to request the vehicle to view the images captured by the image acquisition device in the upper left corner of part C and the image acquisition device in the upper right corner of part D.

[0096] S107: The vehicle generates a first view response based on the first view request.

[0097] The vehicle generates a first viewing response based on the first viewing request, wherein the first viewing response includes an image captured by a first image acquisition device corresponding to the first part. The number of first image acquisition devices can be one or more, and a first image acquisition device can belong to multiple image acquisition devices. Taking the example shown in Figure 2, the first image acquisition device can be one or more of the following: a frontal image acquisition device, a rearal image acquisition device, an upper left image acquisition device, an upper right image acquisition device, a lower left image acquisition device, and a lower right image acquisition device.

[0098] In one specific embodiment, the vehicle generating a first view response based on the first view request can specifically be:

[0099] When the first viewing request is used to request the vehicle to view the image captured by the image acquisition device directly in front of part A, the vehicle generates a first viewing response based on the first viewing request, wherein the first viewing response includes the image captured by the image acquisition device directly in front;

[0100] When the first viewing request is used to request the vehicle to view the image captured by the image acquisition device directly behind Part B, the vehicle generates a first viewing response based on the first viewing request, wherein the first viewing response includes the image captured by the image acquisition device directly behind.

[0101] When the first viewing request is used to request the vehicle to view the image captured by the image acquisition device in the upper left corner corresponding to section C, the vehicle generates a first viewing response based on the first viewing request, wherein the first viewing response includes the image captured by the image acquisition device in the upper left corner;

[0102] When the first viewing request is used to request the vehicle to view the image captured by the image acquisition device in the upper right corner corresponding to section D, the vehicle generates a first viewing response based on the first viewing request, wherein the first viewing response includes the image captured by the image acquisition device in the upper right corner;

[0103] When the first viewing request is used to request the vehicle to view the image captured by the image acquisition device in the lower left corner of section E, the vehicle generates a first viewing response based on the first viewing request, wherein the first viewing response includes the image captured by the image acquisition device in the lower left corner;

[0104] When a first viewing request is used to request the vehicle to view the image captured by the image acquisition device in the lower right corner corresponding to section F, the vehicle generates a first viewing response based on the first viewing request, wherein the first viewing response includes the image captured by the image acquisition device in the lower right corner.

[0105] In the examples above, the first part is a projection image. In practical applications, when there are multiple first parts, the first viewing response includes the images captured by the image acquisition devices corresponding to these multiple parts. For example, when the first viewing request is used to request the vehicle to view the images captured by the image acquisition devices in the upper left and upper right corners, the first viewing response includes the images captured by the image acquisition devices in the upper left and upper right corners.

[0106] S108: The vehicle sends a first viewing response to the terminal device. Accordingly, the terminal device receives the first viewing response sent by the vehicle.

[0107] S109: The terminal device displays the image acquired by the first image acquisition device on the terminal interface.

[0108] The terminal device displays the image captured by the first image acquisition device on its terminal interface. The terminal device can display the image captured by the first image acquisition device in either a first area or a second area of ​​the terminal interface. There can be one or more second areas. When there is only one second area, it can occupy the entire terminal interface or only a portion of it. When there are multiple second areas, they can be combined to occupy the entire terminal interface or to occupy only a portion of it.

[0109] Optionally, if the projected image of the second part includes risk warning icons and / or obstacles, it indicates that there may be a risk at that location. Furthermore, the second part can be displayed using a highlighting method. Highlighting refers to making the second part more visually prominent and noticeable through a specific means to attract attention. Common highlighting methods include color highlighting, flashing, or dynamic effects. Color highlighting uses a bright color different from other parts to highlight content, while flashing or dynamic effects make the second part blink or have dynamic changes to attract attention. Users can input a second viewing operation in the second part to obtain an image captured by the corresponding image acquisition device, facilitating careful observation of the location. Obstacles can be static or dynamic, such as roadblocks or other vehicles, and can be living or non-living things, such as animals or utility poles, etc., without specific limitations. After the user inputs a second viewing operation in the second part of the projected image, the terminal device sends a second viewing request to the vehicle. The second part can be any one or more parts of the projected image other than the first part. Continuing with the example shown in Figure 4, since the projected image comprises multiple parts including part A, part B, part C, part D, part E, and part F, and the first part is part A, the second part can be any one or more of parts B, C, D, E, and F. For example, the second part can be part B, or it can be part E and part F. After receiving the second viewing request, the vehicle generates a second viewing response based on the second viewing request. The second viewing operation can be performed at any location within the second part. The second viewing request is used to request the vehicle to view the image captured by the image acquisition device corresponding to the second part where the second viewing operation is located. The vehicle sends the second viewing response to the terminal device. The second viewing response includes the image captured by the second image acquisition device corresponding to the second part. After receiving the second viewing response, the terminal device replaces the image captured by the first image acquisition device with the image captured by the second image acquisition device. For simplicity, the detailed process is not described here; for specific implementation details, please refer to steps S105 to S109. The above solution is illustrated by the example of a user entering the second viewing operation in the second part only after seeing the risk warning icon in the second part. In actual application, it is not necessary for the user to see the risk warning icon in the second part before entering the second viewing operation. As long as the user wants to observe the image captured by the image acquisition device in the second part in detail, they can enter the second viewing operation in the second part.

[0110] Optionally, when a user inputs a full-screen operation on the terminal interface, the terminal device displays the projected image in full-screen mode on the terminal interface. Specific implementation methods can include two: In the first method, the user needs to click on the panoramic area of ​​the first region. The terminal device responds to the user's click on the panoramic area and displays the projected image in full-screen mode on the terminal interface. In the second method, the user needs to zoom in on the first region. The terminal device responds to the user's zoom-in operation on the first region and displays the projected image in full-screen mode on the terminal interface. It is understood that the above two methods are merely specific examples. In practical applications, other methods can also be used to implement this. For example, pressing the volume up button on the terminal device can achieve full-screen display of the projected image on the terminal interface; this is not specifically limited here.

[0111] Referring to Figure 5, which is a structural schematic diagram of a terminal device provided in this application, the terminal device of this application includes:

[0112] The receiving module 121 is used to receive panoramic images sent by the vehicle, wherein the panoramic images are synthesized from images captured by multiple image acquisition devices of the vehicle;

[0113] Display module 122 is used to display a projection image of the panoramic image in a first area of ​​the terminal interface, wherein the projection image includes multiple parts, and there is a one-to-one correspondence between the multiple parts and the multiple image acquisition devices;

[0114] Sending module 123 is configured to send a first viewing request to the vehicle in response to a first viewing operation input by a user in the first part, wherein the first part belongs to the plurality of parts;

[0115] The receiving module is further configured to receive a first viewing response returned by the vehicle based on the first viewing request, wherein the first viewing response includes an image captured by a first image acquisition device corresponding to the first part, and the first image acquisition device belongs to the plurality of image acquisition devices;

[0116] The image acquired by the first image acquisition device is displayed on the terminal interface.

[0117] The receiving module 121, display module 122, and transmitting module 123 can all be implemented in software or in hardware. For example, the implementation of the receiving module 121 will be described below. Similarly, the implementation of the display module 122 and the transmitting module 123 can refer to the implementation of the receiving module 121.

[0118] As an example of a software functional unit, the receiving module 121 may include code running on a computing instance. The computing instance may include at least one of a physical host (computing device), a virtual machine, or a container. Further, the aforementioned computing instance may be one or more. For example, the receiving module may include code running on multiple hosts / virtual machines / containers. It should be noted that the multiple hosts / virtual machines / containers used to run the code may be distributed in the same region or in different regions. Further, the multiple hosts / virtual machines / containers used to run the code may be distributed in the same availability zone (AZ) or in different AZs, each AZ including one or more geographically proximate data centers. Typically, a region may include multiple AZs.

[0119] Similarly, multiple hosts / virtual machines / containers used to run this code can be distributed within the same Virtual Private Cloud (VPC) or across multiple VPCs. Typically, a VPC is set up within a region. Communication between two VPCs within the same region, as well as between VPCs in different regions, requires a communication gateway to be set up within each VPC to enable interconnection between VPCs.

[0120] As an example of a hardware functional unit, the receiving module 121 may include at least one computing device, such as a server. Alternatively, the receiving module may be implemented using a central processing unit (CPU), an application-specific integrated circuit (ASIC), or a programmable logic device (PLD). The PLD may be implemented using a complex programmable logical device (CPLD), a field-programmable gate array (FPGA), a generic array logic (GAL), a data processing unit (DPU), a neural network processing unit (NPU), a system-on-chip (SoC), an offload card, an accelerator card, or any combination thereof.

[0121] The multiple computing devices included in the receiving module 121 can be distributed in the same region or in different regions. Similarly, the multiple computing devices included in the receiving module can be distributed in the same Availability Zone (AZ) or in different AZs. Likewise, the multiple computing devices included in the receiving module 121 can be distributed in the same Virtual Private Cloud (VPC) or in multiple VPCs. These multiple computing devices can be any combination of computing devices such as servers, ASICs, PLDs, CPLDs, FPGAs, GALs, DPUs, NPUs, SoCs, offloading cards, and accelerator cards.

[0122] It should be noted that in other embodiments, the receiving module 121 can be used to execute any step in the monitoring method, the display module can be used to execute any step in the monitoring method, and the sending module can be used to execute any step in the monitoring method. The steps implemented by the receiving module, the display module, and the sending module can be specified as needed. The receiving module 121, the display module 122, and the sending module 123 respectively implement different steps in the monitoring method to realize all the functions of the terminal device.

[0123] The internal structure of a vehicle will be described in detail below, taking it as an example. Referring to Figure 6, which is an exemplary structural block diagram of a vehicle provided in this application, as shown in Figure 6, components coupled to or included in the vehicle may include a power system 202, a sensor system 204, a control system 206, peripheral devices 208, a power supply 210, a computing system 211, a transceiver 215, and a user interface 212.

[0124] It is understood that the vehicle shown in Figure 6 is merely an example. Optionally, in other examples, the vehicle may include more, fewer, or different systems, and each system may include more, fewer, or different components. Furthermore, the systems and components shown can be combined or divided in any manner, and this application does not limit this. The components and systems of the vehicle can be communicatively linked together via system buses, networks, and / or other connection mechanisms.

[0125] Vehicle components can be configured to operate in a manner that interconnects with each other and / or with other components coupled to various systems. For example, power supply 210 can provide power to all components of the vehicle. Computing system 211 can be configured to receive data from and control the powertrain 202, sensor system 204, control system 206, and peripheral devices 208.

[0126] The power system 202, also known as the drive system, is used to provide power to the vehicle. The power system 202 includes components that provide power to move the vehicle. As shown in Figure 6, the power system 202 includes an engine / motor 218, an energy source 220, a transmission 222, and wheels / tires 224.

[0127] Engine 218 can be any combination of internal combustion engines, electric motors, Stirling engines or other engines, such as a hybrid engine consisting of a gasoline engine and an electric motor, or a hybrid engine consisting of an internal combustion engine and an air compression engine.

[0128] Energy source 220 may supply all or part of the power to engine / moleculars 218, meaning that energy source 220 can be converted into mechanical energy by engine / moleculars 218. Energy source 220 may be, for example, gasoline, diesel, other petroleum-based fuels, propane, other compressed gas-based fuels, ethanol, solar panels, batteries, or other sources of electricity. In some examples, energy source 220 may also provide energy to other systems of the vehicle.

[0129] The transmission 222 can be used to transmit mechanical power from the engine / motor 218 to the wheels / tires 224. For this purpose, the transmission 222 may include a gearbox, a differential, and a drive shaft. In some examples, the transmission 222 may also include a clutch. The drive shaft may include one or more shafts that can be coupled to one or more wheels / tires 224.

[0130] Sensor system 204 includes several sensors for sensing information about the vehicle's environment. As shown in FIG6, sensor system 204 includes a positioning system 226 (e.g., the positioning system may be a Global Positioning System (GPS), a BeiDou system, or other positioning systems), an inertial measurement unit (IMU) 228, a radar 230, a laser rangefinder 232, and a camera 234. In one possible embodiment, sensor system 204 may also include sensors for monitoring vehicle steering, such as yaw rate sensors, lateral acceleration sensors, and steering wheel angle sensors; sensors for monitoring vehicle braking, such as pedal displacement sensors; and sensors for monitoring the vehicle's internal systems, such as fuel gauges, oil temperature gauges, and in-vehicle air quality monitors.

[0131] The positioning system 226 is used to locate the geographical location of the vehicle so as to obtain the vehicle's location coordinates in real time.

[0132] The inertial measurement unit 228 is used to sense changes in the vehicle's position and orientation based on information such as inertial acceleration. In one possible embodiment, the inertial measurement unit 228 may include an accelerometer, a gyroscope, or other sensors. For example, the inertial measurement unit 228 may be used to measure at least one of the vehicle's speed, lateral acceleration, yaw rate, etc.

[0133] Radar 230 can use radio signals to sense objects in the surrounding environment of the intelligent vehicle. In some embodiments, in addition to sensing objects, radar 230 can also be used to sense the speed, direction of travel, height, etc. of objects.

[0134] The laser rangefinder 232 uses lasers to detect objects or measure distances to objects and other information, and includes components such as a laser source, a laser scanner, and receiver electronics.

[0135] Camera 234 can be any camera used to acquire images of the vehicle's surroundings, such as an infrared camera, a 3D camera, etc.

[0136] The control system 206 is used to control the vehicle's motion state. The control system 206 may include a steering unit 236, a throttle 238, and a braking unit 240. The steering unit 236 is used to adjust the vehicle's left and right steering to control its forward direction; for example, the steering unit may be a steering wheel system. The throttle 238 is used to control the operating speed and acceleration of the engine / propeller 218, thereby changing the vehicle's speed. The braking unit 240 is used to decelerate the vehicle. In one example, the braking unit 240 can decelerate the vehicle by increasing the friction between the wheels / tires 224 and the ground.

[0137] In some possible embodiments, the control system 206 may also implement functions for maintaining vehicle stability, such as anti-lock braking system (ABS), traction control system (TCS), electronic stability control system (ESC), and functions for automatic obstacle avoidance, such as automatic emergency braking (AEB) and automatic emergency steering (AES), etc., without being specifically limited here.

[0138] Here, the computing system 211 can analyze information about itself and objects in the environment based on data from various sensors in the sensor system 204, such as its own driving speed, steering wheel speed, brake pedal depressing rate, lateral acceleration, yaw rate, etc., and issue corresponding instructions to the control system 206 based on the analysis results, such as applying braking force, steering, adjusting resistance torque, adjusting steering assist torque, adjusting steer-by-wire ratio, or adjusting road feel feedback torque, etc. The control system 206 controls the corresponding hardware to perform left and right steering and / or acceleration and deceleration to achieve obstacle avoidance operations such as turning and braking based on the received instruction information.

[0139] Peripheral device 208 may be configured to allow the vehicle to interact with external sensors, other vehicles, and / or users. For this purpose, peripheral device 208 may include, for example, a touchscreen 242, a microphone 244, a speaker 246, and status lights 248. The touchscreen 242 may be used by a user to input commands to vehicle 20, the microphone 244 may be configured to receive audio from a user in the vehicle, the speaker 246 may be configured to output audio to a user in the vehicle, and the status lights 248 may be used to characterize the vehicle's driving status and the functional status of various devices within the vehicle. It is understood that peripheral device 208 may additionally or alternatively include components other than those shown.

[0140] Transceiver 215 may include antennas and chipsets for communicating directly or via an air interface with other vehicles, roadside units, sensors, or other entities. Transceiver 215 may be configured to receive and transmit information according to one or more other types of wireless communications (e.g., protocols), such as Bluetooth, IEEE 802.11, cellular technology, Worldwide Interoperability for Microwave Access (WiMAX) or Long Term Evolution (LTE), ZigBee, Dedicated Short Range Communications (DSRC), and Radio Frequency Identification (RFID) communications, etc.

[0141] Power source 210 is used to provide power to some or all of the vehicle's components, and may be a rechargeable lithium-ion or lead-acid battery. One or more such battery packs may be configured to provide power to various components of the vehicle. In some embodiments, power source 210 and energy source 220 may be implemented together, as is the case in some fully electric vehicles.

[0142] User interface 212 is used for users to input information into the vehicle, enabling interaction between the user and the vehicle.

[0143] The computing system 211 further includes a processor 213 and a memory 214.

[0144] The processor 213 may include one or more general-purpose processors and / or one or more special-purpose processors (e.g., image processors, digital signal processors, etc.). The memory 214 may include one or more volatile memory components and / or one or more non-volatile memory components, such as optical, magnetic, and / or organic storage devices. In some possible embodiments, the memory 214 may exist alone, or the memory 214 may be wholly or partially integrated with the processor 213.

[0145] The memory 214 can be used to store program code and data. The program code can be executed by the processor 213 to implement various functions of the vehicle. The program code includes code for coordinating steering and braking to assist the driver in obstacle avoidance, as well as code for sending data to, receiving data from, interacting with, and / or controlling one or more of the aforementioned power system 202, sensor system 204, control system 206, and peripheral devices 208. The data stored in the memory 214 includes detection data from various sensors in the vehicle, such as yaw rate output by the yaw rate sensor, lateral acceleration output by the lateral acceleration sensor, and steering wheel rotation speed output by the steering wheel angle sensor. It also includes data after analysis and processing based on the detection data in the vehicle, such as the vehicle speed obtained from the IMU, the brake pedal depressor rate obtained from the pedal displacement sensor, and the vehicle's braking force.

[0146] In this embodiment, processor 213 is configured to call program code and data in memory 214 to execute the steps performed by the vehicle in the embodiment shown in FIG6.

[0147] The internal structure of a smartphone will be described in detail below, taking the smartphone as an example. Refer to Figure 7, which is an exemplary structural block diagram of a smartphone provided in this application. As shown in Figure 7, the smartphone may include a processor 310, an external memory interface 320, a universal serial bus (USB) interface 330, a charging management module 340, a mobile communication module 350, a wireless communication module 360, an audio module 370, a sensor module 380, and a display screen 390.

[0148] It is understood that the structures illustrated in the embodiments of the present invention do not constitute components of a smartphone, or combine certain components, separate certain components, or arrange different components. The illustrated components can be implemented in hardware, software, or a combination of software and hardware.

[0149] Processor 310 may include one or more processing units, such as an application processor (AP), a modem processor, a graphics processing unit (GPU), an image signal processor (ISP), a controller, memory, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural network processing unit (NPU). Different processing units may be independent devices or integrated into one or more processors. In some embodiments, a smartphone may also include one or more processors 310. Processor 310 may also include memory for storing instructions and data. In some embodiments, the memory in processor 310 is a cache memory. This memory can store instructions or data that the processor 310 has just used or is reusing. If the processor 310 needs to reuse the instruction or data, it can directly retrieve it from the memory. This avoids repeated access, reduces the waiting time of the processor 310, and thus improves the efficiency of the smartphone.

[0150] The external storage interface 320 can be used to connect an external memory card, such as a Micro SD card, to expand the storage capacity of the smartphone. The external memory card communicates with the processor 310 through the external storage interface 320 to perform data storage functions. For example, music, photos, videos, and other data can be stored on the external memory card.

[0151] The USB 330 port is a USB standard compliant interface, which can be a Mini USB port, Micro USB port, USB Type-C port, etc. The USB 330 port can be used to connect a charger to charge a smartphone, and can also be used for data transfer between a smartphone and peripheral devices. It can also be used to connect headphones for audio playback. This port can also be used to connect other smartphones, such as AR devices.

[0152] The charging management module 340 receives charging input from a charger. The charger can be a wireless charger or a wired charger. In some wired charging embodiments, the charging management module 340 receives charging input from the wired charger via a USB interface 330. In some wireless charging embodiments, the charging management module 340 receives wireless charging input via the wireless charging coil of the smartphone. While charging the battery, the charging management module 340 can also supply power to the smartphone via the power management module.

[0153] The mobile communication module 350 can provide solutions for wireless communication applications in smartphones, including 2G / 3G / 4G / 5G. The mobile communication module 350 may include at least one filter, switch, power amplifier, low noise amplifier (LNA), etc. The mobile communication module 350 can receive electromagnetic waves via an antenna, and perform filtering, amplification, and other processing on the received electromagnetic waves before transmitting them to a modem processor for demodulation. The mobile communication module 350 can also amplify the signal modulated by the modem processor and convert it into electromagnetic waves for radiation via the antenna. In some embodiments, at least some functional modules of the mobile communication module 350 may be housed in the processor 310. In some embodiments, at least some functional modules of the mobile communication module 350 and at least some modules of the processor 310 may be housed in the same device. The modem processor may include a modulator and a demodulator. The modulator is used to modulate a low-frequency baseband signal to be transmitted into a mid-to-high frequency signal. The demodulator is used to demodulate the received electromagnetic wave signal into a low-frequency baseband signal. The demodulator then transmits the demodulated low-frequency baseband signal to the baseband processor for processing. The low-frequency baseband signal, after being processed by the baseband processor, is transmitted to the application processor. The application processor outputs sound signals through audio devices (not limited to speakers, receivers, etc.) or displays images or videos through the display screen 390. In some embodiments, the modem processor may be a separate device. In other embodiments, the modem processor may be independent of the processor 310 and may be housed in the same device as the mobile communication module 350 or other functional modules.

[0154] The wireless communication module 360 ​​can provide solutions for wireless communication applications in smartphones, including wireless local area networks (WLAN) (such as wireless fidelity (Wi-Fi) networks), Bluetooth (BT), global navigation satellite system (GNSS), frequency modulation (FM), near field communication (NFC), and infrared (IR) technologies. The wireless communication module 360 ​​can be one or more devices integrating at least one communication processing module. The wireless communication module 360 ​​receives electromagnetic waves via an antenna, modulates and filters the electromagnetic wave signals, and sends the processed signal to the processor 310. The wireless communication module 360 ​​can also receive signals to be transmitted from the processor 310, modulate and amplify them, and then convert them into electromagnetic waves for radiation via the antenna. For example, the wireless communication module 360 ​​may include a Bluetooth module, a Wi-Fi module, etc.

[0155] In some embodiments, a smartphone can communicate with networks and other devices via wireless communication technologies. These wireless communication technologies may include Global System for Mobile Communications (GSM), General Packet Radio Service (GPRS), Code Division Multiple Access (CDMA), Wideband Code Division Multiple Access (WCDMA), Time-Division Code Division Multiple Access (TD-SCDMA), Long Term Evolution (LTE), BT, GNSS, WLAN, NFC, FM, and / or IR technologies. The GNSS may include Global Positioning System (GPS), Global Navigation Satellite System (GLONASS), BeiDou Navigation Satellite System (BDS), Quasi-Zenith Satellite System (QZSS), and / or Satellite Based Augmentation Systems (SBAS).

[0156] In some embodiments, the Bluetooth (BT) module and WLAN module included in the wireless communication module 360 ​​can transmit signals to detect or scan for devices near the smartphone, enabling the smartphone to discover nearby devices using wireless communication technologies such as Bluetooth or WLAN, establish wireless communication connections with these devices, and share data with them through these connections. The Bluetooth (BT) module can provide solutions for one or more Bluetooth communication methods, including classic Bluetooth (Bluetooth 2.1) or Bluetooth Low Energy (BLE). The WLAN module can provide solutions for one or more WLAN communication methods, including Wi-Fi Direct, Wi-Fi LAN, or Wi-Fi SoftAP.

[0157] In some embodiments, the wireless communication solution provided by the mobile communication module 350 enables the smartphone to communicate with devices (such as servers) in the network, and the WLAN wireless communication solution provided by the wireless communication module 360 ​​also enables the smartphone to communicate with devices (such as servers) in the network, and through these devices (such as servers), to communicate with cloud devices. In this way, the smartphone can discover cloud devices and transmit data to them. The smartphone implements display functions through a GPU, a display screen 390, and an application processor. The GPU is a microprocessor for image processing, connecting the display screen 390 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. The processor 310 may include one or more GPUs, which execute instructions to generate or modify display information.

[0158] The audio module 370 is used to convert digital audio information into analog audio signal output, and also to convert analog audio input into digital audio signal. The audio module 370 can also be used for encoding and decoding audio signals. In some embodiments, the audio module 370 may be located in the processor 310, or some functional modules of the audio module 370 may be located in the processor 310.

[0159] The sensor module 380 may include one or more of the following: pressure sensor, gyroscope sensor, accelerometer sensor, barometric pressure sensor, magnetic sensor, distance sensor, proximity sensor, temperature sensor, touch sensor, bone conduction sensor, etc.

[0160] Display screen 390 is used to display images, videos, etc. Display screen 390 includes a display panel. The display panel may be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), a miniature LED, a microLED, a quantum dot light-emitting diode (QLED), etc. In some embodiments, a smartphone may include one or N displays 190, where N is a positive integer greater than 1.

[0161] The processor 310 can execute the aforementioned instructions stored in the internal memory, thereby causing the smartphone to perform the steps executed by the terminal device in the embodiment shown in FIG2. The internal memory may include a program storage area and a data storage area. The program storage area may store the operating system; it may also store one or more applications (such as a gallery, contacts, etc.). The data storage area may store data created during smartphone use (such as photos, contacts, etc.). Furthermore, the internal memory may include high-speed random access memory and may also include non-volatile memory, such as at least one disk storage device, flash memory device, universal flash storage (UFS), etc.

[0162] This application also provides a computer program product containing instructions. The computer program product may be a software or program product containing instructions, capable of running on a computing device or stored on any usable medium. When the computer program product is run on at least one computing device, it causes the at least one computing device to perform the monitoring method shown in FIG2.

[0163] This application also provides a computer-readable storage medium. The computer-readable storage medium can be any available medium that a computing device can store, or a data storage device such as a data center that includes one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid-state drive). The computer-readable storage medium includes instructions that instruct the computing device to execute the monitoring method shown in FIG2.

[0164] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the protection scope of the technical solutions of the embodiments of this application.< / canvas> < / video> < / videoview> < / imgview> < / textview>

Claims

1. A monitoring method, characterized in that, Applied to terminal devices, including: Receive panoramic images sent by a vehicle, wherein the panoramic images are synthesized from images captured by multiple image acquisition devices of the vehicle; The first area of ​​the terminal interface displays a projection image of the panoramic image, wherein the projection image includes multiple parts, and there is a one-to-one correspondence between the multiple parts and the multiple image acquisition devices; In response to a user's first view operation input in the first part, a first view request is sent to the vehicle, wherein the first part belongs to the plurality of parts; Receive a first viewing response returned by the vehicle based on the first viewing request, wherein the first viewing response includes an image captured by a first image acquisition device corresponding to the first part, and the first image acquisition device belongs to the plurality of image acquisition devices; The terminal interface displays the image acquired by the first image acquisition device.

2. The method according to claim 1, characterized in that, After displaying the image acquired by the first image acquisition device on the terminal interface, the method further includes: In response to a second viewing operation input by the user in the second part, a second viewing request is sent to the vehicle, wherein the second part belongs to the plurality of parts; Receive a second viewing response returned by the vehicle based on the second viewing request, wherein the second viewing response includes an image captured by a second image acquisition device corresponding to the second part, and the second image acquisition device belongs to the plurality of image acquisition devices; Replace the image acquired by the first image acquisition device with the image acquired by the second image acquisition device.

3. The method according to claim 2, characterized in that, The method further includes: In response to a user's full-screen input on the terminal interface, the projected image is displayed in full-screen mode on the terminal interface.

4. The method according to claim 3, characterized in that, In response to a user's full-screen input on the terminal interface, the projected image is displayed in full-screen mode on the terminal interface, including: In response to a user's click operation in the panoramic area of ​​the first region, the projected image is displayed in full screen on the terminal interface.

5. The method according to claim 3, characterized in that, In response to a user's full-screen input on the terminal interface, the projected image is displayed in full-screen mode on the terminal interface, including: In response to the user's zoom-in operation in the first area, the projected image is displayed in full-screen mode on the terminal interface.

6. The method according to claim 2, characterized in that, The projected image includes obstacles and / or risk warning icons, which are located in a second part of the projected image and are displayed in a highlighted manner.

7. The method according to claim 6, characterized in that, The risk warning icon is also used to indicate one or more of the types of risks and the distance between the risks and the vehicle.

8. A terminal device, characterized in that, include: A receiving module is used to receive panoramic images sent by a vehicle, wherein the panoramic images are synthesized from images captured by multiple image acquisition devices of the vehicle; The display module is used to display a projected image of the panoramic image in a first area of ​​the terminal interface, wherein the projected image includes multiple parts, and there is a one-to-one correspondence between the multiple parts and the multiple image acquisition devices; A sending module is configured to send a first viewing request to the vehicle in response to a first viewing operation input by a user in the first part, wherein the first part belongs to the plurality of parts; The receiving module is further configured to receive a first viewing response returned by the vehicle based on the first viewing request, wherein the first viewing response includes an image captured by a first image acquisition device corresponding to the first part, and the first image acquisition device belongs to the plurality of image acquisition devices; The image acquired by the first image acquisition device is displayed on the terminal interface.

9. The device according to claim 8, characterized in that, The sending module is used to send a second viewing request to the vehicle in response to a second viewing operation input by the user in the second part, wherein the second part belongs to the plurality of parts; The receiving module is further configured to receive a second viewing response returned by the vehicle based on the second viewing request, wherein the second viewing response includes an image captured by a second image acquisition device corresponding to the second part, and the second image acquisition device belongs to the plurality of image acquisition devices; The display module is further configured to replace the image acquired by the first image acquisition device with the image acquired by the second image acquisition device.

10. The device according to claim 9, characterized in that, The display module is also used to respond to a full-screen operation input by the user on the terminal interface and to display the projected image in full-screen mode on the terminal interface.

11. The device according to claim 10, characterized in that, The display module is also used to respond to the user's click operation in the panoramic area of ​​the first area and display the projected image in full screen on the terminal interface.

12. The device according to claim 10, characterized in that, The display module is also used to respond to the user's zoom-in operation in the first area and display the projected image in full-screen mode on the terminal interface.

13. The device according to claim 9, characterized in that, The projected image includes obstacles and / or risk warning icons, which are located in a second part of the projected image and are displayed in a highlighted manner.

14. The device according to claim 13, characterized in that, The risk warning icon is also used to indicate one or more of the types of risks and the distance between the risks and the vehicle.

15. A monitoring system, characterized in that, include: Terminal equipment and vehicles, The vehicle is used to send panoramic images to the terminal, wherein the panoramic images are synthesized from images captured by multiple image acquisition devices of the vehicle; The terminal device is used to display a projected image of the panoramic image in a first area of ​​the terminal interface, wherein the projected image includes multiple parts, and there is a one-to-one correspondence between the multiple parts and the multiple image acquisition devices; The terminal device is used to send a first viewing request to the vehicle in response to a first viewing operation input by the user in the first part, wherein the first part belongs to the plurality of parts; The vehicle is used to send a first viewing response based on the first viewing request, wherein the first viewing response includes an image captured by a first image acquisition device corresponding to the first part, and the first image acquisition device belongs to the plurality of image acquisition devices; The terminal device is used to display the image acquired by the first image acquisition device on the terminal interface.

16. A terminal device, characterized in that, The terminal device includes a processor and a memory; the processor is configured to execute instructions stored in the memory to cause the terminal device to perform the operational steps of the method as described in any one of claims 1 to 7.

17. A computer program product containing instructions, characterized in that, When the instruction is executed by the terminal device cluster, the terminal device cluster causes the terminal device cluster to perform the operation steps of the method as described in any one of claims 1 to 7.

18. A computer-readable storage medium, characterized in that, The computing device includes program instructions, which, when executed by a terminal device, enable the computing device to perform the operational steps of the method as described in any one of claims 1 to 7.

Images