Data processing method, system and apparatus, and electronic device, storage medium and product

Abstract

A data processing method, system and apparatus, and an electronic device, a storage medium and a product. The method comprises: acquiring eye-tracking data of a user, and on the basis of the eye-tracking data, determining the field of view of the user; on the basis of the field of view, performing screening on a plurality of imaging apparatuses, so as to obtain at least one first imaging apparatus, wherein the first imaging apparatus is an imaging apparatus within the field of view; for any first imaging apparatus, acquiring first real-world image data collected by the first imaging apparatus; and on the basis of the first real-world image data respectively corresponding to each first imaging apparatus, generating a target real-world image. Therefore, unnecessary computing and data processing are avoided, such that the utilization efficiency of system resources is optimized, and the amount of data to be processed is reduced, thereby reducing the system latency, improving the overall performance, and ensuring a smooth user experience.

Description

Data processing methods, systems, devices, electronic equipment, storage media and products

[0001] This application claims priority to Chinese patent application filed on November 7, 2024, with application number 202411587212.4 and title "Data Processing Method, System, Apparatus, Electronic Device, Storage Medium and Product". Technical Field

[0002] This application relates to the field of virtual-real fusion technology, and in particular to a data processing method, system, device, electronic device, storage medium and product. Background Technology

[0003] Mixed reality technology can bring users an immersive experience, and with the continuous development of mixed reality technology, the application scenarios are becoming increasingly diversified. It not only improves the efficiency and quality of various industries, but also provides users with brand-new experiences and services, thus promoting social progress and development.

[0004] Because mixed reality systems require real-time capture, processing, and fusion of real-world scenes and virtual content, their performance and stability are crucial for user experience. However, issues such as large data volumes and high processing complexity lead to low frame rates, high latency, and poor image quality, impacting real-time performance and stability. Summary of the Invention

[0005] This application provides a data processing method, system, apparatus, electronic device, storage medium, and product for reducing data processing volume and increasing processing speed.

[0006] In a first aspect, embodiments of this application provide a data processing method applied to a virtual-real fusion device, the virtual-real fusion device including multiple imaging devices, the method comprising:

[0007] Acquire the user's eye-tracking data and determine the user's field of vision based on the eye-tracking data;

[0008] Based on the field of view, at least one first imaging device is selected from a plurality of imaging devices; the first imaging device is an imaging device located within the field of view.

[0009] For any first imaging device, acquire the first real-world image data collected by the first imaging device;

[0010] A target real-world image is generated based on the first real-world image data corresponding to each first camera.

[0011] Optionally, each imaging device includes multiple sensors, and for any first imaging device, the acquisition of first real-world image data collected by the first imaging device includes:

[0012] For any first imaging device, determine the imaging device viewpoint corresponding to the first imaging device and the user's viewpoint at the first imaging device.

[0013] Determine the overlapping angle between the imaging device's viewpoint and the user's viewpoint;

[0014] Multiple target sensors corresponding to the overlapping viewpoints are determined from the plurality of sensors;

[0015] Acquire first real-world image data, which is composed of data collected by the plurality of target sensors respectively.

[0016] Optionally, the method further includes:

[0017] Based on the field of view, determine the user's central field of view;

[0018] Based on the central field of view, at least one second imaging device is selected from the at least one first imaging device; the second imaging device is used to indicate the first imaging device that is within the central field of view.

[0019] Accordingly, for any first imaging device, acquiring the first real-world image data collected by the first imaging device includes:

[0020] For any second imaging device, acquire second real-world image data captured by the second imaging device at a first preset resolution and / or a first preset frame rate;

[0021] For any third imaging device, acquire third real-world image data acquired by the third imaging device at a resolution lower than the first preset resolution and / or at a frame rate lower than the first preset frame rate; the third imaging device is used to indicate the first imaging device that is not within the central field of view.

[0022] Optionally, the greater the deviation of the third imaging device from the central field of view, the lower the resolution and / or frame rate used by the third imaging device when acquiring third real-world image data.

[0023] Optionally, each imaging device includes multiple sensors, and each sensor contains multiple photoelectric elements. Based on the first real-world image data corresponding to each first camera, a target real-world image is generated, including:

[0024] Obtain the correspondence between the photoelectric components and each pixel in the target real-world image;

[0025] The target real image is generated based on the first real image data corresponding to each first imaging device and the corresponding relationship.

[0026] Optional, also includes:

[0027] Based on the correspondence and the first mapping relationship, the first coordinates of the target point corresponding to each pixel in the target real image in the real world are determined; wherein, the first mapping relationship is used to indicate the correspondence between the coordinates of the point in the real world and the photoelectric element;

[0028] Based on the target real-world image and the first coordinates of the target point corresponding to each pixel in the target real-world image in the real world, a virtual-real fusion image is generated, and the virtual-real fusion image is sent to the display screen for display.

[0029] Optionally, based on the target real-world image and the first coordinates of the target point corresponding to each pixel in the target real-world image in the real world, a virtual-real fusion image is generated, and the virtual-real fusion image is sent to the display screen for display, including:

[0030] Based on the target real image and the first coordinates of the target point corresponding to each pixel in the target real image in the real world, determine the virtual image and the second coordinates of the target point corresponding to each pixel in the virtual image in the real world;

[0031] Based on the first coordinate and the second coordinate, the virtual image and the real image are fused to obtain a virtual-real fused image, and the third coordinate of the target point corresponding to each pixel in the virtual-real fused image in the real world is determined;

[0032] Based on the third coordinates and the second mapping relationship, the display position of each pixel in the virtual-real fusion image on the display screen is determined; the second mapping relationship is used to indicate the correspondence between the display position on the display screen and the coordinates of the point in the real world.

[0033] The virtual-real fusion image and the corresponding display positions of each pixel in the virtual-real fusion image on the display screen are sent to the display screen for display.

[0034] Secondly, embodiments of this application provide a virtual-real fusion device, including:

[0035] Processing apparatus, the processing apparatus being used in the method described in the first aspect;

[0036] An eye-tracking device, wherein the eye-tracking device is used to collect eye-tracking data and send it to the processing device;

[0037] A plurality of imaging devices, at least some of which are used to acquire first real-world image data and send it to the processing device.

[0038] Optional, also includes,

[0039] The display screen is used to display the virtual-real fusion image on the display screen after the virtual-real fusion image sent by the processing device is acquired;

[0040] An optical system, having periodic optical characteristics, is used to process the light emitted from the display screen so that the processed light forms a stereoscopic image on the user's eyeball.

[0041] Optionally, the optical system includes a plurality of optical elements, wherein any one of the plurality of optical elements corresponds to at least one imaging device, or any one of the plurality of imaging devices corresponds to at least two optical elements.

[0042] Optionally, the plurality of optical elements, the plurality of imaging devices, and the display screen are all arranged in a planar structure.

[0043] Optionally, the plurality of optical elements, the plurality of imaging devices, and the display screen are all arranged in a curved structure.

[0044] Thirdly, embodiments of this application provide a data processing apparatus applied to a virtual-real fusion device, the virtual-real fusion device including multiple imaging devices, the apparatus comprising:

[0045] The first acquisition module is used to acquire the user's eye-tracking data and determine the user's field of vision based on the eye-tracking data;

[0046] A filtering module is used to filter at least one first imaging device from a plurality of imaging devices based on the field of view; the first imaging device is an imaging device located within the field of view.

[0047] The second acquisition module is used to acquire first real-world image data collected by the first imaging device for any one of the first imaging devices.

[0048] The generation module is used to generate target real-world images based on the first real-world image data corresponding to each first camera.

[0049] Fourthly, embodiments of this application provide an electronic device, including: a processor, and a memory communicatively connected to the processor;

[0050] The memory stores computer-executed instructions;

[0051] The processor executes computer execution instructions stored in the memory to implement the method described in the first aspect above.

[0052] Fifthly, embodiments of this application provide a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, are used to implement the method described in the first aspect above.

[0053] In a sixth aspect, embodiments of this application provide a computer program product, including a computer program that, when executed by a processor, implements the method described in the first aspect above.

[0054] The data processing method, system, apparatus, electronic device, storage medium, and product provided in this application include: acquiring eye-tracking data of a user and determining the user's visual field range based on the eye-tracking data; selecting at least one first imaging device from a plurality of imaging devices based on the visual field range; the first imaging device being an imaging device located within the visual field range; acquiring first real-world image data collected by the first imaging device for any given first imaging device; and generating a target real-world image based on the first real-world image data corresponding to each first imaging device. This avoids unnecessary calculations and data processing, optimizes the utilization efficiency of system resources, reduces the amount of data to be processed, thereby reducing system latency, improving overall performance, and ensuring a smooth user experience. Attached Figure Description

[0055] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0056] Figure 1 is an application scenario diagram provided by an embodiment of this application;

[0057] Figure 2 is a flowchart illustrating a data processing method provided in an embodiment of this application;

[0058] Figure 3 is a schematic diagram of a method for determining overlapping perspectives provided in an embodiment of this application;

[0059] Figure 4 is a flowchart illustrating another data processing method provided in an embodiment of this application;

[0060] Figure 5 is a schematic diagram illustrating the principle of a mapping relationship provided in an embodiment of this application;

[0061] Figure 6 is a schematic diagram of the arrangement of multiple optical elements and multiple imaging devices provided in an embodiment of this application;

[0062] Figure 7 is a schematic diagram of the structure of a data processing device provided in an embodiment of this application;

[0063] Figure 8 is a schematic diagram of the structure of an electronic device provided in an embodiment of this application.

[0064] The accompanying drawings illustrate specific embodiments of this application, which will be described in more detail below. These drawings and descriptions are not intended to limit the scope of the concept in any way, but rather to illustrate the concept of this application to those skilled in the art through reference to particular embodiments. Detailed Implementation

[0065] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.

[0066] Mixed reality (MR) technology offers users a highly immersive experience by seamlessly blending the virtual and real worlds, creating a more realistic and interactive environment. As MR technology continues to advance, its applications are becoming increasingly diverse. This not only significantly improves work efficiency and product quality across various industries but also brings unprecedented experiences and services to users. The widespread application of MR technology is driving social progress and development, opening up new possibilities and opportunities.

[0067] Because real-time capture, processing, and fusion of real-world scenes and virtual content are required, the performance and stability of mixed reality systems are crucial to user experience. Mixed reality systems need to achieve a good balance between real-time performance and stability. However, problems such as large data volume and high processing difficulty lead to low system frame rate, high latency, and poor image quality, which affect real-time performance and stability.

[0068] In view of this, this application provides a data processing method that can acquire a user's eye-tracking data and determine the user's field of vision based on the eye-tracking data; based on the field of vision, select at least one first imaging device from multiple imaging devices; the first imaging device is an imaging device within the field of vision; for any first imaging device, acquire first real-world image data collected by the first imaging device; and generate a target real-world image based on the first real-world image data corresponding to each first imaging device. In this way, by selecting the first imaging device within the user's field of vision from multiple imaging devices and then using the first real-world image data collected by the first imaging device to generate a virtual-real fusion image, unnecessary calculations and data processing are avoided, the utilization efficiency of system resources is optimized, the amount of data to be processed is reduced, thereby reducing system latency, improving overall performance, and ensuring a smooth user experience.

[0069] Figure 1 is an application scenario diagram provided by an embodiment of this application. As shown in Figure 1, the eye-tracking device is used to collect the user's eye-tracking data and send the eye-tracking data to the control module. The control module determines the user's field of vision based on the eye-tracking data. The determined field of vision is the area between the two rays in the figure. Based on the field of vision, at least one first imaging device is selected from multiple imaging devices. The multiple imaging devices are imaging device 1, imaging device 2, imaging device 3 and imaging device 4. The selected first imaging device is imaging device 2 and imaging device 3. First real image data 1 collected by imaging device 2 is acquired, first real image data 2 collected by imaging device 3 is acquired, and a target real image is generated using first real image data 1 and first real image data 2.

[0070] The technical solution of this application and how the technical solution of this application solves the above-mentioned technical problems are described in detail below with specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments. The embodiments of this application will now be described with reference to the accompanying drawings.

[0071] Figure 2 is a flowchart illustrating a data processing method provided in an embodiment of this application. The executing entity in this embodiment can be any device with data processing capabilities. This application uses the control module in a virtual-real fusion device as the executing entity for specific description. The virtual-real fusion device includes multiple imaging devices and can be any device capable of fusing real and virtual images. For example, it can be a head-mounted display device, augmented reality glasses, a projection system, etc. As shown in Figure 2, a data processing method provided in an embodiment of this application may include:

[0072] Step 201: Obtain the user's eye-tracking data and determine the user's field of vision based on the eye-tracking data.

[0073] The acquisition of eye-tracking data typically relies on eye-tracking devices, which can be dedicated eye-tracking glasses, eye-tracking modules on head-mounted displays (HMDs), or infrared cameras mounted around the display screen.

[0074] Specifically, the eye-tracking device collects the user's eye-tracking data and sends it to the control module. The control module determines the user's gaze point and pupil center position based on the acquired eye-tracking data, and calculates the gaze vector based on the pupil center and gaze point position. The gaze vector is the vector pointing from the pupil center to the gaze point. The visual field is a cone-shaped region with its apex at the pupil center and its axis along the direction of the gaze vector. The size of the visual field can be defined by the field of view (FOV), which is an angular measure representing the maximum range that an observer or camera device can see at a given moment. FOV is usually expressed in degrees and can be divided into horizontal FOV (HFOV), vertical FOV (VFOV), and diagonal FOV (DFOV).

[0075] Step 202: Based on the field of view, select at least one first imaging device from multiple imaging devices; the first imaging device is an imaging device located within the field of view.

[0076] The imaging device can be any device capable of performing imaging functions; for example, it can be a camera, a sensor, etc.

[0077] Specifically, the control module selects at least one first imaging device from multiple imaging devices based on the field of view unit. The first imaging device is the imaging device that is within the field of view.

[0078] Step 203: For any first imaging device, acquire the first real-world image data collected by the first imaging device.

[0079] Optionally, for any first imaging device, the acquisition of the first real-world image data collected by the first imaging device can be carried out in different ways, and this application does not limit this.

[0080] In one alternative implementation, the control module can shut down all imaging devices except the first imaging device among multiple imaging devices. The first imaging device acquires first real-world image data and sends the first real-world image data to the control module.

[0081] In another alternative implementation, the control module may not shut down the imaging devices other than the first imaging device among the multiple imaging devices. For each imaging device, the imaging device sends the acquired real-world image data to the control module. After obtaining the real-world image data acquired by the multiple imaging devices, the control module only performs subsequent processing on the first real-world image data acquired by the first imaging device, and does not perform subsequent processing on the real-world image data acquired by the imaging devices other than the first imaging device.

[0082] Optionally, each imaging device includes multiple sensors, and for any first imaging device, the acquisition of first real-world image data collected by the first imaging device includes:

[0083] For any first imaging device, determine the imaging device viewpoint corresponding to the first imaging device and the user's viewpoint at the first imaging device.

[0084] Determine the overlapping angle between the imaging device's viewpoint and the user's viewpoint;

[0085] Multiple target sensors corresponding to the overlapping viewpoints are determined from the plurality of sensors;

[0086] Acquire first real-world image data, which is composed of data collected by the plurality of target sensors respectively.

[0087] Figure 3 is a schematic diagram of a method for determining the overlapping angle provided in an embodiment of this application. As shown in Figure 3, for imaging device 1, the imaging device angle corresponding to imaging device 1 is the first included angle between two rays whose starting point is the focal point of the lens of imaging device 1 in Figure 3. The user's viewing angle at imaging device 1 is the second included angle between two rays whose starting point is the center of the pupil in Figure 3. The starting point of the overlapping angle is the focal point of the lens of imaging device 1. The intersection points of the two edge rays corresponding to the overlapping angle with imaging device 1 are the same as the intersection points of the two rays corresponding to the user's viewing angle with imaging device 1 in Figure 3. After determining the overlapping angle, the user determines multiple target sensors corresponding to the overlapping angle from multiple sensors.

[0088] Optionally, the control module may use different methods to acquire the first real-world image data, and this application does not limit this method.

[0089] In one feasible approach, all sensors except the target sensor can be turned off. Data collected by multiple target sensors is combined to form a first real-world image, which is then sent to the control module. Alternatively, data collected by multiple target sensors can be sent to the control module, which then combines the data to form the first real-world image.

[0090] In another possible implementation, all sensors except the target sensor can remain on. Multiple sensors collect data separately and send the collected data to the control module. The control module then filters out the data collected by the target sensor and combines the data collected by the target sensor into first real-world image data.

[0091] By determining the overlapping angle between the imaging device's viewpoint and the user's viewpoint, then identifying the target sensor corresponding to the overlapping angle, and combining the data collected by the target sensor into the first real-world image data, not only can the amount of subsequent computational data be reduced, but the problem of positional misalignment between the imaging device and the user's eye can also be solved. This allows for more realistic and natural perspective changes, thereby enhancing the realism of virtual reality and mixed reality experiences. The user's gaze direction directly affects the image content collected by the imaging device, making user interaction in the virtual environment more intuitive and natural. Furthermore, by dynamically adjusting the imaging device's viewpoint to match the user's gaze direction, user dizziness or discomfort can be reduced, improving the comfort of wearing the device.

[0092] Optionally, the data processing method provided in this application embodiment further includes:

[0093] Based on the field of view, determine the user's central field of view;

[0094] Based on the central field of view, at least one second imaging device is selected from the at least one first imaging device; the second imaging device is used to indicate the first imaging device that is within the central field of view.

[0095] Accordingly, for any first imaging device, acquiring the first real-world image data collected by the first imaging device includes:

[0096] For any second imaging device, acquire second real-world image data captured by the second imaging device at a first preset resolution and / or a first preset frame rate;

[0097] For any third imaging device, acquire third real-world image data acquired by the third imaging device at a resolution lower than the first preset resolution and / or at a frame rate lower than the first preset frame rate; the third imaging device is used to indicate the first imaging device that is not within the central field of view.

[0098] The user's central field of vision is typically a subset of their overall field of vision, concentrated in the central area of ​​their gaze. The specific angle of the central field of vision can be determined based on application requirements. Generally, a person's central field of vision is approximately 30 degrees both horizontally and vertically.

[0099] Specifically, when determining the user's central field of view based on the field of view range, the control module can define a cone-shaped region centered on the user's line of sight vector, with the angle of the cone being the angle of the central field of view range.

[0100] The control module selects at least one second imaging device from at least one first imaging device based on the central field of view. The second imaging device is the first imaging device that is within the central field of view, and the third imaging device is the first imaging device that is not within the central field of view.

[0101] For each second imaging device, the control module sends a first control command to the second imaging device. When the first control command is used to instruct the image data to be acquired at a first preset resolution, the second imaging device receives the first control command, acquires second real image data at the first preset resolution, and sends the acquired second real image data to the control module.

[0102] When the first control command is used to instruct image data acquisition at a first preset frame rate, the second imaging device receives the first control command, acquires second real-world image data at the first preset frame rate, and sends the acquired second real-world image data to the control module.

[0103] When the first control command is used to instruct image data acquisition at a first preset resolution and a first preset frame rate, the second imaging device receives the first control command, acquires second real-world image data at the first preset resolution and a first preset frame rate, and sends the acquired second real-world image data to the control module.

[0104] For each third imaging device, the control module sends a second control command to the third imaging device. When the second control command instructs the third imaging device to acquire image data at a resolution lower than the first preset resolution, the third imaging device receives the second control command and acquires third real image data at a resolution lower than the first preset resolution, and sends the acquired third real image data to the control module.

[0105] When the second control command instructs the image data to be acquired at a frame rate lower than the first preset frame rate, the third imaging device receives the second control command and acquires third real-world image data at a frame rate lower than the first preset frame rate, and sends the acquired third real-world image data to the control module.

[0106] When the second control command instructs the third imaging device to acquire image data at a resolution and frame rate lower than the first preset frame rate, the third imaging device acquires third real-world image data at a resolution and frame rate lower than the first preset resolution after receiving the second control command, and sends the acquired third real-world image data to the control module.

[0107] By selecting a second imaging device within the user's central field of view and acquiring image data at a higher preset resolution or frame rate, higher image quality can be ensured for the area the user is focusing on. This method provides a clearer and more detailed visual experience. By distinguishing between imaging devices within and outside the central field of view and acquiring image data at different resolutions or frame rates, computational and storage resources can be effectively utilized. High-resolution or high-frame-rate image data is acquired only in the area of ​​focus, while lower resolution or frame rates are used for peripheral areas. This reduces unnecessary high-resolution or high-frame-rate data processing and storage requirements, contributing to improved overall system performance and responsiveness, especially on resource-constrained devices such as mobile devices and embedded systems.

[0108] Optionally, the greater the deviation of the third imaging device from the central field of view, the lower the resolution and / or frame rate used by the third imaging device when acquiring third real-world image data.

[0109] Specifically, the control module can use different methods to determine the degree of deviation between the third imaging device and the central field of view. This application does not limit this method. For example, it can first determine the center point of the lens corresponding to the third imaging device, then determine the line connecting the center point of the lens and the center of the pupil, and finally determine the angle between the line and the user's line of sight vector. The larger the angle, the greater the degree of deviation.

[0110] The greater the deviation of the third imaging device from the central field of view, the lower the resolution and / or frame rate used by the third imaging device when acquiring third real-world image data.

[0111] By dynamically adjusting the resolution based on the degree of deviation, computing and storage resources can be utilized more effectively. Areas farther from the center of the field of view receive less user attention; therefore, using a lower resolution reduces unnecessary high-resolution data processing and storage requirements. Furthermore, dynamically adjusting the resolution significantly reduces the system's computational load and bandwidth demands. Lower-resolution image data is processed faster and consumes less bandwidth and storage space, thereby improving the overall system performance and responsiveness.

[0112] Step 204: Generate the target real image based on the first real image data corresponding to each first imaging device.

[0113] Specifically, the control module generates a target real-world image based on the first real-world image data corresponding to each of the first imaging devices. The target real-world image is obtained by stitching together images captured by multiple first imaging devices.

[0114] The data processing method provided in this application embodiment can acquire a user's eye-tracking data and determine the user's field of vision based on the eye-tracking data; based on the field of vision, at least one first imaging device is selected from multiple imaging devices; the first imaging device is an imaging device located within the field of vision; for any first imaging device, first real-world image data collected by the first imaging device is acquired; and a target real-world image is generated based on the first real-world image data corresponding to each first imaging device. This avoids unnecessary calculations and data processing, optimizes the utilization efficiency of system resources, reduces the amount of data to be processed, thereby reducing system latency, improving overall performance, and ensuring a smooth user experience.

[0115] Optionally, each imaging device includes multiple sensors, and each sensor contains multiple photoelectric elements. Based on the first real-world image data corresponding to each first imaging device, a target real-world image is generated, including:

[0116] Obtain the correspondence between the photoelectric components and each pixel in the target real-world image;

[0117] The target real image is determined based on the first real image data corresponding to each first imaging device and the corresponding relationship.

[0118] Among them, optoelectronic components are used to convert optical signals into electrical signals, thereby generating image data.

[0119] Specifically, the control module first obtains the correspondence between the photoelectric element and each pixel in the target real image. The target real image is an image obtained by stitching together images captured by multiple first imaging devices. Based on the first real image data and correspondence of each first imaging device, the target real image is determined. The first real image data includes data collected by multiple photoelectric elements.

[0120] In this way, by acquiring the correspondence between the photoelectric element and each pixel in the target real-world image, the system can achieve precise pixel mapping. This precision helps improve the accuracy and realism of the generated target real-world image. Optionally, the data processing method provided in this application further includes:

[0121] Based on the correspondence and the first mapping relationship, the first coordinates of the target point corresponding to each pixel in the target real image in the real world are determined; wherein, the first mapping relationship is used to indicate the correspondence between the coordinates of the point in the real world and the photoelectric element;

[0122] Based on the target real-world image and the first coordinates of the target point corresponding to each pixel in the target real-world image in the real world, a virtual-real fusion image is generated, and the virtual-real fusion image is sent to the display screen for display.

[0123] Among them, virtual-real fusion image refers to the fusion of virtual elements with images of the real world to create an image that contains both real environment and virtual objects.

[0124] Specifically, since the correspondence is between the photoelectric element and each pixel in the target real-world image, and the first mapping relationship is between the coordinates of a point in the real world and the photoelectric element, the control module can determine the first coordinates of the target point in the real world corresponding to each pixel in the target real-world image based on the correspondence and the first mapping relationship. The first mapping relationship is pre-stored in the control module or at a preset address. The control module first obtains the first mapping relationship from the preset address, and then determines the first coordinates of the target point in the real world corresponding to each pixel in the target real-world image based on the correspondence and the first mapping relationship.

[0125] The control module generates a virtual-real fusion image based on the target real image and the first coordinates of the target point corresponding to each pixel in the target real image in the real world, and sends the virtual-real fusion image to the display screen for display.

[0126] In this way, by pre-storing the first mapping relationship and determining the first coordinates of the target point in the real world for each pixel in the target real-world image based on the correspondence and the first mapping relationship, redundant calculations can be avoided during actual operation. This preprocessing method significantly reduces the burden of real-time computation, improves the system's efficiency and image processing speed, and because it reduces the complexity of real-time computation, the system can generate virtual-real fusion images faster and send them to the display screen for display, thus improving the system's real-time performance.

[0127] Optionally, based on the target real-world image and the first coordinates of the target point corresponding to each pixel in the target real-world image in the real world, a virtual-real fusion image is generated, and the virtual-real fusion image is sent to the display screen for display, including:

[0128] Based on the target real image and the first coordinates of the target point corresponding to each pixel in the target real image in the real world, determine the virtual image and the second coordinates of the target point corresponding to each pixel in the virtual image in the real world;

[0129] Based on the first coordinate and the second coordinate, the virtual image and the real image are fused to obtain a virtual-real fused image, and the third coordinate of the target point corresponding to each pixel in the virtual-real fused image in the real world is determined;

[0130] Based on the third coordinates and the second mapping relationship, the display position of each pixel in the virtual-real fusion image on the display screen is determined; the second mapping relationship is used to indicate the correspondence between the display position on the display screen and the coordinates of the point in the real world.

[0131] The virtual-real fusion image and the corresponding display positions of each pixel in the virtual-real fusion image on the display screen are sent to the display screen for display.

[0132] Specifically, scene segmentation, object recognition, and tracking are performed on the target real-world image. Based on the results, preset rules, and the first coordinates of the target points corresponding to each pixel in the target real-world image in the real world, the virtual image and the second coordinates of the target points corresponding to each pixel in the virtual image in the real world are determined.

[0133] The control module fuses the virtual image and the real image based on the first coordinate and the second coordinate to obtain a virtual-real fused image, and determines the third coordinate of the target point corresponding to each pixel in the virtual-real fused image in the real world.

[0134] Since the second mapping relationship is the correspondence between the display position on the screen and the coordinates of the point in the real world, the control module can determine the display position of each pixel in the virtual-real fusion image on the screen based on the third coordinate and the second mapping relationship. The second mapping relationship is pre-stored in the control module or at a preset address. The control module first obtains the second mapping relationship from the preset address, and then determines the display position of each pixel in the virtual-real fusion image on the screen based on the third coordinate and the second mapping relationship.

[0135] Finally, the control module sends the virtual-real fusion image and the corresponding display positions of each pixel in the virtual-real fusion image to the display screen for display.

[0136] In this way, by determining the coordinates of each pixel in the target real-world image and the virtual image in the real world—namely, the first coordinate and the second coordinate—precise alignment between the virtual and real images is achieved. This precise geometric mapping ensures seamless integration of virtual elements with the real-world scene, improving the realism and naturalness of the fused virtual-real image. By pre-storing the second mapping relationship, the complexity and burden of real-time computation are reduced, further improving the efficiency of image processing and enabling the system to generate and display fused virtual-real images more quickly.

[0137] Figure 4 is a flowchart illustrating another data processing method provided in an embodiment of this application, as shown in Figure 4:

[0138] Eye tracking: Using eye tracking technology or other sensors to monitor the user's eye movements in real time, determining the position of the user's pupil center and the position of the fixation point.

[0139] Determine the field of vision: Determine the user's field of vision based on the location of the fixation point and the location of the pupil center.

[0140] Imaging device selection: Select the imaging device within the field of view based on the field of view, and turn off the imaging device outside the field of view, or downsample it.

[0141] Data Acquisition: For the selected imaging device, image data acquisition is performed. This includes capturing the image data acquired by the imaging device and transmitting it to the control module for processing.

[0142] Data processing: The control module processes the acquired image data. This may include steps such as image correction, image stitching, disparity calculation, and depth estimation.

[0143] Data compositing: This involves combining processed image data to generate a final fused image. This may involve stitching or fusing image data acquired by different imaging devices to present a unified visual effect.

[0144] Output Image: The final composite image is output to a display device for user viewing. This can be a head-mounted display, video wall, or other display device.

[0145] Real-time updates: The data processing workflow is continuously updated based on the user's eye movement data. This includes adjusting the selection of the imaging device and the data acquisition method according to the visual field determined by the user's gaze point location to ensure that the user always observes the correct content.

[0146] This method allows for dynamic adjustment of the imaging device's operating state based on the center position of the human eye, thereby enabling effective control over the amount of data. This helps improve system efficiency and performance while reducing the demand for computing resources.

[0147] Figure 5 is a schematic diagram illustrating the principle of a mapping relationship provided in an embodiment of this application. As shown in Figure 5, the key problem addressed in this paper is to ensure that two points in the real world and the virtual world overlap at point A and appear as a single point on the human retina without any offset. An imaging device captures the real scene, while a display screen and lens array display the virtual scene. The real scene at point A is captured by the lens of the imaging device via points A1', A2', and A3' and recorded on the light point elements B1', B2', and B3' in the sensor of the imaging device. The position of point A in the real world relative to the lens can be represented by three points A1', A2', and A3' at infinity. The virtual scene at point A is projected onto the user's retina by the lens array via points C1', C2', and C3' on the display screen, thus overlapping the positions of the virtual and real scenes. This establishes an end-to-end mapping relationship.

[0148] The imaging array also needs to work with a lens array for image rendering. The lens array can generate a virtual 3D scene, which is then used in conjunction with the imaging array for virtual-real fusion. This is a very complex computational process. To reduce the computational load, Table 1 provides a mapping relationship table for the coordinates of points in the real world, their correspondence with the photoelectric elements in the imaging device sensor, and their positions on the display screen. As shown in Table 1, the correspondence between the coordinates of points in the real world and the photoelectric elements is the first mapping relationship, and the correspondence between the display positions on the display screen and the coordinates of points in the real world is the second mapping relationship. Pre-storing the mapping relationships in the mapping relationship table can reduce the subsequent computational load and improve computational efficiency.

[0149] Table 1

[0150] This application also provides a virtual-real fusion device.

[0151] Processing apparatus, the processing apparatus being used in any of the above embodiments of the method;

[0152] An eye-tracking device, wherein the eye-tracking device is used to collect eye-tracking data and send it to the processing device;

[0153] A plurality of imaging devices, at least some of which are used to acquire first real-world image data and send it to the processing device.

[0154] The implementation method and effects of the virtual-real fusion device provided in this application embodiment can be found in the foregoing embodiments, and will not be repeated here.

[0155] Optionally, the virtual-real fusion device provided in this application also includes,

[0156] The display screen is used to display the virtual-real fusion image on the display screen after the virtual-real fusion image sent by the processing device is acquired;

[0157] An optical system, having periodic optical characteristics, is used to process the light emitted from the display screen so that the processed light forms a stereoscopic image on the user's eyeball.

[0158] Optionally, the optical system includes a plurality of optical elements, wherein any one of the plurality of optical elements corresponds to at least one imaging device, or any one of the plurality of imaging devices corresponds to at least two optical elements.

[0159] Specifically, the optical element can be any optical element with periodic optical properties, such as a lens, a diffractive optical element (DOE), a grating, a microlens array, a photonic crystal, a multilayer interference filter, or a liquid crystal optical element.

[0160] Optionally, when the plurality of optical elements correspond one-to-one with the plurality of imaging devices, for any one of the plurality of imaging devices, the line connecting the center of the user's eyeball to the upper edge of the imaging device passes through the upper edge of the optical element corresponding to the imaging device, and the line connecting the center of the user's eyeball to the lower edge of the imaging device passes through the lower edge of the optical element corresponding to the imaging device.

[0161] In this way, each imaging device has a corresponding optical element to project the captured real-world image onto the user's eyes. This configuration is simple and straightforward, and can accurately overlay real-world images onto the virtual environment, enhancing the mixed reality experience.

[0162] Optionally, the optical system includes multiple component groups, with each of the multiple imaging devices corresponding to one of the multiple component groups. Each component group includes multiple optical elements, and each optical component group includes the same number of optical elements.

[0163] Optionally, for any of the plurality of imaging devices, the line connecting the center of the user's eyeball to the upper edge of the imaging device passes through the upper edge of the element group corresponding to the imaging device, and the line connecting the center of the user's eyeball to the lower edge of the imaging device passes through the lower edge of the element group corresponding to the imaging device; wherein, the upper edge of the element group is the upper edge of the uppermost optical element in the element group, and the lower edge of the element group is the lower edge of the lowermost optical element in the element group.

[0164] When an imaging device corresponds to at least two optical elements, the system's field of view or image resolution can be increased. By pairing multiple optical elements with a single imaging device, a wider field of view or higher image quality can be provided while keeping the device size small.

[0165] Optionally, the plurality of optical elements, the plurality of imaging devices, and the display screen are all arranged in a planar structure.

[0166] Figure 6 is a schematic diagram of the arrangement of multiple optical elements and multiple imaging devices provided in an embodiment of this application. As shown in Figure 6, it includes a total of 4 imaging devices and 4 optical elements. All 4 optical elements and 4 imaging devices are arranged in a planar structure. The positional relationship between the imaging devices and the optical elements is shown in Figure 6. For any imaging device, the line connecting the user's eye center to the upper edge of the imaging device passes through the upper edge of the corresponding optical element, and the line connecting the user's eye center to the lower edge of the imaging device passes through the lower edge of the corresponding optical element. The arrangement structure of the display screen can refer to the arrangement structure of multiple optical elements or multiple imaging devices.

[0167] In this way, through this optical design, the system can reduce common visual distortions such as chromatic aberration and distortion while saving space, thereby providing a more natural and comfortable visual experience.

[0168] Optionally, the plurality of optical elements, the plurality of imaging devices, and the display screen are all arranged in a curved structure.

[0169] In this way, curved surface design can help reduce image distortion problems common in traditional graphic design, especially in edge areas. By better matching the natural path of light, curved surface structures can provide more uniform image quality. Furthermore, curved surface structures allow for more compact device designs because they can utilize space more efficiently.

[0170] Corresponding to the above data processing method, this application embodiment also provides a data processing device. Figure 7 is a schematic diagram of the structure of a data processing device provided in this application embodiment. As shown in Figure 7, it is applied to a virtual-real fusion device, which includes multiple imaging devices. The device includes:

[0171] The first acquisition module 701 is used to acquire the user's eye-tracking data and determine the user's field of vision based on the eye-tracking data;

[0172] The filtering module 702 is used to filter at least one first imaging device from a plurality of imaging devices according to the field of view; the first imaging device is an imaging device located within the field of view.

[0173] The second acquisition module 703 is used to acquire first real-world image data collected by the first imaging device for any first imaging device.

[0174] The generation module 704 is used to generate target real images based on the first real image data corresponding to each first camera.

[0175] Optionally, each imaging device includes multiple sensors, and the second acquisition module 703 is specifically used for:

[0176] For any first imaging device, determine the imaging device viewpoint corresponding to the first imaging device and the user's viewpoint at the first imaging device.

[0177] Determine the overlapping angle between the imaging device's viewpoint and the user's viewpoint;

[0178] Multiple target sensors corresponding to the overlapping viewpoints are determined from the plurality of sensors;

[0179] Acquire first real-world image data, which is composed of data collected by the plurality of target sensors respectively.

[0180] Optionally, the first acquisition module 701 is also used for:

[0181] Based on the field of view, determine the user's central field of view;

[0182] Based on the central field of view, at least one second imaging device is selected from the at least one first imaging device; the second imaging device is used to indicate the first imaging device that is within the central field of view.

[0183] Correspondingly, the second acquisition module 703 is specifically used for:

[0184] For any second imaging device, acquire second real-world image data captured by the second imaging device at a first preset resolution and / or a first preset frame rate;

[0185] For any third imaging device, acquire third real-world image data acquired by the third imaging device at a resolution lower than the first preset resolution and / or at a frame rate lower than the first preset frame rate; the third imaging device is used to indicate the first imaging device that is not within the central field of view.

[0186] Optionally, the greater the deviation of the third imaging device from the central field of view, the lower the resolution and / or frame rate used by the third imaging device when acquiring third real-world image data.

[0187] Optionally, each imaging device includes multiple sensors, each sensor containing multiple photoelectric elements, and the generation module 704 is specifically used for:

[0188] The correspondence between the photoelectric element and each pixel in the target real-world image is obtained; the target real-world image is an image obtained by stitching together images captured by multiple first imaging devices.

[0189] The target real image is determined based on the first real image data corresponding to each first imaging device and the corresponding relationship.

[0190] Optionally, the generation module 704 is also used for:

[0191] Based on the correspondence and the first mapping relationship, the first coordinates of the target point corresponding to each pixel in the target real image in the real world are determined; wherein, the first mapping relationship is used to indicate the correspondence between the coordinates of the point in the real world and the photoelectric element;

[0192] Based on the target real-world image and the first coordinates of the target point corresponding to each pixel in the target real-world image in the real world, a virtual-real fusion image is generated, and the virtual-real fusion image is sent to the display screen for display.

[0193] Optionally, when the generation module 704 generates a virtual-real fusion image based on the target real-world image and the first coordinates of the target point corresponding to each pixel in the target real-world image in the real world, and sends the virtual-real fusion image to the display screen for display, it is specifically used for:

[0194] Based on the target real image and the first coordinates of the target point corresponding to each pixel in the target real image in the real world, determine the virtual image and the second coordinates of the target point corresponding to each pixel in the virtual image in the real world;

[0195] Based on the first coordinate and the second coordinate, the virtual image and the real image are fused to obtain a virtual-real fused image, and the third coordinate of the target point corresponding to each pixel in the virtual-real fused image in the real world is determined;

[0196] Based on the third coordinates and the second mapping relationship, the display position of each pixel in the virtual-real fusion image on the display screen is determined; the second mapping relationship is used to indicate the correspondence between the display position on the display screen and the coordinates of the point in the real world.

[0197] The virtual-real fusion image and the corresponding display positions of each pixel in the virtual-real fusion image on the display screen are sent to the display screen for display.

[0198] The specific implementation principle and effects of the data processing device provided in this application embodiment can be found in the foregoing embodiments, and will not be repeated here.

[0199] Figure 8 is a schematic diagram of an electronic device provided in an embodiment of this application. As shown in Figure 8, the electronic device of this embodiment may include:

[0200] At least one processor 801; and

[0201] Memory 802 communicatively connected to the at least one processor;

[0202] The memory 802 stores instructions that can be executed by the at least one processor 801, which, when executed by the at least one processor 801, cause the electronic device to perform the method as described in any of the above embodiments.

[0203] Alternatively, the memory 802 can be either standalone or integrated with the processor 801.

[0204] The implementation principle and technical effects of the electronic device provided in this embodiment can be found in the foregoing embodiments, and will not be repeated here.

[0205] This application also provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, implement the method described in any of the foregoing embodiments.

[0206] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the methods described in any of the foregoing embodiments.

[0207] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative. For instance, the division of modules is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple modules may be combined or integrated into another system, or some features may be ignored or not executed.

[0208] The integrated modules implemented as software functional modules described above can be stored in a computer-readable storage medium. These software functional modules, stored in a storage medium, include several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) or processor to execute some steps of the methods described in the various embodiments of this application.

[0209] It should be understood that the aforementioned processor can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), etc. A general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in the application can be directly implemented by a hardware processor, or implemented by a combination of hardware and software modules within the processor. The memory may include high-speed random access memory (RAM), and may also include non-volatile memory (NVM), such as at least one disk storage device, and may also be a USB flash drive, external hard drive, read-only memory, disk, or optical disc, etc.

[0210] The aforementioned storage media can be implemented from any type of volatile or non-volatile storage device or a combination thereof, such as Static Random-Access Memory (SRAM), Electrically Erasable Programmable Read-Only Memory (EEPROM), Erasable Programmable Read-Only Memory (EPROM), Programmable Read-Only Memory (PROM), Read-Only Memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk. The storage media can be any available medium accessible to general-purpose or special-purpose computers.

[0211] An exemplary storage medium is coupled to a processor, enabling the processor to read information from and write information to the storage medium. Alternatively, the storage medium can be an integral part of the processor. Both the processor and the storage medium can reside in an Application Specific Integrated Circuit (ASIC). Alternatively, the processor and storage medium can exist as discrete components in an electronic device or host device.

[0212] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0213] The sequence numbers of the embodiments in this application are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0214] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes several instructions to cause a terminal device (which may be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0215] The above are merely preferred embodiments of this application and do not limit the patent scope of this application. Any equivalent structural or procedural transformations made using the content of this application's specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the patent protection scope of this application.

Claims

1. A data processing method, characterized by, include: Acquire the user's eye-tracking data and determine the user's field of vision based on the eye-tracking data; Based on the field of view, at least one first imaging device is selected from a plurality of imaging devices; the first imaging device is an imaging device located within the field of view. For any first imaging device, acquire the first real-world image data collected by the first imaging device; A target real image is generated based on the first real image data corresponding to each first imaging device.

2. The method of claim 1, wherein, Each imaging device includes multiple sensors. For any given first imaging device, the system acquires first real-world image data collected by the first imaging device, including: For any first imaging device, determine the imaging device viewpoint corresponding to the first imaging device and the user's viewpoint at the first imaging device. Determine the overlapping angle between the imaging device's viewpoint and the user's viewpoint; Multiple target sensors corresponding to the overlapping viewpoints are determined from the plurality of sensors; Acquire first real-world image data, which is composed of data collected by the plurality of target sensors respectively.

3. The method of claim 1, wherein, The method further includes: Based on the field of view, determine the user's central field of view; Based on the central field of view, at least one second imaging device is selected from the at least one first imaging device; the second imaging device is used to indicate the first imaging device that is within the central field of view. Accordingly, for any first imaging device, acquiring the first real-world image data collected by the first imaging device includes: For any second imaging device, acquire second real-world image data captured by the second imaging device at a first preset resolution and / or a first preset frame rate; For any third imaging device, acquire third real-world image data acquired by the third imaging device at a resolution lower than the first preset resolution and / or at a frame rate lower than the first preset frame rate; the third imaging device is used to indicate the first imaging device that is not within the central field of view.

4. The method of claim 3, wherein, The greater the deviation of the third imaging device from the central field of view, the lower the resolution and / or frame rate used by the third camera imaging device when acquiring third real-world image data.

5. The method of claim 1, wherein, Each imaging device includes multiple sensors, and each sensor contains multiple photoelectric elements. Based on the first real-world image data corresponding to each first camera, a target real-world image is generated, including: Obtain the correspondence between the photoelectric components and each pixel in the target real-world image; The target real image is generated based on the first real image data corresponding to each first imaging device and the corresponding relationship.

6. The method of claim 5, wherein, Also includes: Based on the correspondence and the first mapping relationship, the first coordinates of the target point corresponding to each pixel in the target real image in the real world are determined; wherein, the first mapping relationship is used to indicate the correspondence between the coordinates of the point in the real world and the photoelectric element; Based on the target real-world image and the first coordinates of the target point corresponding to each pixel in the target real-world image in the real world, a virtual-real fusion image is generated, and the virtual-real fusion image is sent to the display screen for display.

7. The method according to claim 6, characterized in that, Based on the target real-world image and the first coordinates of the target point corresponding to each pixel in the target real-world image in the real world, a virtual-real fusion image is generated, and the virtual-real fusion image is sent to the display screen for display, including: Based on the target real image and the first coordinates of the target point corresponding to each pixel in the target real image in the real world, determine the virtual image and the second coordinates of the target point corresponding to each pixel in the virtual image in the real world; Based on the first coordinate and the second coordinate, the virtual image and the real image are fused to obtain a virtual-real fused image, and the third coordinate of the target point corresponding to each pixel in the virtual-real fused image in the real world is determined; Based on the third coordinates and the second mapping relationship, the display position of each pixel in the virtual-real fusion image on the display screen is determined; the second mapping relationship is used to indicate the correspondence between the display position on the display screen and the coordinates of the point in the real world. The virtual-real fusion image and the corresponding display positions of each pixel in the virtual-real fusion image on the display screen are sent to the display screen for display.

8. A virtual-real fusion device, characterized in that, include: Processing apparatus, the processing apparatus being configured to perform the method according to any one of claims 1-7; An eye-tracking device, wherein the eye-tracking device is used to collect eye-tracking data and send it to the processing device; A plurality of imaging devices, at least some of which are used to acquire first real-world image data and send it to the processing device.

9. The virtual-real fusion device according to claim 8, characterized in that, It also includes, The display screen is used to display the virtual-real fusion image after acquiring the virtual-real fusion image sent by the processing device; An optical system, having periodic optical characteristics, is used to process the light emitted from the display screen so that the processed light forms a stereoscopic image on the user's eyeball.

10. The virtual-real fusion device according to claim 9, characterized in that, The optical system includes a plurality of optical elements, wherein any one of the plurality of optical elements corresponds to at least one imaging device, or any one of the plurality of imaging devices corresponds to at least two optical elements.

11. The virtual-real fusion device according to claim 10, characterized in that, The multiple optical elements, multiple imaging devices, and the display screen are all arranged in a planar structure.

12. The virtual-real fusion device according to claim 10, characterized in that, The multiple optical elements, multiple imaging devices, and the display screen are all arranged in a curved structure.

13. A data processing apparatus, characterized in that, The device includes: The first acquisition module is used to acquire the user's eye-tracking data and determine the user's field of vision based on the eye-tracking data; A filtering module is used to filter at least one first imaging device from a plurality of imaging devices based on the field of view; the first imaging device is an imaging device located within the field of view. The second acquisition module is used to acquire first real-world image data collected by the first imaging device for any one of the first imaging devices. The generation module is used to generate target real-world images based on the first real-world image data corresponding to each first camera.

14. An electronic device, characterized in that, include: A processor, and a memory communicatively connected to the processor; The memory stores computer-executed instructions; The processor executes computer execution instructions stored in the memory to implement the method as described in any one of claims 1 to 7.

15. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions, which, when executed by a processor, are used to implement the method as described in any one of claims 1 to 7.

16. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the method as described in any one of claims 1-7.

Images