Video data processing method, electronic device, and storage medium

Abstract

The present application is applicable to the technical field of data processing, and provides a video data processing method and apparatus, an electronic device, and a storage medium. The method comprises: in response to a first operation, a first device acquiring first video, and a first image and a second image related to a first object; a second device displaying second video, the second video being obtained by performing color transformation on the first video according to a first mapping relationship determined by the first image and the second image, the first object in the second video being displayed in a second color, and the second color being a color of the first object under a standard illuminant. In the embodiments of the present application, a first image and a second image corresponding to a first object are acquired by means of modulating an auxiliary illuminator, so as to obtain a mapping relationship for determining an intrinsic color of the first object under a standard illuminant, so that the first object is displayed as the intrinsic color when video is displayed, thereby meeting user needs in scenarios requiring intrinsic colors of objects to be displayed in a video, improving a degree of matching between color adjustments and usage scenarios, and also improving adjustment flexibility.

Description

A method for processing video data, an electronic device, and a storage medium.

[0001] This application claims priority to Chinese Patent Application No. 202510491475.3, filed on April 18, 2025, entitled "A Method for Processing Video Data, Electronic Device and Storage Medium", the entire contents of which are incorporated herein by reference. Technical Field

[0002] This application belongs to the field of multimedia technology, and in particular relates to a method for processing video data, an electronic device, and a storage medium. Background Technology

[0003] With the continuous development of multimedia processing technology, users have increasingly higher requirements for the picture quality of multimedia content. Among these, video, as one of the most widely used multimedia formats today, has received significant attention from users regarding its picture quality. Besides resolution, users are also beginning to consider color deviations in video footage. Therefore, the ability to calibrate colors in video data has become a major factor affecting video picture quality.

[0004] Existing video processing technologies can adjust the colors in video data to match the colors seen by the human eye under ambient light conditions by adjusting camera calibration parameters, thus achieving color calibration. This means the colors of objects in the video data reflect their appearance under ambient light. However, this color calibration method also introduces bias. In some scenarios, such as when users need to know the true color of a product unaffected by ambient light (its intrinsic color), the above method cannot meet the color calibration requirements. Therefore, existing video processing technologies can only calibrate the colors of video images based on ambient light, resulting in a limited calibration method that fails to meet the shooting needs of users in different scenarios and reduces the match between the video's colors and the user's viewing requirements. Summary of the Invention

[0005] This application provides a video data processing method, electronic device, and storage medium, which can solve the problem that existing video processing technologies can only calibrate the color of the video image based on ambient light. The calibration method is singular and cannot meet the shooting needs of users in different scenarios. The matching degree between the color of the video image and the user's viewing needs is low.

[0006] In a first aspect, embodiments of this application provide a method for processing video data, applied to a first device, comprising:

[0007] In the first shooting mode, the auxiliary light is turned on to capture the first image, and the auxiliary light is turned off to capture the second image; the subject of the first image and the second image includes the first object;

[0008] Based on the first image and the second image, a first mapping relationship is determined;

[0009] A first video frame is captured; the subject of the first video frame includes the first object; the first video frame is captured when the auxiliary light is off.

[0010] Based on the first mapping relationship, the first video frame is color-converted to obtain a second video frame; in the second video frame, the first object is displayed in the color under a standard light source.

[0011] The embodiments of this application have the following beneficial effects: In a first shooting mode, the first device can simultaneously acquire a first image of the first object under auxiliary light and a second image of the first object under ambient light (i.e., auxiliary light off). Since the color of the first object in the first image is affected by both ambient light and auxiliary light, the color of the first object under the influence of auxiliary light can be determined using the first and second images. This allows filtering out the influence of ambient light on the first object. Based on the two images, a first mapping relationship for converting the color to the intrinsic color can be obtained. The first video frame acquired under auxiliary light off is then color-converted using the first mapping relationship to obtain the second video frame. This allows the first object in the second video frame to be displayed in its intrinsic color, achieving the goal of calibrating the color of the first object to its intrinsic color in the video. Compared with existing video processing technologies, the embodiments of this application do not adjust the color of the first video frame to a color affected by ambient light. Instead, they can obtain the first image and the second image corresponding to the first object by switching the auxiliary light on and off, and obtain the mapping relationship for determining the intrinsic color of the first object under a standard light source. This allows the first object to be displayed in its intrinsic color in the second video, meeting the user's requirement that the subject in the video be displayed in its intrinsic color. This improves the adaptability between color adjustment and usage scenarios, and also increases the flexibility of adjustment.

[0012] In one possible implementation of the first aspect, before enabling auxiliary light acquisition of the first image and disabling auxiliary light acquisition of the second image in the first shooting mode, the method further includes:

[0013] In response to the first operation, the first shooting mode is activated.

[0014] This method allows users to select from different shooting modes based on their actions. When the first shooting mode is selected, the first object in the captured video frame will be displayed in its intrinsic color, thereby improving the flexibility of the shooting process.

[0015] In some implementations, in response to the second operation, a second shooting mode is initiated, and in the second shooting mode, a first video frame is captured; in the first video frame, the first object is displayed as its color under ambient light.

[0016] This method allows for the selection of a second shooting mode when a color display is required, thereby increasing shooting flexibility.

[0017] In one possible implementation of the first aspect, the step of enabling auxiliary light acquisition of a first image and disabling auxiliary light acquisition of a second image in the first shooting mode includes:

[0018] In the first shooting mode, if the preset triggering conditions are met, the first image and the second image are captured.

[0019] In one possible implementation of the first aspect, the triggering condition includes at least one of the following:

[0020] The ambient light in the environment where the first object is located changes;

[0021] The first object has changed;

[0022] The object change feature of the first object is greater than a preset change threshold; the object change feature is determined based on any two video frames containing the first object in the acquired video frames; the object change feature includes at least one of the following: a first change feature in the shooting angle dimension, a second change feature in the color dimension, and a third change feature in the shape dimension.

[0023] This method sets up a variety of different triggering conditions to automatically trigger the update process of the first mapping relationship. This allows the first mapping relationship to be adjusted according to the changes in the content of the video frame being captured, so that the first mapping relationship matches the captured video frame, thereby improving the accuracy of intrinsic color calibration.

[0024] In one possible implementation of the first aspect, determining the first mapping relationship based on the first image and the second image includes:

[0025] Based on the first image and the second image, determine the second mapping relationship;

[0026] The first mapping relationship is determined based on the preset third mapping relationship and the second mapping relationship; the third mapping relationship is used to map the color under the auxiliary light to the color under the standard light source.

[0027] This method can determine the color of a first object displayed under auxiliary light based on a first image and a second image. Since the intensity of the auxiliary light is predictable—for example, the intensity of a smartphone flash is fixed—a mapping relationship between each color under auxiliary light and each color under a standard light source can be established in advance. Therefore, the first mapping relationship from ambient light to standard light source can be determined through the second and third mapping relationships, thereby enabling subsequent color conversion.

[0028] In one possible implementation of the first aspect, determining the second mapping relationship based on the first image and the second image includes:

[0029] The first image and the second image are subjected to differential processing to determine the second mapping relationship.

[0030] In one possible implementation of the first aspect, the first device is equipped with a color camera and a multispectral sensor; the first image is synthesized based on a first color image captured by the color camera and first spectral information captured by the multispectral sensor; the second image is synthesized based on a second color image captured by the color camera and second spectral information captured by the multispectral sensor.

[0031] Determining the second mapping relationship based on the first image and the second image includes:

[0032] Based on the first color image and the second color image, determine the fourth mapping relationship;

[0033] Based on the first spectral information and the second spectral information, determine the spectral response information corresponding to the first object;

[0034] Based on the spectral response information and the spectral response curves corresponding to all colors, the fourth mapping relationship is extended by color mapping to obtain the second mapping relationship.

[0035] This method is suitable for scenes with a single color, such as when shooting a curtain or a large object where only a corner of the image is visible. If a new color appears, the mapping relationship established solely based on the color image contains limited colors, potentially requiring repeated acquisition of the first and second images to update the mapping relationship, thus impacting the user experience. In contrast, electronic devices can use multispectral sensors to acquire the aforementioned spectral response information and extend the color mapping relationship determined by the color image. This allows for the mapping relationship of various colors under the current ambient light to colors under a standard light source, reducing the need for frequent auxiliary light acquisition and mapping updates, and improving the smoothness of the video recording process.

[0036] In one possible implementation of the first aspect, after performing color conversion on the first video frame according to the first mapping relationship to obtain the second video frame, the method further includes:

[0037] The second video frame is displayed.

[0038] This method allows for local processing and display of video frames after capture, enabling users to immediately view the second video frame, which is now displayed in its intrinsic color after color conversion, thus improving the timeliness of display.

[0039] In one possible implementation of the first aspect, displaying the second video frame includes:

[0040] In response to the second operation of selecting the first display mode, the second video frame is displayed.

[0041] This method can determine the corresponding display mode based on the user's selection, enabling switching between intrinsic color and apparent color, thereby improving the flexibility of the video frame display process and meeting the user's needs in different scenarios.

[0042] In one possible implementation of the first aspect, after performing color conversion on a first video frame containing the first object captured in the first shooting mode according to the first mapping relationship to obtain a second video frame, the method further includes:

[0043] In response to a third operation that selects a second display mode, the first video frame is displayed.

[0044] In one possible implementation of the first aspect, after performing color conversion on the first video frame according to the first mapping relationship to obtain the second video frame, the method further includes:

[0045] Sending second video data to a second device and / or a third device; the second video data is the second video frame or a second video file including the second video frame; the second device is a device for displaying the second video data; the third device is a device for forwarding the second video data.

[0046] This method can send the video data after color conversion to other devices for display, or forward the video data through other devices, thereby expanding the range of intrinsic colors that can be viewed. For example, in live streaming or video sharing scenarios, other devices do not need to have color conversion capabilities, reducing the resource consumption of other devices.

[0047] Secondly, a video data processing apparatus includes:

[0048] An image acquisition unit is configured to, in a first shooting mode, activate auxiliary light to acquire a first image and deactivate the auxiliary light to acquire a second image; the objects captured in the first image and the second image include a first object;

[0049] The first mapping relationship determination unit is used to determine a first mapping relationship based on the first image and the second image;

[0050] A first video frame acquisition unit is used to acquire a first video frame; the subject of the first video frame includes the first object; the first video frame is acquired when the auxiliary light is off.

[0051] The color conversion unit is used to perform color conversion on the first video frame according to the first mapping relationship to obtain a second video frame; in the second video frame, the first object is displayed in the color under a standard light source.

[0052] In one possible implementation of the second aspect, the processing apparatus further includes:

[0053] A shooting activation unit is used to activate the first shooting mode in response to a first operation.

[0054] In one possible implementation of the second aspect, the image acquisition is specifically used for: in the first shooting mode, if a preset triggering condition is met, then acquiring the first image and acquiring the second image.

[0055] In one possible implementation of the second aspect, the triggering condition includes at least one of the following:

[0056] The ambient light in the environment where the first object is located changes;

[0057] The first object has changed;

[0058] The object change feature of the first object is greater than a preset change threshold; the object change feature is determined based on any two video frames containing the first object in the acquired video frames; the object change feature includes at least one of the following: a first change feature in the shooting angle dimension, a second change feature in the color dimension, and a third change feature in the shape dimension.

[0059] In one possible implementation of the second aspect, the first mapping relationship determining unit includes:

[0060] The second mapping relationship determination unit is used to determine a second mapping relationship based on the first image and the second image;

[0061] The mapping relationship conversion unit is used to determine the first mapping relationship based on the preset third mapping relationship and the second mapping relationship; the third mapping relationship is used to map the color under the auxiliary light to the color under the standard light source.

[0062] In one possible implementation of the second aspect, the second mapping relationship determining unit is specifically used to: perform differential processing on the first image and the second image to determine the second mapping relationship.

[0063] In one possible implementation of the second aspect, the first device is equipped with a color camera and a multispectral sensor; the first image is synthesized based on a first color image captured by the color camera and first spectral information captured by the multispectral sensor; the second image is synthesized based on a second color image captured by the color camera and second spectral information captured by the multispectral sensor.

[0064] The second mapping relationship determination unit includes:

[0065] The fourth mapping relationship determination unit is used to determine a fourth mapping relationship based on the first color image and the second color image;

[0066] A spectral information determination unit is used to determine the spectral response information corresponding to the first object based on the first spectral information and the second spectral information.

[0067] The color extension unit is used to perform color mapping extension on the fourth mapping relationship based on the spectral response information and the spectral response curves corresponding to all colors, so as to obtain the second mapping relationship.

[0068] In one possible implementation of the second aspect, the processing apparatus further includes:

[0069] The second video frame display unit is used to display the second video frame.

[0070] In one possible implementation of the second aspect, the second video frame display unit is specifically used to: display the second video frame in response to a second operation that selects a first display mode.

[0071] In one possible implementation of the second aspect, the processing apparatus further includes:

[0072] The second display mode triggering unit is used to respond to a third operation that selects the second display mode and display the first video frame.

[0073] In one possible implementation of the second aspect, the processing apparatus further includes:

[0074] A data transmission unit is configured to transmit second video data to a second device and / or a third device; the second video data is the second video frame or a second video file including the second video frame; the second device is a device for displaying the second video data; and the third device is a device for forwarding the second video data.

[0075] Thirdly, embodiments of this application provide a method for processing video data, applied to a first device, the processing method comprising:

[0076] In the first shooting mode, turn on the auxiliary light and capture the first image;

[0077] The auxiliary light is turned off, and a second image is acquired; the objects captured in both the first and second images include the first object.

[0078] Based on the first image and the second image, a first mapping relationship is determined;

[0079] A first video frame is captured; the subject of the first video frame includes the first object; the first video frame is captured when the auxiliary light is off; the first video frame is captured when the auxiliary light is off.

[0080] The first mapping relationship and the first video frame are sent to a second device or a third device; the first mapping relationship is used by the second device and / or the third device to perform color conversion on the first video frame to obtain a second video frame; in the second video frame, the first object is displayed in the color under a standard light source; the second device is a device for displaying the second video data; the third device is a device for forwarding the second video data.

[0081] The embodiments of this application have the following beneficial effects: In a first shooting mode, the first device can simultaneously acquire a first image of the first object under auxiliary light and a second image of the first object under ambient light (i.e., auxiliary light off). Since the color of the first object in the first image is affected by both ambient light and auxiliary light, the color of the first object under the influence of auxiliary light can be determined using the first and second images. This filters out the influence of ambient light on the first object. Based on the two images, a first mapping relationship for converting the color to the intrinsic color can be obtained. The first mapping relationship and the first video frame are then sent to other devices for processing. The receiving devices can use the first mapping relationship to perform color conversion on the first video frame acquired under auxiliary light off to obtain a second video frame. This allows the first object in the second video frame to be displayed in its intrinsic color, achieving the purpose of calibrating the color of the first object to its intrinsic color in the video. Compared to existing video processing technologies, this application embodiment, when adjusting the color of the first video frame, does not adjust it to a color influenced by ambient light. Instead, it acquires a first image and a second image corresponding to the first object by switching on and off auxiliary light, obtaining a mapping relationship to determine the intrinsic color of the first object under a standard light source. This ensures that the first object is displayed in its intrinsic color in the second video, meeting the user's requirement for the video object to be displayed in its intrinsic color. This improves the adaptability between color adjustment and usage scenarios, and also enhances the flexibility of adjustment. In the first shooting mode, if preset trigger conditions are met, the first image and the second image are acquired. Furthermore, since the conversion process can be implemented on other devices, it reduces the data processing pressure on the first device and also improves the flexibility of the color conversion process.

[0082] In one possible implementation of the third aspect, before activating auxiliary light acquisition of the first image and deactivating auxiliary light acquisition of the second image in the first shooting mode, the method further includes:

[0083] In response to the first operation, the first shooting mode is activated.

[0084] This method allows users to select from different shooting modes based on their actions. When the first shooting mode is selected, the first object in the captured video frame will be displayed in its intrinsic color, thereby improving the flexibility of the shooting process.

[0085] In some implementations, in response to the second operation, a second shooting mode is initiated, and in the second shooting mode, a first video frame is captured; in the first video frame, the first object is displayed as its color under ambient light.

[0086] In one possible implementation of the third aspect, the step of enabling auxiliary light acquisition of the first image and disabling auxiliary light acquisition of the second image in the first shooting mode includes:

[0087] In the first shooting mode, if the preset triggering conditions are met, the first image and the second image are captured.

[0088] In one possible implementation of the third aspect, the triggering condition includes at least one of the following:

[0089] The ambient light in the environment where the first object is located changes;

[0090] The first object has changed;

[0091] The object change feature of the first object is greater than a preset change threshold; the object change feature is determined based on any two video frames containing the first object in the acquired video frames; the object change feature includes at least one of the following: a first change feature in the shooting angle dimension, a second change feature in the color dimension, and a third change feature in the shape dimension.

[0092] In one possible implementation of the third aspect, determining the first mapping relationship based on the first image and the second image includes:

[0093] Based on the first image and the second image, determine the second mapping relationship;

[0094] The first mapping relationship is determined based on the preset third mapping relationship and the second mapping relationship; the third mapping relationship is used to map the color under the auxiliary light to the color under the standard light source.

[0095] In one possible implementation of the third aspect, determining the second mapping relationship based on the first image and the second image includes:

[0096] The first image and the second image are subjected to differential processing to determine the second mapping relationship.

[0097] In one possible implementation of the third aspect, the first device is equipped with a color camera and a multispectral sensor; the first image is synthesized based on a first color image captured by the color camera and first spectral information captured by the multispectral sensor; the second image is synthesized based on a second color image captured by the color camera and second spectral information captured by the multispectral sensor.

[0098] Determining the second mapping relationship based on the first image and the second image includes:

[0099] Based on the first color image and the second color image, determine the fourth mapping relationship;

[0100] Based on the first spectral information and the second spectral information, determine the spectral response information corresponding to the first object;

[0101] Based on the spectral response information and the spectral response curves corresponding to all colors, the fourth mapping relationship is extended by color mapping to obtain the second mapping relationship.

[0102] In one possible implementation of the third aspect, after the acquisition of the first video frame; the subject of the first video frame includes the first object, the method further includes:

[0103] Based on the first mapping relationship, the first video frame is color-converted to obtain the second video frame.

[0104] In one possible implementation of the third aspect, after performing color conversion on the first video frame according to the first mapping relationship to obtain the second video frame, the method further includes:

[0105] The second video frame is displayed.

[0106] In one possible implementation of the third aspect, displaying the second video frame includes:

[0107] In response to the second operation of selecting the first display mode, the second video frame is displayed.

[0108] In one possible implementation of the third aspect, after performing color conversion on a first video frame containing the first object captured in the first shooting mode according to the first mapping relationship to obtain a second video frame, the method further includes:

[0109] In response to a third operation that selects a second display mode, the first video frame is displayed.

[0110] Fourthly, a video data processing apparatus, comprising:

[0111] An image acquisition unit is configured to, in a first shooting mode, activate auxiliary light to acquire a first image and deactivate the auxiliary light to acquire a second image; the objects captured in the first image and the second image include a first object;

[0112] The first mapping relationship determination unit is used to determine a first mapping relationship based on the first image and the second image;

[0113] A first video frame acquisition unit is used to acquire a first video frame; the subject of the first video frame includes the first object; the first video frame is acquired when the auxiliary light is off.

[0114] The sending unit is configured to send the first mapping relationship and the first video frame to a second device or a third device; the first mapping relationship is used by the second device and / or the third device to perform color conversion on the first video frame to obtain a second video frame; in the second video frame, the first object is displayed in the color under a standard light source; the second device is a device for displaying the second video data; the third device is a device for forwarding the second video data.

[0115] In one possible implementation of the fourth aspect, the processing apparatus further includes:

[0116] A shooting activation unit is used to activate the first shooting mode in response to a first operation.

[0117] In one possible implementation of the fourth aspect, the image acquisition unit is specifically used to: in the first shooting mode, if a preset triggering condition is met, acquire the first image and acquire the second image.

[0118] In one possible implementation of the fourth aspect, the triggering condition includes at least one of the following:

[0119] The ambient light in the environment where the first object is located changes;

[0120] The first object has changed;

[0121] The object change feature of the first object is greater than a preset change threshold; the object change feature is determined based on any two video frames containing the first object in the acquired video frames; the object change feature includes at least one of the following: a first change feature in the shooting angle dimension, a second change feature in the color dimension, and a third change feature in the shape dimension.

[0122] In one possible implementation of the fourth aspect, the first mapping relationship determining unit includes:

[0123] The second mapping relationship determination unit is used to determine a second mapping relationship based on the first image and the second image;

[0124] The mapping relationship conversion unit is used to determine the first mapping relationship based on the preset third mapping relationship and the second mapping relationship; the third mapping relationship is used to map the color under the auxiliary light to the color under the standard light source.

[0125] In one possible implementation of the fourth aspect, the second mapping relationship determining unit is specifically used to: perform differential processing on the first image and the second image to determine the second mapping relationship.

[0126] In one possible implementation of the fourth aspect, the first device is equipped with a color camera and a multispectral sensor; the first image is synthesized based on a first color image captured by the color camera and first spectral information captured by the multispectral sensor; the second image is synthesized based on a second color image captured by the color camera and second spectral information captured by the multispectral sensor.

[0127] The second mapping relationship determination unit includes:

[0128] The fourth mapping relationship determination unit is used to determine a fourth mapping relationship based on the first color image and the second color image;

[0129] A spectral information determination unit is used to determine the spectral response information corresponding to the first object based on the first spectral information and the second spectral information.

[0130] The color extension unit is used to perform color mapping extension on the fourth mapping relationship based on the spectral response information and the spectral response curves corresponding to all colors, so as to obtain the second mapping relationship.

[0131] In one possible implementation of the fourth aspect, the processing apparatus further includes:

[0132] The processing unit is configured to perform color conversion on the first video frame according to the first mapping relationship to obtain a second video frame.

[0133] In one possible implementation of the fourth aspect, the processing apparatus further includes:

[0134] The second video frame display unit is used to display the second video frame.

[0135] In one possible implementation of the fourth aspect, the second video frame display unit is specifically used to: display the second video frame in response to a second operation that selects a first display mode.

[0136] In one possible implementation of the fourth aspect, the processing apparatus further includes:

[0137] The second display mode triggering unit is used to respond to a third operation that selects the second display mode and display the first video frame.

[0138] Fifthly, embodiments of this application provide a method for processing video data, applied to a second device, the processing method comprising:

[0139] In the first display mode, a second video frame is displayed; the second video frame is obtained by color conversion of a first video frame containing the first object according to a first mapping relationship; the first mapping relationship is determined based on a first image and a second image containing the first object; the first image is captured when the auxiliary light is turned on in the first device; the second image is captured when the auxiliary light is turned off in the first device; the first video frame is captured by the first device when the auxiliary light is off; in the second video frame, the first object is displayed in the color under a standard light source.

[0140] In this embodiment of the application, the second device obtains a second video frame that has been converted to an intrinsic color, and can display the second video frame in a first display mode, thereby enabling the first object to be displayed in the video as an intrinsic color, so as to meet the user's scenario requirement that the subject in the video be displayed as an intrinsic color, improve the adaptability between color adjustment and usage scenario, and also improve the flexibility of adjustment.

[0141] In one possible implementation of the fifth aspect, prior to displaying the second video frame, the method further includes:

[0142] Receive the first video frame and the first mapping relationship sent by the first device;

[0143] The first video frame is color-converted according to the first mapping relationship to obtain the second video frame.

[0144] In one possible implementation of the fifth aspect, after receiving the first video frame and the first mapping relationship sent by the first device, the method further includes:

[0145] In response to a third operation that selects a second display mode, the first video frame is displayed.

[0146] In one possible implementation of the fifth aspect, displaying the second video frame includes:

[0147] In response to the second operation of selecting the first display mode, the second video frame is displayed.

[0148] Sixthly, embodiments of this application provide a video data processing apparatus, including:

[0149] A first mode display unit is configured to display a second video frame in a first display mode; the second video frame is obtained by color conversion of a first video frame containing the first object according to a first mapping relationship; the first mapping relationship is determined based on a first image and a second image containing the first object; the first image is acquired when the auxiliary light acquisition is turned on in the first device; the second image is acquired when the auxiliary light acquisition is turned off in the first device; the first video frame is acquired by the first device when the auxiliary light is off; in the second video frame, the first object is displayed in the color under a standard light source.

[0150] In one possible implementation of the sixth aspect, the processing apparatus further includes:

[0151] The first mapping relationship receiving unit is used to receive the first video frame and the first mapping relationship sent by the first device;

[0152] A color conversion unit is used to perform color conversion on the first video frame according to the first mapping relationship to obtain the second video frame.

[0153] In one possible implementation of the sixth aspect, the processing apparatus further includes:

[0154] The second mode display unit is used to display the first video frame in response to a third operation that selects a second display mode.

[0155] In one possible implementation of the sixth aspect, the first mode display unit includes a function specifically configured to: display the second video frame in response to a second operation that selects the first display mode.

[0156] In a seventh aspect, embodiments of this application provide an electronic device, including: a memory, a processor, and a program stored in the memory, wherein the processor executes the program to implement the steps of the video data processing method described in any of the first aspects, or the steps of the video data processing method described in any of the third aspects, or the video data processing method described in any of the fifth aspects.

[0157] Eighthly, embodiments of this application provide a readable storage medium storing a program that, when executed by a processor, implements the steps of the video data processing method described in any of the first aspects, the steps of the video data processing method described in any of the third aspects, or the video data processing method described in any of the fifth aspects.

[0158] Ninthly, embodiments of this application provide a program product that, when run on a device, causes the device to perform steps in the video data processing method described in any of the first aspects, or steps in the video data processing method described in any of the third aspects, or the video data processing method described in any of the fifth aspects.

[0159] Other implementations can refer to the aforementioned aspects and will not be elaborated further.

[0160] The beneficial effects of the above aspects can be referenced from each other, and will not be elaborated further here. Attached Figure Description

[0161] Figure 1 is a schematic diagram of color calibration in a live video streaming scenario;

[0162] Figure 2(a) is a schematic diagram of the stages of a video data processing method provided in an embodiment of this application;

[0163] Figure 2(b) is a flowchart illustrating the implementation of each stage of the video data processing method provided in the embodiments of this application;

[0164] Figure 3 is a schematic diagram of the process of acquiring video frames according to an embodiment of this application;

[0165] Figure 4 is a flowchart illustrating the process of acquiring a first image and a second image according to an embodiment of this application;

[0166] Figure 5 is a flowchart illustrating the process of acquiring the first image and the second image according to another embodiment of this application;

[0167] Figure 6 is a timing diagram of the response to changes in ambient light according to an embodiment of this application;

[0168] Figure 7 is a schematic diagram of the change in the state of the shooting object between video frames according to an embodiment of this application;

[0169] Figure 8 is a flowchart illustrating the process of acquiring the first image and the second image according to another embodiment of this application;

[0170] Figure 9 is a flowchart illustrating the determination of a first mapping relationship according to an embodiment of this application;

[0171] Figure 10 is a schematic diagram illustrating the determination of a second mapping relationship provided in an embodiment of this application;

[0172] Figure 11 is a schematic diagram of the generation of a third mapping relationship provided in an embodiment of this application;

[0173] Figure 12 is a flowchart illustrating the implementation of establishing a second mapping relationship based on a color camera and a multispectral sensor according to an embodiment of this application;

[0174] Figure 13 is a timing flowchart of video frame adjustment provided in an embodiment of this application;

[0175] Figure 14 is a comparative schematic diagram of video recording and video display provided in an embodiment of this application;

[0176] Figure 15 is a schematic flowchart of video frame processing provided in an embodiment of this application;

[0177] Figure 16 is a schematic flowchart of video frame processing provided in another embodiment of this application;

[0178] Figure 17 is a schematic flowchart of video frame processing provided in another embodiment of this application;

[0179] Figure 18 is a structural block diagram of a video data processing device provided in an embodiment of this application;

[0180] Figure 19 is a structural block diagram of another video data processing device provided in an embodiment of this application;

[0181] Figure 20 is a structural block diagram of another video data processing device provided in an embodiment of this application;

[0182] Figure 21 is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0183] In the following description, specific details such as particular system architectures and techniques are set forth for illustrative purposes and not for limitation, in order to provide a thorough understanding of the embodiments of this application. However, those skilled in the art will understand that this application may also be implemented in other embodiments without these specific details. In other instances, detailed descriptions of well-known systems, apparatuses, circuits, and methods have been omitted so as not to obscure the description of this application with unnecessary detail.

[0184] It should be understood that, when used in this application specification and the appended claims, the term "comprising" indicates the presence of the described features, integrals, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or a collection thereof.

[0185] It should also be understood that the term “and / or” as used in this application specification and the appended claims means any combination of one or more of the associated listed items and all possible combinations, and includes such combinations.

[0186] As used in this application specification and the appended claims, the term "if" may be interpreted, depending on the context, as "when," "once," "in response to determination," or "in response to detection." Similarly, the phrase "if determined" or "if detected [the described condition or event]" may be interpreted, depending on the context, as meaning "once determined," "in response to determination," "once detected [the described condition or event]," or "in response to detection [the described condition or event]."

[0187] Furthermore, in the description of this application and the appended claims, the terms "first," "second," "third," etc., are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0188] References to "one embodiment" or "some embodiments" as described in this specification mean that one or more embodiments of this application include a specific feature, structure, or characteristic described in connection with that embodiment. Therefore, the phrases "in one embodiment," "in some embodiments," "in other embodiments," "in still other embodiments," etc., appearing in different parts of this specification do not necessarily refer to the same embodiment, but rather mean "one or more, but not all, embodiments," unless otherwise specifically emphasized. The terms "comprising," "including," "having," and variations thereof mean "including but not limited to," unless otherwise specifically emphasized.

[0189] Video, as the most commonly used data type in multimedia today, is widely used in various fields, such as short videos, live video streaming, medical diagnosis, professional design, and remote material selection. As picture quality is a primary factor in judging video quality, providing users with high-quality video images has become a key focus.

[0190] Among the factors affecting video image quality, besides video parameters such as resolution and refresh rate, the accuracy of colors in the image is also related. However, users' needs for color calibration will vary in different scenarios. Color calibration includes at least two calibration methods: calibrating the colors in the image to the apparent colors of objects, and calibrating the colors in the image to the intrinsic colors of objects. The specific definitions of these two types of color can be described as follows:

[0191] Color representation: The color of an object as perceived by the human eye, influenced by the current ambient light source, changes with factors such as the ambient light source and the human's state of adaptation.

[0192] Intrinsic color: The color of an object under a standard light source, namely standard daylight (6500K, D65), which represents its standard state and does not change with ambient light sources or other human factors.

[0193] Users often switch between intrinsic color and representative color when calibrating video data in different scenarios. The following example of a live video streaming scenario illustrates this:

[0194] For example, Figure 1 illustrates a schematic diagram of color calibration in a live video streaming scenario. Referring to Figure 1(a), user A is live streaming a video, and the screen includes the user's face. To improve the quality of the live video, the user turns on a spotlight. Since the spotlight can provide fill light for the scene and increase the brightness of the facial skin, the overall brightness of the screen will increase. That is, the color of the face in the actual scene will be brightened due to the influence of the spotlight. At this time, the user's color calibration requirement is that the color of the face in the video screen is consistent with the color of the face in the scene under the influence of the spotlight, that is, the face in the video screen is displayed as its actual color.

[0195] At a certain moment during a live stream, for example, when the user's view switches from a selfie perspective to a product display perspective, as shown in Figure 1(b), the video frame includes the displayed clothing. Because the user has turned on a spotlight at this time, the actual color of the clothing will appear brighter due to the spotlight, resulting in a difference from the actual color. Since the color of a product is affected by ambient light during a live stream, especially with numerous fill lights present, the difference in actual color becomes even more pronounced. Consumers may experience a discrepancy between the color of the product seen in the live stream and the color of the actual product received, impacting their purchasing experience and leading to numerous returns for merchants due to color differences after shipment. Therefore, when users showcase products during a live stream, their need is often for the video to display the product's intrinsic color, rather than its apparent color.

[0196] Therefore, in the context of live video streaming, the requirements for color calibration will change depending on the subject being filmed. When the subject is a face, the face needs to be displayed in the video as the color of the face; while when the subject is a product, the product needs to be displayed in the video as the color of the product.

[0197] However, existing video technology's color calibration schemes can only calibrate to the representation color, not the character color. The color calibration method is singular. If the video data has different color calibration requirements at the same time, it is necessary to manually adjust the colors of some video frames after the user has completed the video shooting. This not only requires the user to have color adjustment experience, but also adds extra color adjustment operations, increases the user's operating difficulty, and reduces the user's shooting experience.

[0198] To address the limitations of existing video technologies, which can only adjust the image to reflect its color and cannot adjust the colors to their intrinsic colors according to user needs, resulting in a single color calibration method, this application provides a video data processing method. This method acquires a first image and a second image corresponding to a first object by switching on auxiliary light, and obtains a mapping relationship for determining the intrinsic color of the first object under a standard light source. This ensures that the first object is displayed in its intrinsic color when the video is shown, meeting the user's needs for displaying the intrinsic color of an object in the video. This improves the adaptability of color adjustment to the usage scenario and also enhances the flexibility of adjustment.

[0199] The aforementioned video data processing method can be specifically applied to electronic devices, including smartphones, tablets, laptops, desktop computers, and other devices with video processing capabilities. The electronic device can acquire the video data to be processed through its local camera module, or it can receive video data sent by other devices, such as video data transmitted from an external camera via a data cable. The received video data can then be color-calibrated. No limitation is placed on the method by which the electronic device acquires video data.

[0200] Example 1

[0201] In this embodiment, the video data processing method described above may specifically include two stages. For example, Figures 2(a) and 2(b) illustrate the stages of a video data processing method provided in an embodiment of this application. Referring to Figure 2(a), the video data processing method provided in this embodiment may include a video processing stage and a video display stage. Referring to Figure 2(b), the specific operations implemented in the above two stages can be described as follows:

[0202] Corresponding to stage 1, in S201, the video capture device, in the first shooting mode, captures a first video frame and a first image and a second image about a first object. The first video includes the first object; the first video includes the first image and the second image; the first image is an image of the first object captured when the auxiliary light is not turned on; the second image is an image of the first object captured when the auxiliary light is turned on; the first object in the first video is displayed as a first color; the first color is the color of the first object under ambient light; and the first image and the second image can be used to generate a first mapping relationship for color conversion.

[0203] Specifically, the above 201 may include the following steps:

[0204] In S201.1, in the first shooting mode, the auxiliary light is turned on to capture the first image, and the auxiliary light is turned off to capture the second image;

[0205] In S201.2, a first mapping relationship is determined based on the first image and the second image;

[0206] In S201.3, the first video frame is captured;

[0207] In S201.4, the first video frame is color-converted according to the first mapping relationship to obtain the second video frame.

[0208] or

[0209] In S201.5, the first mapping relationship and the first video frame are sent to the second or third device.

[0210] Corresponding to stage 2, in S202, the video display device displays a second video frame. The second video frame is obtained by color conversion of the first video frame based on the first mapping relationship determined by the first image and the second image; the first object in the second video frame is displayed as the second color; the second color is the color of the first object under a standard light source.

[0211] It should be noted that the above two stages can be completed by the same electronic device or by different electronic devices, which can be divided into the following situations:

[0212] Scenario 1: The video acquisition device and the video display device are the same device. For example, the first device acquires video frames, performs color calibration on the video frames, and displays the color-calibrated video frames on the first device.

[0213] Scenario 2: The video capture device and the video display device are different devices. For example, the first device captures video frames, performs color calibration on the video frames, and sends the color-calibrated video frames to another device different from the first device, such as the second device. The second device displays the color-calibrated video locally.

[0214] Alternatively, the first device sends the calibrated video frame to the server, which can then forward the video frame to other devices, such as the second device, that display the calibrated video frame.

[0215] Users can choose from the two methods mentioned above based on their application scenarios. There are no restrictions on the devices used to display or capture video.

[0216] For ease of explanation, the following uses a video recording scenario as an example to describe the specific implementation process of each of the above stages.

[0217] Phase 1: Video processing phase, including: video frame acquisition phase and mapping relationship determination phase.

[0218] Phase 1.1: Video Frame Acquisition Phase

[0219] For example, Figure 3 illustrates a flowchart of acquiring video frames according to an embodiment of this application. A user's electronic device may have an application with video recording functionality installed, such as the camera application 31 shown in Figure 3(a). The user can click on the camera application 31 to open the corresponding camera operation interface and adjust it to video recording mode, as shown in Figure 3(b). This camera operation interface includes a recording start control 32 and may also include a function control 33 for enabling intrinsic color calibration. If the user needs to perform intrinsic color calibration on the subject during video recording, for example, if a product is captured during video recording, they can click on the function control 33 to enable the intrinsic color calibration function. For example, after the user clicks the function control 33, the control will display the interface shown in Figure 3(c); conversely, if the user does not need to enable the above function, they can directly click on the recording start control 32 and begin recording video.

[0220] In some implementations, the electronic device may include different shooting modes, such as a first shooting mode where the subject is displayed in its intrinsic color, and a second shooting mode where the subject is displayed in its representational color. Referring again to Figure 3, the interface includes an intrinsic color calibration function control 33, which can be activated before or during video recording. Continuing with Figure 3 as an example, if the user clicks the activation control 32 when the intrinsic color calibration function is not enabled, they can enter recording mode, as shown in Figure 3(d), where the function control 33 is in an inactive state. If, at some point during recording, the user clicks the function control 33 in Figure 3(d), the intrinsic color calibration function can be activated, resulting in the stage shown in Figure 3(e), where the function control 33 is in an active state.

[0221] In this embodiment, after the electronic device activates the aforementioned intrinsic color calibration function, it can adjust the intrinsic color of each video frame containing the first object in the acquired video, that is, adjust the display color of the first object in the video frame from its apparent color to its intrinsic color. In implementing the above color calibration process, two images containing the first object are required: a first image captured with auxiliary light on, and a second image captured with auxiliary light off.

[0222] In some implementations, the auxiliary light can be a light source equipped on the first device, such as the flash on a smartphone; the auxiliary light can also be an external light source, with the first device connected to the auxiliary light source via a wired or wireless connection, and the external light source can be turned on or off according to usage requirements.

[0223] In some implementations, the timing of acquiring the two images mentioned above can be categorized into three situations: capturing images before video capture, automatically capturing images during video capture, and capturing images in response to user actions during video capture. Let's continue with the video recording process as an example:

[0224] Method 1: Shoot before acquiring the video

[0225] For example, Figure 4 shows a flowchart of the implementation of acquiring a first image and a second image according to an embodiment of this application. Referring to Figure 4, the preview frame of the video recording interface 41 of the smartphone includes a piece of clothing, and the intrinsic color calibration function control 42 is in the activated state. When the user clicks the recording start control 43, they can first activate the flash to take a first image of the clothing under the flash and ambient light, i.e., image 44, and then turn off the flash to take a second image of the clothing under ambient light, i.e., image 45. After obtaining the above two images, recording begins, i.e., video recording interface 46. Since the subsequent adjustment changes the display color of the first object from the color affected by ambient light to the intrinsic color under a standard light source, the auxiliary light is turned off during the video recording process, so that the color displayed by the first object in subsequent video frames is the color affected only by ambient light (i.e., the color of the display).

[0226] In this embodiment, by acquiring the two images mentioned above before acquiring the video, color calibration can be performed on the subsequently acquired video frames without requiring any additional user intervention, thus improving the efficiency of color calibration.

[0227] Method 2: Automatically capture images during video acquisition.

[0228] For example, Figure 5 shows a flowchart of the implementation of acquiring a first image and a second image provided in another embodiment of this application. Referring to Figure 5(a), the smartphone is recording video and the intrinsic color function control 51 is in the active state. At this time, the main subject in the video recording interface is the portrait 52, and it is not necessary to calibrate the video frame containing the portrait 52 to the intrinsic color.

[0229] During the process of acquiring video on a smartphone, the smartphone can perform object recognition on the video. If the main subject in a video frame at a certain moment is detected to be a specified type of object, such as clothing, lipstick, curtains, and shoes, the smartphone can automatically capture the first and second images of that object. This allows for subsequent color calibration of video frames containing that object, adjusting the colors in the subsequently captured video frames to their intrinsic colors. The object recognition can be performed on the main subject in the video. The main subject is defined as an object whose area in the frame occupies a larger than a preset threshold and / or is located in the center of the frame. The number of main subjects can be one or more, depending on the actual scene. For example, in Figure 5(a), the human face 52 is located in the center of the frame and occupies a large area, so the face can be recognized as the main subject; while the clock in the background occupies a smaller area and is located on the edge, so the clock can be recognized as a secondary subject.

[0230] For example, during video recording, if the user moves the smartphone's camera from the person 52 to the clothing 53, as shown in Figure 5(b), the smartphone will detect that the subject in the current video frame is a piece of clothing. This clothing is a commodity-type object. The smartphone can turn on the flash to acquire a first image of the clothing 53 under the influence of the flash and ambient light, as shown in image 54 of Figure 5(c). Then, it can turn off the flash to acquire a second image of the clothing 53 under ambient light, as shown in image 55 of Figure 5(d). Subsequent video frames containing the clothing 53 can be color-converted based on the mapping relationship determined by the two images, so that the color of the clothing is displayed as its intrinsic color in subsequent video frames.

[0231] In some implementations, the electronic device can display a corresponding preview interface during video recording. The preview interface can display either the video frame before it is adjusted to the intrinsic color (i.e., the first video frame) or the video frame after it is adjusted to the intrinsic color according to the mapping relationship (i.e., the second video frame). The specific choice can be made according to the actual situation and is not limited here.

[0232] It should be noted that during the acquisition of images 54 and 55, the smartphone remains in video recording mode. The recorded video may include these two images; that is, images 54 and 55 can be used as two video frames in the recorded video. Alternatively, the video frame corresponding to the activated flash can be removed from the recorded video. For example, if the user stores the recorded video file in their local gallery after recording, the video file stored in the gallery may not contain images 54 and 55. Similarly, in a live streaming scenario, if the recorded video frames need to be sent to other devices watching the live stream in real time, the video frames sent to other devices may also not include these two images. Whether the recorded video file includes these two images can be set according to the actual situation and is not limited here.

[0233] In some implementations, the electronic device can reacquire the two images when a preset trigger condition is detected. Since these two images are used for color calibration of video frames containing the first object, if the shooting scene where the first object is located changes, or if the first object itself changes, it is necessary to reacquire the two images. The trigger condition can include the following three scenarios:

[0234] Scenario 1) When the ambient light changes, the mapping relationship established between the first and second images is used to adjust the first object's color from its apparent color to its intrinsic color. When the ambient light changes—for example, in an outdoor shooting scene where the ambient light is natural light, but clouds block the sun and dim the light—the apparent color of the first object displayed in the video frame will also change. This means the original mapping relationship will become invalid, and a new mapping relationship matching the changed ambient light needs to be established. Therefore, the electronic device can again acquire the first and second images corresponding to the first object under the current ambient light. Scenarios where the ambient light changes can also include: changes in indoor lighting in an indoor scene, or obstructions blocking indoor lighting, etc.

[0235] In this system, electronic devices can determine whether the ambient light has changed based on changes in pixel values ​​within video frames. For example, if the change in pixel values ​​of one or more subjects in any two video frames exceeds a preset threshold, or if the percentage of pixels whose pixel values ​​have changed between two video frames exceeds a certain threshold, then it can be determined that the ambient light has changed, and the two images can be re-acquired.

[0236] For example, Figure 6 shows a timing diagram of the response to changes in ambient light provided in an embodiment of this application. During times 0 to T1, the electronic device can be in a video recording state. At a certain time, the main subject of the video frame is a face. Since the face is not a specified type of object, there is no need to adjust the colors in the video frame to intrinsic colors, nor is it necessary to acquire the first and second images corresponding to the face. At time T1, the main subject of the video frame is clothing 61. At this time, the electronic device can activate the flash to acquire image 62 corresponding to clothing 61, and at time T1+1, acquire image 63 with the flash off. Based on the mapping relationship X1 determined by images 62 and 63, the video frame containing clothing 61 is color-converted so that clothing 61 is displayed in its intrinsic color in subsequent video frames containing clothing 61.

[0237] Between time T1+1 and time T2, the ambient light of the scene where clothing 61 is being filmed remains unchanged. However, at time T2, the ambient light changes. In order to update the mapping relationship, the electronic device can restart the flash to acquire image 64 of clothing 61 under the new ambient light, and acquire image 65 with the flash off at time T2+1. Based on images 64 and 65, the mapping relationship X2 under the new ambient light is determined. After time T2+1, the subsequent video frames containing clothing 61 are color-converted based on the mapping relationship X2.

[0238] Scenario 2) The subject of the photograph changes. If the subject changes from one product to another, the corresponding mapping relationship also needs to be updated. Therefore, the electronic device can acquire the first and second images corresponding to the new subject again, thereby establishing a new mapping relationship for the new subject.

[0239] The electronic device can be configured with an object recognition algorithm. If it detects a change in the main object in the video frame, such as a change from clothes to shoes, or from red clothes to green clothes, it can recognize the change in the object and re-acquire the two images mentioned above. The response timing diagram when the object changes can be the same as the response timing diagram in case 1), which will not be elaborated here.

[0240] In some implementations, if the subject colors of two captured objects are the same, it is not necessary to re-acquire the first and second images. For example, if one captured object is clothing and the other is pants, and both objects are red, the same mapping relationship can be used to achieve color conversion. Specifically, in video frame A containing clothing, the image area corresponding to the clothing is A1; in video frame B containing pants, the image area corresponding to the pants is B1. If the percentage of pixels with changed pixel values ​​between these two image areas is less than a preset percentage threshold, then the two captured objects are identified as having the same color, and it is not necessary to re-acquire the first and second images. Conversely, if the percentage of pixels with changed pixel values ​​between the two image areas is greater than or equal to the preset percentage threshold, then the object has changed, and the mapping relationship needs to be updated.

[0241] Scenario 3) The state of the subject in the video changes. This change in state includes: changes in the shooting angle of the subject in the video frame, changes in the shape of the subject, and changes in the color of the subject.

[0242] In some implementations, the electronic device can be equipped with a motion sensor. Since the movement of the electronic device often leads to a change in the shooting angle, the shooting state of the subject in the video frame will also change. Therefore, the electronic device can determine the current operating data of the electronic device through the motion sensor. If the motion data is greater than a preset motion change threshold, the change in the state of the subject in the video frame is identified. For example, Figure 7 shows a schematic diagram of the change in the state of the subject between video frames provided in an embodiment of this application. Referring to Figure 7(a), the product is a box. The top view of the box is captured at time T1, and the top 71 and front 72 of the box are shown in the video frame. The color of the two sides can be red. At time T2, the user changes the shooting angle and captures the right view of the box, that is, the right side 73 of the box is displayed in the video frame, and the color of the right side 73 is green. Due to the different shooting angles, the area of ​​the subject displayed in the video frame may be different. Different areas can be different colors. Therefore, the original mapping relationship corresponding to the area may not be applicable to the current color of the area, and the mapping relationship needs to be updated, that is, the first image and the second image need to be reacquired.

[0243] In some implementations, electronic devices can use object recognition algorithms to determine the outline of the object in a video frame. If the outline deviation of the object between two video frames is greater than a preset deviation threshold, the camera angle of the object is identified as having changed. Referring to Figure 7(b), the product is a folding mobile phone. The folding phone is in a folded state when photographed at time T1, and in an unfolded state when photographed at time T2. Although the object is the same at both times, its shape displayed in the video frames is different. Therefore, the mapping relationship needs to be updated, i.e., the first and second images need to be re-acquired.

[0244] In some implementations, electronic devices can use object recognition algorithms to determine the outline of the captured object in a video frame. If the change in pixel values ​​within the object outline exceeds a preset threshold, the color of the captured object is identified as having changed. Referring to Figure 7(c), the product is a temperature-sensitive water cup. It appears blue at low temperatures and red at high temperatures, meaning its intrinsic color changes with different temperatures. Therefore, to better reflect the different colors of the same object under different conditions, the mapping relationship can be updated when the object's color changes, i.e., the first and second images can be re-acquired.

[0245] In this embodiment of the application, when the electronic device detects that the preset triggering conditions are met, it can acquire the first image and the second image of the first object to determine the mapping relationship for intrinsic color conversion, which reduces the operations required by the user and can also calibrate the intrinsic color of the first object in the video frame to meet the user's shooting needs.

[0246] Method 3: Respond to user actions and capture video frames during the process.

[0247] For example, Figure 8 shows a flowchart of the implementation of acquiring a first image and a second image provided in another embodiment of this application. Similar to the start-up stage in Figure 5, referring to Figure 8(a), the smartphone is recording video, and the intrinsic color function control 81 is in the activated state. At this time, the main subject in the video recording interface is the portrait 82, and it is not necessary to calibrate the video frame containing the portrait 82 to the intrinsic color.

[0248] During subsequent video recording, the user moves the smartphone's camera from the person 82 to the clothing 83, as shown in Figure 8(b). The smartphone then detects that the subject in the current video frame is clothing, meaning the main subject in the captured video frame has changed. At this time, the electronic device can display a prompt message 84 to ask the user whether they need to adjust the color of the subject to its intrinsic color. If the user wants the clothing 83 to be displayed in its intrinsic color in the captured video frame, they can click the aforementioned "agree" control 85. In response to the user clicking the "agree" control 85, the electronic device can acquire the first image and the second image of the clothing 83, namely, the image 86 taken with the flash on and the image 87 taken with the flash off. The mapping relationship 88 corresponding to the clothing 83 is determined based on the two images. If the user does not want the subject to be displayed in its intrinsic color in the captured video frame, they can click the aforementioned "disagree" control 89, and the clothing 83 can be displayed in its apparent color in subsequent captured video frames.

[0249] If, during video recording, the user moves the smartphone's camera from the clothing 83 to the foldable phone 810, as shown in Figure 8(c), the user will be prompted again whether they need to adjust the color of the subject to its intrinsic color. The above steps can be repeated each time the subject changes, and will not be elaborated further here.

[0250] It should be noted that if the ambient light changes or the state of the subject in the video frame changes, the mapping relationship can be automatically updated, that is, the first and second images can be re-acquired without requiring the user to confirm the above operations again, thus reducing the operations that the user needs to perform.

[0251] Phase 1.2: Determining the Mapping Relationship

[0252] In this embodiment, after acquiring the first image and the second image of the first object, the electronic device can determine the mapping relationship of the first object based on the two images. Through this mapping relationship, the video frames containing the first object can be color-converted so that the first object is displayed in its intrinsic color in the video frame.

[0253] In some implementations, Figure 9 illustrates a flowchart of determining a first mapping relationship according to an embodiment of this application. Specifically, it includes the following steps:

[0254] In S901, a second mapping relationship is determined based on the first image and the second image.

[0255] In this embodiment, the color of the first object displayed in the first image is the color presented under the combined influence of auxiliary light and ambient light; the color of the first object displayed in the second image is the color presented under the influence of ambient light. Through the above two images, the color presented only under the influence of auxiliary light can be determined, thereby determining the color of the first object under ambient light mapped to the color under auxiliary light, that is, obtaining the above-mentioned second mapping relationship.

[0256] In some possible implementations, the aforementioned second mapping relationship is determined by performing a differential calculation on the first image and the second object. For example, Figure 10 shows a schematic diagram illustrating the determination of the second mapping relationship according to an embodiment of this application. Referring to Figure 10, image 101 is a photograph of clothing taken when the auxiliary light is on. The color of the clothing in image 101 is affected by two types of light: ambient light and auxiliary light. Image 102 is a photograph of clothing taken when the auxiliary light is off. The color of the clothing in image 102 is only affected by ambient light. Therefore, by performing a differential calculation on the two images, the color of the clothing under the influence of auxiliary light can be determined, resulting in image 103. Based on images 103 and 102 representing the color of the first object under ambient light, the mapping relationship between the color of the first object under ambient light and its color under auxiliary light can be obtained.

[0257] In step S902, a first mapping relationship is determined based on a preset third mapping relationship and a second mapping relationship. The third mapping relationship is used to map the color under the auxiliary light to the color under the standard light source.

[0258] In this embodiment, the electronic device can store a third mapping relationship, which is used to map the color under the auxiliary light to the color under the standard light source. The standard light source can be D65 or a light source under other standards, and can be set according to the actual situation.

[0259] In some implementations, different electronic devices use different modules to provide auxiliary light. That is, different electronic devices can configure a corresponding third mapping relationship based on the module providing the auxiliary light. For example, if smartphone A uses a flash of model X1, and smartphone B also uses a flash of model X1, then smartphone A and smartphone B can use the same third mapping relationship. However, if smartphone C uses a flash of model X2, then smartphone A and smartphone C can use different third mapping relationships.

[0260] In some implementations, the aforementioned third mapping relationship can be stored on the electronic device at the factory of the smartphone, or it can be obtained by downloading from a server. For example, Figure 11 shows a schematic diagram of the generation of the third mapping relationship provided in an embodiment of this application. Referring to Figure 11, the electronic device can be equipped with color charts of different colors. The camera module of the electronic device acquires an image 111 corresponding to the color chart under a flash, and then a professional colorimetric device acquires an image 112 corresponding to the same color chart under a standard light source D65. Since the two images use the same color chart and the colors at the same positions are the same, the color under the flash can be converted to the color under the standard light source, thus obtaining the aforementioned third mapping relationship.

[0261] In this embodiment, the second mapping relationship determines the color of the first object from ambient light to flash light, and the third mapping relationship determines the color from flash light to standard light source. That is, by combining the above two mapping relationships, the color of the first object from ambient light to standard light source can be obtained. Thus, after color adjustment of the acquired video frame through the third mapping relationship, the first object in the video frame can be displayed as the color under standard light source, that is, as its intrinsic color.

[0262] In one possible implementation, a multispectral sensor can be configured on the electronic device. The multispectral sensor is used to acquire spectra. A spectrum (or optical spectrum) can be understood as a pattern of monochromatic light dispersed by a dispersive system (e.g., a prism, a grating) and arranged sequentially according to wavelength or frequency. For example, a multispectral sensor can acquire spectra in the 350-1000 nanometer wavelength range with a field of view (FOV) of ±35 degrees. The multispectral sensor can also include eight visible light bands and multiple special bands (e.g., a full-spectrum channel, a scintillation frequency detection channel, and / or an infrared channel), or ten visible light bands and multiple special bands. It is understood that the above number of visible light bands is merely an example; in practical applications, there can be fewer or more visible light bands.

[0263] When an electronic device is equipped with a multispectral sensor, the aforementioned first image and second image can be acquired through both a color camera and a multispectral sensor module. Specifically, the first image can include a first color image acquired by the color camera and first spectral information acquired by the multispectral sensor; similarly, the second image can include a second color image acquired by the color camera and second spectral information acquired by the multispectral sensor. The electronic device can perform color augmentation on the mapping relationship based on the spectral information acquired by the multispectral sensor, enabling it to map colors not captured in the image to a standard light source.

[0264] For example, Figure 12 shows a flowchart of an embodiment of this application for establishing a second mapping relationship based on a color camera and a multispectral sensor. At time T1, the smartphone turns on its flash, acquires a color image 121 through the color camera, and acquires spectral information 122 through the multispectral sensor; then, at time T1+1, the smartphone turns off its flash, acquires a color image 123 through the color camera, and acquires spectral information 124 through the multispectral sensor.

[0265] The electronic device can determine the first mapping relationship, namely mapping relationship X1, of the color of the curtain from the ambient light to the color under the flash lamp based on the color image 121 and the color image 123. Then, based on the pre-stored second mapping relationship, mapping relationship X0, of the color from the flash lamp to the color under the standard light source, and the above-determined mapping relationship X1, the fourth mapping relationship (mapping relationship X2) of the curtain can be obtained, so as to realize the mapping of the color of the curtain under the ambient light to the corresponding color under the standard light source in the captured video frame.

[0266] Since the subject of the photograph is a curtain, the captured video frames contain a large area of ​​solid-color fabric. This means that the mapping relationship X2 established by the color image contains a small number of colors. In order to expand the colors in the mapping relationship X2, the spectral information collected by the multispectral sensor can be combined to expand the colors.

[0267] The electronic device can determine the spectral response information 125 corresponding to the current ambient light based on spectral information 122 and spectral information 124. Using spectral response information 125 and a preset spectral response curve 126, it can determine the mapping relationship X3 of the colors not recorded in the above mapping relationship X2 under the current ambient light, thus expanding the color of mapping relationship X2. Combining the recorded color mapping relationship X2 with the expanded mapping relationship X3, a multi-color mapping relationship X4 (i.e., mapping relationship X2 + mapping relationship X4) can be obtained under the current ambient light. Continuing with the example of a curtain as the subject of the photograph, the curtain consists of two parts: the curtain fabric and the curtain rod. Images 121 and 123 only show the curtain fabric; the curtain rod is not shown, meaning that the color of the curtain rod is not recorded in the above mapping relationship X2. If the object captured at a later time includes a curtain rod, since the mapping relationship X2 has already been color-extended to obtain the multi-color mapping relationship X4 for the current environment, it is not necessary to acquire the first and second images again. Instead, the video frame containing the curtain rod can be color-converted through the mapping relationship X4.

[0268] Although the single color displayed on the screen belongs to the visible light band, the information acquired by the multispectral sensor (122 and 124) does not simply include information from the visible light band, but also spectral information from other bands. Therefore, by combining these two spectral information, the spectral information of multiple bands in the current environment can be obtained. The relationship between the spectra of different bands satisfies the spectral response curve 126. Therefore, after determining the spectral information of some spectra, the spectral information of all spectra can be determined according to the spectral response curve 126. This enables the expansion of the color mapping relationship, allowing the mapping relationship of multiple colors under the current ambient light to be obtained even in scenes with a single color. This reduces the need to frequently turn on the auxiliary light to acquire images and update the mapping relationship, improving the smoothness of the video recording process.

[0269] In some implementations, the process of adjusting the acquired first video frame according to the first mapping relationship can be performed after the video file is acquired. The video file includes the first video frame to be color converted. For example, in a video recording scenario, video recording can generate a video file and store it in the gallery. The electronic device can perform color conversion on the first video frame in the video file through the first mapping relationship after the video file is stored in the gallery or after recording is completed.

[0270] In some implementations, the process of adjusting the acquired first video frame according to the first mapping relationship can be performed during the acquisition of the first video frame. That is, during the acquisition of the first video frame, the colors of each first video frame are converted sequentially. For example, in a live video scenario, the live streaming device will send the video frames acquired in real time to the user devices watching the live stream. At this time, before uploading the video frames to the live streaming server, the colors of the first video frames can be converted according to the first mapping relationship to obtain the second video frame, and the second video frame can be sent to the live streaming server. The above process is performed simultaneously with the acquisition of the first video frame while the live streaming is in progress.

[0271] Phase 2: Video Display Phase

[0272] In this embodiment, the electronic device can perform color conversion on the video frame containing the first object through the first mapping relationship determined above, so that when the electronic device displays the adjusted video frame, the first object in the picture is displayed in its intrinsic color.

[0273] For example, Figure 13 shows a timing flowchart of video frame adjustment provided in an embodiment of this application. Referring to Figure 13, during video recording, an image 131 of the clothing taken with the flash on is acquired at time T1, and an image 132 taken with the flash off is acquired at time T1+1. A mapping relationship 133 is determined. Since all video frames between T1 and T2 contain the same object, the clothing, color conversion can be performed on all video frames between T1 and T2 through the mapping relationship 133.

[0274] It should be noted that if different first objects are captured at different times in the same video frame, the electronic device can generate corresponding first mapping relationships for different first objects. That is, when the first object changes, the electronic device will generate a corresponding first mapping relationship for that first object. Through the first mapping relationship of each first object, the corresponding video frame is color-converted. Continuing with Figure 13 as an example, if the object being captured changes from clothes to pants at time T2, then image 134 captured with the flash on can be acquired at time T2, and image 135 captured with the flash off can be acquired at time T2+1, thus determining the correspondence 136. All video frames from time T2 to T3 contain the same object, the pants, so the color of all video frames between T2 and T3 can be converted using the correspondence 135.

[0275] In this embodiment, the first object in the first video frame (i.e. the acquired video frame) can be displayed as its apparent color, i.e., the color affected by ambient light. In the second video frame obtained after color conversion of the first video frame through the first mapping relationship, the first object can be displayed as its intrinsic color, i.e., the color displayed by the object under a standard light source.

[0276] For example, Figure 14 shows a comparative schematic diagram of video recording and video display provided in an embodiment of this application. Referring to Figure 14(a), during video recording, a preview screen can be displayed on the video recording interface of the smartphone. In the preview screen, the clothing is displayed in its intrinsic color, that is, its displayed color is affected by the ambient light. For example, when the clothing is white and the ambient light is red, the color of the clothing displayed in the preview screen is not pure white, but rather mixed with some of the red of the ambient light, that is, it is displayed with a reddish tint in the preview interface. The electronic device can perform color conversion on the video according to the first mapping relationship. For example, after recording is completed, the user opens the recorded video file in the gallery. As shown in Figure 14(b), the video file stored in the gallery is adjusted through the first mapping relationship. Therefore, the clothing in the screen is displayed in its intrinsic color. That is, when the clothing is white, regardless of the ambient light during recording, the clothing is displayed as white in each video frame of the adjusted video file.

[0277] It should be noted that if a video frame contains two or more first objects, such as clothing and shoes, the first mapping relationship is determined based on the first and second images containing the two objects. Therefore, the determined first mapping relationship also applies to the two objects. Thus, in the second video frame obtained after color conversion, the two or more first objects are displayed in their corresponding intrinsic colors, that is, color conversion is performed on multiple objects simultaneously.

[0278] In some implementations, the electronic device adjusts the video frame containing the first object according to the first mapping relationship, including: performing color conversion on the entire video frame containing the first object according to the first mapping relationship, that is, performing color conversion on the entire image of the video frame.

[0279] In some implementations, the electronic device adjusts a video frame containing a first object according to a first mapping relationship, including: performing color conversion on the image region where the first object is located in the video frame containing the first object, i.e., only performing color conversion on the region of the first object in the video frame, and not performing color conversion on regions where the first object is not present. Specifically, the electronic device can use an object recognition algorithm to determine the object outline of the first object from the video frame, and perform color conversion on the pixels contained within the object outline in the video frame according to the first mapping relationship.

[0280] In one possible implementation, color adjustment of the acquired video frames according to the first mapping relationship can be performed after the video frames are acquired, i.e., it can be implemented on the device where the video frames are acquired. For example, Figure 15 shows a schematic flowchart of video frame processing provided in an embodiment of this application. Referring to Figure 15, during the acquisition of video 151, the electronic device can acquire image 152 (with auxiliary light shooting enabled) and image 153 (with auxiliary light shooting disabled) corresponding to the first object, and obtain mapping relationship 154 ​​based on the two images. The video frames containing the first object in video 151 are then color-converted using mapping relationship 154 ​​to obtain video 155. In video 151, the first object is displayed in its apparent color, and in video 155, the first object is displayed in its intrinsic color. The electronic device can then display video 155 locally or send video 155 to other devices for display on those devices.

[0281] Taking a live video streaming scenario as an example, during the live stream, the streaming device acquires a video frame from video 151 at a certain moment, performs color conversion on that video frame according to mapping relationship 154 ​​to generate the corresponding video frame in video 155, and uploads the corresponding video frame in video 155 to the live video streaming server. Other devices watching the live stream can receive each video frame in video 155 from the live video streaming server and display each video frame in video 155 locally. Since the video frame containing the first object in video 155 has already undergone color conversion, the first object is displayed in its intrinsic color on the devices watching the live stream.

[0282] In one possible implementation, the electronic device can simultaneously store a first video frame before color conversion and a second video frame after color conversion. That is, the electronic device can store video frames of both types in a video file, or simultaneously store a first video file and a second video file. The first video file contains all the first video frames before color conversion, and the second video file contains all the second video frames after color conversion. The user can select the video frame to be displayed from the two video frames.

[0283] In some implementations, different video frames can be associated with corresponding display modes. When displaying video on a device, the corresponding video frame can be displayed according to the different display modes selected by the user. For example, in the intrinsic color display mode, the video frame adjusted based on the first mapping relationship (i.e., the second video frame) is displayed; in the non-intrinsic color display mode, the video frame before adjustment (i.e., the first video frame) is displayed.

[0284] For example, Figure 16 illustrates a flowchart of video frame processing provided in another embodiment of this application. Referring to Figure 16(a), during the acquisition of video 161, the electronic device can acquire image 162 (with auxiliary light shooting enabled) and image 163 (with auxiliary light shooting disabled) corresponding to the first object, and obtain a mapping relationship 164 based on the two images. The video frames containing the first object in video 161 are then color-converted using the mapping relationship 164 to obtain video 165. The electronic device can simultaneously store both videos. In video 161, the first object is displayed in its apparent color, while in video 165, the first object is displayed in its intrinsic color. Since the electronic device can store two videos, it can select the video matching the user's selected display mode for display.

[0285] Referring to Figure 16(b), users can click on recorded videos in this map library to display them, such as clicking on video thumbnail 166, and enter the corresponding video playback interface. According to the default display mode, after clicking on video thumbnail 166, the electronic device can display either the generated video 161 or video 165.

[0286] In some implementations, the video playback interface includes an intrinsic color mode control 167, which is enabled by default. When the intrinsic color mode control 167 is enabled, the electronic device displays in the intrinsic color mode. In this case, when the user clicks the video thumbnail 166, the electronic device can display the video 165, which is the video after color conversion according to the mapping relationship 164. The user can also manually disable the intrinsic color mode control 167, as shown in Figure 16(c). When the intrinsic color mode control 167 is disabled, the electronic device displays in the color-based mode, and the video before adjustment, i.e., video 161, can be displayed.

[0287] In some implementations, when the user clicks the aforementioned mode control 167 to switch display modes, the electronic device can replay the video. For example, when the default display mode of the electronic device is the intrinsic color display mode, that is, after the user clicks the aforementioned video thumbnail 166, the electronic device displays video 165 in the intrinsic color display mode; when video 165 has played for 10 seconds, clicking the aforementioned mode control 167 switches from the intrinsic color display mode to the color display mode, at which point the electronic device can replay video 161.

[0288] In some implementations, when a user clicks the mode control 167 to switch display modes, the electronic device can continue playing the video based on the corresponding timestamp. When a user clicks the video thumbnail 166, the electronic device can simultaneously read both video 161 and video 165, for example, storing both videos in a cache area and displaying the corresponding video on the video playback interface according to the display mode. For example, if the default display mode is the intrinsic color display mode, then video 165 will be displayed on the video playback interface. If the user needs to switch display modes, for example, by clicking the mode control 167 at the 10th second, since both videos are already stored in the cache area, the device can jump to the video frame corresponding to the 10th second in video 161 based on the corresponding timestamp and continue playing the video at the original playback progress.

[0289] It should be noted that the above-mentioned mode control 167 will only be displayed if the video has a corresponding first mapping relationship. If a video does not generate a corresponding first mapping relationship when it is acquired, the above-mentioned mode control 167 may not be displayed in the video playback interface.

[0290] Taking a live video streaming scenario as an example, the streaming device can acquire the aforementioned video 1611 and generate video 165 according to mapping relationship 154. Simultaneously, it uploads both video 161 and video 165 to the live video streaming server. Other devices watching the live stream can simultaneously receive both video 161 and video 165 from the live video streaming server and select the corresponding video from 161 and 165 for display based on the display mode. For example, in the intrinsic color display mode, the device watching the live stream displays video 165, and the first object in the video playback interface is displayed in the intrinsic color; in the apparent color display mode, the device watching the live stream displays video 161, and the first object in the video playback interface is displayed in the apparent color.

[0291] In some implementations, when the device for video acquisition and the device for video display are different devices, the device for video acquisition can send both videos, 161 and 165, to the device for video display so that the appropriate video can be selected for display based on the display mode.

[0292] In one possible implementation, color adjustment of the acquired video frame according to the first mapping relationship can be performed before displaying the video frame, i.e., it can be implemented on the video display device. For example, FIG17 shows a schematic flowchart of video frame processing provided in another embodiment of this application. Referring to FIG17, during the acquisition of video frame 171, the electronic device can acquire image 172 (with auxiliary light shooting enabled) and image 173 (with auxiliary light shooting disabled) corresponding to the first object, and obtain mapping relationship 174 based on the two images. The electronic device can store the mapping relationship 174 and the acquired video frame 171. When displaying video frame 171, the electronic device can perform color conversion on the stored video frame 171 according to mapping relationship 174 to obtain video frame 175, and display video frame 175. In video frame 171, the first object is displayed in its apparent color, and in video frame 175, the first object is displayed in its intrinsic color.

[0293] When the device acquiring the video frame and the device displaying the video frame are different devices, the device acquiring the video frame can send the mapping relationship 174 and the video frame 171 to the video display device, so that the video display device can perform color conversion on the video frame 171 to obtain the video frame 175. Taking a live video scenario as an example, the aforementioned video frame 171 can be acquired by the live streaming device. During the live streaming process, the live streaming device can also obtain the aforementioned mapping relationship 174. Other devices watching the live stream can receive the video frame 171 and the mapping relationship 174 sent by the live streaming device. Thus, the devices watching the live stream can perform color conversion on the video frame 171 locally to obtain the video frame 175 and display the video frame 175. Therefore, the first object is displayed in its intrinsic color on the device watching the live stream.

[0294] In some implementations, the above color conversion process can be implemented by a second device or by a cloud server. For example, the live streaming device can send video frame 171 and mapping relationship 174 to the live streaming server. The live streaming server can perform color conversion on video frame 171 according to mapping relationship 174 to obtain video frame 175. The device watching the live stream can obtain the converted video frame 175 from the server and display the video frame 175 locally. This reduces the computing resources occupied by the live streaming device while enabling the device watching the live stream to display the video frame 175 after intrinsic color conversion.

[0295] In this embodiment, since the device displaying the video stores the aforementioned mapping relationship 174 and video frame 171, it can also determine whether the video frame 171 needs to be processed through the mapping relationship 174 based on the display module selected by the user. If the user selects the intrinsic color display mode, the video frame 171 can be color-converted according to the mapping relationship 174 to obtain video frame 175, and then the video frame 175 can be displayed. If the user selects the non-intrinsic color display mode, the video frame 171 can be displayed directly without color conversion, thereby improving the flexibility of the display and reducing the occupation of storage resources.

[0296] As can be seen from the above, in the embodiments of this application, the first device can simultaneously acquire a first image of the first object under the auxiliary light being turned on and a second image of the first object under ambient light (i.e., the auxiliary light being turned off) in the first shooting mode. Since the color of the first object in the first image is affected by both ambient light and auxiliary light, the color of the first object under the influence of auxiliary light can be determined through the first image and the second image, thereby filtering out the influence of ambient light on the first object. Based on the two images, a first mapping relationship for converting the color to the intrinsic color can be obtained. The first video frame acquired under the auxiliary light being turned off is color-converted through the first mapping relationship to obtain the second video frame, thereby making the first object in the second video frame display in the intrinsic color, achieving the purpose of calibrating the color of the first object to the intrinsic color in the video. Compared with existing video processing technologies, the embodiments of this application do not adjust the color of the first video frame to a color affected by ambient light. Instead, they can obtain the first image and the second image corresponding to the first object by switching the auxiliary light on and off, and obtain the mapping relationship for determining the intrinsic color of the first object under a standard light source. This allows the first object to be displayed in its intrinsic color in the second video, meeting the user's requirement that the subject in the video be displayed in its intrinsic color. This improves the adaptability between color adjustment and usage scenarios, and also increases the flexibility of adjustment.

[0297] Example 2:

[0298] The implementation flow of the video acquisition device in the video data processing method corresponding to Embodiment 1 above is shown in Figure 18, which is a structural block diagram of a video data processing device provided in the embodiment of this application. For ease of explanation, only the parts related to the embodiment of this application are shown.

[0299] As shown in Figure 18, the above-mentioned video data processing device includes:

[0300] Image acquisition unit 181 is used to enable auxiliary light acquisition of a first image in a first shooting mode, and to disable the auxiliary light acquisition of a second image; the first image and the second image are both captured by a first object;

[0301] The first mapping relationship determination unit 182 is used to determine a first mapping relationship based on the first image and the second image;

[0302] The first video frame acquisition unit 183 is used to acquire a first video frame; the subject of the first video frame includes the first object; the first video frame is acquired when the auxiliary light is off.

[0303] The color conversion unit 184 is used to perform color conversion on the first video frame according to the first mapping relationship to obtain a second video frame; in the second video frame, the first object is displayed in the color under a standard light source.

[0304] Optionally, the processing apparatus further includes:

[0305] A shooting activation unit is used to activate the first shooting mode in response to a first operation.

[0306] Optionally, the image acquisition is specifically used to: in the first shooting mode, if a preset triggering condition is met, acquire the first image and acquire the second image.

[0307] Optionally, the triggering condition includes at least one of the following:

[0308] The ambient light in the environment where the first object is located changes;

[0309] The first object has changed;

[0310] The object change feature of the first object is greater than a preset change threshold; the object change feature is determined based on any two video frames containing the first object in the acquired video frames; the object change feature includes at least one of the following: a first change feature in the shooting angle dimension, a second change feature in the color dimension, and a third change feature in the shape dimension.

[0311] Optionally, the first mapping relationship determination unit 183 includes:

[0312] The second mapping relationship determination unit is used to determine a second mapping relationship based on the first image and the second image;

[0313] The mapping relationship conversion unit is used to determine the first mapping relationship based on the preset third mapping relationship and the second mapping relationship; the third mapping relationship is used to map the color under the auxiliary light to the color under the standard light source.

[0314] Optionally, the second mapping relationship determination unit is specifically used to: perform differential processing on the first image and the second image to determine the second mapping relationship.

[0315] Optionally, the first device is equipped with a color camera and a multispectral sensor; the first image is synthesized based on a first color image captured by the color camera and first spectral information captured by the multispectral sensor; the second image is synthesized based on a second color image captured by the color camera and second spectral information captured by the multispectral sensor.

[0316] The second mapping relationship determination unit includes:

[0317] The fourth mapping relationship determination unit is used to determine a fourth mapping relationship based on the first color image and the second color image;

[0318] A spectral information determination unit is used to determine the spectral response information corresponding to the first object based on the first spectral information and the second spectral information.

[0319] The color extension unit is used to perform color mapping extension on the fourth mapping relationship based on the spectral response information and the spectral response curves corresponding to all colors, so as to obtain the second mapping relationship.

[0320] Optionally, the processing apparatus further includes:

[0321] The second video frame display unit is used to display the second video frame.

[0322] Optionally, the second video frame display unit is specifically used to: display the second video frame in response to a second operation that selects a first display mode.

[0323] Optionally, the processing apparatus further includes:

[0324] The second display mode triggering unit is used to respond to a third operation that selects the second display mode and display the first video frame.

[0325] Optionally, the processing apparatus further includes:

[0326] A data transmission unit is configured to transmit second video data to a second device and / or a third device; the second video data is the second video frame or a second video file including the second video frame; the second device is a device for displaying the second video data; and the third device is a device for forwarding the second video data.

[0327] Example 3:

[0328] The implementation flow of another video acquisition device in the video data processing method corresponding to the above embodiment 1 is shown in Figure 19. Figure 19 shows a structural block diagram of another video data processing device provided in the embodiment of this application. For ease of explanation, only the parts related to the embodiment of this application are shown.

[0329] Referring to Figure 19, the above-mentioned video data processing device includes:

[0330] Image acquisition unit 191 is used to activate auxiliary light to acquire a first image in a first shooting mode, and deactivate the auxiliary light to acquire a second image; the first image and the second image are both captured by a first object;

[0331] The first mapping relationship determination unit 192 is used to determine a first mapping relationship based on the first image and the second image;

[0332] The first video frame acquisition unit 193 is used to acquire a first video frame; the subject of the first video frame includes the first object; the first video frame is acquired when the auxiliary light is off.

[0333] The sending unit 194 is configured to send the first mapping relationship and the first video frame to a second device or a third device; the first mapping relationship is used by the second device and / or the third device to perform color conversion on the first video frame to obtain a second video frame; in the second video frame, the first object is displayed in the color under a standard light source; the second device is a device for displaying the second video data; the third device is a device for forwarding the second video data.

[0334] Optionally, the processing apparatus further includes:

[0335] A shooting activation unit is used to activate the first shooting mode in response to a first operation.

[0336] Optionally, the image acquisition unit 191 is specifically used to: in the first shooting mode, if a preset triggering condition is met, acquire the first image and acquire the second image.

[0337] Optionally, the triggering condition includes at least one of the following:

[0338] The ambient light in the environment where the first object is located changes;

[0339] The first object has changed;

[0340] The object change feature of the first object is greater than a preset change threshold; the object change feature is determined based on any two video frames containing the first object in the acquired video frames; the object change feature includes at least one of the following: a first change feature in the shooting angle dimension, a second change feature in the color dimension, and a third change feature in the shape dimension.

[0341] Optionally, the first mapping relationship determination unit 192 includes:

[0342] The second mapping relationship determination unit is used to determine a second mapping relationship based on the first image and the second image;

[0343] The mapping relationship conversion unit is used to determine the first mapping relationship based on the preset third mapping relationship and the second mapping relationship; the third mapping relationship is used to map the color under the auxiliary light to the color under the standard light source.

[0344] Optionally, the second mapping relationship determination unit is specifically used to: perform differential processing on the first image and the second image to determine the second mapping relationship.

[0345] Optionally, the first device is equipped with a color camera and a multispectral sensor; the first image is synthesized based on a first color image captured by the color camera and first spectral information captured by the multispectral sensor; the second image is synthesized based on a second color image captured by the color camera and second spectral information captured by the multispectral sensor.

[0346] The second mapping relationship determination unit includes:

[0347] The fourth mapping relationship determination unit is used to determine a fourth mapping relationship based on the first color image and the second color image;

[0348] A spectral information determination unit is used to determine the spectral response information corresponding to the first object based on the first spectral information and the second spectral information.

[0349] The color extension unit is used to perform color mapping extension on the fourth mapping relationship based on the spectral response information and the spectral response curves corresponding to all colors, so as to obtain the second mapping relationship.

[0350] Optionally, the processing apparatus further includes:

[0351] The processing unit is configured to perform color conversion on the first video frame according to the first mapping relationship to obtain a second video frame.

[0352] Optionally, the processing apparatus further includes:

[0353] The second video frame display unit is used to display the second video frame.

[0354] Optionally, the second video frame display unit is specifically used to: display the second video frame in response to a second operation that selects a first display mode.

[0355] Optionally, the processing apparatus further includes:

[0356] The second display mode triggering unit is used to respond to a third operation that selects the second display mode and display the first video frame.

[0357] Example 4:

[0358] In accordance with the implementation flow of another video display device in the video data processing method of Embodiment 1 above, Figure 20 shows a structural block diagram of another video data processing device provided in the embodiment of this application. For ease of explanation, only the parts related to the embodiment of this application are shown.

[0359] Referring to Figure 20, the above-mentioned video data processing device includes:

[0360] The first mode display unit 201 is used to display a second video frame in a first display mode; the second video frame is obtained by color conversion of a first video frame containing the first object according to a first mapping relationship; the first mapping relationship is determined based on a first image and a second image containing the first object; the first image is acquired when the auxiliary light acquisition is turned on in the first device; the second image is acquired when the auxiliary light acquisition is turned off in the first device; the first video frame is acquired by the first device when the auxiliary light is off; in the second video frame, the first object is displayed in the color under a standard light source.

[0361] Optionally, the processing apparatus further includes:

[0362] The first mapping relationship receiving unit is used to receive the first video frame and the first mapping relationship sent by the first device;

[0363] A color conversion unit is used to perform color conversion on the first video frame according to the first mapping relationship to obtain the second video frame.

[0364] Optionally, the processing apparatus further includes:

[0365] The second mode display unit is used to display the first video frame in response to a third operation that selects a second display mode.

[0366] Optionally, the first mode display unit 201 includes a function specifically configured to: display the second video frame in response to a second operation that selects the first display mode.

[0367] Figure 21 is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. As shown in Figure 21, the electronic device 21 of this embodiment includes: at least one processor 210 (only one processor is shown in Figure 21, and the number of processors may match the actual number of chips included in the electronic device in the embodiment), a memory 211, and a program 212 stored in the memory 211 and executable on the at least one processor 210. When the processor 210 executes the program 212, it implements the steps in any of the above-described video data processing method embodiments.

[0368] The electronic device 21 may be a smartphone, tablet computer, laptop computer, desktop computer, etc. This electronic device may include, but is not limited to, a processor 210 and a memory 211. Those skilled in the art will understand that FIG21 is merely an example of the electronic device 21 and does not constitute a limitation on the electronic device 21. It may include more or fewer components than illustrated, or combine certain components, or different components; for example, it may also include input / output electronic devices, network access electronic devices, etc.

[0369] The processor 210 may be a Central Processing Unit (CPU), or it may be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor.

[0370] In some embodiments, the memory 211 may be an internal storage unit of the electronic device 21, such as a hard disk or memory of the electronic device 21. In other embodiments, the memory 211 may be an external storage device of the electronic device 21, such as a plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, etc., equipped on the electronic device 21. Furthermore, the memory 211 may include both internal storage units and external storage devices of the electronic device 21. The memory 211 is used to store the operating system, applications, bootloader, data, and other programs, such as program code. The memory 211 can also be used to temporarily store data that has been output or will be output.

[0371] It should be noted that the information interaction and execution process between the above-mentioned devices / units are based on the same concept as the method embodiments of this application. For details on their specific functions and technical effects, please refer to the method embodiments section, and they will not be repeated here.

[0372] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is merely an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit. Furthermore, the specific names of the functional units and modules are only for easy differentiation and are not intended to limit the scope of protection of this application. The specific working process of the units and modules in the above system can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0373] This application also provides an electronic device, which includes: at least one processor, a memory, and a computer program stored in the memory and executable on the at least one processor, wherein the processor executes the computer program to implement the steps in any of the above method embodiments.

[0374] This application also provides a readable storage medium storing a program that, when executed by a processor, implements the steps described in the various method embodiments above.

[0375] This application provides a program product that, when run on an electronic device, enables the electronic device to perform the steps described in the various method embodiments above.

[0376] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments of this application can be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable medium can include at least: any entity or device capable of carrying computer program code to a photographic device / electronic device, a recording medium, a computer memory, a read-only memory (ROM), a random access memory (RAM), an electrical carrier signal, a telecommunication signal, and a software distribution medium. Examples include USB flash drives, portable hard drives, magnetic disks, or optical disks. In some jurisdictions, according to legislation and patent practice, computer-readable media cannot be electrical carrier signals or telecommunication signals.

[0377] In the above embodiments, the descriptions of each embodiment have different focuses. For parts that are not described in detail or recorded in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0378] The above-described embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application, and should all be included within the protection scope of this application.

Claims

1. A method for processing video data, characterized in that, Applied to the first device, the processing method includes: In the first shooting mode, the auxiliary light is turned on to capture the first image, and the auxiliary light is turned off to capture the second image; the subject of the first image and the second image includes the first object; Based on the first image and the second image, a first mapping relationship is determined; A first video frame is captured; the subject of the first video frame includes the first object; the first video frame is captured when the auxiliary light is off. Based on the first mapping relationship, the first video frame is color-converted to obtain a second video frame; in the second video frame, the first object is displayed in the color under a standard light source.

2. The processing method according to claim 1, characterized in that, Before enabling auxiliary light acquisition of the first image and disabling auxiliary light acquisition of the second image in the first shooting mode, the method further includes: In response to the first operation, the first shooting mode is activated.

3. The processing method according to claim 1, characterized in that, In the first shooting mode, enabling auxiliary light to capture the first image and disabling auxiliary light to capture the second image includes: In the first shooting mode, if the preset triggering conditions are met, the first image and the second image are captured.

4. The processing method according to claim 3, characterized in that, The triggering condition includes at least one of the following: The ambient light in the environment where the first object is located changes; The first object has changed; The object change feature of the first object is greater than a preset change threshold; the object change feature is determined based on any two video frames containing the first object in the acquired video frames; the object change feature includes at least one of the following: a first change feature in the shooting angle dimension, a second change feature in the color dimension, and a third change feature in the shape dimension.

5. The processing method according to any one of claims 1-4, characterized in that, Determining the first mapping relationship based on the first image and the second image includes: Based on the first image and the second image, determine the second mapping relationship; The first mapping relationship is determined based on the preset third mapping relationship and the second mapping relationship; the third mapping relationship is used to map the color under the auxiliary light to the color under the standard light source.

6. The processing method according to claim 5, characterized in that, Determining the second mapping relationship based on the first image and the second image includes: The first image and the second image are subjected to differential processing to determine the second mapping relationship.

7. The processing method according to claim 5 or 6, characterized in that, The first device is equipped with a color camera and a multispectral sensor; the first image is synthesized based on a first color image captured by the color camera and first spectral information captured by the multispectral sensor; the second image is synthesized based on a second color image captured by the color camera and second spectral information captured by the multispectral sensor. Determining the second mapping relationship based on the first image and the second image includes: Based on the first color image and the second color image, determine the fourth mapping relationship; Based on the first spectral information and the second spectral information, determine the spectral response information corresponding to the first object; Based on the spectral response information and the spectral response curves corresponding to all colors, the fourth mapping relationship is extended by color mapping to obtain the second mapping relationship.

8. The processing method according to any one of claims 1-7, characterized in that, After performing color conversion on the first video frame according to the first mapping relationship to obtain the second video frame, the method further includes: The second video frame is displayed.

9. The processing method according to claim 8, characterized in that, The display of the second video frame includes: In response to the second operation of selecting the first display mode, the second video frame is displayed.

10. The processing method according to any one of claims 1-7, characterized in that, After performing color conversion on the first video frame containing the first object captured in the first shooting mode according to the first mapping relationship to obtain the second video frame, the method further includes: In response to a third operation that selects a second display mode, the first video frame is displayed.

11. The processing method according to any one of claims 1-10, characterized in that, After performing color conversion on the first video frame according to the first mapping relationship to obtain the second video frame, the method further includes: Sending second video data to a second device and / or a third device; the second video data is the second video frame or a second video file including the second video frame; the second device is a device for displaying the second video data; the third device is a device for forwarding the second video data.

12. A method for processing video data, characterized in that, Applied to the first device, the processing method includes: In the first shooting mode, turn on the auxiliary light and capture the first image; The auxiliary light is turned off, and a second image is acquired; the objects captured in both the first and second images include the first object. Based on the first image and the second image, a first mapping relationship is determined; A first video frame is captured; the subject of the first video frame includes the first object; the first video frame is captured when the auxiliary light is off; the first video frame is captured when the auxiliary light is off. The first mapping relationship and the first video frame are sent to a second device or a third device; the first mapping relationship is used by the second device and / or the third device to perform color conversion on the first video frame to obtain a second video frame; in the second video frame, the first object is displayed in the color under a standard light source; the second device is a device for displaying the second video data; the third device is a device for forwarding the second video data.

13. A method for processing video data, characterized in that, Applied to a second device, the processing method includes: In the first display mode, a second video frame is displayed; the second video frame is obtained by color conversion of a first video frame containing the first object according to a first mapping relationship; the first mapping relationship is determined based on a first image and a second image containing the first object; the first image is captured when the auxiliary light is turned on in the first device; the second image is captured when the auxiliary light is turned off in the first device; the first video frame is captured by the first device when the auxiliary light is off; in the second video frame, the first object is displayed in the color under a standard light source.

14. The processing method according to claim 13, characterized in that, Before displaying the second video frame, the method further includes: Receive the first video frame and the first mapping relationship sent by the first device; The first video frame is color-converted according to the first mapping relationship to obtain the second video frame.

15. The processing method according to claim 14, characterized in that, After receiving the first video frame and the first mapping relationship sent by the first device, the method further includes: In response to a third operation that selects a second display mode, the first video frame is displayed.

16. The method according to any one of claims 13-15, characterized in that, The display of the second video frame includes: In response to the second operation of selecting the first display mode, the second video frame is displayed.

17. An electronic device, characterized in that, The electronic device includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the steps of the method as claimed in any one of claims 1 to 11, or the steps of the method as claimed in claim 12, or the steps of the method as claimed in any one of claims 13 to 16.

18. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method as claimed in any one of claims 1 to 11, or the steps of the method as claimed in claim 12, or the steps of the method as claimed in any one of claims 13 to 16.

Images