Display device and screen casting processing method therefor

Abstract

The present application relates to the technical field of screen casting, and provides a display device and a screen casting processing method therefor. The display device comprises: a display screen; a decoder, configured to, in response to a mirroring screen casting request from a source-end device, decode an encoded data stream carried in the mirroring screen casting request, and obtain decoded video data; and a processor, separately connected to the display screen and the decoder, the processor being configured to: analyze effective display areas of a plurality of video frames in the decoded video data, and obtain feature information of the plurality of video frames; if it is determined, on the basis of feature information of any one of the video frames, that an effective display area contains sensitive information, perform blurring processing on the sensitive information on the basis of pixel point information corresponding to the sensitive information in the effective display area, and obtain processed video data; and display the processed video data on the basis of the display screen. The application achieves the technical effect of effectively processing sensitive information in mirrored screen casting content from a source-end device by a display device, thereby protecting user privacy.

Description

Display devices and their projection processing methods

[0001] This application claims priority to Chinese Patent Application No. 202510059819.3, filed on January 14, 2025, entitled "Display Device and Projection Processing Method Thereof", the entire contents of which are incorporated herein by reference. Technical Field

[0002] This application belongs to the field of screen projection technology, and more specifically, relates to a display device and a screen projection processing method thereof. Background Technology

[0003] With the widespread adoption of devices such as smartphones, tablets, and smart TVs, screen mirroring technology has gradually become an important means of connecting different devices and enabling large-screen sharing.

[0004] However, screen mirroring may involve sensitive information such as personal privacy or trade secrets. If this information is leaked, it could cause losses to individuals or businesses. Furthermore, as people become more aware of privacy protection, how to protect sensitive information during screen mirroring has become an urgent problem to be solved.

[0005] However, the best solution currently available in the industry is usually to disconnect the screen mirroring when the user is operating on sensitive information, and then reconnect the screen mirroring after the user has finished operating, which is time-consuming and laborious. Summary of the Invention

[0006] The purpose of this application is to provide a display device and a screen projection processing method thereof, which aims to solve the technical problem that the prior art cannot effectively protect sensitive information in the screen projection video data during the screen projection process.

[0007] To achieve the above objectives, according to a first aspect of this application, a display device is provided, the method comprising:

[0008] Display screen;

[0009] The decoder is configured to respond to a mirroring request from the source device by decoding the encoded data stream carried in the mirroring request to obtain decoded video data.

[0010] The processors connected to the display screen and the decoder respectively are configured as follows:

[0011] Analyze the effective display area of ​​multiple video frames in the decoded video data to obtain the feature information of the multiple video frames;

[0012] If it is determined that the sensitive information is contained in the effective display area based on the feature information of any one of the video pictures, the sensitive information is blurred according to pixel point information corresponding to the sensitive information in the effective display area, and processed video data is obtained.

[0013] The processed video data is displayed based on the display screen.

[0014] Optionally, before the processor performs the analysis of the effective display area of the plurality of video pictures in the decoded video data to obtain the feature information of the plurality of video pictures, the processor is further configured to:

[0015] According to a preset interval duration, video pictures are cut from the decoded video data to obtain the plurality of video pictures.

[0016] Optionally, before the processor performs the analysis of the effective display area of the plurality of video pictures in the decoded video data to obtain the feature information of the plurality of video pictures, the processor is further configured to:

[0017] Obtain pixel points at different coordinate positions in each of the video pictures.

[0018] According to the pixel points in each of the video pictures, determine an invalid display area of each of the video pictures.

[0019] Remove the invalid display area of each of the video pictures to obtain an effective display area of each of the video pictures.

[0020] Optionally, in a possible implementation manner of the first aspect, the processor is further configured to:

[0021] Input the feature information of the plurality of video pictures into a video picture analysis model trained in advance, wherein the video picture analysis model is obtained by training a convolutional neural network model based on a plurality of training samples, and each of the training samples includes feature information of a sample video picture and a sensitive information label.

[0022] Analyze the feature information of each of the video pictures by using the video picture analysis model to obtain a comparison analysis result corresponding to each of the video pictures.

[0023] According to the comparison analysis result corresponding to each of the video pictures, determine whether the sensitive information is contained in each of the effective display areas.

[0024] Optionally, in a possible implementation manner of the first aspect, the processor, after performing the blurring processing on the sensitive information according to the pixel point information corresponding to the sensitive information in the effective display area, is further configured to:

[0025] determine, according to the pixel point information corresponding to the sensitive information in the effective display area, a starting coordinate and a region size parameter corresponding to the sensitive information, wherein the starting coordinate includes a starting horizontal coordinate and a starting vertical coordinate, and the region size parameter includes a width and a height;

[0026] determine a to-be-blurred region in the effective display area according to the starting coordinate and the region size parameter;

[0027] perform blurring processing on pixel points in the to-be-blurred region to obtain the processed video data.

[0028] Optionally, in a possible implementation manner of the first aspect, before performing the blurring processing on the pixel points in the to-be-blurred region, the processor is further configured to:

[0029] identify a boundary region between other regions in the effective display area and the to-be-blurred region;

[0030] change a color value of a pixel point on the boundary region, so that the to-be-blurred region and the other regions are smoothly and uniformly transitioned.

[0031] Optionally, in a possible implementation manner of the first aspect, during the process of displaying the processed video data based on the display screen, or after the process of displaying the processed video data based on the display screen, the processor is further configured to:

[0032] continuously analyze feature information of each video frame by using a video frame analysis model, to obtain a contrast analysis result corresponding to each video frame;

[0033] if it is determined that none of the sensitive information is contained in each of the effective display areas based on the contrast analysis result corresponding to each video frame, the blurring processing on the sensitive information is revoked, to obtain recovered video data;

[0034] display the recovered video data based on the display screen.

[0035] According to a second aspect of the present application, a screen projection processing method of a display device is provided, comprising:

[0036] in response to a mirror screen projection request of a source device, decoding processing is performed on encoded data stream carried in the mirror screen projection request, to obtain decoded video data;

[0037] analyze effective display regions of a plurality of video pictures in decoded video data to obtain feature information of the plurality of video pictures;

[0038] if it is determined, based on the feature information of any one of the video pictures, that sensitive information is contained in the effective display region, perform blur processing on the sensitive information according to pixel point information corresponding to the sensitive information in the effective display region to obtain processed video data;

[0039] display the processed video data based on a display screen of the display device.

[0040] Optionally, in a possible implementation manner of the second aspect, the method further includes:

[0041] input the feature information of the plurality of video pictures into a pre-trained video picture analysis model, wherein the video picture analysis model is obtained by training a convolutional neural network model based on a plurality of training samples, and each training sample includes feature information of a sample video picture and a sensitive information label;

[0042] analyze the feature information of each video picture by using the video picture analysis model to obtain a comparison analysis result corresponding to each video picture;

[0043] determine whether the sensitive information is contained in each effective display region according to the comparison analysis result corresponding to each video picture.

[0044] Optionally, in a possible implementation manner of the second aspect, the performing blur processing on the sensitive information according to the pixel point information corresponding to the sensitive information in the effective display region to obtain processed video data includes:

[0045] determine, according to the pixel point information corresponding to the sensitive information in the effective display region, a starting coordinate corresponding to the sensitive information and a region size parameter, wherein the starting coordinate includes a starting horizontal coordinate and a starting vertical coordinate, and the region size parameter includes a width and a height;

[0046] determine a to-be-blurred region in the effective display region according to the starting coordinate and the region size parameter;

[0047] perform blur processing on pixel points in the to-be-blurred region to obtain the processed video data.

[0048] The second aspect and any implementation form of the second aspect each correspond to the first aspect and any implementation form of the first aspect. The technical effects to which the second aspect and any implementation form of the second aspect correspond can be seen with reference to the technical effects to which the first aspect and any implementation form of the first aspect correspond, which will not be described again here.

[0049] According to a third aspect of the present application, a display device is provided, comprising a memory, a processor and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the computer program, causes the display device to implement the method according to any one of the preceding aspects.

[0050] According to a fourth aspect of the present application, a computer readable storage medium is provided, which stores a computer program, wherein the computer program, when executed by a processor, implements the method according to any one of the preceding aspects.

[0051] According to a fifth aspect of the present application, a computer program product is provided, which, when executed on a display device, causes the display device to perform the method according to any one of the preceding aspects.

[0052] It can be understood that the beneficial effects of the second aspect to the fifth aspect described above can be seen with reference to the related description of the first aspect, which will not be described again here.

[0053] The display device and the screen projection processing method thereof provided by the embodiments of the present application achieve the technical effects of effectively processing sensitive information in the mirror projection content of the source device on the display device and protecting user privacy through the cooperative work of the decoder, the processor and the display screen and the like in the display device, decoding the encoded data stream carried in the mirror projection request of the source device, analyzing the effective display area of the plurality of video pictures in the decoded video data to obtain the feature information of the plurality of video pictures, identifying the sensitive information in the effective display area of the video picture based on the feature information of the video picture, and performing blur processing on the sensitive information according to the pixel point information corresponding to the sensitive information in the effective display area to obtain processed video data, and finally displaying the processed video data on the display screen. BRIEF DESCRIPTION OF DRAWINGS

[0054] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed to be used in the embodiments or prior art description will be briefly introduced as follows. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can also be obtained by those skilled in the art without creative labor.

[0055] FIG. 1 is a structural schematic diagram of a display device according to an embodiment of the present application;

[0056] FIG. 2a is a flow schematic diagram of an optional screen projection processing method according to an embodiment of the present application;

[0057] FIG. 2b is a flow schematic diagram of another optional screen projection processing method according to an embodiment of the present application;

[0058] FIG. 3 is a structural schematic diagram of an optional video picture analysis model according to an embodiment of the present application;

[0059] FIG. 4 is a structural schematic diagram of an optional convolution operation module according to an embodiment of the present application;

[0060] FIG. 5 is a flow schematic diagram of an optional screen projection processing method according to another embodiment of the present application;

[0061] FIG. 6 is a structural schematic diagram of a screen projection processing apparatus according to an embodiment of the present application;

[0062] FIG. 7 is a structural schematic diagram of a display device according to an embodiment of the present application. DETAILED DESCRIPTION

[0063] In the following description, specific details are set forth in order to provide a thorough understanding of the embodiments of the present application. However, persons skilled in the art will understand that the present application can be practiced without these specific details. In other instances, well-known structures, circuits, and processes are not shown in order not to obscure the understanding of this application with unnecessary detail.

[0064] It should be understood that the term "comprising" as used in the specification and in the claims indicates the presence of the recited features, integers, steps, operations, elements, and / or components, but does not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof.

[0065] It should also be understood that the term "and / or" as used herein refers to any combination of associated listed items, and all possible combinations thereof, and includes these combinations.

[0066] As used in the specification and the appended claims herein, the term "if' can be interpreted as meaning "when" or "upon" or "in response to determining" or "in response to detecting" depending on the context. Similarly, the phrase "if it is determined" or "if [the described condition or event] is detected" can be interpreted as meaning "upon determining" or "in response to determining" or "upon detecting [the described condition or event]" or "in response to detecting [the described condition or event]" depending on the context.

[0067] In addition, in the description of the specification and the appended claims herein, the terms "first", "second", "third", etc. are only used to distinguish the description and cannot be understood as indicating or implying relative importance.

[0068] In the specification of the present application, the reference "one embodiment" or "some embodiments" and the like means that the specific features, structures or characteristics described in connection with the embodiment are included in one or more embodiments of the present application. Therefore, the statements "in one embodiment", "in some embodiments", "in other some embodiments", "in further some embodiments" and the like appearing in different places in the specification are not necessarily all referring to the same embodiment, but mean "one or more but not all embodiments", unless otherwise specifically emphasized. The terms "include", "contain", "have" and their variants mean "include but not limited to", unless otherwise specifically emphasized.

[0069] First, some terms in the embodiments of the present application are explained and described to facilitate understanding by those skilled in the art.

[0070] Mirror projection: refers to copying and projecting the entire screen content of a source device (such as a mobile phone, tablet, computer, etc.) including interface layout, operation process, display effect, etc. to a display device (such as a smart TV, projector, display screen, etc. receiving device) in real time and as is.

[0071] Adaptive Bayer filter: in the field of image processing, it is a filter used for Raw domain single frame noise reduction, which first removes noise and edges by bilateral filter, and then performs soft threshold processing.

[0072] The above is a brief introduction to the terms involved in the embodiments of the present application, which will not be described hereinafter.

[0073] The present application provides an example of a display device, please refer to Fig. 1, Fig. 1 shows a structural schematic diagram of a display device provided by the present application, the display device 100 includes: display screen 101, decoder 102, processor 103, the processor 103 is connected with display screen 101 and decoder 102 respectively, wherein:

[0074] The decoder 102 is configured to decode the encoded data stream carried in the mirror projection request in response to the mirror projection request of the source device, to obtain decoded video data.

[0075] The processor 103 is configured to analyze the effective display area of the plurality of video pictures in the decoded video data to obtain feature information of the plurality of video pictures, and if it is determined based on the feature information of any one video picture that the sensitive information is contained in the effective display area, to perform blurring processing on the sensitive information according to the pixel point information corresponding to the sensitive information in the effective display area, to obtain processed video data, and to display the processed video data on the display screen.

[0076] Optionally, the display device provided in the examples of the present application can be applied in the following technical scenarios, but is not limited thereto: the display device is in communication connection with the source device, and when the source device uses wireless technology to perform mirror projection on the display device, the sensitive information in the video data of the mirror projection is protected.

[0077] Optionally, in the examples of the present application, the display screen can be understood as a visual window through which the display device is configured to present image content to a user, and the type of the display screen can be, for example, a liquid crystal display screen (LCD), an organic light-emitting diode display screen (OLED), etc.

[0078] Optionally, the decoder is configured to decode the encoded data stream of a specific encoding format (for example, H.264 encoding format, H.265 encoding format, etc.) received by the display device, to obtain decoded video data.

[0079] When the source device (such as a mobile phone, a computer, etc.) initiates a mirror projection request to the display device through any kind of projection application (such as miracast and airpaly, etc.), after the projection application in the display device receives the mirror projection request, the encoded data stream carried in the mirror projection request is forwarded to the decoder, and the decoder decodes the encoded data stream according to the corresponding video encoding rule (for example, H.264 encoding standard, H.265 encoding standard, etc.), to restore the encoded data stream into decoded video data which can be further processed and displayed by the display device.

[0080] In the examples of the present application, as shown in FIG. 2a, the processor in the display device is connected with the display screen and the decoder respectively, and the processor is configured to receive the decoded video data output by the decoder, and analyze the plurality of video pictures contained in the decoded video data to obtain the feature information corresponding to each of the plurality of video pictures.

[0081] It should be understood that the aforementioned multiple video frames refer to multiple static images at different points in time in the decoded video data, while the effective display area of ​​each video frame refers to the part of each video frame that actually carries meaningful image content and / or image content that needs to be displayed to the user. This effective display area is different from the invalid display areas such as black screen areas in the video frame.

[0082] In this application example, the processor can use image analysis algorithms to determine the effective display area in the video frame by analyzing the color, brightness, texture and other features of the pixels in each video frame, and extract the feature information of each video frame based on the effective display area of ​​each video frame. The feature information may include, but is not limited to, software content features (such as whether the application currently displayed in the video frame is a communication software, shopping software, video software or online banking, and the specific text information, control information, etc. involved in the frame), color distribution characteristics (such as whether the overall tone of the video frame involves sensitive information, etc.), and frame layout (such as the positional relationship of each element in the video frame, etc.).

[0083] The processor analyzes the feature information of each video frame obtained from the analysis based on an image feature recognition algorithm to determine whether sensitive information is contained within the effective display area. In this application example, the aforementioned sensitive information may include, but is not limited to, content involving user privacy (such as communication software, unobstructed account password information, private chat logs, etc.), or content that does not comply with relevant regulations, is prohibited, or is inappropriate to display (such as violent, pornographic, or other inappropriate image elements). As shown in Figure 2a, if any type of sensitive information is determined to exist within the effective display area based on the feature information of a certain video frame, the processor performs a blurring operation on the sensitive information according to the pixel information corresponding to the sensitive information within the effective display area.

[0084] It should be understood that the aforementioned pixel information includes the position coordinates of each pixel in the video frame and the color value it represents (for example, in RGB color mode, the specific values ​​of the red, green, and blue color channels for each pixel). Specifically, a specific blurring algorithm can be used to process the pixels in the effective display area where sensitive information is located, changing the original color values ​​and other information of the pixels in the sensitive information area, thereby making the sensitive information blurry and difficult to identify. After the above blurring processing, the processed video data is obtained.

[0085] As shown in Figure 2a, after the processor completes the blurring of sensitive information and obtains the processed video data, it transmits the processed video data to the display screen in the display device. Upon receiving the processed video data, the display screen displays the image content corresponding to the processed video data to the user according to a pre-set display principle and driving mechanism. This ensures that the user ultimately sees a processed video image with sensitive information blurred, guaranteeing that the content displayed by the display device meets user privacy protection and related display requirements.

[0086] As an optional specific embodiment, as shown in Figure 2b, the screen mirroring application of the source device responds to the user's operation, generates a screen mirroring request, and sends the screen mirroring request to the screen mirroring application of the display device. After receiving the screen mirroring request, the screen mirroring application wakes up the image abstraction layer of the display device and the display screen of the display device, and sends the screen mirroring request to the decoder through the screen mirroring application of the display device. The decoder decodes the encoded data stream carried in the screen mirroring request to obtain the decoded video data, and sends the decoded video data to the display screen. The display screen displays the decoded video data through the image display abstraction layer.

[0087] Next, the screen recording module of the display device captures multiple video frames from the decoded video data, and removes invalid display areas from each video frame. Then, the recognition algorithm module in the processor analyzes the valid display areas of the multiple video frames in the decoded video data to obtain feature information of each video frame; based on the feature information of each video frame, it is determined that sensitive information is contained within the valid display area. The processor's recognition algorithm module sends a notification message to the screen mirroring application on the display device to notify that sensitive information is contained within the valid display area. Then, the blurring module in the processor blurs the sensitive information according to the pixel information corresponding to the sensitive information within the valid display area, obtaining processed video data. Finally, the blurring module in the processor sends the processed video data to the display screen, where it is displayed.

[0088] By working collaboratively with components such as the decoder, processor, and display screen in the display device, the encoded data stream carried in the mirroring request from the source device is first decoded. Then, the effective display areas of multiple video frames in the decoded video data are analyzed to obtain the feature information of multiple video frames. Based on the feature information of the video frames, sensitive information within the effective display area of ​​the video frames is identified. According to the pixel information corresponding to the sensitive information within the effective display area, the sensitive information is blurred to obtain the processed video data. Finally, the processed video data is displayed on the display screen, achieving the technical effect of effectively processing sensitive information in the mirrored content of the source device and protecting user privacy.

[0089] It should be noted that this application example does not have specific requirements for the network environment. Since the resolution of the display screen on the source device is generally 1080P, the minimum network bandwidth needs to be sufficient to support the transmission rate of the video stream. Furthermore, this application example does not have specific requirements for the model of the source device; it only needs to support the screen mirroring protocol between the source device and the display device.

[0090] In one possible implementation, during the process of the display device processing the mirrored content projected from the source device, before the processor analyzes the effective display areas of multiple video frames in the decoded video data and obtains the feature information of the multiple video frames, it is further configured as follows:

[0091] Based on a preset interval, video frames are extracted from the decoded video data to obtain multiple video frames.

[0092] Specifically, the aforementioned preset interval duration is a time interval parameter preset by the display device. For example, it can be preset to 500 milliseconds (the specific duration can be determined according to actual needs and display device performance, etc., and this application example does not specifically limit this) as a time interval duration, so as to ensure that the captured video frames can cover a certain range of video content, but are not too frequent (avoiding excessive data processing that would put too much pressure on the display device's performance) or too sparse (ensuring that important screen information is not missed).

[0093] In one example, the processor can perform video frame capture operations on the decoded video data, which has already been processed by the decoder, based on a preset interval and the capture interface in the display device. This is done by transmitting the current frame's layer (which node in the decoded video data it belongs to) and the starting coordinates (x, y), width, and height of the frame to be captured. It should be understood that the decoded video data is a collection of video frame information arranged in chronological order. The quality of the captured frames is consistent with the resolution of the current video stream, and the capture speed is within 100ms / frame, without affecting the performance of the display device. The captured video frame information exists in digital form, containing pixel data, color information, and other content for each video frame.

[0094] For example, assuming the preset interval is 1 second, and the decoded video data corresponds to a 10-second video content, the processor can extract the corresponding video frame from the entire decoded video data at the 1st second, 2nd second, 3rd second, and so on, until the 10th second, to obtain a total of 10 video frames.

[0095] The processor performs video frame capture operations on the decoded video data at preset intervals, providing the necessary data foundation for the processor to conduct in-depth analysis of the video frames and ensure that the content displayed on the screen meets the requirements for sensitive information protection.

[0096] In one possible implementation, before the processor performs analysis on the effective display areas of multiple video frames in the decoded video data to obtain the feature information of the multiple video frames, it is further configured as follows:

[0097] Obtain the pixel points at different coordinates in each video frame;

[0098] Based on the pixels in each video frame, determine the invalid display area of ​​each video frame;

[0099] Remove the invalid display area from each video frame to obtain the valid display area for each video frame.

[0100] In this application example, before analyzing the effective display area of ​​the video frame to obtain feature information, the processor needs to preprocess each video frame. Specifically, by acquiring pixels at different coordinate positions in each video frame, invalid display areas are identified and removed to obtain the effective display area for each video frame. The purpose of this is to eliminate irrelevant parts in each video frame that might interfere with the processor's analysis and judgment, ensuring that feature information can be extracted more accurately based on the accurate effective display area, thereby effectively determining whether sensitive information exists in the video frame.

[0101] It should be understood that, essentially, each video frame is composed of multiple pixels. These pixels are distributed in a certain arrangement (such as a two-dimensional row and column layout) within the video frame, and each pixel has a corresponding coordinate position. For example, in an image coordinate system with the top-left corner as the origin (coordinate value (0,0)), the horizontal axis is the x-axis and the vertical axis is the y-axis. Each pixel can be represented by a unique (x,y) coordinate to indicate its position in the frame.

[0102] The processor obtains all pixels at different coordinate positions by accessing data of each pixel in each video frame, such as the coordinate position of each pixel in the video frame and the color information represented by the pixel. Based on the pixels in each video frame, the processor determines the invalid display area of ​​each video frame.

[0103] Optionally, invalid display areas refer to parts of the video frame that do not carry valid visual content and do not need to participate in subsequent analysis, such as black screen areas or parts without images due to transmission errors or device compatibility issues. As an optional example, the processor may determine invalid display areas based on the acquired pixel information using, but is not limited to, the following two methods:

[0104] Color value-based judgment: Under normal circumstances, the color values ​​of pixels in invalid display areas are relatively simple and conform to specific characteristics. For example, in RGB color mode, the values ​​of the red, green and blue channels of pixels in black screen areas are close to 0 (i.e., they appear black). The processor traverses each pixel in each video frame. If it detects whether the color value of each pixel conforms to the color value range of the pixels in the invalid display area mentioned above, it marks it as an invalid display area.

[0105] Judging by pixel distribution patterns: In addition to color values, the distribution patterns of pixels can also help identify invalid display areas. For example, if the color values ​​of pixels in a certain area show no pattern of change, or if there is no reasonable transition between the pixels and the surrounding normally displayed areas, then this area may be a missing part of the image due to data loss or errors, i.e., an invalid display area. By comprehensively analyzing the color values ​​and distribution patterns of pixels in the video frame, the processor can determine the specific locations and ranges of invalid display areas in each video frame.

[0106] After identifying invalid display areas, the processor performs corresponding operations to remove them, resulting in an area containing only valid visual content, i.e., the valid display area. In this application example, the removal of invalid display areas can be achieved through image cropping, data culling, or other methods. For example, if the upper left corner of a video frame is determined to be an invalid display area, and the coordinates form a rectangular area with an horizontal coordinate from 0 to 100 and a vertical coordinate from 0 to 50, the processor removes the pixel data within the aforementioned rectangular area from the video frame's pixel data set by adjusting the index range of the image data, retaining only the pixel data corresponding to the remaining valid display area. This facilitates further feature analysis and other operations based on the valid display area, ensuring the accuracy and effectiveness of the video frame processing flow.

[0107] In one possible implementation, the processor is further configured as follows:

[0108] The feature information of multiple video frames is input into a pre-trained video frame analysis model. The video frame analysis model is obtained by training a convolutional neural network model based on multiple training samples. Each training sample includes: feature information of the sample video frame and sensitive information labels.

[0109] By using a video frame analysis model, the feature information of each video frame is analyzed to obtain the comparative analysis results for each video frame.

[0110] Based on the comparative analysis results of each video frame, determine whether each effective display area contains sensitive information.

[0111] In the screen projection processing flow of this display device, the processor uses a pre-trained video image analysis model to determine whether the effective display area contains sensitive information. This video image analysis model is obtained by pre-training a Convolutional Neural Network (CNN) model based on a large number of training samples. Each training sample contains two important pieces of information: feature information of the sample video image and a sensitive information label. The feature information of the sample video image is the feature information obtained by feature extraction, including various features such as texture, color distribution, object shape, and contour. The sensitive information label indicates whether the sample video image contains sensitive information; for example, "1" indicates the presence of sensitive information, and "0" indicates the absence of sensitive information. Through training, the video image analysis model can learn the features of sensitive information, thereby enabling it to analyze and judge new video images, improving the accuracy and efficiency of detecting sensitive information in video images.

[0112] It should be understood that the Convolutional Neural Network (CNN) model is a type of deep learning network model, widely used in fields such as image recognition and analysis. Through structures such as convolutional layers, pooling layers, and fully connected layers, it can automatically learn feature information in images, and is particularly adept at extracting local features from images and combining these local features into higher-level feature representations.

[0113] The processor extracts feature information from multiple video frames as input data for a pre-trained video frame analysis model. Using the knowledge learned during training, the model employs any computational analysis method, including but not limited to thresholding, random forests, multilayer perceptrons (MLP), and deep learning, to analyze the feature information of each video frame, yielding comparative analysis results for each frame. As shown in Figure 3, each square represents a computational module, which is a superposition of computational methods such as convolution, depthwise separable convolution, and linear integration units. The connection between adjacent modules is indicated by arrows. Horizontal connections between adjacent modules (horizontal arrows) represent the direct input of the result of the previous computational module into the next.

[0114] In the video frame analysis model, convolutional layers first perform convolution operations on the feature information of the input video frame to automatically extract features at different levels. After multiple convolutions and pooling operations, the extracted features are finally mapped to the final output result through a fully connected layer, thus obtaining the comparative analysis result for each video frame. Optionally, the comparative analysis result can be a probability value representing the likelihood that the video frame contains sensitive information, or a category label, such as "contains sensitive information" or "does not contain sensitive information".

[0115] The processor determines whether the effective display area contains sensitive information based on the comparative analysis results obtained from the video image analysis model. For example, if the comparative analysis result is a probability value, a threshold can be set, such as 0.5. When the probability value is greater than the predetermined threshold, the processor determines that the effective display area of ​​the video image contains sensitive information. As another example, if the comparative analysis result is a category label, the processor directly determines whether sensitive information exists based on the label.

[0116] In one example, the input data of the video image analysis model consists of an I-row, J-column array matrix (input matrix), and the input data consists of an M-row, N-column array matrix and a convolutional kernel. input(m,n) represents the pixel value at the m-th row and n-th column position in the input matrix. Then, the output result output(i,j) is: output(i,j) = ∑ M ∑ N input(m,n)×kernel(im,jn).

[0117] Where m represents the row index of the input matrix (from 1 to M), n represents the column index of the input matrix (from 1 to N) to traverse all positions of the input matrix, i represents the row index of the output matrix (corresponding to the row position of the output), and j represents the column index of the output matrix (corresponding to the column position of the output).

[0118] In video image analysis models, depthwise separable convolution modules are variations of convolution modules. Based on empirical results, the original convolution module can be added to or replaced within the depthwise separable convolution module. For a linear integration unit, all pixels in the intermediate operation matrix undergo a single nonlinear calculation. The nonlinear function f(x) of the linear integration unit is: f(x) = max(0,x).

[0119] In one alternative embodiment, any nonlinear function can replace the linear integration unit, such as the sigmoid function, tanh function, etc. However, in the experimental results of this embodiment, the linear integration unit has the lowest computational cost and the best performance.

[0120] In some embodiments, as shown in Figure 4, the convolution operation module is executed sequentially by convolution calculation, linear integration unit, depthwise separable convolution operation, linear integration unit, and convolution operation. The calculation process is the process of encoding or decoding the input image into abstract features.

[0121] Optionally, in this application example, some sample video frames with the same resolution as the video stream data to be projected later, and labeled with sensitive information values ​​of 0 and 1, can be obtained in advance for pre-training. In the cross-entropy calculation involving the Sigmoid function, the video frame analysis model obtained in the final training takes the maximum value of the two output values ​​[value1, value2] between 0 and 1 as the final output result. Specifically, the output result predict of the video frame analysis model is the predicted binary classification result.

[0122] The training loss, Loss, can be calculated from the cross-entropy, i.e.: Loss = label × ln[σ(predict)] + [1-label] × ln[σ(predict)];

[0123] In the above formula, σ is the Sigmoid function, i.e.:

[0124] Leveraging the powerful feature learning and classification capabilities of deep learning models, processors can more intelligently and accurately identify sensitive information in video footage, enhancing the display device's ability to detect and process sensitive information. Furthermore, based on whether the effective display area contains sensitive information, the processor can take corresponding processing measures for the effective display area containing sensitive information in subsequent screen projection processes, such as blurring or other security protection operations, to ensure that the screen projection content played by the display device meets user information security and privacy requirements.

[0125] In one possible implementation, the processor performs blurring processing on the sensitive information based on the pixel information corresponding to the sensitive information within the effective display area, obtaining processed video data, which is further configured as follows:

[0126] Based on the pixel information corresponding to the sensitive information within the effective display area, determine the starting coordinates and area size parameters corresponding to the sensitive information.

[0127] Based on the starting coordinates and area size parameters, determine the area to be blurred within the effective display area.

[0128] The pixels within the blurred area are blurred to obtain the processed video data.

[0129] Optionally, the starting coordinates include: starting x-coordinate and starting y-coordinate, and the area size parameters include: width and height.

[0130] In this application example, in the processing flow of the display device, after the processor determines that there is sensitive information in the effective display area, it determines the specific area range where the sensitive information is located, and blurs the pixels in that area to finally obtain the processed video data in order to protect privacy or comply with relevant regulations.

[0131] Based on the pixel information corresponding to the sensitive information within the effective display area, the processor first determines the starting x-coordinate and starting y-coordinate of the sensitive information in the video frame. These starting x-coordinate and starting y-coordinate together determine the starting coordinates of the area containing the sensitive information. For example, in an image coordinate system with its origin (0,0) at the top left corner, for a sensitive information located in the video frame, the starting x-coordinate and starting y-coordinate are (300, 200), indicating that the sensitive information area begins at a position with an x-coordinate of 300 pixels and a y-coordinate of 200 pixels.

[0132] Simultaneously, the processor also needs to determine the region size parameters of the area where the sensitive information is located based on the pixel information corresponding to the sensitive information. In this application example, the region size parameters are represented by two parameters: width and height. Specifically, the width represents the number of pixels occupied by the sensitive information region in the horizontal direction, and the height represents the number of pixels occupied by the sensitive information region in the vertical direction. Based on the starting horizontal coordinate (x) and starting vertical coordinate (y) of the sensitive information region, as well as the width and height of the sensitive information region, the extent of the sensitive information region can be accurately described.

[0133] Based on the starting coordinates and region size parameters corresponding to the sensitive information, the processor can define a region within the effective display area that needs to be blurred, thus avoiding interference with other normal display areas. It should be noted that this blurred region is formed by using the starting coordinates as the top-left vertex of a rectangular area, and determining the length and width of the region to be blurred using the width and height parameters. For example, with starting coordinates (300, 200), a width of 100 pixels, and a height of 50 pixels, the region to be blurred can be understood as a rectangular area with its top-left vertex at (300, 200) and its bottom-right vertex at (400, 250).

[0134] In one example, the processor can call the Video Blur interface to blur pixels within a region to be blurred. For instance, it can change the color value of a pixel to blur image details in that area, making sensitive information difficult to identify. An alternative blurring method is to use a weighted average algorithm, calculating a weighted average of the color value of each pixel within the region to be blurred with the color values ​​of its surrounding pixels. For example, in RGB color mode, for any pixel requiring blurring, a new, smoother color value can be obtained by calculating the weighted average based on the red, green, and blue channel color values ​​of the surrounding pixels.

[0135] The processor iterates through all pixels within the region to be blurred, calculating the effect based on a selected blurring algorithm (such as Gaussian blur). This blurs pixels that originally clearly displayed sensitive information, gradually eliminating the image outline and details of the region, thus achieving the desired effect of blurring sensitive information.

[0136] After the processor blurs the pixels within the blurry area, the original sensitive information areas in the video data are blurred. The sensitive information areas in the final processed video data become blurred, while other unprocessed areas remain unchanged. The processor then transmits the processed video data containing the blurred sensitive information to the display screen. When the display screen plays the video data, the user sees the blurred video image. This allows for normal viewing of non-sensitive video data while preventing the leakage of sensitive information in the video data, thus ensuring the security and compliance of screen projection on display devices.

[0137] In one possible implementation, the processor is further configured to: Before performing blurring processing on the pixels within the region to be blurred, the processor is also configured to:

[0138] Identify the boundary region between other areas in the valid display area and the area to be blurred;

[0139] Change the color values ​​of pixels on the boundary area to make the area to be blurred transition smoothly and evenly with other areas.

[0140] When a display device processes the area containing sensitive information in a video frame, it's not enough to simply blur the pixels within that area; the transition between the blurred area and other normally displayed areas in the video frame must also be considered. By adjusting the color values ​​of the pixels in the boundary areas between these areas and other regions, a smooth and uniform transition is achieved, avoiding noticeable visual breaks. This results in a more natural and harmonious video display, enhancing the user's viewing experience and ensuring visual continuity after the sensitive information blurring process.

[0141] Specifically, in this application example, the boundary region is located within the effective display area, situated between the area to be blurred (i.e., the area containing sensitive information that needs to be blurred) and other normally displayed areas in the image that are not affected by the blurring. It can be understood as a "strip" area surrounding the area to be blurred, the width of which can be determined by preset rules or based on the actual situation of the video image; for example, it can be preset to a width range of N pixels.

[0142] The processor calls the Video Blur interface and uses the HW module (hardware) to identify regions in the image where color changes form contour lines. These regions are typically areas with significant color transitions, i.e., boundary or edge regions. It then determines whether each pixel is within a predetermined range extending outward from the edge of the region to be blurred (this predetermined range is set based on the width of the boundary region). If a pixel is within this predetermined range, it is considered to belong to the boundary region.

[0143] For example, assuming the starting coordinates of a region to be blurred are (100, 100), the width of the region is 50 pixels, the height is 30 pixels, and the width of the boundary region is set to 2 pixels, then pixels with horizontal coordinates ranging from 98 to 102 and 148 to 152, and vertical coordinates ranging from 98 to 102 and 128 to 132 will be identified as pixels within the boundary region.

[0144] To achieve a smooth and uniform transition, the hardware module applies a filter (such as an adaptive Bayer filter, ABF) to adjust the color values ​​of pixels within the boundary region. In one example, the processor can determine the new color value of the pixel to be adjusted based on the color difference between its neighboring pixels and its relative position to the blurred area and other areas.

[0145] For example, in RGB color mode, for pixels within a boundary region that are closer to the area to be blurred, the color value of that pixel can be adjusted to gradually approach the color value of the pixels in the area to be blurred after the blurring process. For pixels that are closer to other normal areas, the color value of that pixel can be adjusted to approach the color value of the pixels in the normal area. Through the above color uniformity adjustment method, the boundary region can naturally transition from the color of the area to be blurred to the color of other areas, reducing sharp edges in the image.

[0146] In one optional example, it is assumed that after the region to be blurred is blurred, the RGB color value of the edge pixels of the region to be blurred is (100, 100, 100), while the color value of the pixels of other normal regions adjacent to the region to be blurred is (200, 200, 200), and the width of the boundary region is 3 pixels. For the row of pixels closest to the area to be blurred within the boundary region, the processor can adjust the color value of this row of pixels to be close to (100, 100, 100) but slightly transitioning to (200, 200, 200) according to a predetermined algorithm (such as a linear interpolation algorithm, which assigns color value weights according to the distance ratio to the two adjacent regions), such as (120, 120, 120). For the pixels in the middle of the boundary region, the color value is further adjusted to be closer to the color value of the normal region, such as (150, 150, 150). For the row of pixels closest to the normal region within the boundary region, the color value is adjusted to be close to the color value of the normal region but with a slight influence of the color characteristics of the area to be blurred, such as (180, 180, 180).

[0147] Through the above example, before blurring the pixels in the area to be blurred, the processor adjusts the color values ​​of the pixels in the boundary area row by row and point by point. This can achieve a smooth and uniform color transition between the area to be blurred and other areas, ensuring that when the processed video data is displayed, the transition from the area to be blurred to other normal areas is natural and smooth, and the entire video picture will not have obvious visual disjointness.

[0148] In one possible implementation, the processor is further configured, either during or after executing the video data processed based on the display screen, to:

[0149] By continuously analyzing the feature information of each video frame using a video frame analysis model, comparative analysis results are obtained for each video frame.

[0150] If, based on the comparative analysis results corresponding to each video frame, it is determined that no sensitive information is contained within each effective display area, then the blurring of the sensitive information is removed, and the restored video data is obtained.

[0151] The restored video data is displayed on the screen.

[0152] In the process of processing projected video data on display devices, it is not only necessary to detect and blur any potentially sensitive information in the initial stage and then display the processed video data, but also to continuously monitor the subsequent video frames. A video frame analysis model continuously determines whether sensitive information still exists in the video frame. If no sensitive information is found in any effective display area of ​​subsequent video frames, the blurring of pixels in the previously blurred areas is reversed, restoring the original clear state of the video frame. This ensures that, while meeting information security requirements, clear and complete viewing content is provided to users to the greatest extent possible, thereby improving the user viewing experience.

[0153] During the process of displaying processed video data on the screen, that is, while the video is playing or after the processed video data is displayed, the processor continuously acquires new video frames at regular intervals (the specific time interval can be set according to factors such as device performance and actual needs).

[0154] The processor extracts feature information from multiple video frames as input and passes it to a pre-trained video frame analysis model. The model then analyzes the feature information of each video frame using the knowledge it has learned during training. For example, it compares the feature information of pre-acquired sample video frames with the feature information of each newly acquired video frame to determine the similarity between each video frame and the sample video frames, thereby obtaining the comparative analysis results for each video frame.

[0155] The processor determines whether sensitive information still exists within the effective display area of ​​each video frame based on the comparative analysis results output by the video frame analysis model. If, after analysis, the effective display area of ​​all video frames no longer contains content that is similar to the characteristics of the sample video frames and can be identified as sensitive information, then the current video frame is determined to meet the conditions for canceling the blurring process.

[0156] Since the area containing sensitive information is clearly defined during blurring, the processor can then reverse the process to restore the color values ​​and other information of pixels within that area to their original, unblurred state. For example, if a weighted average algorithm was previously used to change the color values ​​of pixels within the blurring area, the processor can now reverse the process, restoring the color values ​​of these pixels to their original state before the sensitive information blurring was performed, thus obtaining the restored video data.

[0157] After receiving the recovered video data, the processor transmits it to the display screen. The display screen, following display principles and driving mechanisms, displays the image content corresponding to the recovered video data, thus restoring the user's view of the video from a blurred state back to its original clear state. When the projected video no longer contains sensitive information, the display device adjusts its display status in real time to present all video content completely and clearly, ensuring the security and compliance of projected information while optimizing the user's viewing experience as much as possible.

[0158] This application provides an example of a screen projection processing method for a display device. Please refer to Figure 5, which shows a schematic flowchart of a screen projection processing method for a display device provided in this application. This is an example and not a limitation; the method can be applied to or run on any display device. The method includes:

[0159] S501 responds to the mirroring request from the source device by decoding the encoded data stream carried in the mirroring request to obtain the decoded video data.

[0160] S502 analyzes the effective display area of ​​multiple video frames in the decoded video data to obtain the feature information of multiple video frames.

[0161] S503: If it is determined that sensitive information is contained within the effective display area based on the feature information of any video frame, then the sensitive information is blurred according to the pixel information corresponding to the sensitive information within the effective display area to obtain the processed video data.

[0162] S504 displays processed video data on the display screen of a display device.

[0163] It should be understood that the aforementioned multiple video frames refer to multiple static images at different points in time in the decoded video data, while the effective display area of ​​each video frame refers to the part of each video frame that actually carries meaningful image content and / or image content that needs to be displayed to the user. This effective display area is different from invalid black screen areas that may appear in the video frame.

[0164] In this application example, an image analysis algorithm can be used to determine the effective display area of ​​the video frame by analyzing the color, brightness, texture and other features of the pixels in each video frame, and to extract the feature information of each video frame based on the effective display area of ​​each video frame. The feature information may include, but is not limited to, software content features (such as whether the application currently displayed in the video frame is a communication software, shopping software, video software or online banking, and the specific text information, control information, etc. involved in the frame), color distribution characteristics (such as whether the overall tone of the video frame involves sensitive information, etc.), and frame layout (such as the positional relationship of each element in the video frame, etc.).

[0165] In this application example, an image feature recognition algorithm is used to analyze the feature information of each video frame individually to determine whether sensitive information is contained within the effective display area. Optionally, the aforementioned sensitive information may include, but is not limited to, content involving user privacy (such as unobstructed account password information, private chat logs, etc.), or content that does not comply with relevant regulations, is prohibited, or is inappropriate to display (such as violent, pornographic, or other inappropriate visual elements). If any type of sensitive information is determined to exist within the effective display area based on the feature information of a certain video frame, then a blurring operation is performed on the sensitive information according to the pixel information corresponding to the sensitive information within the effective display area.

[0166] It should be understood that the aforementioned pixel information includes the position coordinates of each pixel in the video frame and the color value it represents (for example, in RGB color mode, the specific values ​​of the red, green, and blue channels for each pixel). Specifically, a specific blurring algorithm can be used to process the pixels in the area containing sensitive information within the effective display area, changing the original color values ​​and other information of the pixels in that area, thereby making the sensitive information blurry and difficult to identify. After the above blurring process, the processed video data is obtained.

[0167] After blurring the sensitive information and obtaining the processed video data, the processed video data is transmitted to the display screen in the display device. Upon receiving the processed video data, the display screen displays the image content corresponding to the processed video data to the user according to the pre-set display principles and driving mechanisms. This ensures that the user ultimately sees a processed video image with sensitive information blurred and protected, guaranteeing that the content displayed on the display device meets user privacy protection and related display requirements.

[0168] By first decoding the encoded data stream carried in the mirroring request from the source device, then analyzing the effective display areas of multiple video frames in the decoded video data, feature information of multiple video frames is obtained. Sensitive information within the effective display area of ​​the video frame is identified based on the feature information of the video frame. The sensitive information is then blurred according to the pixel information corresponding to the sensitive information within the effective display area to obtain processed video data. Finally, the processed video data is displayed on the screen, achieving the technical effect of effectively processing sensitive information in the mirrored content of the source device and protecting user privacy.

[0169] In one possible implementation, the above method further includes:

[0170] The feature information of multiple video frames is input into a pre-trained video frame analysis model. The video frame analysis model is obtained by training a convolutional neural network model based on multiple training samples. Each training sample includes: feature information of the sample video frame and sensitive information labels.

[0171] By using a video frame analysis model, the feature information of each video frame is analyzed to obtain the comparative analysis results for each video frame.

[0172] Based on the comparative analysis results of each video frame, determine whether each effective display area contains sensitive information.

[0173] In the screen projection process of this display device, a pre-trained video image analysis model can be used to determine whether sensitive information is contained within the effective display area. This video image analysis model is obtained by pre-training a Convolutional Neural Network (CNN) model based on a large number of training samples. Each training sample contains two important pieces of information: feature information of the sample video image and a sensitive information label. The feature information of the sample video image is obtained by feature extraction, including various features such as texture, color distribution, object shape, and contour. The sensitive information label indicates whether the sample video image contains sensitive information; for example, "1" indicates the presence of sensitive information, and "0" indicates the absence of sensitive information. Through training, the video image analysis model can learn the features of sensitive information, thereby enabling it to analyze and judge new video images, improving the accuracy and efficiency of detecting sensitive information in video images.

[0174] It should be understood that the Convolutional Neural Network (CNN) model is a type of deep learning network model, widely used in fields such as image recognition and analysis. Through structures such as convolutional layers, pooling layers, and fully connected layers, it can automatically learn feature information in images, and is particularly adept at extracting local features from images and combining these local features into higher-level feature representations.

[0175] In one example, feature information extracted from multiple video frames is used as input data for a pre-trained video frame analysis model. Leveraging the knowledge learned during training, the model analyzes the feature information of each video frame to obtain a comparative analysis result for each frame. In the video frame analysis model, convolutional layers first perform convolution operations on the input feature information to automatically extract features at different levels. After multiple convolutions and pooling operations, a fully connected layer maps the extracted features to the final output, yielding the comparative analysis result for each video frame. Optionally, the comparative analysis result can be a probability value representing the likelihood that the video frame contains sensitive information, or a category label, such as "contains sensitive information" or "does not contain sensitive information."

[0176] The system determines whether sensitive information is contained within the effective display area based on the comparative analysis results obtained from the video image analysis model. For example, if the comparative analysis result is a probability value, a threshold can be set, such as 0.5. When the probability value is greater than the predetermined threshold, it is determined that the effective display area of ​​the video image contains sensitive information. Alternatively, if the comparative analysis result is a category label, the presence of sensitive information in the effective display area of ​​the video image can be determined based on the category label.

[0177] Leveraging the powerful feature learning and classification capabilities of deep learning models, sensitive information in video footage can be identified more intelligently and accurately, enhancing the display device's ability to detect and process sensitive information. Furthermore, based on whether the effective display area contains sensitive information, corresponding processing measures can be taken for the effective display area containing sensitive information in subsequent screen projection processes, such as blurring or other security protection operations, to ensure that the screen projection content played by the display device meets user information security and privacy requirements.

[0178] In one possible implementation, the sensitive information is blurred based on the pixel information corresponding to the sensitive information within the effective display area to obtain processed video data, including:

[0179] Based on the pixel information corresponding to the sensitive information within the effective display area, determine the starting coordinates and area size parameters corresponding to the sensitive information.

[0180] Based on the starting coordinates and area size parameters, determine the area to be blurred within the effective display area.

[0181] The pixels within the blurred area are blurred to obtain the processed video data.

[0182] Optionally, the above starting coordinates include: starting x-coordinate and starting y-coordinate, and the area size parameters include: width and height.

[0183] In this application example, in the processing flow of the display device, after it is determined that there is sensitive information in the effective display area, the specific area range where the sensitive information is located is determined, and the pixels in the area are blurred to finally obtain the processed video data in order to protect privacy or comply with relevant regulations.

[0184] Based on the pixel information corresponding to the sensitive information within the effective display area, the starting x-coordinate and starting y-coordinate of the sensitive information in the video frame are first determined. These starting x-coordinate and starting y-coordinate together determine the starting coordinates of the area where the sensitive information is located. For example, in an image coordinate system with the top-left corner as the origin (0,0), for a sensitive information located in the video frame, the starting x-coordinate and starting y-coordinate are (300, 200), indicating that the sensitive information area begins at a position with a x-coordinate of 300 pixels and a y-coordinate of 200 pixels.

[0185] Furthermore, the region size parameters of the area containing the sensitive information can be determined based on the pixel information corresponding to the sensitive information. In this application example, the region size parameters are represented by two parameters: width and height. Specifically, the width represents the number of pixels occupied by the sensitive information region in the horizontal direction, and the height represents the number of pixels occupied by the sensitive information region in the vertical direction. Based on the starting horizontal coordinate (x) and starting vertical coordinate (y) of the sensitive information region, as well as the width and height of the sensitive information region, the extent of the sensitive information region can be accurately described.

[0186] Based on the starting coordinates and area size parameters corresponding to the sensitive information, a region to be blurred can be defined within the effective display area to avoid interference with other normal display areas. It should be noted that this blurred region is formed by using the starting coordinates as the top-left vertex of a rectangular area, and determining the length and width of the region to be blurred using the width and height parameters in the area size parameters. For example, with starting coordinates (300, 200), a width of 100 pixels, and a height of 50 pixels, the region to be blurred can be understood as a rectangular area with its top-left vertex at (300, 200) and its bottom-right vertex at (400, 250).

[0187] In one example, the processor can call the Video Blur interface to blur pixels within a region to be blurred. For instance, it can change the color value of a pixel to blur image details in that area, making sensitive information difficult to identify. An alternative blurring method is to use a weighted average algorithm, calculating a weighted average of the color value of each pixel within the region to be blurred with the color values ​​of its surrounding pixels. For example, in RGB color mode, for any pixel requiring blurring, a new, smoother color value can be obtained by calculating the weighted average based on the red, green, and blue channel color values ​​of the surrounding pixels.

[0188] By iterating through all pixels within the region to be blurred, and calculating the blurring algorithm (such as Gaussian blur), pixels that originally clearly displayed sensitive information become blurred, and the image outline and details of the region gradually disappear, thus achieving the effect of blurring sensitive information.

[0189] After blurring the pixels within the blurred area, the original sensitive information areas in the video data are blurred, resulting in the sensitive information areas in the final processed video data becoming unclear, while other unprocessed areas remain unchanged. The processed video data containing the blurred sensitive information is then transmitted to the display screen, so that when the display screen plays the video data, the user sees the blurred video image. This allows for normal viewing of non-sensitive video data while preventing the leakage of sensitive information in the video data, thus ensuring the security and compliance of screen projection on display devices.

[0190] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application. Industrial applicability

[0191] The solution provided in this application can be used in the field of display technology. Through the collaborative work of components such as decoder, processor and display screen in the display device, the encoded data stream carried in the mirror projection request of the source device is first decoded, and then the effective display area of ​​multiple video frames in the decoded video data is analyzed to obtain the feature information of multiple video frames. Based on the feature information of the video frames, sensitive information in the effective display area of ​​the video frames is identified, and the sensitive information is blurred according to the pixel information corresponding to the sensitive information in the effective display area to obtain the processed video data. Finally, the processed video data is displayed on the display screen, realizing the technical effect of effectively processing sensitive information in the mirror projection content of the source device and protecting user privacy.

[0192] Corresponding to the screen projection processing method of the display device in the above embodiments, Figure 6 is a structural schematic diagram of a screen projection processing device of a display device provided in an embodiment of this application. The device can be implemented by software, hardware or a combination of both as part or all of a computer device, which can be the display device shown in Figure 7.

[0193] Referring to Figure 6, the projection processing device of the display device includes:

[0194] The decoding module 601 is configured to respond to the mirroring request from the source device, decode the encoded data stream carried in the mirroring request, and obtain the decoded video data.

[0195] Analysis module 602 is configured to analyze the effective display area of ​​multiple video frames in the decoded video data to obtain feature information of multiple video frames;

[0196] The blurring module 603 is configured to, if it is determined that sensitive information is contained in the effective display area based on the feature information of any video frame, then blur the sensitive information according to the pixel information corresponding to the sensitive information in the effective display area to obtain the processed video data.

[0197] Display module 604 is configured to display processed video data on the display screen of a display device.

[0198] Furthermore, based on any of the above embodiments, as an example of this application, the apparatus further includes:

[0199] The input module is configured to input the feature information of multiple video frames into a pre-trained video frame analysis model. The video frame analysis model is obtained by training a convolutional neural network model based on multiple training samples. Each training sample includes: feature information of the sample video frame and sensitive information labels.

[0200] The analysis module is configured to analyze the feature information of each video frame using a video frame analysis model, and obtain the comparative analysis results for each video frame.

[0201] The determination module is configured to determine whether each valid display area contains sensitive information based on the comparison and analysis results corresponding to each video frame.

[0202] Furthermore, based on any of the above embodiments, as an example of this application, the fuzzing module includes:

[0203] The first determining submodule is configured to determine the starting coordinates and region size parameters corresponding to the sensitive information based on the pixel information corresponding to the sensitive information within the effective display area. The starting coordinates include the starting horizontal coordinate and the starting vertical coordinate, and the region size parameters include the width and height.

[0204] The second determining submodule is configured to determine the area to be blurred within the effective display area based on the starting coordinates and area size parameters;

[0205] The blur processing submodule is configured to blur the pixels in the area to be blurred, thereby obtaining the processed video data.

[0206] It should be noted that the screen projection processing device of the display device provided in the above embodiments is only an example of the division of the above functional modules. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above.

[0207] The functional units and modules in the above embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit. Furthermore, the specific names of the functional units and modules are only for easy differentiation and are not intended to limit the scope of protection of the embodiments of this application.

[0208] It should be noted that the information interaction and execution process between the above-mentioned devices / units are based on the same concept as the method embodiments of this application. For details on their specific functions and technical effects, please refer to the method embodiments section, and they will not be repeated here.

[0209] This application embodiment also provides a display device, which includes one or more processors and a memory;

[0210] The memory is coupled to one or more processors. The memory is used to store computer program code, which includes computer instructions. One or more processors call the computer instructions to cause the display device to perform the screen projection processing method of the display device described above.

[0211] Figure 7 is a schematic diagram of a display device provided in an embodiment of this application. The display device 700 can be a mobile phone, smart screen, tablet computer, wearable display device, in-vehicle display device, augmented reality (AR) device, virtual reality (VR) device, laptop computer, ultra-mobile personal computer (UMPC), netbook, personal digital assistant (PDA), projector, or a communication device such as a server, storage device, or base station, or a smart car, etc. This application embodiment does not impose any limitations on the specific type of display device.

[0212] The memory 701 can be used to store computer software programs 702 and modules. The processor 703 executes various functional applications and data processing of the display device by running the software programs and modules stored in the memory 701. The memory 701 may mainly include a program storage area and a data storage area. The program storage area may store the operating system, application programs required for at least one function (such as sound playback function, image playback function, etc.), etc.; the data storage area may store data created according to the use of the display device (such as audio data, telephone directory, etc.). In addition, the memory 701 may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device, flash memory device, or other volatile solid-state storage device.

[0213] The processor 703 may include one or more processors such as a central processing unit (CPU), an application processor (AP), and a baseband processor. The processor can serve as the nerve center and command center of the wireless router. The processor 703 can generate operation control signals based on instruction opcodes and timing signals to control instruction fetching and execution. The memory 701 can be used to store executable program code, including instructions. The processor 703 executes various functional applications and data processing of the network device by running the instructions stored in the memory. The memory 701 may include a program storage area and a data storage area, such as storing data for audio signals to be played. For example, the memory may be Double Data Rate Synchronous Dynamic Random Access Memory (DDR) or Flash memory.

[0214] This application also provides a computer-readable storage medium storing computer instructions; when the computer-readable storage medium is run on a display device, the display device executes the aforementioned screen projection processing method.

[0215] The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or can include one or more data storage devices such as servers or data centers that can be integrated with media. The available medium can be magnetic media (e.g., floppy disks, hard disks, magnetic tapes), optical media, or semiconductor media (e.g., solid-state disks (SSDs)).

[0216] This application also provides a computer program product containing computer instructions, which, when run on a display device, enables the display device to execute the aforementioned screen projection processing method.

[0217] The computer storage medium and computer program product provided in the embodiments of this application are used to execute the methods provided above. Therefore, the beneficial effects they can achieve can be referred to the beneficial effects corresponding to the methods provided above, and will not be repeated here.

[0218] In the above embodiments, implementation can also be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product. The computer program product includes one or more computer instructions. When the computer instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another via wired (e.g., coaxial cable, fiber optic, Digital Subscriber Line, DSL) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium accessible to a computer, or a data storage device such as a server or data center that integrates one or more available media. The available media can be magnetic media (e.g., floppy disks, hard disks, magnetic tapes), optical media (e.g., Digital Versatile Discs (DVDs)), or semiconductor media (e.g., Solid State Disks (SSDs)).

[0219] In the above embodiments, the descriptions of each embodiment have different focuses. For parts that are not described in detail or recorded in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0220] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments claimed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0221] In the embodiments provided in this application, it should be understood that the disclosed apparatus / network devices and methods can be implemented in other ways. For example, the apparatus / network device embodiments described above are merely illustrative. For instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0222] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0223] The above-described embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application, and should all be included within the protection scope of this application.

Claims

1. A display device, wherein, include: Display screen; The decoder is configured to respond to a mirroring request from the source device by decoding the encoded data stream carried in the mirroring request to obtain decoded video data. The processors connected to the display screen and the decoder respectively are configured as follows: Analyze the effective display area of ​​multiple video frames in the decoded video data to obtain the feature information of the multiple video frames; If sensitive information is determined to be contained within an effective display area based on the feature information of any one of the video frames, then the sensitive information is blurred according to the pixel information corresponding to the sensitive information within the effective display area to obtain the processed video data. The processed video data is displayed on the screen.

2. The display device according to claim 1, wherein, Before the processor performs the analysis of the effective display areas of multiple video frames in the decoded video data to obtain the feature information of the multiple video frames, it is further configured to: Based on a preset interval, video frames are extracted from the decoded video data to obtain the multiple video frames.

3. The display device according to claim 1, wherein, Before the processor performs the analysis of the effective display areas of multiple video frames in the decoded video data to obtain the feature information of the multiple video frames, it is further configured to: Obtain pixels at different coordinate positions in each of the video frames; Based on the pixels in each video frame, determine the invalid display area of ​​each video frame; Remove the invalid display area of ​​each video frame to obtain the valid display area of ​​each video frame.

4. The display device according to claim 1, wherein, The processor is also configured to: The feature information of the multiple video frames is input into a pre-trained video frame analysis model, wherein the video frame analysis model is obtained by training a convolutional neural network model based on multiple training samples, and each training sample includes: feature information of the sample video frame and sensitive information labels; The video frame analysis model is used to analyze the feature information of each video frame to obtain the comparative analysis results for each video frame. Based on the comparison and analysis results corresponding to each video frame, it is determined whether each effective display area contains the sensitive information.

5. The display device according to claim 1, wherein, The processor executes the process of blurring the sensitive information based on the pixel information corresponding to the sensitive information within the effective display area to obtain processed video data, and is further configured as follows: Based on the pixel information corresponding to the sensitive information within the effective display area, the starting coordinates and area size parameters corresponding to the sensitive information are determined, wherein the starting coordinates include: starting horizontal coordinate and starting vertical coordinate, and the area size parameters include: width and height; Based on the starting coordinates and area size parameters, determine the area to be blurred within the effective display area; The pixels within the area to be blurred are blurred to obtain the processed video data.

6. The display device according to claim 5, wherein, Before the processor performs blurring processing on the pixels within the area to be blurred, it is also configured to: Identify the boundary region between other areas in the effective display area and the area to be blurred; Change the color value of the pixels on the boundary region to make the area to be blurred transition smoothly and evenly with the other regions.

7. The display device according to any one of claims 1 to 5, wherein, The processor is further configured, either during or after executing the process of displaying the processed video data on the display screen: By continuously analyzing the feature information of each video frame using a video frame analysis model, comparative analysis results are obtained for each video frame. If, based on the comparative analysis results corresponding to each video frame, it is determined that no sensitive information is contained in each of the effective display areas, then the blurring processing of the sensitive information is canceled, and the restored video data is obtained; The restored video data is displayed on the screen.

8. A method for screen projection processing of a display device, wherein, include: In response to a mirroring request from the source device, the encoded data stream carried in the mirroring request is decoded to obtain decoded video data. Analyze the effective display area of ​​multiple video frames in the decoded video data to obtain the feature information of the multiple video frames; If sensitive information is determined to be contained within an effective display area based on the feature information of any one of the video frames, then the sensitive information is blurred according to the pixel information corresponding to the sensitive information within the effective display area to obtain the processed video data. The processed video data is displayed on the screen of the display device.

9. The method according to claim 8, wherein, The method further includes: The feature information of the multiple video frames is input into a pre-trained video frame analysis model, wherein the video frame analysis model is obtained by training a convolutional neural network model based on multiple training samples, and each training sample includes: feature information of the sample video frame and sensitive information labels; The video frame analysis model is used to analyze the feature information of each video frame to obtain the comparative analysis results for each video frame. Based on the comparison and analysis results corresponding to each video frame, it is determined whether each effective display area contains the sensitive information.

10. The method according to claim 8, wherein, The step of blurring the sensitive information based on the pixel information corresponding to the sensitive information within the effective display area to obtain processed video data includes: Based on the pixel information corresponding to the sensitive information within the effective display area, the starting coordinates and area size parameters corresponding to the sensitive information are determined, wherein the starting coordinates include: starting horizontal coordinate and starting vertical coordinate, and the area size parameters include: width and height; Based on the starting coordinates and area size parameters, determine the area to be blurred within the effective display area; The pixels within the area to be blurred are blurred to obtain the processed video data.

Images