Video processing apparatus and video processing method
Patent Information
- Application Number
- PCT/CN2026/077321
- Authority / Receiving Office
- WO · WO
- Patent Type
- Applications
- Current Assignee / Owner
- Priority Date
- 2025-03-11
- Filing Date
- 2026-02-05
- Publication Date
- 2026-09-17
Smart Images
Figure CN2026077321_17092026_PF_FP_ABST
Abstract
Description
Image processing device and image processing method
[0001] Cross-references to related applications
[0002] This application claims priority to Japanese Patent Application No. 2025-037918, filed on March 11, 2025, entitled “Video Processing Apparatus and Video Processing Method”, the entire contents of which are incorporated herein by reference. Technical Field
[0003] This application relates to an image processing apparatus and an image processing method. Background Technology
[0004] In video processing devices such as television display units, various image processing techniques (video processing) are performed to improve image quality. Furthermore, videos can contain a wide variety of scenes, such as night scenes, sports, stage performances, and boxing matches. Moreover, it is desirable to adjust parameters related to contrast, color correction, noise removal, and sharpening differently depending on the scene being processed.
[0005] Therefore, in existing technologies, there exists a method such as the following: First, each image constituting the view is classified into one of several pre-defined scene categories using AI (Artificial Intelligence) technology. Then, image processing is performed using parameters corresponding to that scene.
[0006] By doing this, if the scene classification results for each image are correct, good image processing can be achieved.
[0007] Existing technical documents
[0008] Patent documents
[0009] Patent Document 1: Japanese Patent Application Publication No. 2017-92636
[0010] Patent Document 2: Japanese Patent Application Publication No. 2016-52013. Summary of the Invention
[0011] However, in the aforementioned prior art, image processing uses individual images for scene classification. Therefore, there are instances where the scene classification results for each image are incorrect. For example, even for the same scene, the objects contained in the images differ between an image taken just before the camera switches (e.g., a distant image of a large audience and stage during a stage performance) and an image taken immediately after the camera switches (e.g., a close-up image of performers on stage during a stage performance). Consequently, they are sometimes classified as different scenes.
[0012] Specifically, in previous scene determination using AI technology, distant images of numerous audience members and the stage during a stage performance were identified as stage performance scenes because the objects specific to stage performances were within the image. However, when the images are close-up shots of performers on stage, the objects specific to stage performances are not within the image, making it impossible to determine whether the close-up shot of the performers is from a movie or theater scene or from performers on stage.
[0013] Image quality is usually set based on the scene, but there are also standard image quality settings without any particularly emphasized parameters. When the scene cannot be determined, a standard image quality setting has to be used. For example, a close-up shot of an artist on stage during a performance becomes the standard image quality setting, resulting in an image that lacks the sense of being there on stage and is therefore unsatisfactory.
[0014] Therefore, the objective of this implementation is to provide a video processing apparatus and a video processing method capable of determining with high accuracy whether a scene in a video is continuously identical.
[0015] The video processing apparatus of the embodiment includes: a scene classification unit that classifies a first image constituting a video and a subsequent second image into any one of a pre-defined plurality of scene types using a first learning completion model; an object determination unit that determines whether a predetermined object exists in the first image and the second image using a second learning completion model; and a persistence determination unit that determines whether the same scene in the first image and the second image persists based on the classification result of the scene classification unit and the determination result of the object determination unit. Attached Figure Description
[0016] Figure 1 is a diagram showing the structure of the image processing apparatus according to the first embodiment;
[0017] Figure 2 is a flowchart illustrating the processing performed by the image processing apparatus of the first embodiment;
[0018] Figure 3 is an explanatory diagram of an example of the first processing result performed by the image processing apparatus of the first embodiment;
[0019] Figure 4 is an explanatory diagram of an example of the second processing result performed by the image processing apparatus of the first embodiment;
[0020] Figure 5 is a diagram showing the structure of the image processing apparatus according to the second embodiment;
[0021] Figure 6 is a flowchart illustrating the processing performed by the video processing apparatus according to the second embodiment;
[0022] Figure 7 is a diagram showing the structure of the image processing apparatus according to the third embodiment;
[0023] Figure 8 is a flowchart illustrating the processing performed by the video processing apparatus according to the third embodiment.
[0024] Explanation of reference numerals in the attached drawings: 1…Scene classification unit, 2…Object determination unit, 3…Continuity determination unit, 4…Image processing setting unit, 5…Image processing unit, 6…Scene change estimation unit, 7…Image feature detection unit, 100…Image processing device. Detailed Implementation
[0025] Hereinafter, embodiments (first to third embodiments) of the video processing apparatus and video processing method of this application will be described in detail with reference to the accompanying drawings. It should be noted that, from the second embodiment onwards, descriptions of the same matters as previously described will be appropriately omitted.
[0026] First Implementation Method
[0027] Figure 1 is a diagram showing the structure of the video processing apparatus 100 according to the first embodiment. The video processing apparatus 100 includes a scene classification unit 1, an object determination unit 2, a persistence determination unit 3, a video processing setting unit 4, and a video processing unit 5. Each unit 1 to 5 is, for example, composed of a processor using a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a FPGA (Field Programmable Gate Array), an NPU (Neural Processing Unit), or other hardware with equivalent functions.
[0028] The input video is supplied to the scene classification unit 1, the object determination unit 2, and the video processing unit 5. Hereinafter, the input video will also be referred to as "video".
[0029] The scene classification unit 1 classifies the first image and the subsequent second image constituting the view into one of a pre-defined multiple scene categories using a first learning completion model. Furthermore, the second image can be either an image immediately following the first image, or an image located a predetermined number of times after the first image.
[0030] Furthermore, the scenes can be pre-defined, such as night scenes, sports, stage performances, boxing matches, etc., and others. Additionally, the first learned model can employ various inference models constructed from CNNs (Convolutional Neural Networks), Transformers, Random Forests, Support Vector Machines, etc. The scene classification results from the scene classification unit 1 are fed to the continuous decision unit 3.
[0031] Furthermore, the classification results performed by the scene classification unit 1 may also be incorrect. For example, in the case of an image that is a close-up shot of an artist on a stage performing a stage show, as mentioned above, it may be difficult to determine that it is a stage performance scene, and it may be determined to be another scene.
[0032] The object determination unit 2 uses a second learned model to determine whether a specified object (e.g., a person) exists for the first image and the second image, respectively. The second learned model can be any inference model constructed using CNNs, Transformers, Random Forests, Support Vector Machines, etc. Furthermore, in addition to determining whether the specified object exists, the object determination unit 2 can also determine whether it is a "person holding a microphone," a "person holding a musical instrument," a "group wearing dresses," or a "black background," etc.
[0033] The persistence determination unit 3 determines whether the same scene in the first image and the second image persists based on the classification result (scene classification result) of the scene classification unit 1 and the determination result (object determination result) of the object determination unit 2. The following three examples illustrate this.
[0034] The first and second completed learning models can be stored in non-volatile storage media (such as HDDs, SSDs, etc., not shown), or they can be accessed via a network by configuring external communication functions in the video processing device 100. Furthermore, the first and second completed learning models can use the same inference model.
[0035] first case
[0036] Regarding the first image, the scene classification result is "stage performance".
[0037] Regarding the second image, the scene classification result is "stage performance".
[0038] In this case, the scene is determined to be continuous regardless of the object determination results of the first image and the second image.
[0039] Second case
[0040] Regarding the first image, the scene classification result is "stage performance".
[0041] Regarding the second image, the scene classification result is "other", and the object determination result is "no people".
[0042] In this case, it is determined that the scene is not continuous.
[0043] Third case
[0044] Regarding the first image, the scene classification result is "stage performance".
[0045] Regarding the second image, the scene classification result is "other", and the object determination result is "person present".
[0046] In this case, it is determined to be a "continuous scene". However, in this case, if the second image is an image in which people exist outside the stage performance, it is also determined to be a "continuous scene". Therefore, for example, if it is set to determine "continuous scene" when the object determination result satisfies at least one of the following conditions: "people exist", "person holding a microphone", "person holding an instrument", "group wearing women's dresses", and "black background", the possibility of false determination can be reduced.
[0047] The parameters referred to below are those related to contrast, color correction, noise removal, and sharpening during image processing.
[0048] The video processing setting unit 4 obtains the scene continuity determination result from the continuity determination unit 3. Then, the video processing setting unit 4 does not change the parameters when the scene continues. However, when the scene does not continue, the video processing setting unit 4 changes (sets) the parameters to the content corresponding to the new scene.
[0049] The image processing unit 5 performs image processing on the input image using parameters and outputs the output image. For example, the image processing unit 5 performs various image processing operations on the input image, such as contrast adjustment, color correction, noise removal, and sharpening. By supplying the output image from the image processing unit 5 to a display device (not shown), an image based on the output image is displayed on the display screen of the display device.
[0050] Figure 2 is a flowchart illustrating the processing performed by the video processing apparatus 100 according to the first embodiment. In step S11, the scene classification unit 1 classifies the first image constituting the video and the subsequent second image into any one of a pre-defined multiple types of scenes using a first learning completion model.
[0051] Next, in step S12, the object determination unit 2 uses the second learning completion model to determine whether a specified object exists for the first image and the second image, respectively.
[0052] Next, in step S13, the persistence determination unit 3 determines whether the same scene in the first image and the second image persists based on the scene classification result in step S11 and the object determination result in step S12.
[0053] Next, in step S14, the video processing setting unit 4 performs video processing settings corresponding to the scene persistence determination result.
[0054] Next, in step S15, the image processing unit 5 performs image processing on the input image using parameters and outputs the output image.
[0055] Figure 3 is an explanatory diagram of an example of the first processing result performed by the video processing apparatus 100 according to the first embodiment. In this example, the actual stage performance scene continues as (a), (b), and (c).
[0056] In Figure 3, regarding the scene classification results, "stage performance" is found in (a) and (c), and "other" is found in (b). Therefore, with existing technology, when the scene changes from image (a) to image (b), it is mistakenly determined that the scene is not continuous.
[0057] However, according to this embodiment, when the image changes from image (a) to image (b), as explained in the third example above, if the object determination result for image (b) is "a person exists", then it is determined that "the scene continues". Therefore, the possibility of such misdetermination can be reduced.
[0058] Figure 4 is an explanatory diagram of an example of the second processing result performed by the video processing apparatus 100 of the first embodiment. In this example, although the stage performance scene is actually (a) and (b), the scene is not continuous (b) and (c) (a switch has been made). (c) is another (street) scene.
[0059] In Figure 4, the scene classification results are "stage performance" in (a) and "other" in (b) and (c).
[0060] When the image changes from (a) to (b), the result is the same as in Figure 3, and it is determined to be "scene continuous". However, when the image changes from (b) to (c), if the object determination result for image (c) is "no characters", then the scene is "other", meaning it is determined to be "scene not continuous". Therefore, the correct result for scene continuity can be obtained.
[0061] Thus, according to the first embodiment, for both the first image constituting the view and the subsequent second image, it is possible to determine with high accuracy whether the same scene continues, using scene classification results and object determination results. Therefore, for each image, image processing can be performed with appropriate parameters, enabling the production and display of high-quality images.
[0062] Second Implementation Method
[0063] Next, the second embodiment will be described. FIG5 is a diagram showing the structure of the video processing apparatus 100 according to the second embodiment. Compared with the case of the first embodiment (FIG. 1), the video processing apparatus 100 further includes a scene change estimation unit 6.
[0064] The scene change estimation unit 6 calculates the similarity between the first image and the second image, and estimates whether a scene change has occurred based on the similarity. Specifically, the following methods can be considered as methods for calculating the similarity.
[0065] (1) The sum of the absolute values of the differences between each pixel is calculated as the sum of absolute differences (SAD). The smaller the value, the higher the similarity.
[0066] (2) The mean squared error (MSE) is calculated by averaging the squared differences of each pixel. The smaller the value, the higher the similarity.
[0067] (3) The calculation of SSIM (Structural Similarity Index), an index used to evaluate the similarity between two images. SSIM can evaluate the structural similarity of images by using a method that approximates human visual characteristics.
[0068] (4) Obtain the histograms of the two images and calculate the average of the squared errors obtained from the differences in values of each region. Examples of histograms include luminance histograms, RGB (red, green, blue) histograms, and hue histograms. Smaller values indicate higher similarity.
[0069] Alternatively, one could consider using a deep learning model to calculate the similarity between two images and infer whether a scene change has occurred based on the similarity.
[0070] The scene change estimation unit 6 outputs the estimation result of whether a scene change has occurred to the continuous determination unit 3.
[0071] Furthermore, the continuity determination unit 3 determines whether the same scene in the first image and the second image continues based on the classification result of the scene classification unit 1, the determination result of the object determination unit 2, and the estimation result (scene change estimation result) of the scene change estimation unit 6. Compared with the case of the first embodiment, the scene continuity determination result can be improved accordingly by using the scene change estimation result.
[0072] Figure 6 is a flowchart illustrating the processing performed by the video processing apparatus 100 according to the second embodiment. Steps S21 and S22 are the same as steps S11 and S12 in Figure 2.
[0073] After step S22, in step S23, the scene change estimation unit 6 calculates the similarity between the first image and the second image, and estimates whether a scene change has occurred based on the similarity.
[0074] Next, in step S24, the persistence determination unit 3 determines whether the same scene in the first image and the second image persists based on the scene classification result of step S21, the object determination result of step S22, and the scene change estimation result of step S23. Steps S25 and S26 are the same as steps S14 and S15 in FIG2.
[0075] Thus, according to the second embodiment, compared with the first embodiment, by further using the scenario change estimation results, it is possible to determine with higher accuracy whether the same scenario continues. Therefore, for each image, image processing can be performed with more appropriate parameters, and higher quality images can be produced and displayed.
[0076] Third Implementation Method
[0077] Next, the third embodiment will be described. FIG7 is a diagram showing the structure of the video processing apparatus 100 according to the third embodiment. Compared with the case of the second embodiment (FIG. 5), the video processing apparatus 100 further includes an image feature detection unit 7.
[0078] The image feature detection unit 7 detects prescribed image features for the first image and the second image respectively. The image feature detection unit 7 detects image features (indicators) by performing processes such as hue histogram conversion, frequency histogram conversion, brightness histogram conversion, and reduction on the image, and sends the detection results to the continuity determination unit 3.
[0079] Then, the persistence determination unit 3 determines whether the same scene in the first image and the second image persists based on the classification result of the scene classification unit 1, the determination result of the object determination unit 2, the estimation result of the scene change estimation unit 6, and the detection result of the image feature detection unit 7. Compared with the case of the second embodiment, by further using the detection result of image features, the scene persistence determination result can be improved accordingly.
[0080] Figure 8 is a flowchart illustrating the processing performed by the video processing apparatus according to the third embodiment. Steps S31 to S33 are the same as steps S21 to S23 in Figure 6.
[0081] After step S33, in step S34, the image feature detection unit 7 detects specified image features for the first image and the second image respectively.
[0082] Next, in step S35, the persistence determination unit 3 determines whether the same scene in the first image and the second image persists based on the classification result of the scene classification unit 1, the determination result of the object determination unit 2, the estimation result of the scene change estimation unit 6, and the detection result of the image feature detection unit 7. Steps S36 and S37 are the same as steps S25 and S26 in FIG6.
[0083] Thus, according to the third embodiment, compared with the second embodiment, by further utilizing the detection results of image features, the scene persistence determination results can be improved accordingly. Therefore, for each image, image processing can be performed with more appropriate parameters, and higher quality images can be produced and displayed.
[0084] Furthermore, the program executed in the video processing apparatus 100 of this embodiment can be provided as an installable or executable file on a computer-readable storage medium such as a CD (Compact Disc)-ROM (Read Only Memory), floppy disk (FD), CD-R (Recordable), or DVD (Digital Versatile Disk). Alternatively, the program can also be provided or distributed via a network such as the Internet.
[0085] Several embodiments of this application have been described; however, these embodiments are shown as examples and are not intended to limit the scope of the application. These new embodiments can be implemented in various other forms, and various omissions, substitutions, and modifications can be made without departing from the spirit of the application. These embodiments or their variations are included in the scope or spirit of the application, and are also included in the technical solutions described in the claims and their equivalents.
[0086] For example, in the above embodiments, people are used as the object, but it is not limited to this. In addition, it can also be other kinds of objects such as animals, plants, buildings, food and drink.
Claims
1. A video processing apparatus, wherein, The video processing device includes: The scene classification department uses a first learning completion model to classify the first image constituting the view and the subsequent second image into any one of a pre-defined multiple scene categories. The object determination unit uses a second learning completion model to determine whether a specified object exists for the first image and the second image, respectively. as well as The persistence determination unit determines whether the same scene in the first image and the second image is persistent, based on the classification result of the scene classification unit and the determination result of the object determination unit.
2. The video processing apparatus according to claim 1, wherein, The image processing device further includes a scene change estimation unit, which calculates the similarity between the first image and the second image, and estimates whether a scene change has occurred based on the similarity. The persistence determination unit determines whether the same scene in the first image and the second image is persistent based on the classification result of the scene classification unit, the determination result of the object determination unit, and the estimation result of the scene change estimation unit.
3. The video processing apparatus according to claim 2, wherein, The image processing device further includes an image feature detection unit, which detects predetermined image features for both the first image and the second image. The persistence determination unit determines whether the same scene in the first image and the second image is persistent based on the classification result of the scene classification unit, the determination result of the object determination unit, the estimation result of the scene change estimation unit, and the detection result of the image feature detection unit.
4. A video processing method, wherein, The video processing method includes: In the classification step, the scene classification part uses a first learning completion model to classify the first image constituting the view and the subsequent second image into any one of a pre-defined multiple scene categories. In the object determination step, the object determination part uses a second learned model to determine whether a specified object exists for both the first image and the second image; and The continuous determination step involves the continuous determination unit determining, based on the classification result of the classification step and the determination result of the object determination step, whether the same scene in the first image and the second image is continuous.