Video processing system, video processing method, and video processing program

Abstract

According to the present disclosure, it is possible to provide a video processing system, a video processing method, and a video processing program capable of efficiently performing mask processing on an object in a video. The video processing system according to the present disclosure is provided with: an acquisition unit that acquires an instruction sentence in which identification information indicating a type of an object and attribute information indicating an attribute of the object are written in natural language; a detection unit that detects the object in video data on the basis of the identification information and the attribute information; and a mask processing unit that executes mask processing on the object in the video data.

Description

Image Processing System, Image Processing Method, and Image Processing Program

[0001] The present disclosure relates to an image processing system, an image processing method, and an image processing program.

[0002] With the spread of surveillance cameras, surveillance cameras are installed in various places. The images captured by surveillance cameras are used for various purposes. Therefore, from the perspective of personal information protection, mask processing may be performed on the captured images to conceal, for example, the person himself or the windows of vehicles so that the people in the images cannot be identified (for example, Patent Document 1).

[0003] Japanese Patent Application Laid-Open No. 2024-094745

[0004] To perform mask processing on a desired object in an image, it is first necessary to detect the object from the image. For example, to detect a specific object from the same type of objects, the user needs to perform preprocessing such as associating bounding boxes with each other, region division, identifying other elements such as buildings and the ground, and then specifying the position of the object in the image frame. When the number of objects in the image is large or the object is far away in the image frame, the preprocessing becomes complicated and the burden on the user increases.

[0005] The present disclosure aims to provide an image processing system, an image processing method, and an image processing program that can efficiently perform mask processing on objects in an image.

[0006] The image processing system according to the present disclosure includes an acquisition unit that acquires an instruction sentence in which identification information indicating the type of an object and attribute information indicating the attribute of the object are expressed in natural language, a detection unit that detects the object in the image data based on the identification information and the attribute information, and a mask processing unit that executes mask processing on the object in the image data.

[0007] The video processing method relating to this disclosure obtains an instruction statement expressed in natural language, which includes identification information indicating the type of object and attribute information indicating the attributes of the object. Based on the identification information and attribute information, it detects the object in the video data and performs a masking process on the object in the video data.

[0008] The video processing program relating to this disclosure causes a computer to perform the following steps: a process to obtain an instruction statement in which identification information indicating the type of object and attribute information indicating the attributes of the object are expressed in natural language; a process to detect the object in the video data based on the identification information and the attribute information; and a process to perform a masking process on the object in the video data.

[0009] This disclosure provides a video processing system, a video processing method, and a video processing program that can efficiently perform masking on objects within a video.

[0010] This figure shows an example of the configuration of the video processing system related to this disclosure. This figure shows another example of the configuration of the video processing system related to this disclosure. This figure shows an example of video data input to the video processing system related to this disclosure. This figure shows an example of the display screen displayed by the output unit when an extraction instruction is input as instruction information. This figure shows an example of the display screen displayed by the output unit when a mask instruction is input as instruction information. This flowchart shows an example of the video processing method related to this disclosure. This figure shows another example of the configuration of the video processing system related to this disclosure. This flowchart shows another example of the video processing method related to this disclosure. This block diagram shows an example of the hardware configuration that realizes video processing in the video processing system related to this disclosure.

[0011] (Embodiment 1) <Configuration of the video processing system> The configuration example of the video processing system 1 will be described below with reference to Figure 1. Figure 1 is a diagram showing an example of the configuration of the video processing system according to the present disclosure.

[0012] The video processing system 1 according to this disclosure takes video data and command statements as input. The video processing system 1 then performs masking on the input video data for the object (hereinafter referred to as the target object) indicated by the input command statement. As shown in Figure 1, the video processing system 1 comprises an acquisition unit 10, a detection unit 11, and a mask processing unit 12. The video processing system 1 according to this disclosure may be implemented by a single device or by distributed processing using multiple devices.

[0013] The acquisition unit 10 acquires an instruction statement. The instruction statement is text data expressed in natural language. In the instruction statement, identification information indicating the type of object and attribute information indicating the attributes of the object are each expressed in natural language. The type of object is, for example, a car, a person, a bicycle, a motorcycle, etc. In other words, in this disclosure, identification information is, for example, information used to perform object recognition on an image or video and identify "what" the object is. The attributes of the object are, for example, the position of the object, the color of the object, the orientation of the object, etc. The acquisition unit 10 acquires an instruction statement entered by the user of the video processing system 1 via an input interface or the like. The user may enter the instruction statement by text input or voice input using, for example, a keyboard, touch panel, microphone, etc. The acquisition unit 10 inputs the acquired instruction statement to the detection unit 11.

[0014] The detection unit 11 detects objects in the video data based on the command statement. First, the detection unit 11 extracts identification information and attribute information of the objects from the command statement. Then, the detection unit 11 identifies each object in the video data. The detection unit 11 performs object recognition on the identified objects and obtains identification information indicating the type of the identified object and attribute information indicating the attributes of the identified object. Next, the detection unit 11 compares the identification information of the identified objects with the identification information of the target object, and further compares the attribute information of the identified objects with the attribute information of the target object to detect the object in the video data.

[0015] The mask processing unit 12 performs mask processing on objects in the video data detected by the detection unit 11. Examples of mask processing methods include mosaic processing, which reduces the resolution of the object's area; fill processing, which fills the object's area with a specific color; and superimposing processing, which superimposes some kind of image onto the object's area. The mask processing unit 12 may also limit the mask processing to characteristic parts of the object, such as windows if the object is a vehicle, or faces if the object is a person.

[0016] As described above, the video processing system 1 takes video data and command statements as input. The video processing system 1 then performs masking on objects in the input video data. The video processing system 1 detects objects in the video data using command statements in which the identification information and attribute information of the objects are expressed in natural language. By using command statements expressed in natural language in the detection process in this way, the video processing system 1 does not need to perform preprocessing such as associating bounding boxes or dividing regions, identifying other elements such as buildings and the ground, and then specifying the position of objects within the video frame. Therefore, the video processing system 1 can efficiently perform masking on objects in the video data.

[0017] (Embodiment 2) <Configuration of the video processing system> Another example of the video processing system will be described below with reference to Figure 2. Figure 2 is a diagram showing another example of the configuration of the video processing system according to the present disclosure. The video processing system 1 according to the present disclosure takes video data and command statements as input, as described in Embodiment 1. The video processing system 1 then performs masking on the input video data for the object (hereinafter referred to as the target object) indicated by the input command statement. The video processing system 1 outputs at least a masked video. As shown in Figure 2, the video processing system 1 comprises an acquisition unit 10, a detection unit 11, a mask processing unit 12, and an output unit 13.

[0018] The acquisition unit 10 acquires command statements and video data. Command statements are text data expressed in natural language. In the command statements, identification information indicating the type of object, attribute information indicating the attributes of the object, and instruction information indicating the processing content to be instructed to the detection unit 11 or the mask processing unit 12 are each expressed in natural language. The types of objects are, for example, cars, people, bicycles, motorcycles, etc. In other words, in this disclosure, identification information is, for example, information used to perform object recognition on images or videos and identify "what" an object is. Attribute information of an object is, for example, position information indicating the position of the object within the field of view of the video data, color information indicating the color of the object, and direction information indicating the orientation of the object within the field of view of the video data.

[0019] The processing of the instruction information, in the case of an instruction directed only to the detection unit 11, involves causing the detection unit 11 to detect an object in the video data and extract its identification information and attribute information. Hereinafter, this instruction information will be referred to as an extraction instruction. Examples of commands in the case of an extraction instruction include "Please tell me if there is a person facing forward" and "Please tell me the location and color of the car." In the case of an instruction directed to both the detection unit 11 and the mask processing unit 12, the processing involves causing the detection unit 11 to detect an object in the video data and causing the mask processing unit 12 to mask the object. Hereinafter, this instruction information will be referred to as a mask instruction. Examples of commands in the case of a mask instruction include "Please mask cars facing forward" and "Please black out people on the sidewalk."

[0020] The acquisition unit 10 acquires command statements entered by the user of the video processing system 1 via an input interface or the like. The user may input command statements by text input or voice input using, for example, a keyboard, touch panel, or microphone. The acquisition unit 10 may also receive video data directly from, for example, the imaging means, or receive video data stored on a server from the server. The acquisition unit 10 inputs the acquired command statements and video data to the detection unit 11.

[0021] The detection unit 11 detects objects in the video data based on the command statement. The detection process by the detection unit 11 is described below. First, the detection unit 11 extracts identification information indicating the type of object and attribute information indicating the attributes of the object from the command statement. Then, the detection unit 11 identifies each object in the video data. The detection unit 11 performs object recognition on the identified objects and obtains identification information indicating the type of the identified object and attribute information indicating the attributes of the identified object. Next, the detection unit 11 compares the identification information of the identified object with the identification information of the target object, and further compares the attribute information of the identified object with the attribute information of the target object to detect the object in the video data. If the command statement does not contain identification information, the identification information comparison is not performed, and if the command statement does not contain attribute information, the attribute information comparison is not performed. The detection unit 11 may perform the above detection process using a learning model such as VLM (Vision-Language Model) or SAM (Segment Anything Model).

[0022] Furthermore, the detection unit 11 may, during the detection process described above, perform a process to identify objects within the video data using the semantic information of the video data and the distance information of the point cloud data obtained by measuring the shooting area of ​​the video data. Such multimodal 3D analysis improves the detection accuracy of distant objects. The distance information of the point cloud data can be acquired, for example, by LiDAR (Light Detection and Ranging) or a ToF camera (Time of Flight Camera).

[0023] After the detection process, if the instruction information in the command statement is an extraction instruction, the detection unit 11 extracts the identification information and attribute information of the detected object from the video data. Note that the attribute information extracted by the detection unit 11 here may include different types of attribute information than that contained in the command statement. For example, if the attribute information in the command statement contains only color information, the attribute information extracted by the detection unit 11 from the video data may include, for example, position information and direction information. After extraction, the detection unit 11 generates an explanatory text containing an explanation of the detection result. The explanatory text is text data written in natural language. The explanatory text contains the identification information and attribute information extracted by the detection unit 11 from the video data, written in natural language. The detection unit 11 performs the explanatory text generation process using, for example, an LLM (Large Language Model). In addition, along with generating the explanatory text, the detection unit 11 may also perform a marking process to mark the object detected by the detection unit 11 on the input video data. The detection unit 11 inputs the explanatory text and the marked video data to the output unit 13.

[0024] On the other hand, if the instruction information in the command statement is a mask instruction, the detection unit 11 inputs a command to the mask processing unit 12 to perform mask processing on the detected object. The command input to the mask processing unit 12 may include a method for mask processing. Examples of mask processing methods include mosaic processing, which reduces the resolution of the object's area; fill processing, which fills the object's area with a specific color; and processing, which superimposes some kind of image onto the object's area. If the command statement acquired by the acquisition unit 10 includes information indicating a method for mask processing, the detection unit 11 inputs a mask processing command to the mask processing unit 12, including the information indicating a method for mask processing. On the other hand, if the command statement acquired by the acquisition unit 10 does not include information indicating a method for mask processing, the detection unit 11 may, for example, inquire with the user about a method for mask processing and acquire information indicating a method for mask processing. Then, the detection unit 11 inputs a mask processing command including the acquired information to the mask processing unit 12.

[0025] Furthermore, the detection unit 11 may perform the explanatory text generation process in the same way as when the instruction information of the instruction statement is an extraction instruction, even if the instruction information of the instruction statement is a mask instruction. In this case, the detection unit 11 inputs the explanatory text to the output unit 13.

[0026] Furthermore, if the object is a vehicle such as a car or train, the detection unit 11 may detect the windows of the vehicle in the video data. In this case, the detection unit 11 inputs a command to the mask processing unit 12 to perform mask processing on the detected windows. In addition, the detection unit 11 may determine whether a person is reflected in the detected window. In this case, the detection unit 11 inputs a command to the mask processing unit 12 to perform mask processing only on the windows in which it is determined that a person is reflected.

[0027] The mask processing unit 12 performs mask processing on the video data based on the command input by the detection unit 11. The mask processing unit 12 may limit the mask processing to characteristic parts of the object, for example, windows if the object is a vehicle, or the face if the object is a person. The mask processing unit 12 inputs the masked video data to the output unit 13.

[0028] The output unit 13 displays video data, text, etc., on a display means such as a display. First, the output result by the output unit 13 when the instruction information of the command statement is an extraction instruction will be explained. Figure 3 is a diagram showing an example of video data input to the video processing system according to this disclosure. An example will be explained when the command statement input to the video processing system 1 is "Please tell me the location and color of the car" and the video data V is as shown in Figure 3. In this case, the identification information of the object is "car", so the detection unit 11 detects car C from the video data V shown in Figure 3. The detection unit 11 then extracts location information and color information as attribute information of car C. The detection unit 11 generates an explanatory text that includes an explanation of the location information and color information of car C. The detection unit 11 may also perform a marking process to mark the object, car C.

[0029] Figure 4 shows an example of a display screen displayed by the output unit when an instruction statement for extracting instruction information is input. In the example shown in Figure 4, the display screen S shows an explanatory text T and video data V with a marking M1 surrounding the target object, a car C. If the marking process is not performed, only the explanatory text T is displayed on the display screen S. The display screen S shown in Figure 4 allows the user to determine, for example, which target object should be masked. Based on the display screen S shown in Figure 4, the user inputs an instruction statement for masking the instruction information into the video processing system 1.

[0030] In this disclosure, the marking by the detection unit 11 is a bounding box as shown in Figure 4, but is not limited to this. The marking by the detection unit 11 may include a masking process, such as a fill process that fills in a specific color.

[0031] Next, we will explain the output result by the output unit 13 when the instruction information of the command statement is a mask instruction. We will explain using the example where the command statement input to the video processing system 1 is "Apply black to people on the sidewalk" and the video data V is Figure 3. In this case, the identification information of the object is "person", and the attribute information of the object is the location information "on sidewalk F". The mask processing method is black fill processing. First, the detection unit 11 detects person P on sidewalk F from the video data V shown in Figure 3. Then, the detection unit 11 inputs a mask processing command to the mask processing unit 12, including information indicating the mask processing method. The mask processing unit 12 performs mask processing on person P on sidewalk F, which is the object.

[0032] Figure 5 shows an example of a display screen shown by the output unit when a mask instruction command is input. In the example shown in Figure 5, the video data V before masking is displayed on the left side of the display screen S, and the video data V after masking is displayed on the right side. In the video data V after masking, the face of the person P on the sidewalk F, which is the target object, is blacked out M2. The user can check for any omissions in the masking process using the display screen S shown in Figure 5. In addition, an explanatory text may be displayed on the display screen S when a mask instruction command is input. By referring to the explanatory text, the user can more easily check for any omissions in the masking process.

[0033] <Image Processing Method> Next, an example of an image processing method will be explained using Figure 6. Figure 6 is a flowchart of an example of an image processing method according to this disclosure. The flowchart shown in Figure 6 is started when an instruction statement and video data are input to the image processing system 1 as a trigger.

[0034] First, the acquisition unit 10 acquires the command statement and video data. If the instruction information of the command statement is not a mask instruction (step S101: NO), that is, if the instruction information of the command statement is an extraction instruction, the detection unit 11 detects the object in the video data based on at least one of the identification information and attribute information of the command statement (step S102). First, the detection unit 11 extracts the identification information and attribute information of the object from the command statement. Then, the detection unit 11 identifies each object in the video data. The detection unit 11 performs object recognition on the identified objects and acquires the identification information and attribute information of the identified objects. Next, the detection unit 11 compares the identification information of the identified objects with the identification information of the object, and further compares the attribute information of the identified objects with the attribute information of the object to detect the object in the video data. Note that if the command statement does not contain identification information, the identification information comparison is not performed, and if the command statement does not contain attribute information, the attribute information comparison is not performed.

[0035] Next, the detection unit 11 extracts identification information and attribute information of the detected object (step S103). Then, the detection unit 11 generates an explanatory text that includes an explanation of the detection result (step S104). The explanatory text expresses the identification information and attribute information extracted by the detection unit 11 in natural language. In addition, along with generating the explanatory text, the detection unit 11 may also perform a marking process to mark the object detected by the detection unit 11 on the input video data. The detection unit 11 inputs the explanatory text and the marked video data to the output unit 13.

[0036] Next, the output unit 13 causes a display means such as a display to display a screen containing the explanatory text (step S105). If marking processing was performed in step S104, the output unit 13 causes the display means to display a display screen S showing the explanatory text T and video data V with a marking M1 surrounding the object, as shown in Figure 4. The user determines the object to be masked based on the display screen displayed on the display means, and the instruction information inputs a mask instruction command to the video processing system 1.

[0037] Next, the acquisition unit 10 receives an input of a command statement from the user (step S106). If, for example, no command statement is input within a predetermined time (step S106: NO), the video processing system 1 terminates the flowchart shown in Figure 6. If the acquisition unit 10 acquires an command statement (step S106: YES) and the instruction information of the command statement is not a mask instruction (step S107: NO), that is, if the instruction information of the command statement is an extraction instruction, the video processing system 1 executes step 102 again.

[0038] On the other hand, if the instruction information of the command statement is a mask instruction (step S101: YES and step S107: YES), the detection unit 11 detects the object in the video data based on the identification information and attribute information of the command statement (step S108). First, the detection unit 11 extracts the identification information and attribute information of the object from the command statement. Then, the detection unit 11 identifies each object in the video data. The detection unit 11 performs object recognition on the identified objects and obtains the identification information and attribute information of the identified objects. Next, the detection unit 11 compares the identification information of the identified objects with the identification information of the object, and further compares the attribute information of the identified objects with the attribute information of the object to detect the object in the video data. Then, the detection unit 11 inputs a command to the mask processing unit 12 to perform mask processing on the detected object. Note that in step S108, the detection unit 11 may also perform a descriptive text generation process. In this case, the detection unit 11 inputs the descriptive text to the output unit 13.

[0039] Next, the mask processing unit 12 performs mask processing on the object in the video data based on the command input by the detection unit 11 (step S109). The mask processing unit 12 may limit the mask processing to characteristic parts of the object, for example, the windows if the object is a vehicle, or the face if the object is a person. The mask processing unit 12 inputs the video data on which the mask processing has been performed to the output unit 13.

[0040] Finally, the output unit 13 displays a screen on the display means that includes the video data before masking and the video data after masking, for example, as shown in the display screen S in Figure 5 (step S110). The user checks for any omissions in the masking process using the display screen S. If the explanatory text generation process was performed in step S108, the output unit 13 may also display a screen on the display means that shows the video data before masking, the video data after masking, and the explanatory text. The user can more easily check for any omissions in the masking process by referring to the explanatory text.

[0041] As described above, the video processing system 1 takes video data and command statements as input. The video processing system 1 then performs masking on objects in the input video data. The video processing system 1 detects objects in the video data using command statements in which the identification information and attribute information of the objects are expressed in natural language. By using command statements expressed in natural language in the detection process in this way, the video processing system 1 does not need to perform preprocessing such as associating bounding boxes or dividing regions, identifying other elements such as buildings and the ground, and then specifying the position of objects within the video frame. Therefore, the video processing system 1 can efficiently perform masking on objects in the video data.

[0042] (Embodiment 3) <Configuration of the video processing system> Another example of the video processing system will be described below with reference to Figure 7. Figure 7 is a diagram showing another example of the configuration of the video processing system according to the present disclosure. The video processing system 1 according to Embodiment 3 comprises an acquisition unit 10, a detection unit 11, a mask processing unit 12, an output unit 13, and a matching unit 14. The video processing system 1 according to Embodiment 3 differs from the video processing system 1 according to Embodiment 2 in that it comprises a matching unit 14. The other configurations are the same as those of the video processing system 1 described in Embodiment 2, so redundant explanations will be omitted. Note that the output unit 13 is not an essential component in the video processing system 1 according to Embodiment 3.

[0043] The matching unit 14 detects objects in the video data using a different method than the detection unit 11, based on the command statement. For example, if the detection unit 11 performs detection processing using a learned model, the matching unit 14 performs detection processing using a different learned model than the detection unit 11. After performing detection processing, the matching unit 14 compares the detection result from the matching unit 14 with the detection result from the detection unit 11.

[0044] In the video processing system 1 according to Embodiment 2, the output unit 13 causes the display means to display the video data before the masking process and the video data after the masking process, thereby allowing the user to confirm the leakage of the masking process. On the other hand, in the video processing system 1 according to Embodiment 3, the collation unit 14 collates the detection result by the collation unit 14 with the detection result by the detection unit 11 to confirm the leakage of the detection of the object and suppress the leakage of the masking process.

[0045] The collation unit 14 may input the collation result to the output unit 13. The output unit 13 may present the collation result to the user by causing the display means to display the collation result, and for example, may cause the user to determine whether the masking process can be executed.

[0046] <Video processing method> Next, an example of the video processing method will be described with reference to FIG. 8. FIG. 8 is a flowchart showing another example of the video processing method according to the present disclosure. The flowchart shown in FIG. 8 is different from the flowchart shown in FIG. 6 in the steps after step S108. Since the steps before step S108 are the same as those in the flowchart shown in FIG. 6, duplicate descriptions will be omitted.

[0047] After the detection unit 11 detects the object in the video data (step S108), the collation unit 14 detects the object in the video data by a method different from that of the detection unit 11 in step S108 (step S111). Note that this step S110 may be executed simultaneously with step S108 or before step S10 for example.

[0048] Next, the collation unit 14 collates the detection result by the detection unit 11 in step S108 with the detection result by the collation unit 14 in step S111 (step S112). The collation unit 14 may input the collation result to the output unit 13.

[0049] If the matching result indicates that there are no missed detections of objects by the detection unit 11 (step S113: YES), the mask processing unit 12 performs mask processing on the objects in the video data based on the command input by the detection unit 11, similar to the second embodiment (step S109). Also, similar to the second embodiment, the output unit 13 may display both the video data before mask processing and the video data after mask processing on the display means (step S110). The video processing system 1 can further suppress missed mask processing by performing matching in step S112 and further presenting the video data before and after mask processing to the user in step S110. Of course, the video processing system 1 does not have to perform step S110.

[0050] On the other hand, if the matching results indicate that the detection unit 11 has missed detecting an object (step S113: NO), the video processing system 1 accepts the input of a command statement again (step S106). In step S106, the user refers to the matching results and inputs a command statement that includes, for example, new attribute information.

[0051] In step S113, the determination of whether or not the detection unit 11 has missed detecting an object can be made, for example, by checking whether the matching rate obtained as a result of the matching is equal to or greater than a predetermined value, or by presenting the matching result to the user and having them decide whether or not to perform the masking process.

[0052] As described above, the video processing system 1 includes a matching unit 14 that detects objects in the video data in a different way than the detection unit 11, based on the identification information and attribute information of the command statement. The matching unit 14 compares the detection result from the matching unit 14 with the detection result from the detection unit 11. Through the matching by the matching unit 14, the video processing system 1 can suppress the omission of object detection, that is, the omission of mask processing.

[0053] <Hardware Configuration for Realizing the Video Processing Functions of the Video Processing System> Some or all of the video processing realized by the video processing system 1 can be realized by a general-purpose computer system. This will be briefly explained below using Figure 9.

[0054] Figure 9 is a block diagram showing an example of a hardware configuration for realizing video processing in the video processing system according to this disclosure. The computer 20 includes, for example, a CPU (Central Processing Unit) 21 which is a control device, RAM (Random Access Memory) 22, and ROM (Read Only Memory) 23. The computer 20 further includes an IF (Interface) 24 which is an interface to the outside world, and an HDD (Hard Disk Drive) 25 which is an example of a non-volatile storage device. Furthermore, the computer 20 may also include other configurations not shown, such as input devices like a keyboard or mouse, and display devices like a display.

[0055] HDD 25 stores an OS (Operating System) and a control program 26, which are not shown in the diagram. The control program 26 is a computer program (image processing program) that implements the image processing of the image processing system 1.

[0056] The CPU 21 controls various processes in the computer 20, including access to RAM 22, ROM 23, IF 24, and HDD 25. The computer 20 reads and executes the OS and control program 26 stored in HDD 25 by the CPU 21. As a result, the computer 20 realizes the video processing of the video processing system 1.

[0057] The program described above includes a set of instructions (or software code) that, when loaded into a computer, causes the computer to perform one or more of the functions described in this disclosure. The program may be stored in a non-temporary computer-readable medium or a physical storage medium. Examples, but not limited to, include RAM, ROM, flash memory, SSD (Solid-State Drive), or other memory technologies, CD-ROM, DVD (Digital Versatile Disc), Blu-ray® disc, or other optical disc storage, magnetic cassette, magnetic tape, magnetic disk storage, or other magnetic storage devices. The program may be transmitted over a temporary computer-readable medium or a communication medium. Examples, but not limited to, include, a temporary computer-readable medium or a communication medium that includes electrically, optically, acoustically, or otherwise propagating signals.

[0058] Although the present disclosure has been described above with reference to embodiments, the present disclosure is not limited to the embodiments described above. Various modifications to the structure and details of the present disclosure can be made as can be understood by those skilled in the art within the scope of the present disclosure. Furthermore, each embodiment can be combined with other embodiments as appropriate.

[0059] Each drawing is merely illustrative to illustrate one or more embodiments. Each drawing may be associated with one or more other embodiments, rather than being associated with only one specific embodiment. As those skilled in the art will understand, various features or steps described with reference to any one drawing can be combined with features or steps shown in one or more other drawings, for example, to create embodiments not explicitly shown or described. Not all features or steps shown in any one drawing to illustrate an exemplary embodiment are necessarily required, and some features or steps may be omitted. The order of steps described in any of the drawings may be changed as appropriate.

[0060] Some or all of the above embodiments may also be described as follows, but are not limited to the following:

[0061] (Note 1) A video processing system comprising: an acquisition unit that acquires an instruction statement expressed in natural language, which includes identification information indicating the type of object and attribute information indicating the attributes of the object; a detection unit that detects the object in video data based on the identification information and the attribute information; and a masking unit that performs masking on the object in the video data.

[0062] (Note 2) The video processing system according to Note 1, wherein the command statement is expressed in natural language and includes the identification information, the attribute information, and instruction information indicating the processing content to be instructed to the detection unit or mask processing unit, and the instruction information includes an extraction instruction causing the detection unit to detect the object in the video data and to extract the identification information and attribute information of the object from the video data, and a mask instruction causing the detection unit to detect the object in the video data and to cause the mask processing unit to mask the object.

[0063] (Note 3) The video processing system according to Note 1 or 2, wherein the attribute information includes positional information of the object within the field of view of the video data.

[0064] (Note 4) The video processing system according to any one of Notes 1 to 3, wherein the detection unit detects the windows of the vehicle in the video data when the type of object is a vehicle, and the mask processing unit performs mask processing on the windows.

[0065] (Note 5) The video processing system according to any one of Notes 1 to 4, wherein the detection unit, when the type of object is a vehicle, detects the windows of the vehicle in the video data and further determines whether a person is reflected in the windows, and the masking unit performs masking only on the windows in which it is determined that a person is reflected.

[0066] (Note 6) The video processing system according to any one of Notes 1 to 5, further comprising an output unit for displaying the video data before masking and the video data after masking on a display means.

[0067] (Note 7) The video processing system according to any one of Notes 1 to 6, wherein the detection unit generates an explanatory text in natural language that includes an explanation of the detection result, and the output unit causes the explanatory text to be displayed on the display means.

[0068] (Note 8) The video processing system according to any one of Notes 1 to 7, further comprising a comparison unit that detects the object in the video data in a method different from that of the detection unit based on the identification information and the attribute information, and compares the result of the detection with the result of the detection by the detection unit.

[0069] (Note 9) An image processing method comprising: obtaining an instruction statement expressed in natural language, which includes identification information indicating the type of object and attribute information indicating the attributes of the object; detecting the object in the image data based on the identification information and attribute information; and performing a masking process on the object in the image data.

[0070] (Note 10) A video processing program that causes a computer to perform the following steps: a process to obtain an instruction statement in which identification information indicating the type of object and attribute information indicating the attributes of the object are expressed in natural language; a process to detect the object in video data based on the identification information and the attribute information; and a process to perform a masking process on the object in the video data.

[0071] (Note 11) An image processing apparatus comprising: an acquisition unit that acquires an instruction statement expressed in natural language, which includes identification information indicating the type of object and attribute information indicating the attributes of the object; a detection unit that detects the object in the image data based on the identification information and the attribute information; and a masking unit that performs masking on the object in the image data.

[0072] (Note 12) The image processing system according to any one of Notes 1 to 8, wherein the detection unit identifies an object in the image data using the semantic information of the image data and the distance information of the point cloud data obtained by measuring the shooting area of ​​the image data, and detects the target object from the object based on the identification information and the attribute information.

[0073] Some or all of the elements (e.g., configuration and function) described in Appendices 2 to 8 and 12, which are dependent on Appendice 1, may also be dependent on Appendices 9, 10, and 11 in the same manner as those described in Appendices 2 to 8 and 12. Some or all of the elements described in any appendice may be applied to various hardware, software, recording means, systems, and methods for recording software.

[0074] This application claims priority based on Japanese Patent Application No. 2025-024040, filed on 18 February 2025, and incorporates all of its disclosures herein.

[0075] 1. Video Processing System 10. Acquisition Unit 11. Detection Unit 12. Mask Processing Unit 13. Output Unit 14. Matching Unit 20. Computer 21. CPU 22. RAM 23. ROM 24. IF 25. HDD 26. Control Program C. Car F. Sidewalk M1. Marking M2. Blackout P. Person S. Display Screen T. Explanation V. Video Data

Claims

1. A video processing system comprising: an acquisition unit that acquires an instruction statement expressed in natural language, which includes identification information indicating the type of object and attribute information indicating the attributes of the object; a detection unit that detects the object in video data based on the identification information and the attribute information; and a masking unit that performs masking on the object in the video data.

2. The video processing system according to claim 1, wherein the instruction statement is expressed in natural language and includes the identification information, the attribute information, and instruction information indicating the processing content to be instructed to the detection unit or the mask processing unit, and the instruction information comprises: an extraction instruction causing the detection unit to detect the object in the video data and to extract the identification information and attribute information of the object from the video data; and a mask instruction causing the detection unit to detect the object in the video data and to cause the mask processing unit to mask the object.

3. The video processing system according to claim 1 or 2, wherein the attribute information includes positional information of the object within the field of view of the video data.

4. The image processing system according to claim 1 or 2, wherein the detection unit detects the windows of the vehicle in the image data when the type of object is a vehicle, and the mask processing unit performs mask processing on the windows.

5. The video processing system according to claim 4, wherein the detection unit, when the type of object is a vehicle, detects the windows of the vehicle in the video data and further determines whether a person is reflected in the windows, and the masking unit performs masking only on the windows in which it is determined that a person is reflected.

6. The video processing system according to claim 1 or 2, further comprising an output unit for displaying the video data before masking and the video data after masking on a display means.

7. The video processing system according to claim 6, wherein the detection unit generates an explanatory text in natural language that includes an explanation of the detection result, and the output unit causes the explanatory text to be displayed on the display means.

8. The video processing system according to claim 1 or 2, further comprising a comparison unit that detects the object in the video data in a method different from that of the detection unit based on the identification information and the attribute information, and compares the result of the detection with the result of the detection by the detection unit.

9. The image processing system according to claim 1 or 2, wherein the detection unit identifies an object in the image data using the semantic information of the image data and the distance information of the point cloud data obtained by measuring the shooting area of ​​the image data, and detects the target object from the object based on the identification information and the attribute information.

10. A video processing method comprising: obtaining an instruction statement expressed in natural language, which includes identification information indicating the type of object and attribute information indicating the attributes of the object; detecting the object in video data based on the identification information and attribute information; and performing a masking process on the object in the video data.

11. A video processing program that causes a computer to perform the following steps: a process to obtain an instruction statement expressed in natural language, which includes identification information indicating the type of object and attribute information indicating the attributes of the object; a process to detect the object in video data based on the identification information and the attribute information; and a process to perform a masking process on the object in the video data.

Images