Information processing device, information processing method, and program

Abstract

An information processing device according to the present invention comprises: a detection unit that obtains position information of a facial region of a person detected from an image and acquires the surface area of the facial region from the position information; a determination unit that determines the scene where the image is captured on the basis of the surface area of the facial region and the positional relationship of the facial region; an identification unit that identifies the main subject in the image on the basis of the scene determined by the determination unit; and an evaluation unit that evaluates the facial expression of the main subject identified by the identification unit.

Description

Information Processing Apparatus, Information Processing Method, and Program

[0001] The present disclosure relates to a technical field of estimating and evaluating a main subject from a scene of a photographed image.

[0002] Functions for evaluating the expression and quality of an image through image analysis using AI (Artificial Intelligence) technology are being realized. Elements for evaluating an image include expressions such as smiling and winking based on the faces of people in the image, and out-of-focus, camera shake, or subject blur. Also, when evaluating an image, for example, when one image contains the faces of multiple people with different expressions for each person, it is necessary to determine which person's face to use as a reference for evaluating the image.

[0003] Patent Document 1 describes a photo classification method for classifying whether a given photo is a solo photo of one predetermined reference person or a group photo of multiple people including the reference person, and determining whether it includes unspecified people other than the reference person in the case of a group photo. Also, Patent Document 2 describes a method for detecting a person commonly included in multiple images and estimating a main subject based on the position or line-of-sight direction of the person in the multiple images.

[0004] Japanese Patent No. 7522161 Japanese Patent No. 5246275

[0005] In Patent Document 1, it is necessary to register the face image of that person in advance to identify the reference person. Also, in Patent Document 2, multiple images are required to estimate the main subject, and it is necessary to perform individual identification of the people included in the multiple images. However, the personal feature information of a person's face may be regarded as sensitive personal information, and in some countries and regions, strict restrictions are imposed on the acquisition and management of such information, making it difficult to evaluate images by individual identification. Also, even for an image with a person as the subject, the person may not be someone known to the photographer, and in such a case, it is difficult to evaluate the image by specifying in advance the person to be evaluated.

[0006] The present disclosure has been made in view of the above problems, and provides a technical advantage of estimating the person to be evaluated from the scene of a photographed image and evaluating the expression of the face without identifying a specific individual.

[0007] To solve the above problems, the information processing device of the present disclosure includes: detection means for obtaining location information of a person's face region detected from an image and obtaining the area of ​​the face region from the location information; determination means for determining the scene in which the image was taken based on the area of ​​the face region and the positional relationship of the face region; determination means for determining the main subject in the image based on the scene determined by the determination means; and evaluation means for evaluating the facial expression of the main subject determined by the determination means.

[0008] According to this disclosure, it is possible to estimate the person being evaluated from the scene of a captured image and evaluate their facial expression without identifying a specific individual.

[0009] Other features and advantages of the technical ideas derived from this disclosure will become apparent from the following description with reference to the attached drawings. In the attached drawings, the same or similar components are given the same reference numeral.

[0010] The attached drawings are included in the specification and constitute a part thereof, illustrating embodiments in this disclosure and used to explain the technical ideas derived from this disclosure together with their descriptions. Figure 1 is a system configuration diagram of this embodiment. Figure 2 is a block diagram illustrating the hardware configuration of the information processing device and server device of this embodiment. Figure 3 is a block diagram illustrating the hardware configuration of the imaging device of this embodiment. Figure 4 is a block diagram illustrating the functional configuration of the facial expression evaluation device in the server device of this embodiment. Figure 5 is a diagram illustrating an image to be evaluated in this embodiment. Figure 6 is a diagram illustrating the configuration of the face region management table of this embodiment. Figure 7 is a diagram illustrating the focus position and gaze direction in the image to be evaluated in this embodiment. Figure 8 is a diagram illustrating a method for calculating the variation deviation of the distance of the face region in the image to be evaluated in this embodiment. Figure 9 is a diagram illustrating a method for calculating the variation deviation of the alignment of the face region in the image to be evaluated in this embodiment. Figure 10 is a diagram illustrating the configuration of the scene determination criterion table of this embodiment. Figure 11 is a diagram illustrating the scene determination method of this embodiment. Figure 12 is a diagram illustrating the scene determination method of this embodiment. Figure 13 is a diagram illustrating the scene determination method of this embodiment. Figure 14 is a diagram illustrating the configuration of the evaluation target determination table in this embodiment. Figure 15 is a flowchart illustrating the image evaluation process in this embodiment. Figure 16 is a flowchart illustrating the face position detection process in this embodiment. Figure 17 is a flowchart illustrating the face region information analysis process in this embodiment. Figure 18 is a flowchart illustrating the scene determination process in this embodiment.

[0011] The embodiments will be described in detail below with reference to the attached drawings. Note that the following embodiments do not limit the scope of the claims. While the embodiments describe multiple features, not all of these features are necessary, and the features may be combined in any way. Furthermore, in the attached drawings, identical or similar configurations are given the same reference numerals, and redundant descriptions are omitted.

[0012] In this embodiment, we describe an example in which an imaging device, an information processing device, and a server device are connected to each other via a network, and an image captured by the imaging device is uploaded to the server device, and the server device performs image evaluation processing.

[0013] <System Configuration> First, the system configuration of this embodiment will be described with reference to Figure 1.

[0014] Figure 1 is a diagram illustrating the system configuration of this embodiment.

[0015] In this embodiment, the user is a person who uses the system of this embodiment.

[0016] The system of this embodiment includes an imaging device 101, an information processing device 102, and a server device 103. The imaging device 101, the information processing device 102, and the server device 103 are communicated together via a network 104.

[0017] The imaging device 101 captures the image data to be processed in this embodiment and uploads it to the server device 103 from the user's local environment via the network 104. The imaging device 101 in this embodiment is a digital camera, but it is widely applicable to electronic devices that have an imaging function. Such electronic devices include computer devices (personal computers, tablet computers, smartphones, etc.). The image data to be processed in this embodiment is not limited to a specific file format or encoding format.

[0018] The information processing device 102 is a terminal device used by a user of the system in this embodiment. The information processing device 102 transmits processing instructions to the server device 103 via the network 104, receives information transmitted from the server device 103, and displays the information received from the server device 103. The information processing device 102 in this embodiment is a general-purpose computer such as a personal computer (PC) or tablet computer, or a smartphone.

[0019] The server device 103 performs evaluation processing on image data uploaded from the imaging device 101 via the network 104 and transmits the evaluation results to the information processing device 102. In this embodiment, the server device 103 is an information processing device such as a cloud server that provides services through cloud computing.

[0020] The communication method of network 104 may be either wireless communication or wired communication.

[0021] The system of this embodiment is not limited to a configuration in which image data is transmitted to the server device 103, and may be transmitted in a form other than communication over a network. For example, image data may be uploaded directly from the imaging device 101 to the server device 103, or it may be uploaded from the imaging device 101 to the server device 103 via the information processing device 102.

[0022] The server device 103 is not limited to the form of a server; it may also be an electronic device having functions similar to the information processing functions of a server.

[0023] Furthermore, in this embodiment, the output destination for the image evaluation results from the server device 103 is the information processing device 102, but this configuration is not limited to this depending on the format of the output data. For example, if a server of another system is connected, it is not necessary to output data in text file format or displayed in a UI, and the data may be output in a format such as a database.

[0024] Furthermore, although the server device 103 performs the image evaluation processing in this embodiment, the system is not limited to this configuration, and the information processing device 102 may also perform the image evaluation processing.

[0025] <Device Configuration> Next, with reference to Figures 2 to 4, the configurations of the imaging device 101, information processing device 102, and server device 103 that constitute the system of this embodiment will be described.

[0026] Figure 2 is a block diagram illustrating the configuration of the information processing device 102 and server device 103 in this embodiment.

[0027] The information processing device 102 and server device 103 of this embodiment include a display 201, VRAM 202, BMU 203, keyboard 204, pointing device 205, CPU 206, ROM 207, RAM 208, HDD 209, removable storage device 210, network I / F 211, and bus 212.

[0028] The display 201 shows, for example, images, icons, messages, menus, and other user interface information for managing the information processing device 102 and the server device 103.

[0029] VRAM 202 is a video memory where data for display on the display 201 is drawn. The data generated in VRAM 202 is transferred to the display 201 according to predetermined rules, thereby displaying an image on the display 201.

[0030] The BMU (Bit Move Unit) 203 is a circuit that controls, for example, data transfer between memories (e.g., between VRAM 202 and other memories) and data transfer between memory and each I / O device (e.g., network I / F 211).

[0031] The keyboard 204 is an operating unit having various keys for inputting characters and other data.

[0032] The pointing device 205 is used, for example, to point to icons, menus, or other content displayed on the display 201, or for dragging and dropping objects.

[0033] The CPU 206 is a control unit that controls each component based on the OS (operating system) and control programs (applications) stored in the ROM 207 and HDD 209.

[0034] ROM207 is a non-volatile memory that stores control programs and data executed by CPU206.

[0035] RAM 208 is a work memory that includes the CPU 206's work area, a data storage area for error handling, and a control program load area.

[0036] HDD209 is an auxiliary storage device that stores control programs, content, and data executed by CPU206.

[0037] The removable storage device 210 is a large-capacity storage device that stores the OS and control programs.

[0038] The network interface 211 is a communication unit that communicates with external devices such as the imaging device 101 and the information processing device 102. Furthermore, if the external device is a communication device such as a smartphone, the network interface 211 can perform public wireless communication such as 4G or 5G.

[0039] Bus 212 includes an address bus, a data bus, and a control bus.

[0040] The control program executed by the CPU 206 can be provided from the ROM 207, HDD 209, removable storage device 210, or from an external device connected via the network interface 211.

[0041] The control program includes the operating system (OS), which is the basic software executed by the CPU 206, and applications that work in cooperation with the OS to realize advanced functions. The applications include those that allow the information processing device 102 and the server device 103 of this embodiment to execute the flowchart processing described later.

[0042] The flowchart processing in this embodiment is realized by loading software provided by an application. The application is assumed to have software for utilizing the basic functions of the operating system installed on the information processing device 102 and the server device 103. The operating systems of the information processing device 102 and the server device 103 may also have software for realizing the processing in this embodiment.

[0043] Figure 3 is a block diagram illustrating the configuration of the imaging device 101 in this embodiment.

[0044] The imaging lens 301 is a lens group including a zoom lens and a focus lens. The shutter 302 has an aperture function. The imaging unit 303 is an imaging device composed of an imaging element such as a CCD or a CMOS that converts an optical image of a subject into an electrical signal. The A / D converter 304 converts an analog signal into a digital signal. The A / D converter 304 is used to convert the analog signal output from the imaging unit 303 into a digital signal. The barrier 305 covers the imaging system including the imaging lens 301 of the imaging device 101 to prevent dirt and damage to the imaging system including the imaging lens 301, the shutter 302, and the imaging unit 303.

[0045] The image processing unit 306 performs resizing processes such as predetermined pixel interpolation and reduction, and color conversion processes on the data from the A / D converter 304 or the data from the memory control unit 307. Also, in the image processing unit 306, predetermined arithmetic processing is performed using the captured image data, and based on the obtained arithmetic result, the system control unit 308 performs exposure control and distance measurement control.

[0046] The output data from the A / D converter 304 is written into the memory 309 via the image processing unit 306 and the memory control unit 307, or via the memory control unit 307. The memory 309 stores the image data obtained by the imaging unit 303 and converted into digital data by the A / D converter 304, and the image data for display on the display unit 310. The memory 309 has a storage capacity sufficient to store a predetermined number of still images, a moving image for a predetermined time, and audio. Also, the memory 309 doubles as a memory for image display (video memory).

[0047] The D / A converter 311 converts the image display data stored in the memory 309 into an analog signal and supplies it to the display unit 310. Thus, the image data for display written in the memory 309 is displayed by the display unit 310 via the D / A converter 311. The display unit 310 performs display according to the analog signal from the D / A converter 311 on a display device such as a liquid crystal panel. By once performing A / D conversion by the A / D converter 304, analog-converting the digital signal accumulated in the memory 309 in the D / A converter 311, and sequentially transferring and displaying it on the display unit 310, a live view image can be displayed.

[0048] The nonvolatile memory 312 is a memory that can be electrically erased and stored, and for example, an EEPROM or the like is used. In the nonvolatile memory 312, constants, programs, etc. for the operation of the system control unit 308 are stored.

[0049] The system control unit 308 controls the entire imaging device 101. The system control unit 308 controls each component by executing the program stored in the nonvolatile memory 312. The system memory 313 uses a RAM, and constants, variables, programs read from the nonvolatile memory 312, etc. for the operation of the system control unit 308 are loaded. Also, the system control unit 308 performs display control by controlling the memory 309, the D / A converter 311, the display unit 310, etc.

[0050] The mode switch 314, the first shutter switch 315, the second shutter switch 316, and the operation unit 317 are operation means for inputting various operation instructions to the system control unit 308.

[0051] The mode switch 314 switches the operation mode of the system control unit 308 to any one of a still image shooting mode, a moving image recording mode, a playback mode, etc.

[0052] The first shutter switch 315 turns on during the operation of the shutter button on the imaging device 101, specifically when it is half-pressed (instruction to prepare for shooting), and generates a first shutter switch signal SW1. The first shutter switch signal SW1 initiates shooting preparation processes such as AF processing, AE processing, AWB processing, and EF processing.

[0053] The second shutter switch 316 turns on when the shutter button operation is completed, so-called full press (shooting instruction), and generates the second shutter switch signal SW2. The system control unit 308 starts a series of shooting processes, from reading the signal from the imaging unit 303 to writing the image data to the recording medium 325, in response to the second shutter switch signal SW2.

[0054] Each operating element of the control unit 317 is assigned a function as appropriate for each scene by selecting various function icons displayed on the display unit 310, and acts as various function buttons. Examples of function buttons include an exit button, back button, image advance button, jump button, filter button, and attribute change button. For example, when the menu button is pressed, various configurable menu screens are displayed on the display unit 310. The user can intuitively make various settings using the menu screen displayed on the display unit 310 and the four directional buttons (up, down, left, and right) and the SET button.

[0055] The controller wheel 318 is a rotatable operating member included in the operating unit 317 and is used together with the directional buttons to indicate selection items. When the controller wheel 318 is rotated, an electrical pulse signal is generated according to the amount of rotation, and the system control unit 308 controls each component of the imaging device 101 based on this pulse signal. This pulse signal allows the system control unit 308 to determine the angle by which the controller wheel 318 has been rotated and how many rotations it has made. The controller wheel 318 can be any operating member that can detect rotation. For example, it could be a dial operating member in which the controller wheel 318 itself rotates in response to the user's rotation and generates a pulse signal. Alternatively, it could be an operating member consisting of a touch sensor, in which the controller wheel 318 itself does not rotate, but detects the rotational movement of the user's finger on the controller wheel 318.

[0056] The controller ring 319 is a rotatable operating member included in the operating unit 317, and is an operating member that can be rotated around the optical axis of the lens barrel. When the controller ring 319 is operated, an electrical pulse signal corresponding to the amount of rotation (operation amount) is generated, and the system control unit 308 controls each component of the imaging device 101 based on this pulse signal. In addition, when the function switching button of the controller ring 319 is pressed, a menu screen that allows the function assigned to the controller ring 319 to be changed is displayed on the display unit 310.

[0057] The controller wheel 318 and controller ring 319 are used to select setting items and change their values.

[0058] The power switch 320 is an operating component that switches the power of the imaging device 101 on or off.

[0059] The power control unit 321 consists of a battery detection circuit, a DC-DC converter, a switch circuit for switching which blocks are energized, and the like, and detects whether a battery is installed, the type of battery, and the remaining battery level. The power control unit 321 also controls the DC-DC converter based on the detection results and instructions from the system control unit 308, and supplies the necessary voltage to each component, including the recording medium 325, for the required period of time.

[0060] The power supply unit 322 consists of primary batteries such as alkaline batteries and lithium batteries, secondary batteries such as NiCd batteries, NiMH batteries and Li-ion batteries, and an AC adapter. The recording medium I / F 324 is an interface with the recording medium 325, such as a memory card or hard disk. The recording medium 325 is a recording medium such as a memory card for recording captured images, and is composed of semiconductor memory, magnetic disks, etc.

[0061] The communication unit 323 is connected to an external device via a wireless antenna or wired cable for communication and transmits and receives image data. The communication unit 323 can also connect to a wireless LAN (Local Area Network) or the internet. The communication unit 323 can transmit image data (including live view images) captured by the imaging unit 303 and image files recorded on the recording medium 325 to an external device, and can also receive image data and other various information from an external device. The communication unit 323 is not limited to wireless LAN; it may also use wireless communication modules such as infrared communication, Bluetooth®, Bluetooth® Low Energy, WirelessUSB, or wired connection means such as a USB cable, HDMI®, or IEEE 1394.

[0062] The network interface 326 is controlled by the system control unit 308 and communicates with external devices via the network.

[0063] Note that the configuration of the imaging device 101 in this embodiment is just one example, and among the components in Figure 3, the shutter 302, mode switching switch 314, display unit 310, etc., are not essential.

[0064] Figure 4 is a block diagram illustrating the functional configuration of the facial expression evaluation device 400 in the server device 103 of this embodiment.

[0065] Each function of the facial expression evaluation device 400 in this embodiment is realized by utilizing the hardware resources and control program (application) described in Figure 2.

[0066] The facial expression evaluation device 400 receives image data transmitted via the network 104 with the data receiving unit 401 and stores it in the image data storage 402.

[0067] The image data reading unit 403 reads the instructed image data from the multiple image data stored in the image data storage 402 and extracts metadata such as shooting information recorded in that image data.

[0068] The face position detection unit 405 detects whether a person's face is included in the image read by the image data reading unit 403. It obtains the coordinate information of the detected face region, calculates the area ratio, which is the ratio of the area of ​​the face region to the total area of ​​the image, and stores it in the face region information database (DB) 404.

[0069] The face region information analysis unit 406 obtains the coordinate information and area ratio of the face region detected by the face position detection unit 405 from the face region information DB 404, sorts the face region information in descending order of area ratio, and calculates the difference in area ratio between preceding and succeeding face regions.

[0070] The scene determination unit 407 estimates the scene in which the image was taken based on the area and positional relationship of the faces of people included in the image. The scene determination unit 407 determines the scene in which the image was taken by comparing the face area information recorded in the face area information DB 404 with predetermined determination criteria.

[0071] The evaluation target determination unit 408 determines one or more facial regions to be evaluated based on the determination criteria defined for each scene determined by the scene determination unit 407.

[0072] The image evaluation unit 409 performs facial expression evaluations, such as smiles, closed eyes, and blur / blur, on the facial region to be evaluated, as determined by the evaluation target determination unit 408, and outputs an evaluation of the image's overall quality. The image evaluation unit 409 excludes facial expression evaluations if it determines that the person is not the main subject in the image.

[0073] Figure 5 illustrates the image 500 to be evaluated in this embodiment.

[0074] The image 500 captured by the imaging device 101 and transmitted to the server device 103 is recorded in the image data storage 402 as an image in which a person 501, which is the main subject, and a person 502 in the background area are visible.

[0075] The facial expression evaluation device 400 detects the face region 503 of person 501 in the face position detection unit 405, approximates the detected face region 503 as a rectangular region, and obtains the upper-left coordinate 504 and lower-right coordinate 505, which represent the rectangular region, as position information for the face region 503. Similarly, it detects the face region 506 of person 502 in the background region, approximates the detected face region 506 as a rectangular region, and obtains the upper-left coordinate 507 and lower-right coordinate 508, which represent the rectangular region, as position information for the face region 506. Coordinate 509 indicates the image size of image 500.

[0076] Figure 6 illustrates a face region management table 600 containing face region information for each image to be evaluated in this embodiment.

[0077] The facial region management table 600 is stored in the facial region information DB 404 of the facial expression evaluation device 400.

[0078] When the face position detection unit 405 detects a face region from an image, it assigns a unique face region ID 601 to that region, and adds the coordinates of the upper left corner 602 and lower right corner 603 of a rectangular region that approximates the detected face region to the face region management table 600 as face region information. Since multiple face regions are allowed to exist in a single image, face region information is added to the face region management table 600 for each detected face region.

[0079] Next, the face region information analysis unit 406 calculates the ratio (area ratio) of the number of pixels in each face region obtained from the coordinate information 602 and 603 for each face region ID 601 to the total number of pixels in the entire image, and records it in the region area ratio 604. The total number of pixels in the entire image corresponds to, for example, the coordinate information 509 shown in Figure 5.

[0080] Furthermore, the face region information analysis unit 406 uses the center or focus position of the image as a reference position to calculate the distance from the reference position in the image to each face region, and records it as face region information at the reference distance 605.

[0081] The priority coefficient 606 is a coefficient used by the image data reading unit 403 to assign priority to each face region based on the metadata of the image. The metadata includes, for example, the focus position (AF point) for AF processing and the line of sight vector of the person's face region.

[0082] After reading all the face region information contained in the image, the face region information analysis unit 406 sorts this face region information in descending order of area ratio 604. For the face region information before and after sorting, the difference in area ratio 604 is calculated, and the area ratio threshold flag 607 is set to true and recorded for face regions where the difference changes rapidly. In this embodiment, face region information for which the area ratio threshold flag 607 is set to true becomes a threshold for determining whether the person can be considered the main subject in the image or a person that is in the background.

[0083] Figure 7 illustrates an image 700 of a person 701 that includes focus position information and gaze information detected by the image data reading unit 403 of the facial expression evaluation device 400 of this embodiment.

[0084] The imaging device 101 records information on all focus positions 702 and information on the focus position 703 that was focused by AF as metadata for the image 700.

[0085] The image data reading unit 403 of the facial expression evaluation device 400 acquires metadata of the image 700 or coordinate information of the focus position 703, and the face position detection unit 405 determines whether the focus position 703 is within the range specified by the face region coordinate information 602, 603. If the focus position 703 is included in the face region coordinate information 602, 603, it can be determined that the photographer considers that face region to be the main subject, and the priority coefficient 606 of that face region ID 601 is recorded. Alternatively, the face position detection unit 405 analyzes the gaze vector of the face region of person 701 in the image 700 and records a priority coefficient 606 based on the gaze direction 704 of the face region of person 701.

[0086] Figure 8 illustrates a method for calculating the distance variation deviation of the face region in the image 800 to be evaluated in this embodiment.

[0087] The face position detection unit 405 detects the face regions 804 and 805 of people 802 and 803 from the image 800 and records the coordinate information 602 and 603 in the face region management table 600.

[0088] The face region information analysis unit 406 determines the center coordinates of the image 800, calculates the distance between the center coordinates of the image 800 and the center coordinates of each face region 804, 805 recorded in the face region management table 600, and records this as the deviation of the distance variation of the face regions in the reference distance 605 of the face region management table 600. The reference position can be the coordinate information of the center of the image, or the coordinate information of the focus position 703 shown in Figure 7.

[0089] Figure 9 illustrates a method for calculating the deviation of the alignment of the face region in the image 900 to be evaluated in this embodiment.

[0090] The face position detection unit 405 detects the face regions 904 and 905 of people 902 and 903 from the image 900 and records the coordinate information 602 and 603 in the face region management table 600.

[0091] The face region information analysis unit 406 obtains the center coordinates of the face regions 904 and 905 detected by the face position detection unit 405, and calculates the difference 906 in the Y direction of these center coordinates as the deviation of alignment variation in the direction in which the face regions are aligned. Note that instead of calculating the difference in the Y direction of the face regions, the difference in the X direction may be calculated, or the difference in both the X and Y directions may be calculated.

[0092] Figure 10 illustrates a scene determination criteria table 1000, which records the criteria for determining the scene in which the image to be evaluated in this embodiment was taken.

[0093] The scene determination unit 407 of the facial expression evaluation device 400 determines the scene in which the image to be evaluated was taken by comparing the information obtained from the facial region information recorded in the facial region management table 600 with each item of the scene determination criteria 1000.

[0094] Scene type 1001 defines a shooting scene to be estimated in advance. In this embodiment, examples of shooting scenes include "portrait," "small group scene," "large group scene," and "crowd scene."

[0095] The scene determination unit 407 reads the face region management table 600 of the image to be evaluated from the face region information DB 404 and compares the area ratio 604 with the maximum face region area ratio 1002 in the scene determination criteria table 1000. The scene determination unit 407 also calculates the number of face regions in which the area ratio 604 exceeds the threshold, using the face region information in the face region management table 600 for which the area ratio threshold flag 607 is set to true as a threshold, and compares this with the number of face regions 1003 in the scene determination criteria table 1000 that exceeds the area ratio threshold. The scene determination unit 407 also compares the deviation of the variation in face region distance calculated from the reference distance 605 in the face region management table 600 with the deviation of the variation in reference distance 1004 in the scene determination criteria table 1000. Furthermore, the scene determination unit 407 compares the deviation of the alignment of the face region calculated from the difference 906 in the Y direction of the face region in Figure 9 with the deviation of the alignment of the standard in the scene determination standard table 1000.

[0096] Based on these comparison results, the scene determination unit 407 determines which of the predefined scene types 1001 the scene in which the image to be evaluated was taken corresponds to.

[0097] Each judgment criterion defined in the scene judgment criterion table 1000 can be set using a combination of numerical values, default values, and thresholds.

[0098] Figure 11 illustrates an image 1100 that is determined to be a scene with a small number of people by the scene determination unit 407 of the facial expression evaluation device 400 of this embodiment.

[0099] The face position detection unit 405 detects the face regions 1105, 1106, 1107, and 1108 of four people, 1101, 1102, 1103, and 1104, as subjects in the image, and records the detected face region information in the face region management table 600. The face region information analysis unit 406 sorts the face region information recorded in the face region management table 600 in descending order of area ratio, rearranges it according to priority, and then compares the value obtained from each face region information in the face region management table 600 with the judgment criteria in the scene judgment criteria table 1000. If the result satisfies the following conditions, it is determined to be a scene with a small number of people. - The maximum value of the area ratio 604 is sufficiently large at the maximum face area ratio 1002 of the scene determination criteria table 1000 (the second largest value in the example of Figure 10). - The number of face areas where the area ratio 604 exceeds the area ratio threshold is sufficiently small at the number of face areas exceeding the area ratio threshold 1003 of the scene determination criteria table 1000 (the second smallest value in the example of Figure 10). - The deviation of the variation in the distance of the face areas is sufficiently small at the deviation of the variation in the reference distance of the scene determination criteria table 1000 1004 (the second smallest value in the example of Figure 10). - The deviation of the variation in the alignment of the face areas is sufficiently small at the deviation of the variation in the reference alignment of the scene determination criteria table 1000 1005 (the second smallest value in the example of Figure 10). Figure 12 illustrates an image 1200 that is determined to be a scene with many people by the scene determination unit 407 of the facial expression evaluation device 400 of this embodiment.

[0100] The face position detection unit 405 detects multiple face regions 1202, including person 1201, as subjects in the image, and records the detected face region information in the face region management table 600. The face region information analysis unit 406 sorts the face region information recorded in the face region management table 600 in descending order of area ratio, rearranges them according to priority, and then compares the values ​​obtained from each face region information in the face region management table 600 with the judgment criteria in the scene judgment criteria table 1000. If the result satisfies the following conditions, it determines that it is a scene with many people. - The maximum value of the area ratio 604 is sufficiently small in the largest face area ratio 1002 of the scene determination criteria table 1000 (the third largest value in the example of Figure 10). - The number of face areas in which the area ratio 604 exceeds the area ratio threshold is sufficiently large in the number of face areas exceeding the area ratio threshold 1003 of the scene determination criteria table 1000 (the third largest value in the example of Figure 10). - The deviation of the distance variation of the face areas is sufficiently large in the deviation of the distance variation 1004 of the reference distance of the scene determination criteria table 1000 (the third largest value in the example of Figure 10). - The deviation of the alignment of the face areas is sufficiently large in the deviation of the alignment variation 1005 of the reference alignment of the scene determination criteria table 1000 (the third largest value in the example of Figure 10). Figure 13 illustrates an image 1300 that is determined by the scene determination unit 407 of the facial expression evaluation device 400 of this embodiment to be a crowded scene or a landscape scene including a person.

[0101] The face position detection unit 405 detects the face regions 1308, 1309, 1310, 1311, 1312, 1313, and 1314 of the people 1301 and 1304 located relatively close to the viewer, the people 1305, 1302, and 1303 located behind them, and the people 1310 and 1311 located further away, as subjects in the image, and records the detected face region information in the face region management table 600. The face region information analysis unit 406 sorts the face region information recorded in the face region management table 600 in descending order of area ratio, rearranges it according to priority, and then compares the value obtained from each face region information in the face region management table 600 with the judgment criteria in the scene judgment criteria table 1000. If the result satisfies the following conditions, it determines that there are people in the image, but they are not considered the main subjects in a crowd scene or landscape scene. - The maximum value of the area ratio 604 is very small at the maximum face area ratio 1002 in the scene determination criteria table 1000 (the smallest value in the example of Figure 10). - The number of face areas where the area ratio 604 exceeds the area ratio threshold is very large at the number of face areas exceeding the area ratio threshold 1003 in the scene determination criteria table 1000 (the largest value in the example of Figure 10). - The deviation of the variation in face area distance is very large at the standard deviation of distance variation 1004 in the scene determination criteria table 1000 (the largest value in the example of Figure 10). - The deviation of the variation in face area alignment is very large at the standard deviation of alignment variation 1005 in the scene determination criteria table 1000 (the largest value in the example of Figure 10). In this embodiment, the maximum area ratio of the face area, the number of face areas where the area ratio exceeds the threshold, the deviation of the variation in face area distance, and the deviation of the variation in face area alignment are given as examples of items for determining a scene, but a scene may be determined based on at least one of these items.

[0102] Figure 14 illustrates an evaluation target determination table 1400 used by the evaluation target determination unit 408 of the facial expression evaluation device 400 in this embodiment to determine which facial region in an image will be evaluated based on the shooting scene determined by the scene determination unit 407.

[0103] In this embodiment, an example of evaluating facial expressions as "smile," "closed eyes," and "blur / shake" is described. Based on the shooting scene determination determined by the scene determination unit 407, the criteria for the facial area to be evaluated when determining smile 1402, closed eyes 1403, and blur / shake 1404 are defined for each predefined shooting scene 1401.

[0104] In this embodiment, smile detection is performed in portraits and small group scenes, but not in large group scenes, crowded scenes, or landscape scenes. Furthermore, closed eyes detection is performed in portraits, small group scenes, and large group scenes, but not in crowded scenes or landscape scenes. In addition, blur / blur detection is performed in portraits, small group scenes, and large group scenes, but not in crowded scenes or landscape scenes.

[0105] In portrait scenes, one or more facial regions whose area ratio exceeds a threshold are targeted to determine if a smile is occurring. In scenes with a small group of people, a predetermined percentage or more of the facial regions whose area ratio exceeds the threshold are targeted to determine if a smile is occurring.

[0106] Furthermore, in portrait scenes, all facial regions exceeding the threshold in area ratio are targeted for eye-closing detection; in scenes with a small number of people, one or more facial regions exceeding the threshold in area ratio are targeted; and in scenes with a large number of people, all facial regions are targeted for eye-closing detection.

[0107] Furthermore, in portrait scenes, all face regions whose area ratio exceeds the threshold are targeted for blur / blur determination; in small group scenes, one or more face regions whose area ratio exceeds the threshold are targeted; and in large group scenes, all face regions are targeted.

[0108] For example, in a portrait with one or two main subjects, a smile detection requires that at least one of the main subjects is smiling, a closed eyes detection requires that everyone has their eyes open, and a blur / out-of-focus detection requires that everyone is not blurred or out of focus.

[0109] In scenes with a small group of 3 to 5 people, the following conditions must be met: for smile detection, a separately specified percentage of the facial area larger than a certain predetermined value must be smiling; for eye-closed detection, at least one person must not have their eyes closed; and for blur / out-of-focus detection, at least one person must not be blurry or out of focus.

[0110] In scenes with more than 10 people, smiles are not expected, so smile detection will be excluded from evaluation. On the other hand, eye-closed detection requires that the eyes be open in all facial regions recognized as face regions, and similarly, blur / blur detection requires that all facial regions are not blurred or blurred.

[0111] In crowded or landscape scenes, people are not considered the primary subject of the image, and facial expression detection is excluded from evaluation in all evaluation criteria.

[0112] Figure 15 is a flowchart illustrating the image evaluation process of the facial expression evaluation device 400 in the server device 103 of this embodiment.

[0113] In Figure 15, the CPU 206 of the server device 103 executes a program to control each component and realize the functions of the facial expression evaluation device 400.

[0114] In step S1501, the CPU 206 of the server device 103, acting as the data receiving unit 401 of the facial expression evaluation device 400, receives image data uploaded from the imaging device 101 or the information processing device 102 into the memory 208 or storage 209.

[0115] In step S1502, the CPU 206 of the server device 103 saves the image data received in step S1501 to the image data storage 402 of the facial expression evaluation device 400.

[0116] In step S1503, the CPU 206 of the server device 103 reads the image data saved in step S1502, acquires metadata attached to the image data, and records it in the face area information DB 404.

[0117] In step S1504, the CPU 206 of the server device 103, acting as the face position detection unit 405 of the facial expression evaluation device 400, performs face position detection on the image data read in step S1503. The face position detection unit 405 of the facial expression evaluation device 400 acquires the coordinate information 504, 505 of the face region, the coordinate information of the focus position 703, and the gaze vector 704, and records them in the face region management table 600 of the face region information DB 404. The face position detection unit 405 also updates the face region management table 600 by setting a priority coefficient 606 from the coordinate information of the focus position 703 and the gaze vector 704.

[0118] Furthermore, if no area is found to be a person's face during face position detection, the image data received in step S1501 is determined not to contain a person, and processing is terminated.

[0119] In step S1505, the CPU 206 of the server device 103, acting as the face region information analysis unit 406 of the facial expression evaluation device 400, reads the coordinate information 602 and 603 of the face region from the face region management table 600 updated in step S1505, calculates the area ratio 604, the reference distance 605, and the difference 906 in the Y direction of the face region, sets the area ratio threshold 607, and records it in the face region management table 600.

[0120] In step S1506, the CPU 206 of the server device 103, acting as the scene determination unit 407 of the facial expression evaluation device 400, obtains the number of face regions exceeding the area ratio threshold, the variation deviation of the distance between face regions, and the variation deviation of the alignment of face regions from the face region management table 600 recorded in step S1505, and determines the scene in which the image was taken by comparing it with the corresponding determination criteria 1002 to 1005 in the scene determination criterion table 1000.

[0121] In step S1507, the CPU 206 of the server device 103, acting as the evaluation target determination unit 408 of the facial expression evaluation device 400, reads facial region information from the facial region management table 600 recorded in step S1504, and determines the facial region to be evaluated by comparing it with the evaluation target determination table 1400 based on the shooting scene determined in step S1506, and then performs the facial expression evaluation.

[0122] In step S1508, the CPU 206 of the server device 103, acting as an image evaluation unit 409, transmits the image evaluation results performed in step S1507 to the information processing device 102. The image evaluation results may be used as parameters in other processes in the server device 103.

[0123] Figure 16 illustrates the face position detection process in step S1504 of Figure 15.

[0124] In step S1601, the face position detection unit 405 detects a person's face in the image data read from the image data storage 402 of the facial expression evaluation device 400 in step S1503.

[0125] In step S1602, the face position detection unit 405 generates a face region ID 601 for the face region detected in step S1601.

[0126] In step S1603, the face position detection unit 405 acquires coordinate information for the upper left corner coordinate 504 and the lower right corner coordinate 505, which determine the rectangular area of ​​the face region detected in step S1601.

[0127] In step S1604, the face position detection unit 405 adds the face region ID generated in step S1602 and the face region coordinate information acquired in step S1603 to the face region management table 600.

[0128] In step S1605, the face position detection unit 405 determines whether the face region detected in step S1601 coincides with the focus position 703. The face position detection unit 405 determines whether the coordinate information of the focus position 703 from the metadata of the image data acquired in step S1503 is included in the rectangular area of ​​the face region acquired in step S1603. If it is included, the process proceeds to step S1606; otherwise, the process proceeds to step S1607.

[0129] In step S1606, the face position detection unit 405 sets the priority coefficient 606 of the face region management table 600. In this embodiment, the priority coefficient is set so that face regions that match the focus position 703 of the image or face regions whose gaze direction 704 is facing forward have a higher priority.

[0130] In step S1607, the face position detection unit 405 determines whether or not there are unprocessed face regions in the image data. If there are no unprocessed face regions, it terminates the process. If there are unprocessed face regions, it returns to step S1602 and records all face regions of multiple people included in the image in the face region management table 600.

[0131] Figure 17 illustrates the facial region information analysis process in step S1505 of Figure 15.

[0132] In step S1701, the face region information analysis unit 406 sorts the face region information recorded in the face region management table 600 in step S1604 of Figure 16 in descending order of area ratio 604.

[0133] In step S1702, the face region information analysis unit 406 performs a process to rearrange the face region information in the face region management table 600, which was sorted in step S1701, in order of highest priority coefficient 606.

[0134] In step S1703, the face region information analysis unit 406 calculates the difference in area ratio 604 before and after sorting the information for each face region in the face region management table 600, which was sorted in step S1701 based on the area ratio 604.

[0135] In step S1704, the face region information analysis unit 406 calculates the amount of change in the difference of the area ratio obtained in step S1703, and sets the area ratio threshold flag 607 to true for face regions where the difference changes rapidly, and records it.

[0136] In step S1705, the face area information analysis unit 406 calculates the reference distance 605 and records it in the face area management table 600.

[0137] In step S1706, the face region information analysis unit 406 calculates the deviation of the distance variation of the face region based on the reference distance 605 obtained in step S1705.

[0138] In step S1708, the face region information analysis unit 406 calculates the deviation of the alignment of the face region from the coordinates 602 and 603 of the face region acquired in step S1603.

[0139] Figure 18 illustrates the scene determination process in step S1506 of Figure 15.

[0140] In step S1801, the scene determination unit 407 obtains the maximum value of the area ratio 604 from the face area management table 600, which was processed in step S1702 according to the priority coefficient 606.

[0141] In step S1802, the scene determination unit 407 calculates the number of face regions with an area ratio 604 that is greater than the face regions for which the area ratio threshold flag 607 is set to true in step S1704.

[0142] In step S1803, the scene determination unit 407 obtains the deviation of the distance variation of the face region obtained in step S1706.

[0143] In step S1804, the scene determination unit 407 obtains the deviation of the alignment of the face regions obtained in step S1708.

[0144] In steps S1805 and S1806, the scene determination unit 407 compares each value obtained in steps S1801 to S1804 with the corresponding item in the scene determination criterion table 1000 to determine the scene in which the image was taken.

[0145] As described above, according to this embodiment, without using personal information such as individual facial feature information, it is possible to determine which person in the image should be evaluated by estimating the scene in which the image was taken based on the area ratio and positional relationship of the facial regions of the people in the image. This makes it possible to evaluate the facial expressions of people in images containing multiple people more appropriately.

[0146] Furthermore, since there is no need to register standard facial images or feature information in advance, and there is no need to store and manage sensitive personal information, it leads to improved information management and avoidance of legal risks. In addition, the matching process for facial feature information is unnecessary, which reduces the load on the system.

[0147] [Other Embodiments] The present invention can also be realized by supplying a program that implements one or more of the functions of the above-described embodiments to a system or device via a network or storage medium, and by having one or more processors in the computer of that system or device read and execute the program. It can also be realized by a circuit (e.g., ASIC) that implements one or more functions.

[0148] The technical ideas derived from this disclosure are not limited to the exemplary embodiments disclosed, but are intended to encompass various modifications of the exemplary embodiments, or substitutions with equivalent structures or functions. The scope of the following claims should be interpreted in the broadest way to encompass all such modifications and equivalent structures and functions.

[0149] This application claims priority based on Japanese Patent Application No. 2025-010638, filed on 24 January 2025, and all of its contents are incorporated herein by reference.

[0150] 101...Imaging device, 102...Information processing device, 103...Server device, 206...CPU, 400...Facial expression evaluation device, 405...Face position detection unit, 406...Face region information analysis unit, 407...Scene determination unit, 408...Evaluation target determination unit, 409...Image evaluation unit

Claims

1. An information processing device comprising: detection means for obtaining location information of a person's face region detected from an image and obtaining the area of ​​the face region from the location information; determination means for determining the scene in which the image was taken based on the area of ​​the face region and the positional relationship of the face region; determination means for determining the main subject in the image based on the scene determined by the determination means; and evaluation means for evaluating the facial expression of the main subject determined by the determination means.

2. The information processing apparatus according to claim 1, characterized in that the detection means approximates the face region as a rectangular region and determines the area ratio of the face region to the whole image from the coordinate information of the rectangular region.

3. The information processing apparatus according to claim 2, further comprising analysis means for determining the difference in area ratios of preceding and succeeding face regions in order of the face region having the largest area ratio, and for determining the deviation of the variation in the distance to the reference position of the face region and the deviation of the variation in alignment with respect to the direction in which the face regions are lined up.

4. The information processing apparatus according to claim 3, characterized in that the reference position is the center or focus position of the image.

5. The information processing apparatus according to claim 3 or 4, characterized in that the detection means sets a priority for the face regions, and the analysis means calculates the difference in area ratio between the face regions before and after the face region with the highest priority, and calculates the deviation of the distance variation of the face regions and the deviation of the alignment variation of the face regions.

6. The information processing apparatus according to claim 5, characterized in that the detection means gives higher priority to a face region including the focus position of the image or a face region whose gaze direction is facing forward.

7. The information processing apparatus according to any one of claims 3 to 6, wherein the analysis means sets a threshold for the area ratio based on the difference, and the determination means determines the number of face regions in which the area ratio exceeds the threshold.

8. The information processing apparatus according to claim 7, further comprising storage means for storing the position information and priority of the face regions, the threshold of the area ratio, the variation deviation of the distance of the face regions, and the variation deviation of the alignment of the face regions as a table for each face region.

9. The information processing apparatus according to claim 8, characterized in that the determination means determines the scene in which the image was taken based on at least one of the area ratio of the largest face area, the number of face areas whose area ratio exceeds a threshold, the deviation of the variation in the distance of the face areas, and the deviation of the variation in the alignment of the face areas.

10. The information processing apparatus according to claim 9, further comprising storage means for storing, as a table, judgment criteria that define, for each scene, the area ratio of the largest face area, the number of face areas whose area ratio exceeds a threshold, the deviation of the variation in the distance of the face areas, and the deviation of the variation in the alignment of the face areas.

11. The information processing apparatus according to claim 10, characterized in that the determination means determines the scene by comparing the area ratio of the largest face area, the number of face areas whose area ratio exceeds a threshold, the deviation of the variation in the distance of the face areas, and the deviation of the variation in the alignment of the face areas with the determination criteria.

12. The information processing apparatus according to any one of claims 9 to 11, characterized in that the scene includes a portrait, a scene with a small number of people, a scene with a large number of people, a crowd scene, or a landscape scene.

13. The information processing apparatus according to claim 12, characterized in that the evaluation means determines, for each scene, one of the following as the facial expression: smiling, closed eyes, blurred / out of focus, or not subject to evaluation.

14. The information processing apparatus according to claim 13, characterized in that the evaluation means determines whether a smile is present in the portrait and the small group scene, but does not determine whether a smile is present in the large group scene, the crowd scene, or the landscape scene.

15. The information processing apparatus according to claim 13, characterized in that the evaluation means determines whether eyes are closed in the portrait, the small group scene, and the large group scene, but does not determine whether eyes are closed in the crowd scene or the landscape scene.

16. The information processing device according to claim 13, characterized in that the evaluation means determines blur / blur in the portrait, the small group scene, and the large group scene, but does not determine blur / blur in the crowd scene or landscape scene.

17. The information processing apparatus according to any one of claims 1 to 16, characterized in that the evaluation means excludes from evaluation when it is determined that a person is not the main subject in the image.

18. The information processing apparatus according to claim 14, characterized in that the evaluation means targets one or more face regions whose area ratio exceeds a threshold in the portrait scene, and targets a predetermined proportion or more of the face regions whose area ratio exceeds a threshold in the small group scene to determine that a face is smiling.

19. The information processing apparatus according to claim 15, characterized in that the evaluation means targets all face regions whose area ratio exceeds a threshold in the portrait scene, targets one or more face regions whose area ratio exceeds a threshold in the small group scene, and targets all face regions in the large group scene to determine if the eyes are closed.

20. The information processing device according to 16, characterized in that the evaluation means targets all face regions whose area ratio exceeds a threshold in the portrait scene, targets one or more face regions whose area ratio exceeds a threshold in the small group scene, and targets all face regions in the large group scene to determine blur / blur.

21. An information processing method performed by an information processing device, comprising: a detection step of obtaining location information of a person's face region detected from an image and obtaining the area of ​​the face region from the location information; a determination step of determining the scene in which the image was taken based on the area of ​​the face region and the positional relationship of the face region; a decision step of determining the main subject in the image based on the determined scene; and an evaluation step of evaluating the facial expression of the determined main subject.

22. A program for causing a computer to function as one of the means of an information processing device according to any one of claims 1 to 20.

Images