Information processing device and information processing method

Abstract

The present disclosure relates to an information processing device and an information processing method that make it possible to improve the accuracy of estimating the shape of a person's hand estimated from a captured image of the person who is a subject. In the present invention, a mesh is estimated from an image of the person who is the subject, the mesh is used to set a hand joint parameter to 0 and determine the size from the wrist to the tip of the middle finger when the palm is opened as the size of a hand region, the skeleton of the person is estimated from the image, a center position of the hand region is determined from the joint group corresponding to the skeleton of the person, and the hand shape is estimated on the basis of a hand image obtained by cutting out, with the center position of the hand region as a reference, the size range obtained by multiplying the size of the hand region by a prescribed coefficient. The present disclosure can be applied to virtual viewpoint image reproduction apparatuses.

Description

Information Processing Apparatus and Information Processing Method

[0001] The present disclosure relates to an information processing apparatus and an information processing method, and particularly to an information processing apparatus and an information processing method capable of improving the estimation accuracy of the hand shape of a person estimated from an image of the person serving as a subject.

[0002] A technique has been proposed for imaging a person serving as a subject and estimating the shape of the hand based on the captured image.

[0003] For example, there is a technique for detecting two-dimensional joint positions and three-dimensional joint positions including the hand from an image of a person serving as a subject, and generating three-dimensional motion data of the whole body including fine finger movements from the detected joint positions (see Patent Document 1).

[0004] Japanese Patent Application Laid-Open No. 2024-080532

[0005] However, in the technique of Patent Document 1, although motion data including finger movements is generated from the two-dimensional joint positions and three-dimensional joint positions estimated from the entire image, there is a limit to the estimation accuracy of the joint positions obtained from the entire image.

[0006] In particular, the joint positions of the hand are often in a small range when viewed from the entire image, and as the hand becomes more complex in shape with a part of the hand hidden by a part of the body, another hand, or a finger, the estimation accuracy of the joint positions of the hand may decrease, and a decrease in the estimation accuracy of the hand shape associated with this is a concern.

[0007] The present disclosure has been made in view of such a situation, and particularly aims to improve the estimation accuracy of the hand shape of a person estimated from an image of the person serving as a subject.

[0008] An information processing device in one aspect of the present disclosure is an information processing device comprising: a size determination unit that estimates the mesh of a person from an image of the person to be the subject, processes the hand region from the estimated mesh to determine the size of the hand region; a position determination unit that estimates the skeleton of the person from the image, determines the center position of the hand region from the estimated skeleton of the person; and a hand image cropping unit that crops the hand image from the image based on the size of the hand region and the center position of the hand region.

[0009] One aspect of the information processing method of this disclosure is an information processing method that includes: estimating a mesh of a person from an image of a person, processing the hand region from the estimated mesh to determine the size of the hand region; estimating the skeleton of the person from the image, determining the center position of the hand region from the estimated skeleton of the person; and cutting out the hand image from the image based on the size of the hand region and the center position of the hand region.

[0010] In one aspect of this disclosure, a mesh of a person is estimated from an image of the person being photographed, a hand region is processed from the estimated mesh to determine the size of the hand region, the skeleton of the person is estimated from the image, the central position of the hand region is determined from the estimated skeleton of the person, and the hand image is cropped from the image based on the size of the hand region and the central position of the hand region.

[0011] This is a diagram illustrating the outline of this disclosure. This is a block diagram illustrating an example of the hardware configuration of the information processing device of this disclosure. This is a functional block diagram illustrating the functions realized by the information processing device in Figure 2. This is a diagram illustrating the processing of the size determination unit. This is a diagram illustrating the processing of the position determination unit. This is a diagram illustrating the processing of the hand image cropping unit. This is a flowchart illustrating the hand shape recognition operation processing by the information processing device in Figure 3. This is a flowchart illustrating the size determination process. This is a flowchart illustrating the position determination process. This is a flowchart illustrating the hand image cropping process. This shows an example of the computer configuration.

[0012] Preferred embodiments of this disclosure will be described in detail below with reference to the attached drawings. In this specification and the drawings, components having substantially the same functional configuration are denoted by the same reference numerals, and redundant descriptions will be omitted.

[0013] The following describes embodiments for implementing the technology of this disclosure. The description will be in the following order: 1. Overview of this disclosure 2. Preferred embodiments 3. Description of a computer to which this technology is applied

[0014] <<1. Overview of this Disclosure>> <Generation of virtual viewpoint images> This disclosure aims to improve the accuracy of estimating the hand shape of a person, which is estimated from an image of a person being photographed.

[0015] First, I will explain the overview of the technology described in this disclosure.

[0016] It has been found that when estimating the shape of a hand based on an image of a person, the estimation accuracy can be improved by appropriately extracting the image of the region in which the hand is captured from the image of the person.

[0017] Therefore, in this disclosure, first, a mesh model of the human body is estimated from an image of the subject by performing mesh estimation, and the corresponding joint positions of the person's skeleton are estimated by skeletal estimation.

[0018] Next, an image region suitable for estimating the hand shape within the image is extracted as a hand image based on the mesh and joint positions, and the hand shape is then estimated using this extracted hand image.

[0019] More specifically, as shown in Figure 1, for example, a mesh model of the human body is estimated by mesh estimation based on an image PI consisting of a full-body image of the subject, and a mesh image PM is generated. In addition, a skeletal image PB showing the joint positions corresponding to the person's skeleton is generated by skeletal estimation based on an image PI consisting of a full-body image of the subject.

[0020] In Figure 1, the mesh image PM is colored gray, representing the human body model within the image PI. The mesh is a human body model corresponding to a person, and its structure allows for adjustment of parameters such as the size of parts (hands, feet, face, etc.) and the angles of each joint. In estimating the mesh based on the image PI, the size of each part and the degree of curvature of each joint are reflected in the shape by adjusting parameters based on the information of the person's entire body in the image PI. Therefore, the size of each part, including the hands, can be determined with a certain degree of accuracy.

[0021] Furthermore, in the skeletal image PB of Figure 1, the estimated positions of the joints corresponding to the estimated positions of the skeleton are represented by black dots. Skeletal estimation estimates the positions of the skeleton and the joints that connect the skeleton, but the positions of each part estimated from the image, especially the joints, may not be determined with sufficient accuracy when parts are hidden or when the shape is complex depending on the body's pose. In particular, the hands have more bones and joints compared to other parts of the body, so the accuracy of individual positions tends to decrease depending on the pose.

[0022] However, the position of the hand can be determined with a certain degree of accuracy from the positions of the joints, which are a group of bones and joints that make up the hand.

[0023] Therefore, based on the hand size identified from the mesh image PM and the hand position identified from the skeletal image PB, the position and size of the hand image necessary for estimating the hand shape are determined, and as shown in Figure 1, hand images Ph1 and Ph2 with the position and size identified from image PI are extracted.

[0024] Based on the hand images Ph1 and Ph2 extracted in this way, the hand shape is estimated, and processing is performed based on the estimation result.

[0025] Since hand images Ph1 and Ph2 are extracted as images of an appropriate position and size for hand shape estimation, it becomes possible to improve the accuracy of hand shape estimation.

[0026] As a result, it becomes possible to improve the processing accuracy performed by applications based on hand gestures that utilize hand shapes.

[0027] <<2. Preferred Embodiments>> <Example of Hardware Configuration of Information Processing Device> Next, an example of the hardware configuration of the information processing device of the present disclosure will be described with reference to Figure 2. The information processing device 31 in Figure 2 captures an image of a person who is a user, estimates the user's hand gestures as a hand shape, and performs various processes based on the estimated hand shape.

[0028] The information processing device 31 consists of a processing circuit 51, an input unit 52, an output unit 53, a storage unit 54, a communication unit 55, a drive 56, and a removable storage medium 57, which are interconnected via a bus 58, enabling the transmission and reception of data and programs. An imaging unit 61 is also connected to the information processing device 31, via the bus 58.

[0029] The processing circuit 51 consists of a processor that executes programs such as a CPU (Central Processing Unit) and a DSP (Digital Signal Processor), as well as memory, and controls the overall operation of the information processing device 31. The processing circuit 51 also includes a human detection unit 71, a mesh estimation unit 72, a size determination unit 73, a skeleton estimation unit 74, a position determination unit 75, a hand image cropping unit 76, a hand shape estimation unit 77, and an motion processing unit 78.

[0030] The functions of the information processing device 31, which is realized by a processing circuit 51 comprising a person detection unit 71, a mesh estimation unit 72, a size determination unit 73, a skeleton estimation unit 74, a position determination unit 75, a hand image cropping unit 76, a hand shape estimation unit 77, and a motion processing unit 78, will be described in detail later with reference to Figure 3.

[0031] The input unit 52 consists of input devices such as buttons, switches, operation knobs, keyboards, pointing devices, and touch panels for inputting various types of information, and supplies the input information and corresponding signals to the processing circuit 51.

[0032] The output unit 53 is controlled by the processing circuit 51 and includes a display unit and an audio output unit. The display unit is composed of, for example, an LCD (Liquid Crystal Display) or an organic EL (Electro Luminescence) and displays the operation input from the input unit 52 and the various processing results from the processing circuit 51.

[0033] Furthermore, the audio output section consists of audio output devices such as speakers, and outputs various voices, music, sound effects, etc. as audio.

[0034] The storage unit 54 consists of an HDD (Hard Disk Drive), an SSD (Solid State Drive), or semiconductor memory, and is controlled by the processing circuit 51 to write or read various types of data and programs.

[0035] The communication unit 55 is controlled by the processing circuit 51 and enables communication via wired or wireless means, such as LAN (Local Area Network) or Bluetooth (registered trademark), and transmits and receives various data and programs with other information processing devices via the network as needed.

[0036] The drive 56 reads and writes data to removable storage media 57 such as magnetic disks (including flexible disks), optical disks (including CD-ROMs (Compact Disc-Read Only Memory) and DVDs (Digital Versatile Discs)), magneto-optical disks (including MDs (Mini Discs)), or semiconductor memory.

[0037] The imaging unit 61 is an image sensor composed of a CMOS (Complementary Metal Oxide Semiconductor) sensor or the like. The imaging unit 61 captures images with a field of view that includes not only the user's hand but also other parts of the body, and supplies the captured images to the processing circuit 51.

[0038] In Figure 2, the imaging unit 61 is shown as being built into the information processing device 31, but it may be provided outside the information processing device 31 and connected by wire or wireless. Furthermore, the imaging unit 61 does not necessarily have to be connected inside or outside the information processing device 31; the information processing device 31 may acquire images captured by imaging equipment such as the imaging unit 61 and perform processing based on the acquired images. In cases where the imaging unit 61 is provided outside the information processing device, this disclosure can be considered to be implemented by an information processing system consisting of the information processing device 31 and the imaging unit 61.

[0039] <Example of the configuration of functions realized by the information processing device> Next, referring to Figure 3, the functions of the information processing device 31 realized by the processing circuit 51 in Figure 2 will be explained.

[0040] The processing circuit 51 is equipped with a human detection unit 71, a mesh estimation unit 72, a size determination unit 73, a skeleton estimation unit 74, a position determination unit 75, a hand image cropping unit 76, a hand shape estimation unit 77, and an motion processing unit 78, and these units enable the functions of the information processing device 31.

[0041] The person detection unit 71 detects people in the image, generates a person image with the area of ​​the person cut out, and outputs it to the mesh estimation unit 72 and the skeleton estimation unit 74. The person detection algorithm of the person detection unit 71 is not particularly limited and may be a known algorithm using a machine learning model such as ViTDet (see https: / / arxiv.org / abs / 2203.16527 for details).

[0042] The mesh estimation unit 72 estimates a mesh of the entire body (including hands) from a human image and outputs the estimated mesh to the size determination unit 73. The mesh estimation algorithm of the mesh estimation unit 72 is not particularly limited and may be a known algorithm using a machine learning model such as SMPLer-X (for details, see https: / / arxiv.org / abs / 2309.17448).

[0043] The size determination unit 73 calculates the hand size based on the estimated mesh and outputs it to the hand image extraction unit 76.

[0044] More specifically, for example, as shown in the image Pm1 of FIG. 4, when the mesh M is estimated, the size determination unit 73 sets the joint parameters of the hand mesh to 0 and opens the palm as shown in the image Pm2 of FIG. 4. Then, as shown in the image Pm3 of FIG. 5, the size determination unit 73 calculates the length of the palm in the image (the number of pixels from the base of the wrist to the tip of the middle finger) Dh as the hand size and outputs it to the hand image extraction unit 76.

[0045] Generally, at the tip of the body such as the hand or foot, the position of the estimated mesh may shift. This is because the mesh is realized by forward kinematics starting from the pelvis, so the position shift accumulates as the distance from the pelvis increases. For example, the position of joints farther in front of the pelvis than the elbow or knee is more likely to shift. Therefore, the position of the hand is separately calculated by skeletal estimation by the skeletal estimation unit 74 described later. However, regarding the joint angles, especially when the shape of the hand is complex or a part of the hand is hidden, both skeletal estimation and mesh estimation are often inaccurate, and errors cannot be corrected even in mesh estimation.

[0046] Therefore, the size determination unit 73 sets the hand joint parameters to 0 and widens the palm to reduce the influence of the error related to the hand joint angle that occurs in mesh estimation and make it suitable for calculating the hand size. This is because the mesh estimation takes into account the sizes of other body parts besides the hand, and the size of the hand itself ignoring the joint angle can be calculated more accurately than skeletal estimation. As a result, the size determination unit 73 can calculate the hand size with higher accuracy than a predetermined accuracy based on the mesh.

[0047] The skeleton estimation unit 74 estimates the full-body skeleton (including the hands) from a human image and outputs information on the joint positions corresponding to the estimated skeleton to the positioning unit 75. The skeleton estimation algorithm of the skeleton estimation unit 74 is not particularly limited, and for example, a known algorithm using a machine learning model such as ViTPose++ (for details, refer to https: / / arxiv.org / abs / 2212.04246v3) may be used.

[0048] The positioning unit 75 calculates the center position of the hand based on the information on the joint positions corresponding to the estimated skeleton and outputs it to the hand image extraction unit 76.

[0049] More specifically, for example, when information on the joint positions corresponding to the skeleton shown by the black dot in the person H in the image Pb1 of FIG. 5 is supplied, the positioning unit 75 calculates the smallest rectangular region Zg that encloses the hand skeleton (joint positions) as shown in the image Pb2 of FIG. 5. Next, the positioning unit 75 calculates the center of the rectangular region Zg as the center position Pc of the hand and outputs it to the hand image extraction unit 76, as shown in the image Pb3 of FIG. 5.

[0050] Generally, when the shape of the hand is complex or a part of the hand is hidden, the position of the estimated skeleton and the joint positions corresponding to the skeleton may shift.

[0051] However, unlike mesh estimation, in skeleton estimation, the positional shift does not accumulate even when moving away from the pelvis. Therefore, by calculating the center of the hand joint group as the center position of the hand by the above-described method, it is possible to obtain the center position of the hand with higher accuracy than a predetermined accuracy.

[0052] The hand image extraction unit 76 outputs an image obtained by cutting out the hand region from the image PI as a hand image suitable for hand shape estimation to the hand shape estimation unit 77 based on the calculated center position Pc of the hand and the size Dh.

[0053] More specifically, the hand image extraction unit 76 determines the size of the image to be cut out as a rectangular size consisting of the length obtained by multiplying the size Dh of the hand by a predetermined coefficient C (for example, 1.5 times, etc.), generates a hand image by cutting it out with the center position Pc of the hand as a reference, and outputs it to the hand shape estimation unit 77.

[0054] More specifically, as shown in the left part of Figure 6, in the case of image PI' (a peripheral image of the hand in image PI of Figure 1), the hand image cutout section 76 cuts out a rectangular region based on the center position Pc, with a length of (Dh × C) × (Dh × C) obtained by multiplying the hand size Dh by a coefficient C, as the hand image Pzc shown in the right part of Figure 6.

[0055] By generating a hand image using the method described above, it becomes possible to extract an appropriate range of the hand area as a hand image, even when the hand shape is complex or when part of the hand is hidden. Note that the coefficient C multiplied by the hand size Dh will differ depending on the type of hand shape estimation unit 77, so it is desirable to tune it in advance. More specifically, for example, if the hand shape estimation unit 77 is generated by machine learning, the coefficient C may be set to a value such that when the hand size Dh is multiplied by the coefficient C, the size of the hand in the extracted hand image matches the size of the hand in the image used in machine learning.

[0056] In the above, we have described an example in which the joint parameters are set to 0 by the size determination unit 73, and the palm is in an open position, and the palm length Dh, which is the number of pixels from the base of the wrist to the tip of the middle finger, is calculated as the hand size.

[0057] However, as mentioned above, the hand size Dh is a criterion for determining the size of the image to be cropped as a hand image, so it does not have to be the length from the base of the wrist to the tip of the middle finger. For example, it could be the length from the tip of the little finger to the tip of the thumb when the palm is open. However, in this case, the coefficient C will need to be adjusted according to the length that serves as the basis for the hand size Dh.

[0058] The hand shape estimation unit 77 estimates the shape of the hand based on the hand image and outputs the estimation result to the motion processing unit 78. The hand shape estimation algorithm of the hand shape estimation unit 77 is not particularly limited and may be a known algorithm using a machine learning model such as HaMeR (for details, see https: / / arxiv.org / abs / 2312.05251).

[0059] The motion processing unit 78 executes a predetermined operation based on the hand shape estimation result supplied by the hand shape estimation unit 77.

[0060] The operations performed in the operation processing unit 78 are, for example, operations implemented by application programs installed on the information processing device. For example, in the case of an application program that reflects the poses actually taken by the user onto an avatar in a virtual space, the operation involves recognizing the hand shape as a hand gesture demonstrated by the user, and displaying the avatar in a pose corresponding to the recognition result.

[0061] Furthermore, if the application program functions as a remote control for electronic devices (such as televisions, air conditioners, and fluorescent lights), the operation performed by the operation processing unit 78 will be to assign hand gestures to each button operation on the remote control, thereby executing button operations according to the recognition results.

[0062] Furthermore, in the case of application programs such as games, the actions performed by the operation processing unit 78 are actions that realize interaction related to game operation using hand gestures.

[0063] <Hand Shape Recognition Operation Processing> Next, the hand shape recognition operation processing by the information processing device 31 shown in Figures 2 and 3 will be explained with reference to the flowchart in Figure 7.

[0064] In step S31, the human detection unit 71 acquires an image captured by the imaging unit 61, or an image that has already been captured.

[0065] In step S32, the person detection unit 71 detects a person in the acquired image, generates a person image with the area of ​​the person cut out, and outputs it to the mesh estimation unit 72 and the skeleton estimation unit 74.

[0066] In step S33, the mesh estimation unit 72 estimates the mesh of the entire body (including the hands) from the human image and outputs the estimated mesh to the size determination unit 73.

[0067] In step S34, the size determination unit 73 performs a size determination process based on the estimated mesh to calculate the size of the hand and output it to the hand image cropping unit 76.

[0068] <Size Determination Process> Now, with reference to the flowchart in Figure 8, the size determination process by the size determination unit 73 will be explained.

[0069] In step S51, the size determination unit 73 sets the joint parameters of the hand mesh to 0, causing the palm to be in an open position.

[0070] In step S52, the size determination unit 73 calculates the palm length Dh in the image as the hand size and outputs it to the hand image cropping unit 76.

[0071] Through the above process, the joint parameters of the hand mesh are set to 0, the size of the hand in an open palm state is calculated, and this is output to the hand image cropping unit 76.

[0072] This makes it possible to calculate hand size with a higher level of accuracy than a predetermined one, without being affected by inaccurate joint angles in the estimated mesh.

[0073] Now, let's return to the explanation of the flowchart in Figure 7.

[0074] In step S35, the skeleton estimation unit 74 estimates the entire skeleton (including the hands) from the image of the person and outputs information on the estimated skeleton and the corresponding joint positions to the position determination unit 75.

[0075] In step S36, the position determination unit 75 performs a position determination process based on the estimated skeleton and corresponding joint position information to calculate the center position of the hand and output it to the hand image cropping unit 76.

[0076] <Position Determination Process> Now, with reference to the flowchart in Figure 9, the position determination process by the position determination unit 75 will be explained.

[0077] In step S71, the position determination unit 75 calculates the smallest rectangular region Zg that encompasses the bones (joint positions) of the hand.

[0078] In step S72, the position determination unit 75 calculates the center of the rectangular region Zg as the center position Pc of the hand and outputs it to the hand image cropping unit 76.

[0079] Through the above processing, the smallest rectangular region Zg that encompasses the bones (joint positions) of the hand is calculated, and the center of the calculated rectangular region Zg is calculated as the center position Pc of the hand and output to the hand image cropping unit 76.

[0080] This means that even when the hand shape is complex or part of the hand is hidden, there is no accumulation of errors due to forward kinematics, making it possible to calculate the center position of the hand with a higher accuracy than a predetermined level.

[0081] Now, let's return to the explanation of the flowchart in Figure 7.

[0082] In step S37, the hand image cropping unit 76 performs hand image cropping processing based on the calculated hand center position Pc and size Dh, generates an image with the hand region cropped from the input image, and outputs it to the hand shape estimation unit 77.

[0083] <Hand Image Cutout Processing> Now, referring to the flowchart in Figure 10, the hand image cutout processing by the hand image cutout unit 76 will be explained.

[0084] In step S91, the hand image cutout unit 76 determines the size of the rectangular region to be cut out, (Dh × C) × (Dh × C), by multiplying the hand size Dh by a predetermined coefficient C (for example, 1.5 times).

[0085] In step S92, the hand image cropping unit 76 crops an image of size (Dh × C) × (Dh × C) with respect to the center position Pc of the hand, and outputs it as a hand image to the hand shape estimation unit 77.

[0086] The above process generates a hand image, making it possible to generate a hand image within a range appropriate for estimating the hand shape, even when the hand shape is complex or when part of the hand is hidden.

[0087] Now, let's return to the explanation of the flowchart in Figure 7.

[0088] In step S38, the hand shape estimation unit 77 estimates the shape of the hand based on the hand image supplied by the hand image cropping unit 76, and outputs the estimation result to the motion processing unit 78.

[0089] In step S39, the operation processing unit 78 performs a predetermined operation based on the hand shape estimation result supplied by the hand shape estimation unit 77.

[0090] Through the above processing, a hand image with the optimal position and size for recognizing the hand shape is generated from the image of the subject person, making it possible to improve the accuracy of hand shape recognition.

[0091] Furthermore, the above process makes it possible to generate a hand image with the optimal position and size for hand shape recognition even from a single frame of image. Therefore, it is possible to improve the accuracy of hand shape recognition, regardless of whether the image is moving or still.

[0092] As a result, it becomes possible to improve the accuracy of recognizing the hand shape of a person, which is estimated from images of the subject.

[0093] In the above explanation, we have proceeded under the assumption that the image captured by the imaging unit 61 is a whole-body image. As mentioned above, it is desirable that the image captured by the imaging unit 61 be a whole-body image that includes the parts of the hand necessary for estimating the position and size of the hand, as this allows for more accurate estimation. However, the image captured by the imaging unit 61 does not necessarily have to be a whole-body image, as long as some parts, including the hand, are captured to the extent that the size and position of the hand can be estimated with a predetermined accuracy. For example, even if the image captured by the imaging unit 61 does not show the parts above the neck or half of the body, the mesh and skeleton can be estimated with a higher accuracy than the predetermined value, so the mesh and skeleton can be estimated from such an image, and the hand image can be extracted from the estimation result.

[0094] <<3. Description of a computer using this technology>>

[0095] The series of processes described above can be executed by hardware or by software. When the series of processes are executed by software, the programs that make up that software are installed on a computer. Here, "computer" includes computers built into dedicated hardware, as well as general-purpose personal computers, for example, that can perform various functions by installing various programs.

[0096] Figure 11 is a block diagram showing an example of the hardware configuration of a computer that executes the series of processes described above by a program.

[0097] In a computer, the processing circuit 1001, ROM (Read Only Memory) 1002, and RAM (Random Access Memory) 1003 are interconnected by a bus 1004.

[0098] An input / output interface 1005 is further connected to the bus 1004. An input / output interface 1005 is connected to an input unit 1006, an output unit 1007, a storage unit 1008, a communication unit 1009, and a drive 1010.

[0099] The input unit 1006 may include physical or virtual operating means that the user operates to input information, such as a keyboard, mouse, or touch panel, as well as means that the user inputs information through voice, eye gaze, etc. Furthermore, the input unit 1006 may include sensors for inputting various physical quantities into the computer. For example, the input unit 1006 may include sensors that acquire physical quantities such as light (including infrared light other than visible light) or sound, such as a camera or microphone. Also, for example, the input unit 1006 may include sensors that acquire other physical quantities such as temperature, moisture content, acceleration, and distance. The output unit 1007 may include means that present information to the user by stimulating the user's perception, such as a display, speaker, or haptic device. The storage unit 1008 is composed of a hard disk, non-volatile or volatile memory, etc., and stores various information (including programs). The communication unit 1009 is a network interface, etc., and performs wired or wireless communication with the outside. The drive 1010 drives removable media 1011 such as a magnetic disk, optical disk, magneto-optical disk, or semiconductor memory.

[0100] The processing circuit 1001 includes a processor that executes programs such as a CPU (Central Processing Unit) and a DSP (Digital Signal Processor). The processing circuit 1001 (its processor) performs the series of processes described above by loading the program stored in the memory unit 1008 into the RAM 1003 via the input / output interface 1005 and the bus 1004 and executing it. The processing circuit 1001 can output the processing results of the series of processes from the output unit 1007, for example, via the bus 1004 and the input / output interface 1005, as needed. The processing circuit 1001 can also store the processing results in the memory unit 1008 or transmit them from the communication unit 1009.

[0101] The program executed by the computer (processing circuit 1001) can be provided by recording it on a removable medium 1011, such as a package medium. The program can also be provided via wired or wireless transmission media, such as a local area network, the internet, or digital satellite broadcasting.

[0102] In a computer, a program can be installed in the storage unit 1008 via the input / output interface 1005 by inserting the removable media 1011 into the drive 1010. Alternatively, a program can be received by the communication unit 1009 from another device, such as a server, via a wired or wireless transmission medium, and installed in the storage unit 1008. Furthermore, programs can be pre-installed in the ROM 1002 or the storage unit 1008.

[0103] The programs executed by the computer may be programs that are processed chronologically in the order described herein, or they may be programs that are processed in parallel or at necessary times, such as when a call is made.

[0104] The processes that a computer performs according to a program do not necessarily have to follow the order described in the flowchart. In other words, the processes that a computer performs according to a program include processes that are executed in parallel or individually (e.g., parallel processing and object-based processing).

[0105] The program may be processed by a single computer (processor), or it may be processed in a distributed manner by multiple computers. Furthermore, the program may be transferred to a remote computer and executed there.

[0106] When the computer executes a program to perform the series of processes described above, the processing circuit 1001 (its processor) functions as the human detection unit 71, mesh estimation unit 72, size determination unit 73, skeleton estimation unit 74, position determination unit 75, hand image cropping unit 76, hand shape estimation unit 77, and motion processing unit 78 of the processing circuit 51 by executing the program.

[0107] In this specification, a system means one component or a collection of multiple components (devices, modules (parts), etc.). Therefore, one or more components of a computer, for example, only the processor, or a combination of the processor and memory, for example, only the processing circuit 1001, or a combination of the processing circuit 1001 to the bus 1004, etc., constitute a system. Regarding a collection of multiple components, it is not necessary whether all components reside in the same enclosure or not. Therefore, multiple devices housed in separate enclosures and connected via a network, or a single device containing multiple modules within a single enclosure, are all systems. Furthermore, for example, the entire computer, or a combination of a computer and other devices such as a server (not shown), also constitute a system.

[0108] Furthermore, for example, each step of a flowchart may be executed by one device, or it may be divided among multiple devices. Additionally, if a single step includes multiple processes, these processes may be executed by one device, or they may be divided among multiple devices. In other words, multiple processes included in a single step can be executed as multiple steps. Conversely, processes described as multiple steps can be combined and executed as a single step.

[0109] Furthermore, for example, a program executed by a computer may be structured so that the steps of the program are executed chronologically in the order described herein, or they may be executed in parallel or individually at necessary times, such as when a call is made. In other words, the steps may be executed in an order different from the order described above, as long as no inconsistencies arise. Moreover, the steps of this program may be executed in parallel with the processing of other programs, or in combination with the processing of other programs.

[0110] Furthermore, for example, the various technologies relating to this disclosure can be implemented independently, as long as they do not conflict with each other. Of course, any combination of the disclosures can also be implemented. For example, some or all of the disclosures described in one embodiment can be implemented in combination with some or all of the disclosures described in another embodiment. Also, some or all of the aforementioned disclosures can be implemented in combination with other technologies not described above.

[0111] Furthermore, this disclosure may also take the following configurations: <1> An information processing device comprising: a size determination unit that estimates the mesh of a person from an image of the person to be the subject, processes the hand region from the estimated mesh, and determines the size of the hand region; a position determination unit that estimates the skeleton of the person from the image, and determines the center position of the hand region from the estimated skeleton of the person; and a hand image cropping unit that crops the hand image from the image based on the size of the hand region and the center position of the hand region. <2> The information processing device according to <1>, wherein the image is a full-body image. <3> The information processing device according to <1>, wherein the size determination unit processes the hand region by adjusting the hand joint parameters from the mesh, and determines the size of the hand region. <4> The information processing device according to <3>, wherein the size determination unit processes the hand region by adjusting the hand joint parameters to 0, so that the palm in the mesh is open, and determines the size of the hand region. <5> The size determination unit processes the hand region by adjusting the hand joint parameters to 0 and opening the palm in the mesh, and determines the length from the base of the wrist to the tip of the middle finger, or the length from the tip of the little finger to the tip of the thumb, as the size of the hand region, as described in <4>. <6> The position determination unit determines the center position of the hand region from the information of the joints corresponding to the skeleton of the person, as described in <1>. <7> The position determination unit calculates the smallest rectangular region that encloses the hand joint group from the joint group corresponding to the skeleton of the person, and determines the center position of the hand region from the rectangular region, as described in <6>. <8> The position determination unit determines the center of the rectangular region as the center position of the hand region, as described in <7>. <9> The hand image cropping unit determines the size and center position of the hand image in the image based on the size of the hand region and the center position of the hand region, and crops the hand image from the image, as described in <1>.<10> The hand image cropping unit determines the size of the hand image by multiplying the size of the hand region by a predetermined coefficient, and crops the hand image from the image by cropping a range of the hand image size with respect to the center position of the hand region. The information processing device according to <9>. <11> An information processing method comprising: a size determination process that estimates the mesh of a person from an image of a person to be the subject, processes the hand region from the estimated mesh, and determines the size of the hand region; a position determination process that estimates the skeleton of the person from the image, determines the center position of the hand region from the estimated skeleton of the person; and a hand image cropping process that crops the hand image from the image based on the size of the hand region and the center position of the hand region. <12> The information processing method according to <11>, wherein the image is a full-body image. <13> The size determination process that processes the hand region by adjusting the joint parameters of the hand from the mesh, and determines the size of the hand region. The information processing method according to <11>. <14> The size determination process involves processing the hand region by setting the hand joint parameters to 0 and adjusting the mesh to open the palm, thereby determining the size of the hand region, as described in <13>. <15> The size determination process involves processing the hand region by setting the hand joint parameters to 0 and adjusting the mesh to open the palm, thereby determining the size of the hand region as the length from the base of the wrist to the tip of the middle finger, or the length from the tip of the little finger to the tip of the thumb, as described in <14>. <16> The position determination process involves determining the center position of the hand region from the information of the joints corresponding to the skeleton of the person, as described in <11>. <17> The position determination process involves calculating the smallest rectangular region that encloses the hand joint group from the joint group corresponding to the skeleton of the person, and determining the center position of the hand region from the rectangular region, as described in <16>. <18> The position determination process determines the center of the rectangular area as the center position of the hand area, as described in <17>.<19> The hand image cropping process determines the size and center position of the hand image in the image based on the size of the hand region and the center position of the hand region, and crops the hand image from the image. The information processing method according to <11>. <20> The hand image cropping process determines the size of the hand image by multiplying the size of the hand region by a predetermined coefficient, and crops the hand image from the image by cropping a range of the hand image size with respect to the center position of the hand region. The information processing method according to <19>.

[0112] 31 Information processing device, 71 Human detection unit, 72 Mesh estimation unit, 73 Size determination unit, 74 Skeleton estimation unit, 75 Position determination unit, 76 Hand image cropping unit, 77 Hand shape estimation unit, 78 Motion processing unit

Claims

1. An information processing device comprising: a size determination unit that estimates the mesh of a person from an image of the person to be the subject, processes the hand region from the estimated mesh, and determines the size of the hand region; a position determination unit that estimates the skeleton of the person from the image, and determines the center position of the hand region from the estimated skeleton of the person; and a hand image cropping unit that crops the hand image from the image based on the size of the hand region and the center position of the hand region.

2. The information processing apparatus according to claim 1, wherein the image is a whole-body image.

3. The information processing apparatus according to claim 1, wherein the size determination unit processes the hand region from the mesh by adjusting the hand joint parameters to determine the size of the hand region.

4. The information processing apparatus according to claim 3, wherein the size determination unit processes the hand region by adjusting the hand joint parameters to 0 and opening the palm in the mesh, thereby determining the size of the hand region.

5. The information processing device according to claim 4, wherein the size determination unit processes the hand region by adjusting the hand joint parameters to 0 and opening the palm in the mesh, and determines the length from the base of the wrist to the tip of the middle finger, or the length from the tip of the little finger to the tip of the thumb, as the size of the hand region.

6. The information processing device according to claim 1, wherein the position determination unit determines the central position of the hand region from the information of the human skeleton and the corresponding joints.

7. The information processing apparatus according to claim 6, wherein the position determination unit calculates the smallest rectangular region that encloses the hand joint group from the joint group corresponding to the skeleton of the person, and determines the center position of the hand region from the rectangular region.

8. The information processing apparatus according to claim 7, wherein the position determination unit determines the center of the rectangular area as the center position of the hand area.

9. The information processing apparatus according to claim 1, wherein the hand image cropping unit determines the size and center position of the hand image in the image based on the size of the hand region and the center position of the hand region, and crops the hand image from the image.

10. The information processing apparatus according to claim 9, wherein the hand image cropping unit determines the size of the hand image by multiplying the size of the hand region by a predetermined coefficient, and crops the hand image from the image by cropping a range of the hand image size with respect to the center position of the hand region.

11. An information processing method comprising: estimating the mesh of a person from an image of the person to be the subject, processing the hand region from the estimated mesh to determine the size of the hand region; estimating the skeleton of the person from the image, determining the center position of the hand region from the estimated skeleton of the person; and cutting out the hand image from the image based on the size of the hand region and the center position of the hand region.

12. The information processing method according to claim 11, wherein the image is a whole-body image.

13. The information processing method according to claim 11, wherein the size determination process involves processing the hand region from the mesh by adjusting the hand joint parameters to determine the size of the hand region.

14. The information processing method according to claim 13, wherein the size determination process involves adjusting the hand region by setting the hand joint parameters to 0 and opening the palm in the mesh, thereby processing the hand region and determining the size of the hand region.

15. The information processing method according to claim 14, wherein the size determination process processes the hand region by adjusting the hand joint parameters to 0 and opening the palm in the mesh, and determines the size of the hand region as the length from the base of the wrist to the tip of the middle finger, or the length from the tip of the little finger to the tip of the thumb.

16. The information processing method according to claim 11, wherein the position determination process determines the central position of the hand region from the information of the skeleton and corresponding joints of the person.

17. The information processing method according to claim 16, wherein the position determination process calculates the smallest rectangular region that encloses the hand joint group from the joint group corresponding to the skeleton of the person, and determines the center position of the hand region from the rectangular region.

18. The information processing method according to claim 17, wherein the position determination process determines the center of the rectangular area as the center position of the hand area.

19. The information processing method according to claim 11, wherein the hand image cropping process determines the size and center position of the hand image in the image based on the size of the hand region and the center position of the hand region, and crops the hand image from the image.

20. The information processing method according to claim 19, wherein the hand image cropping process determines the size of the hand image by multiplying the size of the hand region by a predetermined coefficient, and crops the hand image from the image by cropping a range of the hand image size with respect to the center position of the hand region.

Images