Information processing device for facilitating setting of object to be focused, method for controlling information processing device, and program
The information processing device addresses usability issues in photography by using main object and gesture detection to adjust focus, ensuring clear images without obstructions, thereby improving user experience.
Patent Information
- Application Number
- PCT/JP2026/016893
- Authority / Receiving Office
- WO · WO
- Patent Type
- Applications
- Current Assignee / Owner
- Priority Date
- 2025-05-13
- Filing Date
- 2026-04-28
- Publication Date
- 2026-11-19
AI Technical Summary
Conventional electronic devices face challenges in improving usability when taking photos, particularly in adjusting focus based on user gestures and preventing obstructions by main objects.
An information processing device equipped with a main object detection unit, gesture detection, imaging, determination, and focus adjustment means to adjust focus to a subject position based on user gestures, ensuring the main object no longer obstructs the subject.
Enables easy and accurate setting of the focus point by detecting user gestures, preventing main objects from obstructing the subject, and enhancing user experience in photography.
Smart Images

Figure JP2026016893_19112026_PF_FP_ABST
Abstract
Description
Information processing device for facilitating the setting of the object to be focused, a control method for the information processing device, and a program.
[0001] This disclosure relates to an information processing device, a method for controlling the information processing device, and a program.
[0002] In conventional electronic devices, there is a known technology that performs actions related to photography based on the hand movements of the photographer or other user (Patent Document 1).
[0003] Japanese Patent Publication No. 2008-17501
[0004] There is still room for improvement in terms of usability when taking photos.
[0005] The information processing apparatus of the present disclosure includes: a main object detection means for detecting a predetermined object as a main object using a detection unit; a gesture detection means for detecting a predetermined gesture by the main object; an imaging means for performing photography using an imaging unit; a determination means for determining the position of a subject to be focused on in the photography, using a main object position indicating the position of the main object and a reference position of the shooting range of the imaging means, when a predetermined gesture is performed by the main object; and a focus adjustment means for adjusting the focus to the subject. If the main object is obstructing the subject because it is at the subject position, the focus adjustment means adjusts the focus to the subject after the main object no longer obstructs the subject.
[0006] This is a block diagram showing an example of the functional configuration of a head-mounted display device (HMD). This is a flowchart showing an example of HMD operation. This is a diagram showing an example of an object detection image of an HMD. This is a diagram showing an example of an object detection image of an HMD. This is a diagram showing an example of an object detection image of an HMD. This is a diagram showing an example of a live view image of an HMD. This is a diagram showing an example of a live view image of an HMD. This is a diagram showing an example of a live view image of an HMD. This is a diagram showing an example of a live view image of an HMD. This is a diagram showing an example of an HMD configuration. This is a diagram showing an example of an HMD configuration. This is a diagram showing an example of an HMD configuration. This is a diagram showing an example of a position detection device. This is a block diagram showing an example of the functional configuration of an information processing system.
[0007] The embodiments of this disclosure will be described in detail below with reference to the drawings. However, the configurations described in the following embodiments are merely illustrative, and the scope of this disclosure is not limited by the configurations described in each embodiment. For example, each part constituting this disclosure can be replaced with any configuration that can perform a similar function. In addition, any additional components may be added. Furthermore, any two or more configurations (features) from each embodiment can be combined.
[0008] <First Embodiment> The first embodiment will be described below with reference to Figures 1 to 6B. Figure 1 is a block diagram showing an example of the functional configuration of the HMD 100 (information processing device). The HMD 100 has an imaging unit 101, a processing unit 102, an operation unit 103, a display unit 104, a storage unit 105, and a communication unit 106. The imaging unit 101 acquires imaging data by imaging the real space and outputs the imaging data to the processing unit 102. The processing unit 102 controls the entire HMD 100.
[0009] The operation unit 103 is an interface for receiving user input. The operation unit 103 includes a main object detection unit 111 and a button 112. The main object detection unit 111 detects the position and shape of a predetermined object and outputs information including the position and shape of the detected predetermined object to the processing unit 102. The predetermined object may be the user's hand, a ring-type wearable device worn on the user's finger, a wrist-type wearable device worn on the user's wrist, or a pointing tool held in the user's hand. In the first embodiment, the user's hand is used as the predetermined object. Hereinafter, the predetermined object will be referred to as the "main object".
[0010] Button 112 is pressed by the user when switching the power ON / OFF state of the HMD 100, setting the mode of the HMD 100, or taking a picture with the HMD 100, and also outputs to the processing unit 102 that the button has been pressed by the user. The display unit 104 is made up of a liquid crystal panel or an organic EL display. Images output from the processing unit 102 are displayed on the display unit 104. Therefore, the HMD 100 is a video see-through type. The display unit 104 also serves as an interface for the user to check images before and after taking a picture.
[0011] The memory unit 105 consists of a main memory device (memory) and an auxiliary storage device (storage), and transmits and receives data with the processing unit 102. The memory temporarily stores data during image processing and values necessary for various calculations. The storage device stores various programs such as the shooting application, and saves image files acquired through shooting. The storage device may be an external storage device that can be inserted and removed, such as a memory card. The communication unit 106 will be mentioned in the description of Modification 1 below.
[0012] Figure 2 is a flowchart illustrating an example of the operation of the HMD 100. The flowchart in Figure 2 (control method of the information processing device) is realized when the processing unit 102 (computer) loads the program stored in the storage of the storage unit 105 into the memory of the storage unit 105 and executes it. When the user turns on the power of the HMD 100 using the button 112, the HMD 100 starts the operation shown in the flowchart of Figure 2.
[0013] In step S201, the processing unit 102 starts the shooting application stored in the storage unit 105. When the shooting application is started, a live view image is displayed on the display unit 104, and this displayed live view image continues to be updated until the shooting operation in step S215 is completed. The live view image is an image that the user can see to confirm the image to be taken, and it is generated by image processing performed by the processing unit 102 on the imaging data acquired by the imaging unit 101.
[0014] In step S202, the processing unit 102 checks the setting value stored in the storage unit 105 to determine whether the gesture AF mode is ON. The gesture AF mode is a mode in which the object to be focused on is set based on a gesture made by the main object. Hereinafter, the object to be focused on will be referred to as the "subject". The user can switch the gesture AF mode ON / OFF by changing the setting value using the button 112 in advance. If the processing unit 102 determines that the gesture AF mode is not ON, the process proceeds to step S214, which will be described later. On the other hand, if the processing unit 102 determines that the gesture AF mode is ON, the process proceeds to step S203.
[0015] In step S203, the processing unit 102 determines whether the button 112 has been pressed. If the processing unit 102 determines that the button 112 has been pressed, the process proceeds to step S215, which will be described later. If the processing unit 102 determines that the button 112 has not been pressed, the process proceeds to step S204. In step S204, the processing unit 102 determines whether the main object has been detected by the main object detection unit 111. If the processing unit 102 determines that the main object has not been detected, the process returns to step S203. If the processing unit 102 determines that the main object has been detected, the process proceeds to step S205.
[0016] Here, the method for detecting the main object in step S204 will be described. The main object detection unit 111 is a sensor that can acquire a distance map using LiDAR (Light Detection and Ranging). Figures 3A to 3C show an example of an object detection image 301 generated by the main object detection unit 111. The object detection image 301 is an image generated based on the distance map acquired by the main object detection unit 111, and is a binarized image composed of a main object 302 and a background 303.
[0017] The main object detection unit 111 (detection unit) recognizes objects within a distance close enough to be within reach of the user's hand (specifically, for example, 0.5 m from the main object detection unit 111) (within a predetermined distance) as the main object 302. Furthermore, the main object detection unit 111 displays the recognized main object 302 in white. At this time, areas other than the main object 302 are recognized as the background 303 and are displayed in black. The main object detection unit 111 outputs the object detection image 301 to the processing unit 102. The processing unit 102 (main object detection means) determines that the main object has been detected if the area of the main object 302 relative to the area of the object detection image 301 is equal to or greater than a predetermined ratio (main object detection step). Figure 3A is a diagram showing an example of the object detection image 301 before a release gesture by the user's hand, which is the main object 302.
[0018] Returning to the explanation of Figure 2, in step S205, the processing unit 102 obtains the main object position. The main object position refers to the position of the main object's representative point. In the example of the object detection image 301 shown in Figure 3A, the processing unit 102 calculates the approximate circle 304 closest to the boundary line of the background 303 region surrounded by the main object 302 using the least squares method or the like. Furthermore, the processing unit 102 obtains the center position of the calculated approximate circle 304 as the main object position 305. The main object position 305 is represented in the xy coordinate system shown in Figure 3A.
[0019] The xy coordinates shown in Figure 3A represent a coordinate system where the center of the object detection image 301 is the origin, the longitudinal direction is the x-axis, and the transverse direction is the y-axis. This is also true for the xy coordinates shown in Figures 3B and 3C, which will be described later. Furthermore, the same applies to the xy coordinates shown in Figures 4A to 4E, which will be described later. In other words, the xy coordinates shown in Figures 4A to 4E represent a coordinate system where the center of the live view image 401 is the origin, the longitudinal direction is the x-axis, and the transverse direction is the y-axis.
[0020] Returning to the explanation of Figure 2, in step S206, the processing unit 102 acquires the planned setting position within the shooting range. The coordinates of the planned setting position (X Q , YQ ), where the coordinates of the main object position are defined as (x P , y P ), are calculated by the following Equation (1) and Equation (2). X Q =m×x P +X O ... Equation (1) Y Q =m×y P +Y O ... Equation (2)
[0021] The coefficient m in Equation (1) and Equation (2) is a coefficient indicating sensitivity, and is stored in the storage of the storage unit 105. The coefficient m is, for example, a constant determined by the detection range of the main object detection unit 111, the shooting angle of view of the imaging unit 101, and the like, and can be changed by user settings using the button 112. X used in Equation (1) and Equation (2) O and Y O the coordinates (X O , Y O ) indicated by are the coordinates of the reference position of the shooting range. In the first embodiment, it is assumed that m=1, X O =0, Y O =0 is set.
[0022] In step S207, the processing unit 102 (display means) displays the setting frame (index) on the display unit 104 so as to be superimposed on the live view image. FIGS. 4A to 4E are diagrams showing an example of the live view image 401 displayed on the display unit 104. FIG. 4A is a diagram showing an example of the live view image 401 when no main object exists within the shooting range. The setting frame 402 is displayed such that its center is located at the coordinates (X Q , Y Q ) of the scheduled setting position. Note that the live view image 401 includes a subject 403.
[0023] Returning to the explanation of Figure 2, in step S208, the processing unit 102 determines whether a predetermined gesture by the main object has been detected by the main object detection unit 111. If the processing unit 102 determines that the predetermined gesture by the main object has not been detected, the process returns to step S203. On the other hand, if the processing unit 102 determines that the predetermined gesture by the main object has been detected, the process proceeds to step S209.
[0024] Here, the method for detecting a predetermined gesture in step S208 will be described. Figure 3B shows an example of an object detection image 301 after a release gesture by the user's hand, which is the main object 302. A release gesture is an action that transitions from a state in which a roughly circular shape is formed by the contact between the fingertips of the thumb and index finger of the hand, as shown in Figure 3A, to a state in which the contact points between the fingertips are separated, as shown in Figure 3B. In the first embodiment, the release gesture is a predetermined gesture. The processing unit 102 (gesture detection means) determines that a release gesture by the main object 302 has been detected when the background 303 area surrounded by the main object 302 transitions from a state in which there is an area of the background 302 that does not exist (gesture detection step).
[0025] Returning to the explanation of Figure 2, in step S209, the processing unit 102 (determination means) determines the position of the subject (hereinafter also referred to as "subject position") (determination step). For example, if the processing unit 102 determines in step S208 that a predetermined gesture by the main object has been detected, it determines the coordinates (X) of the planned position acquired immediately before the detection. Q , Y Q ) and the coordinates of the subject's position (X' Q , Y' Q )
[0026] In step S210, the processing unit 102 determines whether the subject is occluded by the main object as viewed from the imaging unit 101. The determination of whether the subject is occluded by the main object as viewed from the imaging unit 101 may be performed based on a live view image 401 generated by the imaging unit 101 or the like, or may be estimated based on a distance map acquired by the main object detection unit 111. The processing unit 102 uses the coordinates (X' of the subject position Q , Y' Q ), and determines that the subject is not occluded by the main object as viewed from the imaging unit 101 when the main object is not present at these coordinates. In contrast, the processing unit 102 determines that the subject is occluded by the main object as viewed from the imaging unit 101 when the main object is present at the coordinates (X' of the subject position Q , Y' Q ).
[0027] When the processing unit 102 determines that the subject is occluded by the main object as viewed from the imaging unit 101, the process repeats step S210. In contrast, when the processing unit 102 determines that the subject is not occluded by the main object as viewed from the imaging unit 101, the process proceeds to step S211. FIG. 4B is a diagram showing an example of the live view image 401 in the case where the main object is present at the coordinates (X' Q , Y' Q ) of the subject position within the imaging range. Note that the main object 404 in the live view image 401 corresponds to the main object 302 in the object detection image 301. The determination in step S210 can prevent the focus adjustment operation described later from being performed on the main object against the user's intention in the next step S211.
[0028] The description returns to FIG. 2. In step S211, the processing unit 102 (focus adjustment means) executes a focus adjustment operation via the imaging unit 101. The focus adjustment operation refers to the operation for the coordinates (X' of the subject position Q , Y' QThis is the operation of focusing on the subject 403 located in the image. Based on a command from the processing unit 102, the imaging unit 101 performs the focus adjustment operation by moving the focus lens, described later, in the optical axis direction. After the focus adjustment operation is performed, the processing unit 102 may change the color or thickness of the setting frame 402 on the display unit 104 so that the user can visually confirm that the focus adjustment operation has been performed. The focus adjustment operation is performed by known methods such as a contrast method or a phase difference detection method based on the image generated by the imaging unit 101. Alternatively, the focus adjustment operation may be performed using the distance to the subject calculated based on the output of the main object detection unit 111.
[0029] In step S212, the processing unit 102 checks the setting value stored in the storage unit 105 to determine whether the gesture shutter mode is ON. The gesture shutter mode is a mode in which, when a predetermined gesture by the main object is detected, the system automatically transitions to a shooting operation without requiring any additional input from the user. The user can switch the gesture shutter mode ON / OFF by changing the setting value using the button 112 in advance. If the processing unit 102 determines that the gesture shutter mode is not ON, the process returns to step S203. On the other hand, if the processing unit 102 determines that the gesture shutter mode is ON, the process proceeds to step S213.
[0030] In step S213, the processing unit 102 determines whether a main object is present within the imaging range. The determination of whether a main object is present within the imaging range may be performed based on the live view image 401 generated by the imaging unit 101 or the like, or may be estimated based on the distance map acquired by the main object detection unit 111. If the processing unit 102 determines that the main object is present within the imaging range, the process repeats step S213. In contrast, if the processing unit 102 determines that no main object is present within the imaging range, the process proceeds to step S215 described later. FIG. 4C is a diagram illustrating an example of the live view image 401 when a main object 404 is present within the imaging range. Through the determination in step S213, it is possible to prevent the main object from being captured in the image file acquired by imaging against the user's intention in step S215 described later.
[0031] Returning to the description of FIG. 2. In step S214, the processing unit 102 determines whether the button 112 has been pressed. If the processing unit 102 determines that the button 112 has not been pressed, the process repeats step S214. In contrast, if the processing unit 102 determines that the button 112 has been pressed, the process proceeds to step S215. In step S215, the processing unit 102 (imaging unit) causes the imaging unit 101 to execute an imaging operation (imaging step). At this time, the imaging unit 101 acquires imaging data and outputs the imaging data to the processing unit 102. The processing unit 102 converts the imaging data into an image file by performing image processing thereon. The processing unit 102 stores the converted image file in the storage of the storage unit 105. When the processing unit 102 stores the image file, the flowchart of FIG. 2 ends.
[0032] Figures 5A to 5C show examples of the configuration of the HMD 100. Figures 5A and 5B show the external appearance of the HMD 100. The HMD 100 has a main body 501 and a belt 502. The HMD 100 is used with the main body 501 in contact with the front of the user's head and secured by the belt 502 so that the display unit 104 can be seen by the user. In the main body 501, the imaging unit 101 is composed of a stereo camera consisting of a right imaging unit 503R and a left imaging unit 503L. The functions of the imaging unit 101 in the operation described with reference to Figures 1 to 4E are performed by at least one of the right imaging unit 503R and the left imaging unit 503L for the other or both.
[0033] The main body 501 is provided with a main object detection unit 111 and a button 112. The main object detection unit 111 is positioned to face downward relative to the orientation of the imaging unit 101 in order to easily detect predetermined gestures made by the user. Alternatively, the main object detection unit 111 may be positioned so that its orientation is parallel to the orientation of the imaging unit 101. The display unit 104 consists of a right eyepiece lens 504R and a right display panel 505R, and a left eyepiece lens 504L and a left display panel 505L. This allows the display unit 104 to display images and the like to the user's right and left eyes, respectively.
[0034] Figure 5C is a cross-sectional view illustrating the configuration of the imaging unit 101. The configuration of the right imaging unit 503R of the imaging unit 101 will be described below with reference to Figure 5C, but the configuration of the left imaging unit 503L of the imaging unit 101 can be described similarly. The right imaging unit 503R includes a photographic lens unit 511 and an image sensor 512. The photographic lens unit 511 includes a fixed lens 513, a focusing lens 514, a lens drive unit 515, and an aperture 516.
[0035] The image sensor 512 is composed of, for example, a CCD or a CMOS sensor. The image sensor 512 converts the optical image formed by the fixed lens 513, the focusing lens 514, and the aperture 516 into an optical image and outputs the image data acquired by the photoelectric conversion to the processing unit 102. The lens drive unit 515 drives the focusing lens 514 in the optical axis direction for adjusting the focus using an actuator (not shown) based on a command from the processing unit 102. The aperture 516 changes the size of the aperture diameter using an aperture drive unit (not shown) based on a command from the processing unit 102.
[0036] Based on the above, in the HMD 100 of the first embodiment, the position of the subject 403, which is the object to be focused on, is determined based on a predetermined gesture made by the user's hand, which is the main object 404. This allows the user to easily set the object to be focused on in the HMD 100.
[0037] Furthermore, the operation of the HMD 100 can also be applied when shooting video. In other words, even during video recording, steps S204 to S211 are performed, allowing the user to easily set the object to focus on. However, even during video recording, the main object may appear in the video file acquired by the recording. In this case, the processing unit 102 (completion means) deletes the main object in each frame of the video file acquired by the recording, and then fills in the deleted area with background images that are behind the main object in the frames before and after the frame in which the deletion occurred.
[0038] Alternatively, after video recording, the main object may be deleted by a program launched on the HMD 100 or a PC. To do this, the processing unit 102 (storage means) stores information indicating the area where the main object is located within each frame of the video file, along with the video file, in the storage (storage unit) of the memory unit 105. This allows the processing unit 102 to generate a video file that excludes main objects that are not the subject the user wants to film.
[0039] (Modification 1) As modification 1 of the first embodiment, various modifications relating to the main object detection unit 111 will be described. In the first embodiment, the main object detection unit 111 is a sensor that can acquire a distance map using LiDAR, and detects objects near the main object detection unit 111 as the main object. The function of such a main object detection unit 111 can also be realized by a stereo camera composed of a right imaging unit 503R and a left imaging unit 503L in the imaging unit 101 creating a distance map. In this case, in the HMD 100, the imaging unit 101 also serves as the main object detection unit 111, so there is no need to have a main object detection unit 111, resulting in a simpler configuration.
[0040] Furthermore, unlike the stereo camera modification described above, it is also possible to implement the functions of the main object detection unit 111 with an imaging unit 101 consisting of at least one camera. In this case, the processing unit 102 recognizes the main object based on the imaging data acquired by the imaging unit 101, which consists of at least one camera, using a predetermined algorithm. Image recognition may also be performed by AI (Artificial Intelligence). It is preferable that the imaging unit 101 is composed of a camera with a deep depth of field.
[0041] These image recognition methods also enable the recognition of a hand of a predetermined size (including one wearing gloves) and a pointing tool such as a pointer as the main object. In this way, the HMD 100, which has an imaging unit 101 consisting of at least one camera, does not need to have a main object detection unit 111, resulting in a simple configuration. Furthermore, when a pointing tool is used, the accuracy of image recognition can be improved by saving an image of the pointing tool to the storage of the memory unit 105 through prior registration.
[0042] Furthermore, Figures 4D and 4E show examples of live view images 401 when the main object 404 is captured within the shooting range. If the aperture diameter of the diaphragm 516 of the imaging lens unit 511 of the imaging unit 101 is large, out-of-focus images tend to become blurred, and as shown in Figure 4D, the blurring of the main object 404 increases. Therefore, when the processing unit 102 (aperture adjustment means) determines in step S204 that the main object 404 has been detected, it adjusts the aperture diameter of the diaphragm 516 of the imaging lens unit 511 to be smaller by the imaging unit 101, thereby increasing the depth of field.
[0043] In this way, the processing unit 102 reduces the blurring of the main object 404, as shown in Figure 4E. This prevents a decrease in the visibility of the main object 404 to the user, and further prevents a decrease in the accuracy of the main object position acquired in step S205. The processing unit 102 considers the coordinates (X') of the subject position of the main object. Q , Y' Q If the subject is obscured by the presence of the main object, the aperture diameter of the aperture 516 may be returned to its original value to reduce the depth of field after the main object no longer obstructs the subject. The adjustment of the aperture diameter of the aperture 516 of the photographic lens unit 511 as described above is also the same for the optical imaging device (an electronic device that allows the user to visually confirm the shooting range by an optical image) described later.
[0044] Furthermore, if the imaging unit 101 also functions as the main object detection unit 111, the processing unit 102 may focus on the object closest to the imaging unit 101 before the main object 404 is detected. Even in this case, the processing unit 102 may be able to reduce the blurring of the main object 404, as shown in Figure 4E. This is also true for the optical imaging device (an electronic device that allows the user to visually confirm the shooting range using an optical image) described later.
[0045] Furthermore, the main object detection unit 111 may recognize a position detection device as the main object based on the output of a position detection device having an IMU (Internal Measurement Unit) sensor capable of measuring acceleration, etc. Figures 6A and 6B show examples of position detection devices. Figure 6A shows a ring-shaped position detection device 600A. The position detection device 600A has a main body 601A, a device button 602A, and an IMU sensor 603A (detection unit). If the user attaches the main body 601A to the index finger of their hand, they can press the device button 602A with the thumb of that hand. The position detection device 600A outputs information such as acceleration acquired by the IMU sensor 603A, that is, information necessary to acquire the position of the position detection device 600A, to the HMD 100 via a device communication unit (not shown).
[0046] The HMD 100 acquires information from the position detection device 600A via its communication unit 106 and outputs the acquired information to the main object detection unit 111 and the processing unit 102. Based on the information from the position detection device 600A, the main object detection unit 111 recognizes the user's hand, which is wearing the position detection device 600A, as the main object. The processing unit 102 also acquires the position of the position detection device 600A as the main object position based on the information from the position detection device 600A. Subsequently, when the processing unit 102 acquires the planned position in step S206, and subsequently determines the subject position in step S209, it treats the position of the position detection device 600A as the main object position. The user shall, if necessary, perform calibration in advance to determine the initial position and orientation of the position detection device 600A relative to the HMD 100.
[0047] Figure 6B shows a stick-type position detection device 600B. The position detection device 600B is an example of an instruction tool and has a grip portion 604 in addition to a main body portion 601B, a device button 602B, and an IMU sensor 603B. The user can grasp the grip portion 604 with their hand and press the device button 602B with their thumb. The main body portion 601B, the device button 602B, and the IMU sensor 603B have the same functions as the main body portion 601A, the device button 602A, and the IMU sensor 603A described above. The main object detection unit 111 recognizes the position detection device 600B, held in the user's hand, as the main object based on information from the position detection device 600B. The processing unit 102 also acquires the tip position of the position detection device 600B as the main object position based on information from the position detection device 600B.
[0048] In this way, when the position detection devices 600A and 600B are used, the recognition of the main object and the acquisition of the main object's position can be performed with higher accuracy than when distance sensors or image recognition are used. Therefore, by using the position detection devices 600A and 600B, the user can input their intentions into the HMD 100 with greater accuracy.
[0049] (Modification 2) As a second modification of the first embodiment, various modifications of the method for detecting a predetermined gesture will be described. In the first embodiment, the main object detection unit 111 detected a release gesture as a predetermined gesture in the gesture detection in step S208. In this respect, the main object detection unit 111 may also detect a circle gesture as a predetermined gesture in the gesture detection in step S208. A circle gesture is the reverse of the transition of a release gesture, and refers to an action that transitions from a state in which the thumb and index finger of the hand are separated, as shown in Figure 3B, to a state in which a substantially circular shape is formed by the contact between the fingertips of the thumb and index finger of the hand, as shown in Figure 3A. A circle gesture is effective, for example, when the user's hand (including one wearing gloves) is detected as the main object.
[0050] Furthermore, the main object detection unit 111 may detect a pinching motion as a predetermined gesture in the gesture detection in step S208. A pinching motion is the same as the circle gesture described above. However, in this case, the processing unit 102 acquires the main object position in step S205 and the planned setting position in step S206 so that the position of the point where the user's fingertips touch can be determined as the subject position in step S209.
[0051] Furthermore, the main object detection unit 111 may detect a tap gesture as a predetermined gesture in the gesture detection in step S208. A tap gesture is an action in which all or part of the main object moves in a first direction at a predetermined speed or faster, and then transitions to a second action in which it remains stationary or moves in a second direction different from the first direction. In this case, in step S205, the processing unit 102 treats, for example, the position of the user's fingertips or the position of the tip of a pointer held in the user's hand as the position of a representative point of the main object and acquires it as the main object position.
[0052] Figure 3C shows an example of an object detection image 301 before a tap gesture by the user's hand, which is the main object 302. In the example in Figure 3C, the main object position 305 is the position of the user's fingertip. Furthermore, if the processing unit 102 determines in step S208 that a predetermined gesture has been detected, in step S209, it obtains the coordinates (X) of the planned position to be set, which were acquired immediately before the start or end of the first operation. Q , Y Q ) to the coordinates of the subject's position (X' Q , Y' Q ) Note that the main object position when a tap gesture is detected may be one obtained based on information from position detection devices 600A and 600B. Also, the first and second directions may include not only the xy plane (i.e., the x-axis and y-axis directions) but also the z-axis direction, which is the depth direction.
[0053] Furthermore, when position detection devices 600A and 600B are used, the main object detection unit 111 may detect a push gesture as a predetermined gesture in the gesture detection in step S208. A push gesture is an action in which the user presses the device buttons 602A and 602B. In this case, when the user presses the device buttons 602A and 602B, the position detection devices 600A and 600B output information that the device buttons 602A and 602B have been pressed to the HMD 100 via a device communication unit (not shown). Subsequently, when the processing unit 102 obtains information that the device buttons 602A and 602B have been pressed via the communication unit 106, it determines in step S208 that a predetermined gesture has been detected.
[0054] In the modified version 2 described above, the user can set the object to be focused on using gestures that are easy to input into the HMD 100. Therefore, modified version 2 can improve the usability of the HMD 100.
[0055] (Modification 3) As modification 3 of the first embodiment, various modifications relating to the imaging unit 101 will be described. In the first embodiment, an example was described in which the imaging unit 101 images the real space. In this regard, the function of the imaging unit 101 may be realized by a VR shooting program executed by the processing unit 102 (virtual shooting means). A VR shooting program is an application that allows a user to photograph a virtual space. In other words, a VR shooting program can provide a user with the experience of photographing a virtual view seen from a virtual user who has entered the virtual space in the program. The virtual view changes in conjunction with the position and orientation of the HMD 100 and is displayed as a live view image 401 on the display unit 104. The user can photograph the virtual space by looking at the live view image 401 in the same way as photographing the real space. Note that in an HMD 100 that executes a VR shooting program that does not photograph the real space, the imaging unit 101 is not an essential component. Therefore, in this case, the HMD 100 is fully immersive.
[0056] Furthermore, the functions of the imaging unit 101 may be realized together with an MR imaging program executed by the processing unit 102. An MR imaging program is an application that allows the user to photograph a mixed reality space in which the real and virtual spaces are merged. In other words, an MR imaging program can provide the user with the experience of photographing a space in which virtual objects and backgrounds are superimposed on the real space. Furthermore, the functions of the imaging unit 101 may be realized together with an AR imaging program executed by the processing unit 102. An AR imaging program is an application that allows the user to photograph an augmented reality space in which virtual objects are superimposed on the real space. In other words, an AR imaging program can provide the user with the experience of photographing an augmented reality space.
[0057] Based on the above, in the third modified version, the HMD100, the user can easily set the object to be focused on, even when shooting in a space other than the real world.
[0058] (Modification 4) As modification 4 of the first embodiment, various modifications relating to the form of the HMD 100 will be described. In the first embodiment, the HMD 100 was a head-mounted display device, in which the main body 501 was in contact with the front of the user's head and fixed by a belt 502. In this regard, an electronic device that is not mounted on the head (hereinafter referred to as a "gripping device") may be used instead of the HMD 100. Examples of gripping devices include smartphones, tablet PCs, notebook PCs, and digital cameras. In this case, the user, for example, grips the gripping device with one hand while pointing it in the direction they want to photograph, and sets the object to be focused on with a predetermined gesture using the other hand.
[0059] Incidentally, some gripping devices are equipped with a touch panel on the display unit that shows the live view image, allowing the user to easily set the object to be focused on by touch operation. With such a function, if the user's hands are wet, they are wearing gloves, or they are looking through the optical viewfinder of a digital camera, they will not be able to perform touch operation and will not be able to set the object to be focused on. However, even in these cases, gripping devices allow the user to easily set the object to be focused on by the operation shown in the flowchart of Figure 2.
[0060] Alternatively, instead of the HMD 100, an electronic device that allows the user to visually confirm the shooting range using an optical image (hereinafter referred to as "optical image device") may be used. Examples of optical image devices include optical see-through type HMDs (including smart glasses) and digital cameras with optical viewfinders. In the optical image device, the live view image 401 and step S207 are not essential components. In this case, the optical image device has a simpler functional configuration, although the information that can be conveyed to the user is limited. However, it is preferable that the processing unit 102 (display means) of the optical image device displays the setting frame 402 (indicator) superimposed on the subject position of the optical image in the transmissive display unit 104 of the optical image device. This allows the optical image device to convey to the user the object that the imaging unit 101 can focus on.
[0061] <Second Embodiment> The second embodiment will be described below with reference to Figure 7. In the second embodiment, the differences from the first embodiment will be described. Also, components identical to those in the first embodiment will be denoted by the same reference numerals and their description will be omitted. Figure 7 is a block diagram showing an example of the functional configuration of the information processing system 700. The information processing system 700 consists of an imaging device 701 (second electronic device) and a detection device 702 (first electronic device).
[0062] The imaging device 701 is, for example, an electronic device such as a digital camera. The imaging device 701 has an imaging unit 101, an imaging processing unit 711, an imaging storage unit 712, and an imaging communication unit 713. The imaging processing unit 711 has the function of controlling the imaging unit 101 and the function of controlling the communication unit 106 (i.e., the imaging communication unit 713), which are among the functions of the processing unit 102 described in the first embodiment. The imaging storage unit 712 has the function related to the imaging unit 101, which are among the functions of the storage unit 105 described in the first embodiment.
[0063] The detection device 702 is, for example, an electronic device such as an HMD. The detection device 702 has an operation unit 103, a display unit 104, a detection processing unit 721, a detection storage unit 722, and a detection communication unit 723. The detection processing unit 721 has the function of controlling the operation unit 103 and the display unit 104, and the function of controlling the communication unit 106 (i.e., the detection communication unit 723), which are among the functions of the processing unit 102 described in the first embodiment. The detection storage unit 722 has the function related to the operation unit 103 and the display unit 104, which are among the functions of the storage unit 105.
[0064] In the second embodiment, the operation of the HMD 100 described with reference to Figures 1 to 6B is performed in the information processing system 700. In this case, the processing unit 102 of the HMD 100 is replaced by the imaging processing unit 711 or the detection processing unit 721, and the storage unit 105 of the HMD 100 is replaced by the imaging storage unit 712 or the detection storage unit 722. The imaging communication unit 713 and the detection communication unit 723 send and receive information to each other that is necessary for the imaging processing unit 711 and the detection processing unit 721 to perform the operation of the information processing system 700 based on commands from the imaging processing unit 711 and the detection processing unit 721. As a result, both the imaging processing unit 711 and the detection processing unit 721 can, for example, determine the subject position in step S209.
[0065] As described above, even in the information processing system 700 of the second embodiment, in which the imaging unit 101 and the main object detection unit 111 are mounted on different electronic devices, the user can easily set the object to be focused on, in the same manner as in the first embodiment. Furthermore, modifications 1 to 4 of the first embodiment are also applicable to the information processing system 700 of the second embodiment.
[0066] <Other> Although preferred embodiments of the present disclosure have been described above, the present disclosure is not limited to the embodiments described above, and various modifications and changes are possible within the scope of its gist. For example, the processing unit 102 may measure the elapsed time after a predetermined gesture is performed by the main object using a timer (not shown), and perform the shooting operation of step S210 after the measured time reaches a predetermined time. Alternatively, the processing unit 102 may perform the shooting operation of step S210 continuously after a predetermined gesture is performed by the main object. In this case, it is preferable that the processing unit 102 displays a plurality of images acquired by continuous shooting on the display unit 104 so that the user can select an image from among the plurality of images.
[0067] Furthermore, in the first embodiment, if the HMD 100 is made smaller and lighter, the processing system and other components of the HMD 100 may be separated, with the other components being located in the HMD 100, while the processing system is located in a small external box computer as an external component. In addition, at least one of the imaging system or detection system of the HMD 100 may be separated, and at least one of the imaging system or detection system may be located in a digital camera or a wearable device. In these configurations, the small external box computer corresponds to an information processing device. Note that the external component is not limited to a small external box computer; for example, it may be a portable computer such as a notebook PC, tablet PC, or smartphone, or a stationary computer such as a desktop PC.
[0068] This disclosure can also be implemented by supplying a program that implements one or more functions of each of the embodiments described above to a system or device via a network or storage medium, and by having one or more processors in the computer of that system or device read and execute the program. Furthermore, this disclosure can also be implemented by a dedicated processor (e.g., an ASIC, FPGA, or other circuit) that implements one or more functions. Moreover, this disclosure can also be implemented by a combination of a general-purpose processor and a dedicated processor. Note that "processor" refers to a processor in a broad sense and includes general-purpose processors and dedicated processors. Furthermore, the operation of a processor may not only be performed by a single processor, but may also be performed by multiple processors located in physically separate locations working together.
[0069] This application claims priority based on Japanese Patent Application No. 2025-080469, filed on 13 May 2025, and all of its contents are incorporated herein by reference.
[0070] 100 HMD (Human-Mounted Display) 101 Imaging Unit 111 Main Object Detection Unit (Detection Unit) 102 Processing Unit (Main Object Detection Means) (Gesture Detection Means) (Shooting Means) (Decision Means)
Claims
1. An information processing device comprising: a main object detection means for detecting a predetermined object as a main object using a detection unit; a gesture detection means for detecting a predetermined gesture by the main object; an imaging means for performing photography using an imaging unit; a determination means for determining the position of a subject to be focused on in the photography, using a main object position indicating the position of the main object and a reference position of the shooting range of the imaging means when a predetermined gesture is performed by the main object; and a focus adjustment means for adjusting the focus to the subject, wherein the focus adjustment means adjusts the focus to the subject after the main object no longer obstructs the subject when the main object is at the subject position and obstructs the subject.
2. The information processing apparatus according to claim 1, characterized in that the shooting means performs the shooting after the focus has been adjusted to the subject.
3. The information processing apparatus according to claim 2, characterized in that the imaging means performs imaging when the main object is not in the imaging range.
4. The information processing apparatus according to claim 1, characterized in that the main object detection means detects an object that is within a predetermined distance from the detection unit as the main object.
5. The information processing apparatus according to claim 1, characterized in that the determination means treats the position of the position detection device obtained from the output of the position detection device held or attached to the user's hand as the main object position.
6. The information processing apparatus according to claim 1, characterized in that the main object detection means detects the user's hand as the main object, and the gesture detection means detects a release gesture or a circle gesture made by the hand detected as the main object as a predetermined gesture made by the main object.
7. The information processing apparatus according to claim 1, characterized in that, when a video file is acquired during the aforementioned shooting, it includes a means for deleting the main object in each frame of the video file and for filling in the deleted area with an image of the background that is behind the main object.
8. The information processing apparatus according to claim 1, characterized in that, when a video file is acquired during the aforementioned shooting, it is provided with a storage means for storing in a storage unit information indicating the region in which the main object is located in each frame of the video file.
9. The information processing apparatus according to claim 1, characterized in that it includes a display means for displaying an indicator superimposed on the subject position of the live view image shown on the display unit when displaying the live view image indicating the shooting range on the display unit.
10. The information processing apparatus according to claim 1, comprising: a display means for displaying a live view image or optical image indicating the shooting range on a display unit; and an aperture adjustment means for adjusting the aperture diameter of the imaging unit when the main object is detected.
11. The information processing apparatus according to claim 10, characterized in that, when the main object is obstructing the subject by being at the subject position, the aperture adjustment means returns to its original aperture diameter after the main object no longer obstructs the subject.
12. The information processing apparatus according to claim 1, wherein, when the imaging unit also functions as the detection unit, the focus adjustment means adjusts the focus to the object closest to the imaging unit before the main object is detected.
13. The information processing apparatus according to claim 1, wherein, when the imaging unit is a stereo camera having two cameras, the determination means determines the subject position in at least the other camera of the two cameras using the main object position in one of the two cameras and the reference position of the shooting range.
14. The information processing apparatus according to claim 1, characterized in that, when a predetermined gesture by the main object is detected, the determination means determines the position calculated based on the position of the main object and the reference position of the shooting range as the subject position immediately before the detection of the predetermined gesture.
15. The information processing apparatus according to claim 2, characterized in that the imaging means automatically performs the imaging without requiring any additional input from the user in response to the detection of a predetermined gesture by the main object.
16. The information processing apparatus according to 15, characterized in that the imaging means does not perform imaging when the main object is within the imaging range, and performs imaging when the main object is not within the imaging range.
17. The information processing apparatus according to claim 6, wherein the gesture detection means detects the release gesture or the circle gesture in a binarized image consisting of a region of a main object and a background region, based on a transition from a state in which the background region enclosed by the region of the main object exists to a state in which it does not exist, or a transition from a state in which it does not exist to a state in which it exists.
18. The determining means sets the coordinates of the main object position as (x P , y P ), sets the coordinates of the reference position of the shooting range as (X O , Y O ), and uses the coefficient m held in the storage unit to determine the coordinates (X Q , Y Q ) represented by (m×x P +X O , m×y P +Y O ) as the subject position, the information processing apparatus according to claim 1. Q , Y Q ) as the subject position, the information processing apparatus according to claim 1.
19. A control method for an information processing device, comprising: a main object detection step of detecting a predetermined object as a main object using a detection unit; a gesture detection step of detecting a predetermined gesture by the main object; a shooting step of taking a photograph using an imaging unit; a determination step of determining the position of a subject to be focused on in the shooting, using a main object position indicating the position of the main object and a reference position of the shooting range of the shooting step when a predetermined gesture is performed by the main object; and a focus adjustment step of focusing on the subject, wherein the focus adjustment step, if the main object is obstructing the subject by being at the subject position, focuses on the subject after the main object no longer obstructs the subject.
20. A program for causing a computer to execute each step of the control method described in claim 19.
