Object detection device and program

Abstract

An object detection device 40 connected to a camera-equipped device having a left camera and a right camera is provided with: an object recognition processing unit 54 that recognizes objects from images captured by a left camera 16L and a right camera 16R; an identification unit 55 that identifies the objects recognized by the object recognition processing unit as objects to be detected; and a selection processing unit 58 that selects, on the basis of the images captured by the left camera 16L and the images captured by the right camera 16R, which camera, from the left camera 16L and the right camera 16R, will capture images for which object recognition will be performed preferentially.

Description

Object detection apparatus and program

[0001] The present invention relates to an object detection apparatus and a program.

[0002] The miniaturization of cameras has made it possible to install cameras in various small devices. As an example of a small device, a technology related to an apparatus in which an earphone is provided with a camera has been disclosed (see, for example, Patent Document 1).

[0003] Japanese Unexamined Patent Publication No. 2022-178851

[0004] An earphone provided with a camera includes a microphone and a speaker in addition to the camera. It is possible to perform object recognition from an image captured by a camera disposed on the earphone and perform a process of audibly notifying a user wearing the earphone of information related to the recognized object. However, because the camera is mounted on the earphone, the direction in which the user is looking may not match the direction in which the camera is capturing an image due to differences in the shape of the user's head or ear, or a state in which the earphone is not properly worn.

[0005] The present disclosure has been made in view of the above, and an object thereof is to appropriately specify an object to be processed from an image captured by a camera mounted on an earphone.

[0006] In order to solve the above-described problem and achieve the object, an object detection apparatus according to the present disclosure is an object detection apparatus connected to a device with a camera including a left camera and a right camera, comprising: an object recognition processing unit that recognizes an object from images captured by the left camera and the right camera; a specifying unit that specifies the object recognized by the object recognition processing unit as a detection target object; and a selection processing unit that selects, based on the image captured by the left camera and the image captured by the right camera, a camera that captures the image on which object recognition is to be preferentially performed among the left camera and the right camera.

[0007] The program relating to this disclosure causes an object detection device connected to a camera-equipped device having a left camera and a right camera to perform the following processes: recognize an object from the images captured by the left camera and the right camera; identify the recognized object as a target for detection; and select, based on the images captured by the left camera and the right camera, which camera will capture the image for which object recognition will be performed preferentially.

[0008] According to this disclosure, the effect is that the object to be processed can be appropriately identified from the image of the camera attached to the earphone.

[0009] Figure 1 is a schematic diagram showing an example configuration of an object detection system according to the first embodiment. Figure 2 is a schematic front view of an earphone according to the first embodiment, showing the earphone 10 as seen from the front of the user when worn by the user. Figure 3 is a block diagram showing an example configuration of an earphone according to the first embodiment. Figure 4 is a block diagram showing an example configuration of an information terminal device according to the first embodiment. Figure 5 is a schematic diagram showing the relationship between the camera's shooting angle of view, effective angle of view, and the position of an object. Figure 6 is a diagram showing an example of the detection target object recognition range and object recognition range in an image captured by the camera. Figure 7 is a flowchart showing an example of the image selection processing flow in an object detection system according to the first embodiment. Figure 8 is a flowchart showing another example of the image selection processing flow in an object detection system according to the first embodiment. Figure 9 is a flowchart showing another example of the processing flow in an object detection system according to the first embodiment. Figure 10 is a flowchart showing another example of the image selection processing flow in an object detection system according to the second embodiment. Figure 11 is a diagram showing another example of the detection target object recognition range and object recognition range in an image captured by the camera. Figure 12 is a flowchart showing another example of the image selection processing flow in an object detection system according to the third embodiment. Figure 13 is a flowchart showing another example of the image selection processing flow in an object detection system according to the fourth embodiment. Figure 14 is a block diagram showing an example configuration of an information terminal device according to the fifth embodiment. Figure 15 is a schematic diagram showing the relationship between the camera's shooting angle of view, effective angle of view, and the position of an object. Figure 16 is a diagram showing another example of the detection target object recognition range and object recognition range in the image captured by the camera. Figure 17 is a flowchart showing the processing flow in the object detection system according to the fifth embodiment. Figure 18 is a diagram showing another example of the detection target object recognition range and object recognition range in the image captured by the camera. Figure 19 is a flowchart showing the processing flow in the object detection system according to the fifth embodiment. Figure 20 is a flowchart showing the processing flow in the object detection system according to the fifth embodiment.

[0010] Embodiments of the object detection system 1 according to this disclosure will be described in detail below with reference to the attached drawings. However, the present invention is not limited to the following embodiments.

[0011] [First Embodiment] <Object Detection System> Figure 1 is a schematic diagram showing an example configuration of an object detection system according to the first embodiment. The object detection system 1 includes an earphone 10, which is an example of a camera-equipped device, and an information terminal device 40, which is an example of an object detection device. The earphone 10 and the information terminal device 40 are connected by wire or wireless communication. In the following description, the earphone 10 and the information terminal device 40 will be described as being connected by wireless communication. When the object detection system 1 recognizes the same object from the left image captured by the left camera 16L and the right image captured by the right camera 16R, it selects the camera 16 that will capture the image for which object recognition is preferentially performed, based on the difference between the images in which the same object is recognized. In this embodiment, the object detection system 1 selects the camera that captured the image in which the range in which the object is recognized is closer to the reference position in the image, from the image captured by the left camera and the image captured by the right camera, as the camera that will capture the image for which object recognition is preferentially performed.

[0012] <Earphone (Camera-equipped device)> The earphone 10 has multiple independent housings 11 that are worn on the head, more specifically on the ears, of a person who is a user. The earphone 10 can be, for example, a wireless earphone or a wired earphone. The form of the earphone 10 can be any form, for example, an open-ear type or a canal type. Figures 1 and 2 show an example of an open-ear type earphone with TWS (True Wireless Stereo). The earphone 10 has a left earphone 10L for the left ear, which is worn on the left ear, and a right earphone 10R for the right ear, which is worn on the right ear. The left earphone 10L outputs left channel data of the audio signal. The right earphone 10R outputs right channel data of the audio signal. In the following description, when it is not necessary to distinguish between the left earphone 10L and the right earphone 10R, they will be described as earphone 10.

[0013] The earphone 10 will be explained using Figures 1 to 3. Figure 1 is a schematic side view of the earphone 10 according to the embodiment, showing how it looks when the user is wearing the earphone 10 and is viewed from the left or right direction of the user. In Figure 1, the left earphone 10L is shown as viewed from the left side of the user when it is worn on the user's left ear, and the right earphone 10R is shown as viewed from the right side of the user when it is worn on the user's right ear. Figure 2 is a schematic front view of the earphone according to the first embodiment, showing how it looks when the user is wearing the earphone 10 and is viewed from the front of the user. In Figure 2, the right earphone 10R is shown, and the left earphone 10L has a similar configuration. Figure 3 is a block diagram showing an example of the configuration of the earphone according to the first embodiment. The earphone 10 acquires data, including music data, from the information terminal device 40. The earphone 10 includes a touch operation unit 15, a camera 16, a microphone (voice acquisition unit) 17, an amplification unit 18, an audio output unit 19, a communication unit 27, a battery 28, a power supply unit 29, and a control unit 30.

[0014] The housing 11 of the earphone 10 is worn on the user's ear. The housing 11 defines the external shape of the earphone 10. The housing 11 comprises a main body 11a, a first ear hook 11b, and a second ear hook 11c. The main body 11a is the part that is positioned inside the user's auricle when the user wears the earphone 10. The first ear hook 11b is positioned between the helix of the auricle and the head when the user wears the earphone 10. The second ear hook 11c is the part that is positioned behind the auricle when the user wears the earphone 10. The main body 11a, the first ear hook 11b, and the second ear hook 11c are integrally formed. The housing 11 houses a touch operation unit 15, a camera 16, a microphone 17, an amplification unit 18, an audio output unit 19, a communication unit 27, a battery 28, a power supply unit 29, and a control unit 30. The arrangement of the touch operation unit 15, camera 16, microphone 17, amplification unit 18, audio output unit 19, communication unit 27, battery 28, power supply unit 29, and control unit 30 in the housing 11 is just one example.

[0015] The earphone 10 has a left housing 11L that is worn on the user's left ear and a right housing 11R that is worn on the right ear. When it is not necessary to distinguish between the left housing 11L and the right housing 11R, they will be described as housing 11.

[0016] The touch operation unit 15 is a touch sensor located on the main body 11a. The touch operation unit 15 can receive various operations, such as playing and stopping music data on the earphones 10. The touch operation unit 15 can receive various operations, such as voice recognition and video recognition on the earphones 10. The touch operation unit 15 outputs an operation signal indicating the received operation to the operation control unit 32.

[0017] Camera 16 captures images of the area around the earphone 10. Camera 16's shooting is controlled by the shooting control unit 33. Camera 16 outputs the captured image to the shooting control unit 33. Camera 16 comprises a left camera 16L located in the left housing 11L of the left earphone 10L, and a right camera 16R located in the right housing 11R of the right earphone 10R. In the description of this embodiment, when it is not necessary to distinguish between the left camera 16L and the right earphone 10R, they will be described as camera 16.

[0018] The left camera 16L, located on the left earphone 10L, is positioned on the main body 11aL of the left housing 11L. The left camera 16L is positioned on the main body 11aL so that it faces forward when the user wears the left earphone 10L on their left ear; in other words, the shooting direction is in front of the user.

[0019] The right camera 16R, located on the right earphone 10R, is positioned on the main body 11aR of the right housing 11R. The right camera 16R is positioned on the main body 11aR so that it faces forward when the user wears the right earphone 10R in their right ear; in other words, the camera is positioned so that the area in front of the user is the direction of shooting.

[0020] The microphone 17 picks up sounds from the vicinity of the earphone 10. The microphone 17 can pick up various sounds, including, for example, voice commands to the earphone 10. The microphone 17 outputs the picked-up sounds to the voice input control unit 34. The microphone 17 is located in at least one of the left housing 11L and the right housing 11R. The microphone 17 is located in the main body 11a.

[0021] The amplification unit 18 amplifies the audio output from the audio output unit 19. The amplification unit 18, for example, performs D / A conversion on audio channel data, including music data acquired from the information terminal device 40, and amplifies it. The audio output unit 19 is located in the main unit 11a.

[0022] The audio output unit 19 outputs sound for the user to hear. The audio output unit 19 outputs sound based on a music signal, for example. The audio output control unit 35 controls the output of information acquired regarding the detected object as sound in the information terminal device 40, which will be described later, for example. The audio output unit 19 outputs an audio signal amplified by the amplification unit 18. The audio output unit 19 is located in at least one of the left housing 11L and the right housing 11R. The audio output unit 19 is located in the main body 11a.

[0023] The communication unit 27 is a communication unit. The communication unit 27 is capable of wireless communication including, for example, Bluetooth®, Wi-Fi®, or NFMI (Near Field Magnetic Induction). In this embodiment, the communication unit 27 is connected to the information terminal device 40 using the Bluetooth method. The communication unit 27 is located, for example, in the second ear hook portion 11c.

[0024] The communication unit 27 is connected to the information terminal device 40 so as to be able to send and receive data. The communication unit 27 receives data, for example, music data, from the information terminal device 40. In this embodiment, the communication unit 27 transmits to the information terminal device 40 video footage captured by the camera 16 acquired by the shooting control unit 33, and audio data based on audio signals picked up by the microphone 17 acquired by the audio input control unit 34. In this embodiment, the communication unit 27 receives audio data of information acquired regarding the object to be detected from the information terminal device 40.

[0025] The battery 28 supplies power for the earphone 10 to operate. The battery 28 is a rechargeable battery that can be repeatedly charged and discharged. The battery 28 is, for example, a nickel-metal hydride rechargeable battery, a lithium-ion rechargeable battery, or a lithium polymer rechargeable battery. The battery 28 is built into the earphone 10. The charging and discharging of the battery 28 is controlled by the power control unit 39. The battery 28 is located, for example, in the second ear hook portion 11c.

[0026] The power supply unit 29 can be connected to the power supply of an earphone case (not shown) and supplies power to the battery 28. The power supply unit 29 is connected to the power control unit 39. The power supply unit 29 may also be capable of supplying power from a DC power source such as a mobile battery or from a contactless charging device.

[0027] <Earphone Control Unit> The control unit 30 is an arithmetic processing unit composed of, for example, a CPU (Central Processing Unit). The control unit 30 loads a program stored in a storage unit (not shown) into memory and executes the instructions contained in the program. The control unit 30 includes an internal memory (not shown) which is used for temporary data storage, etc. The control unit 30 has an operation control unit 32, a shooting control unit 33, an audio input control unit 34, an audio output control unit 35 which is an example of an output control unit, a communication control unit 37, and a power supply control unit 39.

[0028] The operation control unit 32 acquires operation signals from the touch operation unit 15 in response to operations performed on the touch operation unit 15. The operation control unit 32 outputs control signals corresponding to the acquired operation signals to each part of the earphone 10 and the information terminal device 40.

[0029] The shooting control unit 33 controls the shooting by the camera 16. The shooting control unit 33 acquires the video captured by the camera 16. In this embodiment, the shooting control unit 33 uses the communication unit 27 via the communication control unit 37 to transmit the captured video to the information terminal device 40.

[0030] The audio input control unit 34 acquires the audio signal picked up by the microphone 17. The audio input control unit 34 performs A / D conversion on the audio signal picked up by the microphone 17 and acquires it as audio data.

[0031] The audio output control unit 35 controls the output of audio from the audio output unit 19. The audio output control unit 35 performs processing to output audio to the audio output unit 19. The audio output control unit 35 controls the output of audio data, such as music data, from the audio output unit 19. The audio output control unit 35 controls the output of audio data, such as information acquired regarding the detected object in the information terminal device 40 (described later), from the audio output unit 19. The audio output control unit 35 outputs information regarding the detected object as an audio signal to the audio output unit 19. The audio output control unit 35 outputs the audio data acquired from the information terminal device 40 to the amplification unit 18.

[0032] The communication control unit 37 communicates wirelessly with the information terminal device 40 by controlling the communication unit 27. For example, the communication control unit 37 controls the information terminal device 40 to receive data including music data. In this embodiment, the communication control unit 37 transmits to the information terminal device 40 video footage captured by the camera 16 and audio data based on audio signals picked up by the microphone 17 acquired by the audio input control unit 34. In this embodiment, the communication control unit 37 controls the information terminal device 40 to receive audio data indicating information acquired regarding the detected object.

[0033] The power control unit 39 is a charging circuit that charges the battery 28. The power control unit 39 is connected to the battery 28 and the power supply unit 29. The power control unit 39 supplies power from the power supply unit 29 to the battery 28.

[0034] <Information Terminal Device (Object Detection Device)> The information terminal device 40 will be described using Figure 4. Figure 4 is a block diagram showing an example configuration of the information terminal device according to the first embodiment. The information terminal device 40 is a portable electronic device used by the user of the earphone 10, such as a smartphone or tablet terminal. When the information terminal device 40 recognizes the same object from the left image captured by the left camera 16L and the right image captured by the right camera 16R, it selects the camera 16 that will capture the image for object recognition preferentially from the left camera 16L and the right camera 16R based on the difference between the images in which the same object is recognized. In this embodiment, the information terminal device 40 selects the camera that captured the image in which the range in which an object is recognized is closer to the reference position in the image, from the image captured by the left camera and the image captured by the right camera, as the camera that will capture the image for object recognition preferentially. The information terminal device 40 includes a communication unit 41 and a control unit 50.

[0035] The communication unit 41 is a communication unit. The communication unit 41 is connected to the earphone 10 so as to be able to send and receive data. The communication unit 41 transmits data, including music data, to the earphone 10, for example. In this embodiment, the communication unit 41 receives video footage captured by the camera 16 and audio data based on audio signals picked up by the microphone 17 acquired by the audio input control unit 34 from the earphone 10. In this embodiment, the communication unit 41 transmits to the AI ​​(Artificial Intelligence) server, along with the video footage captured by the camera 16 from the earphone 10, including instructions such as prompts corresponding to audio commands for executing object recognition processing. In this embodiment, the communication unit 41 receives text data from the AI ​​server indicating information about the object to be detected. In this embodiment, the communication unit 41 converts the text data indicating information acquired regarding the object to be detected into audio data and transmits it to the earphone 10. The communication unit 41 is capable of short-range wireless communication, including Bluetooth, for example. In this embodiment, the communication unit 41 is connected to the earphone 10 using the Bluetooth method.

[0036] The video acquired by the information terminal device 40 from the earphone 10 can be used for various purposes. For example, the video may be recorded. For example, an object may be recognized from the video, and audio data indicating information about the detected object, which is the recognition result, may be fed back to the earphone 10. The object recognition process may use an external AI server provided by the AI ​​generation service provider. In this case, for example, the video acquired from the earphone 10 is sent to the external AI server along with instructions such as prompts corresponding to voice commands that are trigger words for executing the object recognition process. The AI ​​server performs the object recognition process and sends text data, audio data, etc., indicating information about the detected object, which is the object recognition result, to the information terminal device 40. The information terminal device 40 converts the text data received from the AI ​​server into audio data and sends it to the earphone 10.

[0037] The user's spoken voice, which the information terminal device 40 acquires from the earphone 10, can be used for various purposes. For example, the voice may be recorded. For example, if the voice recognized by the speech recognition process is a trigger word, which is a voice command to execute object recognition processing, it may be used as a trigger to start the process of recognizing an object from the video acquired from the earphone 10 and outputting the information as voice.

[0038] A trigger word is an utterance that inquires about object information, such as "What is that?", "What is that?", or "Tell me its name." A trigger word is an instruction that, for example, recognizes an object from an image and searches for and outputs the name or other information about the recognized object. In other words, a trigger word is an utterance that means an instruction to recognize an object from an image.

[0039] <Control Unit of Information Terminal Device> The control unit 50 is a processing unit composed of, for example, a CPU. The control unit 50 loads a program stored in a storage unit (not shown) into memory and executes the instructions contained in the program. The control unit 50 includes an internal memory (not shown) which is used for temporary data storage, etc. In this embodiment, the control unit 50 implements the function of a control unit of the object detection system 1. When the control unit 50 recognizes the same object from the left image captured by the left camera 16L and the right image captured by the right camera 16R, it selects the camera 16 that will capture the image for object recognition preferentially from the left camera 16L and the right camera 16R based on the difference between the images in which the same object is recognized. In this embodiment, the control unit 50 selects the camera that captured the image in which the range in which the object is recognized is closer to the reference position in the image, from the image captured by the left camera and the image captured by the right camera, as the camera that will capture the image for object recognition preferentially. The control unit 50 includes a communication control unit 51, a speech recognition processing unit 52, a speech synthesis processing unit 53, an object recognition processing unit 54, a specific unit 55, an information acquisition control unit 56, and an image selection processing unit (selection processing unit) 58.

[0040] The communication control unit 51 communicates wirelessly with the earphone 10 by controlling the communication unit 41. For example, the communication control unit 51 controls the earphone 10 to transmit data including music data. In this embodiment, the communication control unit 51 controls the earphone 10 to receive video footage captured by the camera 16 and audio data based on audio signals picked up by the microphone 17 acquired by the audio input control unit 34. In this embodiment, the communication control unit 51 transmits instructions to the AI ​​server corresponding to an audio command to execute object recognition processing along with the video footage captured by the camera 16 from the earphone 10. In this embodiment, the communication control unit 51 receives text data indicating the object recognition result from the AI ​​server. In this embodiment, the communication control unit 51 controls the earphone 10 to transmit audio data indicating information acquired regarding the detected object, which has been converted by the speech synthesis processing unit 53.

[0041] The speech recognition processing unit 52 performs speech recognition processing on speech data based on an audio signal collected by a microphone 17 arranged in the earphone 10. More specifically, the speech recognition processing unit 52 performs speech recognition processing for recognizing utterance content by natural language processing or the like from speech data based on an audio signal collected by the microphone 17 arranged in the earphone 10 via the communication unit 41.

[0042] The speech recognition processing unit 52 recognizes, for example, an utterance having a meaning of inquiring about object information such as "What is this?" as a trigger word (voice command).

[0043] The speech synthesis processing unit 53 converts text data indicating information about a detection target object into speech data. For example, the speech synthesis processing unit 53 converts text data acquired by the communication control unit 51 from an AI server or the like into speech data.

[0044] The object recognition processing unit 54 recognizes an object from the left video and the right video captured by the left camera 16L and the right camera 16R. In the present embodiment, the object recognition processing unit 54 transmits, via the communication control unit 51, a command for causing an AI server to execute object recognition processing together with the left video and the right video captured by the left camera 16L and the right camera 16R to the AI server.

[0045] The specifying unit 55 specifies the object recognized by the object recognition processing unit 54 as a detection target object.

[0046] When an operation unit receives an operation requesting processing for specifying the object recognized by the object recognition processing unit 54, the specifying unit 55 specifies the object recognized by the object recognition processing unit 54 as a detection target object. In the present embodiment, when the speech recognition processing unit 52 recognizes a predetermined voice command, the specifying unit 55 specifies the object recognized by the object recognition processing unit 54 as a detection target object. For example, the object recognition processing unit 55 recognizes an object from an image captured at the timing when a user utters a trigger word such as "What is that?", and specifies the object as a detection target object.

[0047] The operation for starting an inquiry about information on a detection target object identified by the identifying unit 55 has been described as an operation performed by uttering a trigger word using speech recognition. This operation may also be performed by a touch operation on the touch operation unit 15. Therefore, the touch operation unit 15 and the operation control unit 32 may together be referred to as an operation unit, and the microphone 17 and the voice input control unit 34 may also together be referred to as an operation unit.

[0048] It has been described that the speech synthesis processing unit 53 converts information about the detection target object identified by the identifying unit 55 into speech data, and the speech output control unit 35 outputs the speech data to the speech output unit 19. In addition, when a smart glass or the like having a function equivalent to that of the earphone 10 is used, information about the detection target object identified by the identifying unit 55 may be output as text or an image by being displayed on a display unit (not shown). Therefore, the speech output unit 19, the amplification unit 18, and the speech output control unit 35 may together be referred to as an output unit, and a display unit (not shown) and a display control unit may also together be referred to as an output unit.

[0049] The information acquisition control unit 56 acquires information about a detection target object. The information acquisition control unit 56 acquires a name as information about the object recognized by the object recognition processing unit 54, and uses the acquired name in the video selection processing unit 58 described later. The information acquisition control unit 56 acquires information about an object identified as a detection target object, and notifies a person of the acquired information.

[0050] The information acquisition control unit 56 acquires, from the video captured by the camera 16 selected by the video selection processing unit 58, information about the object identified as a detection target object by the identifying unit 55, and notifies the user of the earphone 10 of the information about the object.

[0051] The information acquisition control unit 56 acquires, from the AI server, a name related to the object recognized by the object recognition processing unit 54. The AI server transmits the name related to the object to the video selection processing unit 58 of the information terminal device 40.

[0052] The information acquisition control unit 56 acquires information about the identified detection target from the AI ​​server. The AI ​​server transmits the information about the detection target identified by the identification unit 55 to the information terminal device 40. The speech synthesis processing unit 53 converts the text data indicating the information about the detection target acquired from the AI ​​server into speech data. The communication control unit 51 transmits the converted speech data indicating the information about the detection target to the earphone 10. The earphone 10 outputs the speech data received by the communication control unit 37 from the information terminal device 40 as speech from the speech output unit 19.

[0053] The video selection processing unit 58 selects, based on the left video and the right video, the camera 16 that will capture the video for object recognition preferentially from among the left camera 16L and the right camera 16R. In other words, based on the left video and the right video, it selects either the left camera 16L or the right camera 16R as the camera 16 that will capture the video for object recognition preferentially.

[0054] Prioritizing here means increasing the priority of the camera 16 that captures images used for object recognition. Prioritizing means having the camera 16 with the highest priority capture images and using the recognition results from those images. If camera 16 is not capturing images, the camera 16 with the highest priority captures images and uses those images for object recognition.

[0055] The video selection processing unit 58 selects the video that will be preferentially used for object recognition based on the left video and the right video. In other words, based on the left video and the right video, it selects either the left video or the right video as the video that will be preferentially used for object recognition.

[0056] Prioritization here refers to increasing the priority of the video used for object recognition. Prioritization means using the recognition results of the video captured by the camera 16 with the highest priority. If camera 16 is already capturing video, such as by being in a continuous recording state, the video captured by the camera 16 with the highest priority will be used for object recognition.

[0057] When the video selection processing unit 58 recognizes the same object from both the left and right video, it selects the camera 16 that will capture the video for which object recognition will be preferentially performed, based on the differences between the videos in which the same object was recognized.

[0058] When the same object is recognized from the left and right images, this includes cases where the name of the object identified from the speech content recognized by the speech recognition processing unit 52 matches the name of the object recognized by the object recognition processing unit 54.

[0059] The difference between images recognizing the same object is the result of comparing the left and right images that recognize the same object. For example, this could be a comparison of the position or area occupied by the same object in the images, a comparison of the recognition score (confidence score) of the same object, or a comparison of the brightness of the recognition area in the images of the same object. In this embodiment, the difference between images recognizing the same object is the result of comparing the position of the same object in the images.

[0060] The video selection processing unit 58 prioritizes the camera 16 that captures the image whose object recognition range (the range in which an object is recognized) is closer to the reference position in the image, out of the right and left images, as the camera 16 that captures the image for which object recognition is performed.

[0061] The reference position is the position or range set within the shooting range of the left camera 16L and the right camera 16R. The reference position is, for example, the position or range set assuming that the camera 16 is shooting centered on the direction the user's face is facing when the user wears the earphones 10. The reference position may be changed for each user or each time the earphones 10 are worn. As a product, the reference position is set so that the orientation of the camera 16 coincides on average with the direction in front of the user's face. In the following description, the object detection range will be described as a reference position with a range. Therefore, in the following description, the object detection range can be rephrased as the reference position.

[0062] For example, the image with a smaller discrepancy between the object recognition range and the reference position may be given higher priority as the image that is closer to the reference position.

[0063] For example, if the object recognition range of the object recognized in the left image and the object recognition range of the object recognized in the right image are both within the reference position, in other words, within the detection target recognition range, then prioritization of the left and right images may not be performed. In this case, either the objects recognized in the left and right images will be treated as detection targets, or the object recognized from either the left or right image (whichever is pre-configured) will be treated as the detection target.

[0064] For example, if the object recognition range of an object recognized in the left image and the object recognition range of an object recognized in the right image are both within the reference position, and the other is partially or entirely outside the reference position, the image in which the object recognition range of the recognized object is within the reference position will be given higher priority as the image closer to the reference position.

[0065] For example, if the object recognition range of an object recognized in the left image and the object recognition range of an object recognized in the right image are both partially and entirely outside the reference position, the image in which the object recognition range of the recognized object is partially outside the reference position may be given higher priority as the image closer to the reference position. Alternatively, the image in which the object recognition range of the recognized object is closer to the reference position may be given higher priority as the image closer to the reference position.

[0066] For example, if the object recognition range of an object recognized in the left image and the object recognition range of an object recognized in the right image are both entirely outside the reference position, the image whose object recognition range is closer to the reference position may be given higher priority as the image closer to the reference position.

[0067] The image closer to the reference position is, for example, the image where the distance (number of pixels) between the center of the object recognition area and the reference position or the center of the detected object recognition area is shorter. Alternatively, the image closer to the reference position is, for example, the image where the pixels constituting the object recognition area are more numerous within the detected object recognition area.

[0068] The video selection processing unit 58 selects the video that is closer to the reference position. When a user requests information about an object, the system identifies the object from the video of the camera 16 that is appropriate in terms of the position from which the user views the object and the position that the camera 16 is facing, particularly the reference position which is the detection target recognition range. This reduces misrecognition. In particular, when multiple objects are present, the object for which the user requests information can be appropriately identified.

[0069] The video selection processing unit 58 may prioritize the video in which the object recognition range is closer to the center of the video, between the left and right videos, as the video for which object recognition is performed.

[0070] By prioritizing the image closer to the center, the image captured by camera 16 that is appropriately oriented towards the object for which the user wants to acquire information is prioritized. This assumes that the object near the center of the image from the prioritized camera 16 will be the object to be detected. In this case, if the name of the object identified from the speech content recognized by the speech recognition processing unit 52 matches the name of the object recognized by the object recognition processing unit 54, the image selection processing unit 58 will prioritize the image whose object recognition range is closer to the center of the image, between the right and left images, as the image for which object recognition will be performed.

[0071] The video selection processing unit 58 may determine that the same object has been recognized from the left and right video if the name of the object identified from the speech content recognized by the speech recognition processing unit 52 matches the name of the object recognized by the object recognition processing unit 54.

[0072] For example, if the name of an object identified from the speech content recognized by the speech recognition processing unit 52 matches the name of an object recognized by the object recognition processing unit 54, the video selection processing unit 58 may prioritize the camera 16 that captured the video in which the object recognition range is closer to the reference position in the video, between the right video and the left video.

[0073] An example of identifying a target object will be explained using Figures 5 and 6. Figure 5 is a schematic diagram showing the relationship between the camera's shooting angle of view, effective angle of view, and the position of the object. As shown in Figure 5, the right camera 16R located in the right earphone 10R and the left camera 16L located in the left earphone 10L have shooting angles of view A1 and A2, respectively, of approximately 120° to 150° in the horizontal direction. They have similar shooting angles of view in the vertical direction. The horizontal shooting angles of view A1 and A2 have areas that cannot be captured by the user's head. The areas of shooting angles A1 and A2 that are not obstructed by the user's head are defined as the effective angle of view B1 and B2.

[0074] As shown in Figure 5, suppose a user wearing the left earphone 10L and the right earphone 10R faces object 301 and starts calibration. Calibration can be started by the user uttering a trigger word such as "start calibration" or by operating the touch operation part 15 of the earphone 10.

[0075] Figure 6 shows an example of the object recognition range and object recognition range in the video captured by the camera. Figure 6 shows the recognition results of video captured simultaneously by the left camera 16L and the right camera 16R. In Figure 6, the shooting range of the left camera 16L is shifted to the upper right due to misalignment of the left earphone 10L or the shape of the user's ear. The object recognition range L1 and the object recognition range R1 are pre-set.

[0076] In the example shown in Figure 6, the object recognition processing unit 54 determines that the recognized object is an "apple," or that the recognized object includes an "apple." The speech recognition processing unit 52 also shows the object recognition range L2 and object recognition range R2 in the video at the time it detects that the user has spoken "apple." The video selection processing unit 58 calculates the amount of deviation between the object recognition range L2 and the detection target object recognition range L1, and the amount of deviation between the object recognition range R2 and the detection target object recognition range R1. The video selection processing unit 58 then prioritizes the video with the smallest deviation, in other words, the video closer to the reference position, as the video for which object recognition will be performed. In the example shown in Figure 6, the object recognition range R2 in the right video is included in the detection target object recognition range R1, while the object recognition range L2 in the left video is not partially included in the detection target object recognition range L1. Therefore, the right video is selected as the video that is closer to the reference position. The object recognition area is the frame surrounding the recognized object within the image, and is generally a rectangular frame, as shown in the object recognition area L2 and object recognition area R2 in Figure 6.

[0077] <Object Detection Method (Image Selection Process)> Next, the image selection process in the object detection system 1 will be explained using Figure 7. Figure 7 is a flowchart showing an example of the flow of the image selection process in the object detection system according to the first embodiment. When the earphones 10 are first used, or each time they are put on, the process shown in the flowchart in Figure 7 is executed as a calibration process. While the process shown in the flowchart in Figure 7 is being executed, the image captured by the camera 16 is transmitted to the information terminal device 40. When calibration is started, the audio output unit 19 of the earphones 10 outputs an audio instruction, such as "Please look straight at something easy to see."

[0078] The control unit 50, using the object recognition processing unit 54, determines whether the same object is recognized in the left and right images (step S101). The control unit 50, using the object recognition processing unit 54, determines whether the same object is recognized in the left image captured by the left camera 16L and the right image captured by the right camera 16R. If the control unit 50 determines, using the object recognition processing unit 54, that the same object is recognized in the left and right images (step S101; Yes), it proceeds to step S102. If the control unit 50 does not determine, using the object recognition processing unit 54, that the same object is not recognized in the left and right images (step S101; No), it terminates the processing of this flowchart. If, for example, no object is recognized for 30 seconds after the voice instruction to start calibration (S101; No), the processing of this flowchart is terminated.

[0079] If the control unit 50 determines that the same object is recognized from both the left and right images (step S101; Yes), the control unit 50 acquires recognized object information using the information acquisition control unit 56 (step S102). The control unit 50, using the information acquisition control unit 56, sends an instruction to the AI ​​server via the communication control unit 51 to acquire the name of the recognized object along with the left image captured by the left camera 16L and the right image captured by the right camera 16R. The control unit 50, using the information acquisition control unit 56, acquires the name of the recognized object from the AI ​​server via the communication control unit 51. The control unit 50 proceeds to step S103.

[0080] The control unit 50, using the video selection processing unit 58, determines whether there is a difference in distance between the object recognition range and the area outside the reference position (step S103). The control unit 50, using the video selection processing unit 58, determines whether there is a difference in distance between the object recognition range of the object recognized in the left video and the reference position, and between the object recognition range of the object recognized in the right video and the reference position. Preferably, the difference in distance is a predetermined distance or greater such that one of the object recognition ranges is not included in the reference position. The video selection processing unit 58 may also determine whether at least a portion of the object recognition range of the object recognized in the left video or the object recognition range of the object recognized in the right video extends beyond the reference position. If the control unit 50 determines, using the video selection processing unit 58, that there is a difference in distance between the object recognition range and the area outside the reference position (step S103; Yes), it proceeds to step S104. If the control unit 50 does not determine, using the video selection processing unit 58, that there is a difference in distance between the object recognition range and the area outside the reference position (step S103; No), it terminates the processing of this flowchart.

[0081] If it is determined that there is a difference in distance between the object recognition range and the reference position (step S103; Yes), the control unit 50 performs priority setting using the video selection processing unit 58 (step S104). The control unit 50, using the video selection processing unit 58, prioritizes the video in which the object recognition range is closer to the reference position in the video, between the right video and the left video, as the video for which object recognition is performed. The control unit 50 performs priority setting.

[0082] Steps S101 and S102 may be executed as a single step without being separated.

[0083] <Object Detection Method (Modified Image Selection Process)> Next, a modified image selection process in the object detection system 1 will be described using Figure 8. Figure 8 is a flowchart showing another example of the flow of the image selection process in the object detection system according to the first embodiment. During the execution of the flowchart shown in Figure 8, the audio acquired by the microphone 17 and the video captured by the camera 16 are transmitted to the information terminal device 40. When calibration is started, the audio output unit 19 of the earphone 10 outputs an audio instruction, such as "Look straight at an easily identifiable object and say its name." The processes in steps S201, S202, S206, and S207 are the same as the processes in steps S101, S102, S103, and S104 of the flowchart shown in Figure 7.

[0084] The control unit 50 determines whether or not speech has been detected by the speech recognition processing unit 52 (step S203). If the control unit 50 determines that speech has been detected by the speech recognition processing unit 52 (step S203; Yes), it proceeds to step S204. If the control unit 50 does not determine that speech has been detected by the speech recognition processing unit 52 (step S203; No), it terminates the processing of this flowchart. After the voice instruction to start calibration, the speech recognition processing unit 52 determines whether or not speech of an object name has been detected for, for example, 30 seconds. If speech of an object name is detected (S203; Yes), it will be S203; No.

[0085] The control unit 50 recognizes the content of the speech uttered by the speech recognition processing unit 52 (step S204). The control unit 50 recognizes the content of what the user has said using the speech recognition processing unit 52. Since the user speaks the name of the object from which they want to obtain information, the speech recognition processing unit 52 recognizes the name of the object from which the user wants to obtain information. The control unit 50 then proceeds to step S205.

[0086] The control unit 50 determines, using the video selection processing unit 58, whether the object names match (step S205). The control unit 50 determines, using the video selection processing unit 58, whether the name of the object identified from the speech content recognized by the speech recognition processing unit 52 matches the name of the object recognized by the object recognition processing unit 54. If the control unit 50 determines, using the video selection processing unit 58, that the object names match (step S205; Yes), it proceeds to step S206. If the control unit 50 does not determine, using the video selection processing unit 58, that the object names do not match (step S205; No), it terminates the processing of this flowchart.

[0087] If it is determined that the same object is not recognized from the left and right images (step S201; No), the control unit 50 acquires recognized object information via the information acquisition control unit 56 (step S202). The control unit 50, via the information acquisition control unit 56, sends an instruction to the AI ​​server via the communication control unit 51 to acquire the name of the object recognized from the left image captured by the left camera 16L and the right image captured by the right camera 16R. The control unit 50, via the information acquisition control unit 56, acquires the name of the recognized object from the AI ​​server via the communication control unit 51. The control unit 50 proceeds to step S203.

[0088] Steps S209 to S211 perform the same processing as in steps S203 to S205.

[0089] If it is determined that the object names match (step S205; Yes), the control unit 50 prioritizes the matching camera 16 using the video selection processing unit 58 (step S212). The control unit 50, using the video selection processing unit 58, prioritizes the video in which the object name matches between the right and left videos for object recognition.

[0090] The order in which the processes in steps S201 and S202, and steps S203 and S204 are performed is not limited and they may be executed in parallel.

[0091] Steps S201 and S202 may be executed as a single step without being separated.

[0092] In the flowchart of Figure 8, the priority is determined in step S201 by "recognizing the same object from both the left and right images." However, if an object is recognized from one of the left or right images, but the same object is not recognized from the other, and the name of the object recognized from one image matches the spoken name of the object, then the image from which the object was recognized is given priority.

[0093] Before executing the processes shown in the flowcharts of Figures 7 and 8, a process is performed to detect whether the left earphone 10L and the right earphone 10R are in the ears. If the left earphone 10L and the right earphone 10R are in the ears, the processes shown in the flowcharts of Figures 7 and 8 may be executed.

[0094] <Object Detection Method (Object Recognition Processing)> Next, the object recognition processing in the object detection system 1 will be explained using Figure 9. Figure 9 is a flowchart showing another example of the processing flow in the object detection system according to the first embodiment. Before the processing shown in the flowchart in Figure 9 is executed, the processing shown in the flowchart in Figure 7 or Figure 8 is executed.

[0095] The control unit 50 determines whether or not a voice command has been acquired by the voice recognition processing unit 52 (step S301). The control unit 50 determines whether or not a trigger word (voice command), which is an utterance that has meaning in questioning object information such as "What is this?", has been recognized from the sound picked up by the microphone 17 placed on the earphone 10 by the voice recognition processing unit 52. If the control unit 50 determines that a voice command has been acquired by the voice recognition processing unit 52 (step S301; Yes), it proceeds to step S302. If the control unit 50 does not determine that a voice command has been acquired by the voice recognition processing unit 52 (step S301; No), it proceeds to step S306.

[0096] If the control unit 50 determines that it has received a voice command (step S301; Yes), the control unit 50 uses the object recognition processing unit 54 to recognize an object from the video captured by the preferred camera 16 (step S302). The control unit 50 uses the object recognition processing unit 54 to send a command to the AI ​​server via the communication control unit 51 to perform object recognition processing along with the video captured by the preferred camera 16, which was selected according to the flowchart in Figure 7 or Figure 8. The control unit 50 uses the object recognition processing unit 54 to obtain the AI ​​server recognition result via the communication control unit 51. The control unit 50 then proceeds to step S303.

[0097] The control unit 50 determines whether or not there is an object recognized by the object recognition processing unit 54 (step S303). If the control unit 50 determines that there is an object recognized by the object recognition processing unit 54 (step S303; Yes), it proceeds to step S304. If the control unit 50 does not determine that there is an object recognized by the object recognition processing unit 54 (step S303; No), it proceeds to step S306.

[0098] If it is determined that there is a recognized object (step S303; Yes), the control unit 50, using the information acquisition control unit 56, acquires information about the detected object as the detected object (step S304). The control unit 50 identifies the recognized object as the detected object using the identification unit 55. The control unit 50, using the information acquisition control unit 56, acquires information about the detected object from the AI ​​server as the recognized object. The control unit 50 proceeds to step S305.

[0099] The control unit 50 outputs audio data for the detected object to the earphone 10 (step S305). The control unit 50 converts the information about the detected object obtained from the AI ​​server into audio data using the speech synthesis processing unit 53. The control unit 50 transmits the converted audio data to the earphone 10 using the communication control unit 51. The control unit 50 proceeds to step S306.

[0100] The control unit 50 determines whether or not to terminate the process (step S306). More specifically, the control unit 50 determines to terminate the process, for example, when an operation to turn off the object recognition function is detected, or when the power of the earphone 10 is turned off. If the control unit 50 determines to terminate the process (step S306; Yes), it terminates the process in this flowchart. If the control unit 50 does not determine to terminate the process (step S306; No), it executes the process in step S301 again.

[0101] In this way, after calibration is complete, the user, while wearing the left earphone 10L and right earphone 10R that constitute the object detection system 1, speaks a trigger word, which causes the system to recognize an object detected from the image of the higher-priority camera 16 among the left camera 16L and right camera 16R, and outputs information about the object as audio data to the earphone 10.

[0102] Information regarding objects recognized within a reference position in the video feed of the high-priority camera 16 may be output as audio data to the earphone 10.

[0103] <Effects> As described above, in this embodiment, if the name of an object identified from the spoken content matches the name of the object recognized by the object recognition processing unit 54, the image on the left or right of the two images, the one in which the range in which the object is recognized is closer to the reference position in the image, can be preferentially used as the image for object recognition. In this embodiment, by selecting the image closer to the reference position, when a user requests information about an object, the object to be detected can be identified from the image of the camera 16 that has an appropriate position for the object being viewed, particularly the reference position, which is the detection target object recognition range. According to this embodiment, misrecognition can be reduced. In this embodiment, when there are multiple objects, the object for which the user wants information can be appropriately identified. In this way, according to this embodiment, the object to be processed can be appropriately identified from the image of the camera 16 attached to the earphone 10.

[0104] In this embodiment, of the left and right images, the image in which the area where an object is recognized is closer to the center of the image can be prioritized as the image for object recognition. By prioritizing the image closer to the center, this embodiment can prioritize the image captured by the camera 16 that is appropriately pointed towards the object for which the user wants to acquire information.

[0105] In this embodiment, information about an object identified as a target for detection can be acquired, and the acquired information can be notified to the object.

[0106] [Second Embodiment] The object detection system 1 according to this embodiment will be described with reference to Figure 10. Figure 10 is a flowchart showing another example of the video selection processing flow in the object detection system according to the second embodiment. The basic configuration of the object detection system 1 is the same as that of the object detection system 1 of the first embodiment. In the following description, components similar to those of the object detection system 1 are denoted by the same reference numerals or corresponding reference numerals, and their detailed descriptions are omitted. In this embodiment, the processing of the video selection processing unit 58 of the control unit 50 differs from that of the first embodiment.

[0107] In this embodiment, the control unit 50 selects the camera 16 that captured the image from the left camera 16l and the right camera 16R, whichever camera captures the image from the right camera 16R, as the camera 16 that will be preferentially used for capturing the image for object recognition, because the range in which objects are recognized is included more in the reference range (detection target object recognition range) of the image. In the following description, the detection target object recognition range will be described as the reference range. Therefore, in the following description, the detection target object recognition range can be rephrased as the reference range.

[0108] In this embodiment, the video selection processing unit 58 uses the comparison result of the range occupied by the reference range in the video of the same object as the difference between videos in which the same object is recognized.

[0109] The video selection processing unit 58 selects the camera 16 that captured the video from the left camera 16l and the right camera 16R, whichever camera captures the video from

[0110] In the example shown in Figure 6, there is a difference between the proportion of the object recognition range L2 in the left image, where "apple" is recognized, within the reference range (detection target recognition range L1), and the proportion of the object recognition range R2 in the right image, where "apple" is recognized, within the reference range (detection target recognition range R1). In the example shown in Figure 6, the object recognition range R2 in the right image is included in the detection target recognition range R1, while the object recognition range L2 in the left image is not partially included in the detection target recognition range L1. The proportion in the right image is greater than the proportion in the left image. Therefore, the right image is selected as the image that includes more of the reference range.

[0111] <Object Detection Method (Image Selection Process)> Next, another example of the image selection process in the object detection system 1 will be explained using Figure 10. Steps S401, S402, and S404 are performed in the same way as steps S101, S102, and S104 in the flowchart shown in Figure 7.

[0112] The control unit 50, using the image selection processing unit 58, determines whether there is a difference in the proportion of the object recognition range within the reference range (step S403). The control unit 50, using the image selection processing unit 58, determines whether there is a difference between the proportion of the object recognition range of an object recognized in the left image that is included in the reference range and the proportion of the object recognition range of an object recognized in the right image that is included in the reference range. The image selection processing unit 58 may also determine which image contains more of the object recognition range of an object recognized in the left image and which image contains more of the object recognition range of an object recognized in the right image. If the control unit 50 determines, using the image selection processing unit 58, that there is a difference in the proportion of the object recognition range within the reference range (step S403; Yes), it proceeds to step S404. If the control unit 50 does not determine, using the image selection processing unit 58, that there is a difference in the proportion of the object recognition range within the reference range (step S403; No), it terminates the processing of this flowchart.

[0113] In step S404, the left image and the right image are selected, with priority given to selecting the image whose reference range includes a larger portion of the object recognition area.

[0114] This embodiment is also applicable to Figure 8. In this case, the process in step S206 can be replaced with the process in step S403 described above.

[0115] <Effects> As described above, in this embodiment, the camera 16 that captures the image in which the range in which an object is recognized is included more in the reference range in the image, between the left image and the right image, can be selected as the camera 16 that captures the image for which object recognition is performed preferentially. According to this embodiment, when a user requests information about an object, even if a situation occurs where the object cannot be properly recognized due to the orientation of the camera or other reasons, the object for which the user wants information can be properly identified.

[0116] [Third Embodiment] The object detection system 1 according to this embodiment will be described with reference to Figures 11 and 12. Figure 11 is a diagram showing another example of the detection target object recognition range and object recognition range in the image captured by the camera. Figure 12 is a flowchart showing another example of the image selection processing flow in the object detection system according to the third embodiment. In this embodiment, the processing of the image selection processing unit 58 of the control unit 50 differs from that of the first embodiment.

[0117] In this embodiment, the control unit 50 selects the camera 16 that captures the image with the higher brightness in the reference range of the left and right images as the camera 16 that will preferentially capture the image for object recognition.

[0118] In this embodiment, the video selection processing unit 58 uses the comparison result of the brightness of the reference range (detection target object recognition range) in the video of the same object as the difference between videos in which the same object is recognized.

[0119] The video selection processing unit 58 selects the camera 16 that captured the video with the higher brightness in the reference range of the left video and the right video as the camera 16 that will be preferentially used to capture the video for object recognition. The video selection processing unit 58 selects the camera 16 that captured the video with the higher brightness in the reference range of the left video and the right video as the video that will be preferentially used to capture the video for object recognition.

[0120] If hair is placed in front of the lens (not shown) of camera 16, the brightness of the image captured by camera 16 decreases. Therefore, the image with higher brightness is selected. Even if there is no hair in front of either the left camera 16L or the right camera 16R, a difference in brightness may occur between the left and right images. Accordingly, if the difference between the brightness of the reference range in the left image and the brightness of the reference range in the right image is 30-50% or more, the image with higher brightness may be preferentially selected as the image for object recognition. The brightness of the reference range refers to the average brightness value of the pixels in the reference range, or the sum of the brightness values ​​of the pixels in the reference range.

[0121] Figure 11 shows another example of the detection target recognition range and object recognition range in video footage captured by the camera. In Figure 11, an "apple" is recognized in both the left and right images. Also, in Figure 11, the user's hair is visible in the center of the detection target recognition range L1 in the left image, and in the right image, the user's hair is visible at the left edge of the detection target recognition range R1. In this case, the area of ​​the user's hair visible in the detection target recognition range L1 of the left image is larger than that of the detection target recognition range R1 of the right image. Therefore, the brightness of the reference range in the left image is lower than that of the right image. In such cases, the video selection processing unit 58 prioritizes the right image for object recognition.

[0122] <Object Detection Method (Image Selection Process)> Next, another example of the image selection process in the object detection system 1 will be explained using Figure 12. Steps S501, S502, and S504 are performed in the same way as steps S101, S102, and S104 in the flowchart shown in Figure 7.

[0123] The control unit 50, using the video selection processing unit 58, determines whether there is a difference in brightness within the reference range (step S503). The control unit 50, using the video selection processing unit 58, determines whether there is a difference between the brightness of the reference range in the left video and the brightness of the reference range in the right video. The video selection processing unit 58 may also determine which video has the higher brightness within the reference range. The control unit 50 may also determine whether the difference in brightness within the reference range is greater than or equal to a predetermined value. If the control unit 50 determines, using the video selection processing unit 58, that there is a difference in brightness within the reference range (step S503; Yes), it proceeds to step S504. If the control unit 50 does not determine, using the video selection processing unit 58, that there is a difference in brightness within the reference range (step S503; No), it terminates the processing of this flowchart.

[0124] In step S504, the left and right images are selected with priority given to the image with higher brightness in the reference range. The image with higher brightness in the reference range is the image in which the user's hair is not reflected in the reference range, or is reflected in less of it.

[0125] This embodiment is also applicable to Figure 8. In this case, the process in step S206 can be replaced with the process in step S503 described above.

[0126] <Effects> As described above, this embodiment allows the camera 16 that captures the image with higher brightness in the reference range of the image to be selected as the camera 16 that captures the image for which object recognition is performed preferentially. According to this embodiment, when a user requests information about an object, even if the object cannot be properly recognized due to obstructions such as hair or the orientation of the camera, the object for which the user wants information can be properly identified.

[0127] [Fourth Embodiment] The object detection system 1 according to this embodiment will be described with reference to Figure 13. Figure 13 is a flowchart showing another example of the video selection processing flow in the object detection system according to the fourth embodiment. In this embodiment, the processing of the video selection processing unit 58 of the control unit 50 differs from that of the first embodiment.

[0128] In this embodiment, the camera 16 that captures the video with the higher recognition score for object recognition among the left and right videos is selected as the camera 16 that will preferentially capture the video for object recognition.

[0129] In this embodiment, the video selection processing unit 58 uses the comparison result of the recognition scores of the same object as the difference between videos in which the same object is recognized.

[0130] The recognition score is a score (confidence score) that indicates the reliability of object recognition.

[0131] The video selection processing unit 58 selects the camera 16 that captured the video with the higher recognition score for recognizing an object from the left video and the right video, as the camera 16 that will be preferentially used to capture the video for object recognition. The video selection processing unit 58 selects the camera that captured the video with the higher recognition score for recognizing an object from the left video and the right video, as the video that will be preferentially used to capture the video for object recognition.

[0132] If the name of an object identified from the speech content recognized by the speech recognition processing unit 52 matches the name of an object recognized by the object recognition processing unit 54, the video selection processing unit 58 may prioritize the video with the higher recognition score for object recognition among the right and left videos.

[0133] For example, suppose the recognition score indicating object similarity when recognizing an object is expressed on a scale of 0.00 to 1.00, and an object is determined to be an object if its recognition score is 0.70 or higher. If the difference in recognition scores between the left and right images is 0.10 or more, it is determined that there is a difference in recognition scores.

[0134] For example, in video footage where part of an object is obscured by the user's face or hair, the recognition score will be lower for the user whose part of the object is obscured is larger.

[0135] Figure 11 shows another example of the detection target recognition range and object recognition range in the video captured by the camera. The object recognition processing unit 54 determines that the recognized object is an "apple," or that the recognized object includes an "apple." The video selection processing unit 58, from the video at the time it detects that the user has said "apple," prioritizes the video with the higher recognition score when the object recognition processing unit 54 recognizes an "apple" as the video for object recognition.

[0136] In Figure 11, "apple" is recognized in both the left and right images. The recognition score indicating object-likeness in the right image is, for example, 0.95. In the left image, the recognition score indicating object-likeness is, for example, 0.75, due to reasons such as the user's hair obscuring the entire image of the object. In such cases, the image selection processing unit 58 prioritizes the right image for object recognition.

[0137] <Object Detection Method (Image Selection Process)> Next, another example of the image selection process in the object detection system 1 will be explained using Figure 13. Steps S601, S602, and S604 are performed in the same way as steps S101, S102, and S104 in the flowchart shown in Figure 7.

[0138] The control unit 50, using the video selection processing unit 58, determines whether or not there is a difference in recognition scores (step S603). A difference in recognition scores is a difference in recognition accuracy between the recognition score of one video and the recognition score of the other video, and it may also be determined whether or not the difference in recognition scores is above a predetermined level. The control unit 50, using the video selection processing unit 58, determines whether or not there is a difference in recognition scores from the video at the time the user uttered the object name, when the object recognition processing unit 54 recognized the object. If the control unit 50 determines that there is a difference in recognition scores (step S603; Yes), it proceeds to step S604. If the control unit 50 does not determine that there is a difference in recognition scores (step S603; No), it terminates the processing of this flowchart.

[0139] This embodiment is also applicable to Figure 8. In this case, the process in step S206 can be replaced with the process in step S603 described above.

[0140] <Effects> As described above, in this embodiment, the camera 16 that captures the video with the higher recognition score for recognizing an object among the left and right videos can be selected as the camera 16 that captures the video for which object recognition is performed preferentially. According to this embodiment, when a user requests information about an object, even if the object cannot be properly recognized due to obstructions such as hair or the orientation of the camera, the object for which the user's information is desired can be properly identified.

[0141] [Fifth Embodiment] The object detection system 1A according to this embodiment will be described with reference to Figures 14 to 20. Figure 14 is a block diagram showing an example configuration of an information terminal device according to the fifth embodiment. Figure 15 is a schematic diagram showing the relationship between the camera's shooting angle of view, effective angle of view, and the position of an object. Figure 16 is a diagram showing another example of the detection target object recognition range and object recognition range in the image captured by the camera. Figure 17 is a flowchart showing the processing flow in the object detection system according to the fifth embodiment. Figure 18 is a diagram showing another example of the detection target object recognition range and object recognition range in the image captured by the camera. Figure 19 is a flowchart showing the processing flow in the object detection system according to the fifth embodiment. Figure 20 is a flowchart showing the processing flow in the object detection system according to the fifth embodiment. The basic configuration of the object detection system 1A is the same as that of the object detection system 1 of the first embodiment. In the following description, components similar to those of the object detection system 1 are denoted by the same reference numerals or corresponding reference numerals, and their detailed descriptions are omitted. In this embodiment, the control unit 50A is equipped with an adjustment control unit 57A, which is different from the first embodiment.

[0142] When the object detection system 1A finds that the name of an object identified from the spoken content matches the name of an object recognized from the video, it adjusts the object recognition range (reference position), which is the range in which the object to be detected is recognized, based on the object recognition range in the image captured by the camera 16.

[0143] In this embodiment, the camera 16 may be either the left camera 16L or the right camera 16R, or it may be configured to include both the left camera 16L and the right camera 16R.

[0144] When the name of an object identified from the speech content recognized by the speech recognition processing unit 52 matches the name of an object recognized by the object recognition processing unit 54, the adjustment control unit 57A adjusts the detection target object recognition range, which is the range in which the detection target object is recognized, based on the object recognition range in the image captured by the camera 16.

[0145] The adjustment control unit 57A adjusts the detection target object recognition range using the central part of the object recognition range in the image captured by the camera 16 as the reference point.

[0146] The processing of the adjustment control unit 57A will be explained using Figures 15 and 16. Figure 15 is an explanatory diagram showing the case where the camera 16 is the right camera 16R and there is no left camera 16L. The object recognition processing unit 54 determines that the recognized object is an "apple," or determines that the recognized object includes an "apple." In addition, the speech recognition processing unit 52 adjusts the position of the detected object recognition range, centering on the range in the video where an "apple" was recognized at the time it was detected that the user had said "apple."

[0147] In Figure 16, the shooting range of the right camera 16R is shifted downward and to the right due to misalignment of the right earphone 10R or the shape of the user's ear. Therefore, the adjustment control unit 57A shifts the detection target recognition range R1 of the right camera 16R to a position centered on the range where the "apple" was recognized (upper right).

[0148] Figure 16 shows that the object recognition range R1 adjusted by the adjustment control unit 57A was adjusted from the initial value of the object recognition range R1i to the central part of the object recognition range R1, because the voice recognition processing unit 52 recognized the sound "apple" and the name of the object in the object recognition range R2 in which "apple" was recognized matched.

[0149] Through this process, when a user requests information about an object, the position from which the user views the object is aligned with the position that the camera 16 is facing, particularly within the detection target recognition range. This ensures that the detected object is properly identified. In particular, when multiple objects are present, the object for which the user desires information is properly identified.

[0150] <Object Detection Method (Range Adjustment Process)> Next, the information processing in the object detection system 1A will be explained using Figure 17. Steps S703 to S705 of the flowchart shown in Figure 17 perform the same processing as steps S103 to S105 of the flowchart shown in Figure 7. When the earphones 10 are first used, or each time they are put on, the processing in the flowchart shown in Figure 17 is executed as a calibration process. While the processing in the flowchart shown in Figure 17 is being executed, the audio acquired by the microphone 17 and the video captured by the camera 16 are transmitted to the information terminal device 40. When calibration is started, the audio output unit 19 of the earphones 10 outputs an audio instruction, such as "Look straight at an easy-to-understand object and say its name."

[0151] The control unit 50A determines, using the object recognition processing unit 54, whether or not an object has been recognized from the video (step S701). The control unit 50A determines, using the object recognition processing unit 54, whether or not an object has been recognized from the video captured by the camera 16. If the control unit 50A determines, using the object recognition processing unit 54, that an object has been recognized from the video (step S701; Yes), it proceeds to step S702. If the control unit 50A determines, using the object recognition processing unit 54, that an object has not been recognized from the video (step S701; No), it terminates the processing of this flowchart. If, for example, no object is recognized for 30 seconds after the voice instruction to start calibration (S701; No), the processing of this flowchart is terminated.

[0152] If it is determined that an object has been recognized from the video (step S701; Yes), the control unit 50A acquires recognized object information via the information acquisition control unit 56 (step S702). The control unit 50A, via the information acquisition control unit 56, sends an instruction to the AI ​​server via the communication control unit 51 to acquire information about the object identified as a detection target along with the video captured by the camera 16. The control unit 50A, via the information acquisition control unit 56, acquires information about the object identified as a detection target from the AI ​​server via the communication control unit 51. The control unit 50A proceeds to step S703.

[0153] The control unit 50A adjusts the detection target recognition range with the adjustment control unit 57A (step S706). The control unit 50A adjusts the detection target recognition range with the adjustment control unit 57A using the center of the object recognition range in the image captured by the camera 16 as the reference point.

[0154] The order in which the processes in steps S701 and S702, and steps S703 and S704 are performed is not limited and they may be executed in parallel.

[0155] Steps S701 and S702 may be executed as a single step without being separated.

[0156] Before executing the process shown in the flowchart of Figure 17, a process may be performed to detect whether the earphone 10 is being worn in the ear.

[0157] Figure 18 is an explanatory diagram showing the case where camera 16 is equipped with a right camera 16R and a left camera 16L.

[0158] If the object recognized in the left image is the same object as the object recognized in the left image, and the name of the object identified from the speech content recognized by the speech recognition processing unit 52 matches the name of the object recognized by the object recognition processing unit 54, the adjustment control unit 57A adjusts the range for recognizing the target object based on the object recognition range in the image captured by the camera 16.

[0159] Figure 18 shows that in the right image, when the voice recognition processing unit 52 recognizes the word "apple" and the name of the object in the object recognition range R2 where "apple" was recognized matches, the object recognition range R2 is included in the detection target recognition range R1, and therefore it is determined that no adjustment is needed for the detection target recognition range R1. In other words, the right earphone 10R is properly fitted. Also, Figure 18 shows that in the left image, when the voice recognition processing unit 52 recognizes the word "apple" and the name of the object in the object recognition range L2 where "apple" was recognized matches, the detection target recognition range L1 is adjusted from the initial value of the detection target recognition range L1i to the center of the detection target recognition range. In other words, the left earphone 10L is misaligned, or the shape of the user's ear may be causing the shooting range of the left camera 16L to shift downwards and to the left.

[0160] <Object Detection Method (Range Adjustment Process)> Next, the information processing in the object detection system 1A will be explained using Figure 19. Steps S801 to S804 and S806 to S807 of the flowchart shown in Figure 19 perform the same processing as steps S701 to S704 and S705 to S706 of the flowchart shown in Figure 17.

[0161] The adjustment control unit 57A determines whether the object recognized in the left image is the same object (step S805). If the adjustment control unit 57A does not determine that the object recognized in the left image is the same object (step S805; No), it terminates the process of this flowchart. If the adjustment control unit 57A determines that the object recognized in the left image is the same object (step S805; Yes), it proceeds to step S806.

[0162] <Object Detection Method (Object Recognition Processing)> Next, the object recognition processing in the object detection system 1A will be explained using Figure 20. Steps S901, S904 to S906 are the same as steps S301, S304 to S306 in the flowchart shown in Figure 9.

[0163] If the control unit 50A determines that it has received a voice command (step S901; Yes), the control unit 50A uses the object recognition processing unit 54 to recognize an object from the image captured by the camera 16 (step S902). The control unit 50A uses the object recognition processing unit 54 to send a command to the AI ​​server via the communication control unit 51 to perform object recognition processing along with the image captured by the camera 16. At this time, the image used for object recognition processing may be either the right image captured by the right camera 16R, or the left image captured by the left camera 16L, or both. The control unit 50A uses the object recognition processing unit 54 to obtain the AI ​​server recognition result via the communication control unit 51. The control unit 50A then proceeds to step S903.

[0164] The control unit 50A determines, based on the object recognition processing unit 54, whether or not there is an object recognized within the detection target recognition range (step S903). If the control unit 50A determines, based on the object recognition processing unit 54, that there is an object recognized within the detection target recognition range (step S903; Yes), the control unit 50A proceeds to step S904. If the control unit 50A does not determine, based on the object recognition processing unit 54, that there is an object recognized within the detection target recognition range (step S903; No), the control unit 50A proceeds to step S906.

[0165] In this way, after calibration is complete, when the user wears the left earphone 10L and right earphone 10R that constitute the object detection system 1A, they can speak a trigger word to recognize an object detected in the center of the shooting range or object recognition range of the left camera 16L and right camera 16R, and output information about the object as audio data to the earphone 10.

[0166] If the camera 16 is provided only in the left earphone 10L or the right earphone 10R, or if the camera 16 is provided in both the left earphone 10L and the right earphone 10R, and one of the cameras 16 is configured to function as an object recognition camera, steps S902 and S903 in Figure 20 are reinterpreted as, for example, "Object recognition from (right) image" and "Is there an object recognized within the detection target object recognition range (of the (right) image)?" and processed accordingly.

[0167] If the left earphone 10L and the right earphone 10R are equipped with cameras 16, steps S902 and S903 in Figure 19 are reinterpreted as, for example, "object recognition from (left and right) images" and "are there any (identical) objects recognized within the detection target object recognition range (of the right and left images)?" and processed accordingly.

[0168] <Effects> As described above, in this embodiment, when the name of an object identified from the spoken content matches the name of an object recognized by the object recognition processing unit 54, the range for recognizing the object can be adjusted based on the range in which the object is recognized in the camera's captured image. According to this embodiment, when a user requests information about an object, the position where the user views the object and the position the camera 16 is facing, particularly the position of the object recognition range, can be aligned. According to this embodiment, the object to be detected can be appropriately identified. According to this embodiment, when there are multiple objects, the object for which the user wants information can be appropriately identified. In this way, according to this embodiment, the object to be processed can be appropriately identified from the image of the camera 16 attached to the earphone 10.

[0169] In this embodiment, the central part of the area in the camera's captured image where an object is recognized can be used as a reference point to adjust the range for recognizing the object to be detected.

[0170] In this embodiment, if the object recognized in the left image and the object recognized in the right image are the same object, and the name of the object identified from the speech content matches the name of the object recognized by the object recognition processing unit, the range for recognizing the target object can be adjusted based on the range in which the object was recognized in the image captured by the camera 16.

[0171] Each component of the illustrated object detection system is a functional concept and does not necessarily have to be physically configured as shown. In other words, the specific form of each device is not limited to that shown, and all or part of them may be functionally or physically distributed or integrated in any unit depending on the processing load and usage conditions of each device.

[0172] The object detection system is implemented, for example, as software, such as a program loaded into memory. In the above embodiment, these were described as functional blocks implemented through the cooperation of hardware or software. That is, these functional blocks can be implemented in various ways using hardware alone, software alone, or a combination thereof.

[0173] The components described above include those that are easily conceivable by those skilled in the art, and those that are substantially identical. Furthermore, the components described above can be combined as appropriate. In addition, various omissions, substitutions, or modifications of the components are possible without departing from the spirit of the present invention.

[0174] In the above embodiment, the touch operation unit 15 and the operation control unit 32 are not essential components.

[0175] In the above description, the earphone 10 was described as having a left earphone 10L and a right earphone 10R that can communicate with the information terminal device 40, but it is not limited to this. The earphone 10 may also be a master unit in which the left earphone 10L or the right earphone 10R can communicate with the portable electronic device 100, and a slave unit in which the right earphone 10R or the left earphone 10L can communicate with the left earphone 10L or the right earphone 10R.

[0176] In the above example, the control unit 50 of the information terminal device 40 implements the function of identifying the object to be detected, but this is not limited to this. For example, the control unit 30 of the earphone 10 may implement the function of identifying the object to be detected. In this case, the left earphone 10L or the right earphone 10R, which has the control unit 30 that implements the function of identifying the object to be detected, receives the video captured from the other right earphone 10R or left earphone 10L.

[0177] The object detection device and program of this disclosure can be used, for example, in an information terminal device connected to a camera-equipped device such as a camera-equipped earphone.

[0178] 1 Object detection system 10 Earphone (device with camera) 11 Housing 15 Touch operation unit 16 Camera 17 Microphone (sound acquisition unit) 18 Amplifier unit 19 Sound output unit (output unit) 27 Communication unit 28 Battery 29 Power supply unit 30 Control unit 32 Operation control unit 33 Shooting control unit 34 Sound input control unit 35 Sound output control unit (output control unit) 37 Communication control unit 39 Power supply control unit 40 Information terminal device (object detection device) 41 Communication unit 50 Control unit 51 Communication control unit 52 Speech recognition processing unit 53 Speech synthesis processing unit 54 Object recognition processing unit 55 Identification unit 56 Information acquisition control unit 58 Video selection processing unit (selection processing unit)

Claims

1. An object detection device connected to a camera-equipped device having a left camera and a right camera, comprising: an object recognition processing unit that recognizes an object from images captured by the left camera and the right camera; an identification unit that identifies the object recognized by the object recognition processing unit as a target object for detection; and a selection processing unit that selects, from the left camera and the right camera, which camera captures images for which object recognition is to be performed preferentially.

2. The object detection device according to claim 1, wherein, when the selection processing unit recognizes the same object from the image captured by the left camera and the image captured by the right camera, it selects, based on the difference between the images in which the same object is recognized, which camera will capture the image for which object recognition will be performed preferentially from the left camera and the right camera.

3. The object detection device according to claim 2, wherein the selection processing unit preferentially selects the camera that captured the image from the left camera and the right camera, whose image shows an area in which the object is recognized that is closer to the reference position in the image, as the camera that captures the image for which object recognition is performed.

4. The object detection device according to claim 2, wherein the selection processing unit selects, among the image captured by the left camera and the image captured by the right camera, the camera that captured the image in which the range in which the object is recognized is included in a larger portion of the reference range in the image, as the camera that will preferentially capture the image for object recognition.

5. The object detection device according to claim 2, wherein the selection processing unit selects the camera that captured the image with the higher brightness in the reference range of the image from the image captured by the left camera and the image captured by the right camera as the camera that will preferentially capture the image for object recognition.

6. The object detection device according to claim 2, wherein the selection processing unit selects, from among the images captured by the left camera and the images captured by the right camera, the camera that captured the image with a larger recognition score for recognizing the object, as the camera that will preferentially capture the image for object recognition.

7. The object detection device according to claim 2, further comprising a speech recognition processing unit that recognizes speech content from speech acquired by a speech acquisition unit provided in the camera-equipped device, wherein the selection processing unit determines that the same object has been recognized from the video captured by the left camera and the video captured by the right camera when the name of the object identified from the speech content recognized by the speech recognition processing unit matches the name of the object recognized by the object recognition processing unit.

8. The object detection device according to claim 1, further comprising: an information acquisition control unit that acquires information about an object identified as the object to be detected by the identification unit in the video footage captured by the camera selected by the selection processing unit, and notifies the wearer of the camera-equipped device of the information about the object.

9. A program that causes an object detection device connected to a camera-equipped device having a left camera and a right camera to execute the following: a process to recognize an object from the images captured by the left camera and the right camera; a process to identify the recognized object as a target for detection; and a process to select which of the left and right cameras will capture the image for priority object recognition based on the image captured by the left camera and the image captured by the right camera.

Images