Object detection device and program

Abstract

An object detection device 40 is an information terminal device that, when worn on the head of a user, connects to an earphone 10 that comprises a camera 16 that can capture images of what is ahead of the user. The object detection device 40 comprises an object recognition processing unit 54 that recognizes objects from video captured by the camera 16, an audio recognition processing unit 52 that recognizes the content of speech spoken by the user, and an adjustment control unit 57 that, when the name of an object identified from the content of speech recognized by the audio recognition processing unit 52 coincides with the name of an object recognized by the object recognition processing unit 54, adjusts a range for recognizing objects to be detected on the basis of the range in which objects in images captured by the camera 16 have been recognized.

Description

Object detection apparatus and program

[0001] The present invention relates to an object detection apparatus and a program.

[0002] Due to the miniaturization of cameras, cameras can be installed in various small devices. As an example of a small device, a technology related to an apparatus in which an earphone is provided with a camera has been disclosed (see, for example, Patent Document 1).

[0003] Japanese Unexamined Patent Application Publication No. 2022-178851

[0004] An earphone provided with a camera includes a microphone and a speaker in addition to the camera. It is possible to perform a process of performing object recognition from video captured by a camera disposed in the earphone, and notifying a user wearing the earphone of information related to the recognized object by voice. However, since the camera is mounted on the earphone, the direction the user is looking may not match the direction in which the camera is capturing an image due to differences in the shape of the user's head or ears, or a state where the earphone is not properly worn.

[0005] The present disclosure has been made in view of the above, and an object thereof is to appropriately specify an object to be processed from video captured by a camera mounted on an earphone.

[0006] In order to solve the above-described problem and achieve the object, an object detection apparatus according to the present disclosure is an object detection apparatus connected to a camera-equipped device including a camera capable of capturing an image in front of a user when the object detection apparatus is worn on the user's head, the object detection apparatus comprising: an object recognition processing unit that recognizes an object from video captured by the camera; a speech recognition processing unit that recognizes utterance content from utterance speech of the user; and an adjustment control unit that, when a name of an object specified from the utterance content recognized by the speech recognition processing unit matches a name of an object recognized by the object recognition processing unit, adjusts a range for recognizing a detection target object based on a range in which the object is recognized in a captured image of the camera.

[0007] The object detection device according to this disclosure is an object detection device connected to a camera-equipped device that includes a left camera worn on the user's left ear to capture a left image and a right camera worn on the user's right ear to capture a right image, and comprises an object recognition processing unit that recognizes an object from the image captured by the camera, and an adjustment control unit that, when the object recognized in the left image and the object recognized in the right image are the same object, adjusts the range for recognizing the object to be detected based on the range in which the object is recognized in the image captured by the camera.

[0008] The program relating to this disclosure causes an object detection device connected to a camera-equipped device that, when worn on the user's head, is capable of capturing images in front of the user, to perform the following processes: recognize an object from the image captured by the camera; acquire the user's spoken voice and recognize the content of the speech; and, if the name of an object identified from the content of the speech recognized from the user's spoken voice matches the name of an object recognized from the image, adjust the range in which the object is recognized in the image captured by the camera to be recognized as the range for detecting the object.

[0009] The program relating to this disclosure causes an object detection device connected to a camera-equipped device, which includes a left camera worn on the user's left ear to capture a left image and a right camera worn on the user's right ear to capture a right image, to perform the following processes: to recognize an object from the images captured by the cameras; and, if the object recognized in the left image and the object recognized in the right image are the same object, to adjust the range for recognizing the object based on the range in which the object is recognized in the camera's captured image.

[0010] According to this disclosure, the effect is that the object to be processed can be appropriately identified from the image of the camera attached to the earphone.

[0011] Figure 1 is a schematic diagram showing an example configuration of an object detection system according to the first embodiment. Figure 2 is a schematic front view of an earphone according to the first embodiment, showing the earphone 10 as seen from the front of the user when worn by the user. Figure 3 is a block diagram showing an example configuration of an earphone according to the first embodiment. Figure 4 is a block diagram showing an example configuration of an information terminal device according to the first embodiment. Figure 5 is a schematic diagram showing the relationship between the camera's shooting angle of view, effective angle of view, and the position of an object. Figure 6 is a diagram showing an example of the detection target object recognition range and object recognition range in an image captured by the camera. Figure 7 is a flowchart showing another example of the video selection processing flow in the object detection system according to the first embodiment. Figure 8 is a diagram showing another example of the detection target object recognition range and object recognition range in an image captured by the camera. Figure 9 is a flowchart showing another example of the video selection processing flow in the object detection system according to the first embodiment. Figure 10 is a flowchart showing another example of the processing flow in the object detection system according to the first embodiment. Figure 11 is a block diagram showing an example configuration of an information terminal device according to the second embodiment. Figure 12 is a schematic diagram showing the relationship between the camera's shooting angle of view, effective angle of view, and the position of an object. Figure 13 is a diagram showing another example of the detection target recognition range and object recognition range in the image captured by the camera. Figure 14 is a flowchart showing the processing flow in the object detection system according to the second embodiment. Figure 15 is a diagram showing another example of the detection target recognition range and object recognition range in the image captured by the camera. Figure 16 is a flowchart showing the processing flow in the object detection system according to the third embodiment. Figure 17 is a flowchart showing the processing flow in the object detection system according to the fourth embodiment. Figure 18 is a flowchart showing the processing flow in the object detection system according to the fourth embodiment.

[0012] Embodiments of the object detection system 1 according to this disclosure will be described in detail below with reference to the attached drawings. However, the present invention is not limited to the following embodiments.

[0013] [First Embodiment] <Object Detection System> Figure 1 is a schematic diagram showing an example of the configuration of an object detection system according to the first embodiment. The object detection system 1 includes an earphone 10, which is an example of a camera-equipped device, and an information terminal device 40, which is an example of an object detection device. The earphone 10 and the information terminal device 40 are connected by wire or wireless communication. In the following description, the earphone 10 and the information terminal device 40 will be described as being connected by wireless communication. When the name of an object identified from the content of speech matches the name of an object recognized from the video, the object detection system 1 will prioritize using the video in which the range in which the object is recognized (hereinafter referred to as the "object recognition range") is closer to the reference position in the video, between the video captured by the right camera 16R (hereinafter referred to as the "right video") and the video captured by the left camera 16L (hereinafter referred to as the "left video").

[0014] <Earphone (Camera-equipped device)> The earphone 10 has multiple independent housings 11 that are worn on the head, more specifically on the ears, of a person who is a user. The earphone 10 can be, for example, a wireless earphone or a wired earphone. The form of the earphone 10 can be any form, for example, an open-ear type or a canal type. Figures 1 and 2 show an example of an open-ear type earphone with TWS (True Wireless Stereo). The earphone 10 has a left earphone 10L for the left ear, which is worn on the left ear, and a right earphone 10R for the right ear, which is worn on the right ear. The left earphone 10L outputs left channel data of the audio signal. The right earphone 10R outputs right channel data of the audio signal. In the following description, when it is not necessary to distinguish between the left earphone 10L and the right earphone 10R, they will be described as earphone 10.

[0015] The earphone 10 will be explained using Figures 1 to 3. Figure 1 is a schematic side view of the earphone 10 according to the embodiment, showing how it looks when the user is wearing the earphone 10 and is viewed from the left or right direction of the user. In Figure 1, the left earphone 10L is shown as viewed from the left side of the user when it is worn on the user's left ear, and the right earphone 10R is shown as viewed from the right side of the user when it is worn on the user's right ear. Figure 2 is a schematic front view of the earphone according to the first embodiment, showing how it looks when the user is wearing the earphone 10 and is viewed from the front of the user. In Figure 2, the right earphone 10R is shown, and the left earphone 10L has a similar configuration. Figure 3 is a block diagram showing an example of the configuration of the earphone according to the first embodiment. The earphone 10 acquires data, including music data, from the information terminal device 40. The earphone 10 includes a touch operation unit 15, a camera 16, a microphone (voice acquisition unit) 17, an amplification unit 18, an audio output unit 19, a communication unit 27, a battery 28, a power supply unit 29, and a control unit 30.

[0016] The housing 11 of the earphone 10 is worn on the user's ear. The housing 11 defines the external shape of the earphone 10. The housing 11 comprises a main body 11a, a first ear hook 11b, and a second ear hook 11c. The main body 11a is the part that is positioned inside the user's auricle when the user wears the earphone 10. The first ear hook 11b is positioned between the helix of the auricle and the head when the user wears the earphone 10. The second ear hook 11c is the part that is positioned behind the auricle when the user wears the earphone 10. The main body 11a, the first ear hook 11b, and the second ear hook 11c are integrally formed. The housing 11 houses a touch operation unit 15, a camera 16, a microphone 17, an amplification unit 18, an audio output unit 19, a communication unit 27, a battery 28, a power supply unit 29, and a control unit 30. The arrangement of the touch operation unit 15, camera 16, microphone 17, amplification unit 18, audio output unit 19, communication unit 27, battery 28, power supply unit 29, and control unit 30 in the housing 11 is just one example.

[0017] The earphone 10 has a left housing 11L that is worn on the user's left ear and a right housing 11R that is worn on the right ear. When it is not necessary to distinguish between the left housing 11L and the right housing 11R, they will be described as housing 11.

[0018] The touch operation unit 15 is a touch sensor located on the main body 11a. The touch operation unit 15 can receive various operations, such as playing and stopping music data on the earphones 10. The touch operation unit 15 can receive various operations, such as voice recognition and video recognition on the earphones 10. The touch operation unit 15 outputs an operation signal indicating the received operation to the operation control unit 32.

[0019] Camera 16 captures images of the area around the earphone 10. Camera 16's shooting is controlled by the shooting control unit 33. Camera 16 outputs the captured image to the shooting control unit 33. Camera 16 comprises a left camera 16L located in the left housing 11L of the left earphone 10L, and a right camera 16R located in the right housing 11R of the right earphone 10R. In the description of this embodiment, when it is not necessary to distinguish between the left camera 16L and the right earphone 10R, they will be described as camera 16.

[0020] The left camera 16L, located on the left earphone 10L, is positioned on the main body 11aL of the left housing 11L. The left camera 16L is positioned on the main body 11aL so that it faces forward when the user wears the left earphone 10L on their left ear; in other words, the shooting direction is in front of the user.

[0021] The right camera 16R, located on the right earphone 10R, is positioned on the main body 11aR of the right housing 11R. The right camera 16R is positioned on the main body 11aR so that it faces forward when the user wears the right earphone 10R in their right ear; in other words, the camera is positioned so that the area in front of the user is the direction of shooting.

[0022] The microphone 17 picks up sounds from the vicinity of the earphone 10. The microphone 17 can pick up various sounds, including, for example, voice commands to the earphone 10. The microphone 17 outputs the picked-up sounds to the voice input control unit 34. The microphone 17 is located in at least one of the left housing 11L and the right housing 11R. The microphone 17 is located in the main body 11a.

[0023] The amplification unit 18 amplifies the audio output from the audio output unit 19. The amplification unit 18, for example, performs D / A conversion on audio channel data, including music data acquired from the information terminal device 40, and amplifies it. The audio output unit 19 is located in the main unit 11a.

[0024] The audio output unit 19 outputs sound for the user to hear. The audio output unit 19 outputs sound based on a music signal, for example. The audio output control unit 35 controls the output of information acquired regarding the detected object as sound in the information terminal device 40, which will be described later, for example. The audio output unit 19 outputs an audio signal amplified by the amplification unit 18. The audio output unit 19 is located in at least one of the left housing 11L and the right housing 11R. The audio output unit 19 is located in the main body 11a.

[0025] The communication unit 27 is a communication unit. The communication unit 27 is capable of wireless communication including, for example, Bluetooth®, Wi-Fi®, or NFMI (Near Field Magnetic Induction). In this embodiment, the communication unit 27 is connected to the information terminal device 40 using the Bluetooth method. The communication unit 27 is located, for example, in the second ear hook portion 11c.

[0026] The communication unit 27 is connected to the information terminal device 40 so as to be able to send and receive data. The communication unit 27 receives data, for example, music data, from the information terminal device 40. In this embodiment, the communication unit 27 transmits to the information terminal device 40 video footage captured by the camera 16 acquired by the shooting control unit 33, and audio data based on audio signals picked up by the microphone 17 acquired by the audio input control unit 34. In this embodiment, the communication unit 27 receives audio data of information acquired regarding the object to be detected from the information terminal device 40.

[0027] The battery 28 supplies power for the earphone 10 to operate. The battery 28 is a rechargeable battery that can be repeatedly charged and discharged. The battery 28 is, for example, a nickel-metal hydride rechargeable battery, a lithium-ion rechargeable battery, or a lithium polymer rechargeable battery. The battery 28 is built into the earphone 10. The charging and discharging of the battery 28 is controlled by the power control unit 39. The battery 28 is located, for example, in the second ear hook portion 11c.

[0028] The power supply unit 29 can be connected to the power supply of an earphone case (not shown) and supplies power to the battery 28. The power supply unit 29 is connected to the power control unit 39. The power supply unit 29 may also be capable of supplying power from a DC power source such as a mobile battery or from a contactless charging device.

[0029] <Earphone Control Unit> The control unit 30 is an arithmetic processing unit composed of, for example, a CPU (Central Processing Unit). The control unit 30 loads a program stored in a storage unit (not shown) into memory and executes the instructions contained in the program. The control unit 30 includes an internal memory (not shown) which is used for temporary data storage, etc. The control unit 30 has an operation control unit 32, a shooting control unit 33, an audio input control unit 34, an audio output control unit 35 which is an example of an output control unit, a communication control unit 37, and a power supply control unit 39.

[0030] The operation control unit 32 acquires operation signals from the touch operation unit 15 in response to operations performed on the touch operation unit 15. The operation control unit 32 outputs control signals corresponding to the acquired operation signals to each part of the earphone 10 and the information terminal device 40.

[0031] The shooting control unit 33 controls the shooting by the camera 16. The shooting control unit 33 acquires the video captured by the camera 16. In this embodiment, the shooting control unit 33 uses the communication unit 27 via the communication control unit 37 to transmit the captured video to the information terminal device 40.

[0032] The audio input control unit 34 acquires the audio signal picked up by the microphone 17. The audio input control unit 34 performs A / D conversion on the audio signal picked up by the microphone 17 and acquires it as audio data.

[0033] The audio output control unit 35 controls the output of audio from the audio output unit 19. The audio output control unit 35 performs processing to output audio to the audio output unit 19. The audio output control unit 35 controls the output of audio data, such as music data, from the audio output unit 19. The audio output control unit 35 controls the output of audio data, such as information acquired regarding the detected object in the information terminal device 40 (described later), from the audio output unit 19. The audio output control unit 35 outputs information regarding the detected object as an audio signal to the audio output unit 19. The audio output control unit 35 outputs the audio data acquired from the information terminal device 40 to the amplification unit 18.

[0034] The communication control unit 37 communicates wirelessly with the information terminal device 40 by controlling the communication unit 27. For example, the communication control unit 37 controls the information terminal device 40 to receive data including music data. In this embodiment, the communication control unit 37 transmits to the information terminal device 40 video footage captured by the camera 16 and audio data based on audio signals picked up by the microphone 17 acquired by the audio input control unit 34. In this embodiment, the communication control unit 37 controls the information terminal device 40 to receive audio data indicating information acquired regarding the detected object.

[0035] The power control unit 39 is a charging circuit that charges the battery 28. The power control unit 39 is connected to the battery 28 and the power supply unit 29. The power control unit 39 supplies power from the power supply unit 29 to the battery 28.

[0036] <Information Terminal Device (Object Detection Device)> The information terminal device 40 will be described using Figure 4. Figure 4 is a block diagram showing an example configuration of the information terminal device according to the first embodiment. The information terminal device 40 is a portable electronic device used by the user of the earphone 10, such as a smartphone or a tablet device. When the name of an object identified from the spoken content matches the name of the recognized object, the information terminal device 40 prioritizes using the image on the right or left where the object recognition range is closer to the reference position in the image as the image for object recognition. The information terminal device 40 comprises a communication unit 41 and a control unit 50.

[0037] The communication unit 41 is a communication unit. The communication unit 41 is connected to the earphone 10 so as to be able to send and receive data. The communication unit 41 transmits data, including music data, to the earphone 10, for example. In this embodiment, the communication unit 41 receives video footage captured by the camera 16 and audio data based on audio signals picked up by the microphone 17 acquired by the audio input control unit 34 from the earphone 10. In this embodiment, the communication unit 41 transmits to the AI ​​(Artificial Intelligence) server, along with the video footage captured by the camera 16 from the earphone 10, including instructions such as prompts corresponding to audio commands for executing object recognition processing. In this embodiment, the communication unit 41 receives text data from the AI ​​server indicating information about the object to be detected. In this embodiment, the communication unit 41 converts the text data indicating information acquired regarding the object to be detected into audio data and transmits it to the earphone 10. The communication unit 41 is capable of short-range wireless communication, including Bluetooth, for example. In this embodiment, the communication unit 41 is connected to the earphone 10 using the Bluetooth method.

[0038] The video acquired by the information terminal device 40 from the earphone 10 can be used for various purposes. For example, the video may be recorded. For example, an object may be recognized from the video, and audio data indicating information about the detected object, which is the recognition result, may be fed back to the earphone 10. The object recognition process may use an external AI server provided by the AI ​​generation service provider. In this case, for example, the video acquired from the earphone 10 is sent to the external AI server along with instructions such as prompts corresponding to voice commands that are trigger words for executing the object recognition process. The AI ​​server performs the object recognition process and sends text data, audio data, etc., indicating information about the detected object, which is the object recognition result, to the information terminal device 40. The information terminal device 40 converts the text data received from the AI ​​server into audio data and sends it to the earphone 10.

[0039] The user's spoken voice, which the information terminal device 40 acquires from the earphone 10, can be used for various purposes. For example, the voice may be recorded. For example, if the voice recognized by the speech recognition process is a trigger word, which is a voice command to execute object recognition processing, it may be used as a trigger to start the process of recognizing an object from the video acquired from the earphone 10 and outputting the information as voice.

[0040] A trigger word is an utterance that inquires about object information, such as "What is that?", "What is that?", or "Tell me its name." A trigger word is an instruction that, for example, recognizes an object from an image and searches for and outputs the name or other information about the recognized object. In other words, a trigger word is an utterance that means an instruction to recognize an object from an image.

[0041] <Control Unit of Information Terminal Device> The control unit 50 is a processing unit composed of, for example, a CPU. The control unit 50 loads a program stored in a storage unit (not shown) into memory and executes the instructions contained in the program. The control unit 50 includes an internal memory (not shown) which is used for temporary data storage, etc. In this embodiment, the control unit 50 implements the function of a control unit of the object detection system 1. When the name of an object identified from the speech content matches the name of the recognized object, the control unit 50 prioritizes the image in which the object recognition range is closer to the reference position in the image, between the right image and the left image, as the image for which object recognition is performed. The control unit 50 includes a communication control unit 51, a speech recognition processing unit 52, a speech synthesis processing unit 53, an object recognition processing unit 54, a identification unit 55, an information acquisition control unit 56, and an image selection processing unit 58.

[0042] The communication control unit 51 communicates wirelessly with the earphone 10 by controlling the communication unit 41. For example, the communication control unit 51 controls the earphone 10 to transmit data including music data. In this embodiment, the communication control unit 51 controls the earphone 10 to receive video footage captured by the camera 16 and audio data based on audio signals picked up by the microphone 17 acquired by the audio input control unit 34. In this embodiment, the communication control unit 51 transmits instructions to the AI ​​server corresponding to an audio command to execute object recognition processing along with the video footage captured by the camera 16 from the earphone 10. In this embodiment, the communication control unit 51 receives text data indicating the object recognition result from the AI ​​server. In this embodiment, the communication control unit 51 controls the earphone 10 to transmit audio data indicating information acquired regarding the detected object, which has been converted by the speech synthesis processing unit 53.

[0043] The speech recognition processing unit 52 performs speech recognition processing on the audio data based on the audio signal picked up by the microphone 17 located in the earphone 10. More specifically, the speech recognition processing unit 52 performs speech recognition processing to recognize the content of the spoken words from the audio data based on the audio signal picked up by the microphone 17 located in the earphone 10 via the communication unit 41, using natural language processing or the like.

[0044] The speech recognition processing unit 52 recognizes, for example, an utterance having a meaning of inquiring about object information such as "What is this?" as a trigger word (voice command).

[0045] The speech synthesis processing unit 53 converts text data indicating information related to a detection target object into speech data. For example, the speech synthesis processing unit 53 converts text data acquired by the communication control unit 51 from an AI server or the like into speech data.

[0046] The object recognition processing unit 54 recognizes an object from the left video and the right video captured by the left camera 16L and the right camera 16R. In the present embodiment, the object recognition processing unit 54 causes the AI server to execute object recognition processing together with the left video and the right video captured by the left camera 16L and the right camera 16R via the communication control unit 51, and transmits a command for this purpose.

[0047] The identifying unit 55 identifies the object recognized by the object recognition processing unit 54 as a detection target object.

[0048] The identifying unit 55 identifies the object recognized by the object recognition processing unit 54 as a detection target object when the operation unit receives an operation requesting processing for identifying the object recognized by the object recognition processing unit 54. In the present embodiment, the identifying unit 55 identifies the object recognized by the object recognition processing unit 54 as a detection target object when the speech recognition processing unit 52 recognizes a predetermined speech command. For example, the identifying unit 55 causes the object recognition processing unit 54, which has recognized an object from a video captured at the timing when a user utters a trigger word such as "What is that?", to identify the object as the detection target object.

[0049] The operation for starting an inquiry about information related to the detection target object identified by the identifying unit 55 has been described as an operation performed by uttering a trigger word using speech recognition. This operation may also be performed by a touch operation on the touch operation unit 15. Therefore, the touch operation unit 15 and the operation control unit 32 may be collectively referred to as an operation unit, and the microphone 17 and the speech input control unit 34 may also be collectively referred to as an operation unit.

[0050] In this explanation, the information regarding the detected object identified by the identification unit 55 is converted into audio data by the speech synthesis processing unit 53, and the audio output control unit 35 outputs this audio data to the audio output unit 19. However, when using smart glasses or the like with functions equivalent to the earphones 10, the information regarding the detected object identified by the identification unit 55 may be output as text or an image by displaying it on a display unit (not shown). Therefore, the audio output unit 19, the amplification unit 18, and the audio output control unit 35 may be collectively referred to as the output unit, or the display unit (not shown) and the display control unit may be collectively referred to as the output unit.

[0051] The information acquisition control unit 56 acquires information about the object to be detected. The information acquisition control unit 56 acquires the name of the object recognized by the object recognition processing unit 54, and uses the acquired name in the video selection processing unit 58, which will be described later. The information acquisition control unit 56 acquires information about the object identified as the object to be detected, and notifies the person of the acquired information.

[0052] The information acquisition control unit 56 obtains the name of the object recognized by the object recognition processing unit 54 from the AI ​​server. The AI ​​server transmits the name of the object to the video selection processing unit 58 of the information terminal device 40.

[0053] The information acquisition control unit 56 acquires information about the identified detection target from the AI ​​server. The AI ​​server transmits the information about the detection target identified by the identification unit 55 to the information terminal device 40. The speech synthesis processing unit 53 converts the text data indicating the information about the detection target acquired from the AI ​​server into speech data. The communication control unit 51 transmits the converted speech data indicating the information about the detection target to the earphone 10. The earphone 10 outputs the speech data received by the communication control unit 37 from the information terminal device 40 as speech from the speech output unit 19.

[0054] If the name of an object identified from the speech content recognized by the speech recognition processing unit 52 matches the name of an object recognized by the object recognition processing unit 54, the video selection processing unit 58 prioritizes the video in which the object recognition range is closer to the reference position in the video, between the right video and the left video, as the video for which object recognition will be performed.

[0055] Prioritizing means increasing the priority. Prioritizing means that when the same object is detected by both the left camera 16L and the right camera 16R, the recognition result of the image captured by the camera with the higher priority 16 will be used.

[0056] The reference position is the position or range set within the shooting range of the left camera 16L and the right camera 16R. The reference position is, for example, the position or range set assuming that the camera 16 is shooting centered on the direction the user's face is facing when the user wears the earphones 10. The reference position may be changed for each user or each time the earphones 10 are worn. As a product, the reference position is set so that the orientation of the camera 16 coincides on average with the direction in front of the user's face. The reference position with a range is called the object detection recognition range. Therefore, in the following explanation, the reference position can be rephrased as the object detection recognition range.

[0057] For example, the image with a smaller discrepancy between the object recognition range and the reference position may be given higher priority as the image that is closer to the reference position.

[0058] For example, if the object recognition range of the object recognized in the left image and the object recognition range of the object recognized in the right image are both within the reference position, in other words, within the detection target recognition range, then prioritization of the left and right images may not be performed. In this case, either the objects recognized in the left and right images will be treated as detection targets, or the object recognized from either the left or right image (whichever is pre-configured) will be treated as the detection target.

[0059] For example, if the object recognition range of an object recognized in the left image and the object recognition range of an object recognized in the right image are both within the reference position, and the other is partially or entirely outside the reference position, the image in which the object recognition range of the recognized object is within the reference position will be given higher priority as the image closer to the reference position.

[0060] For example, if the object recognition range of an object recognized in the left image and the object recognition range of an object recognized in the right image are both partially and entirely outside the reference position, the image in which the object recognition range of the recognized object is partially outside the reference position may be given higher priority as the image closer to the reference position. Alternatively, the image in which the object recognition range of the recognized object is closer to the reference position may be given higher priority as the image closer to the reference position.

[0061] For example, if the object recognition range of an object recognized in the left image and the object recognition range of an object recognized in the right image are both entirely outside the reference position, the image whose object recognition range is closer to the reference position may be given higher priority as the image closer to the reference position.

[0062] The image closer to the reference position is, for example, the image where the distance (number of pixels) between the center of the object recognition area and the reference position or the center of the detected object recognition area is shorter. Alternatively, the image closer to the reference position is, for example, the image where the pixels constituting the object recognition area are more numerous within the detected object recognition area.

[0063] The video selection processing unit 58 selects the video that is closer to the reference position. When a user requests information about an object, the system identifies the object from the video of the camera 16 that is appropriate in terms of the position from which the user views the object and the position that the camera 16 is facing, particularly the reference position which is the detection target recognition range. This reduces misrecognition. In particular, when multiple objects are present, the object for which the user requests information can be appropriately identified.

[0064] The video selection processing unit 58 may prioritize the video in which the object recognition range is closer to the center of the video, between the left and right videos, as the video for which object recognition is performed.

[0065] By prioritizing the image closer to the center, the image captured by camera 16 that is appropriately oriented towards the object for which the user wants to acquire information is prioritized. This assumes that the object near the center of the image from the prioritized camera 16 will be the object to be detected. In this case, if the name of the object identified from the speech content recognized by the speech recognition processing unit 52 matches the name of the object recognized by the object recognition processing unit 54, the image selection processing unit 58 will prioritize the image whose object recognition range is closer to the center of the image, between the right and left images, as the image for which object recognition will be performed.

[0066] If the name of an object identified from the speech content recognized by the speech recognition processing unit 52 matches the name of an object recognized by the object recognition processing unit 54, the video selection processing unit 58 may prioritize the video with the higher recognition score for object recognition among the right and left videos.

[0067] For example, suppose the recognition score indicating object similarity when recognizing an object is expressed on a scale of 0.00 to 1.00, and an object is determined to be an object if its recognition score is 0.70 or higher. If the difference in recognition scores between the left and right images is 0.10 or more, it is determined that there is a difference in recognition scores.

[0068] For example, in video footage where part of an object is obscured by the user's face or hair, the recognition score will be lower for the user whose part of the object is obscured is larger.

[0069] An example of identifying a target object will be explained using Figures 5 and 6. Figure 5 is a schematic diagram showing the relationship between the camera's shooting angle of view, effective angle of view, and the position of the object. As shown in Figure 5, the right camera 16R located in the right earphone 10R and the left camera 16L located in the left earphone 10L have shooting angles of view A1 and A2, respectively, of approximately 120° to 150° in the horizontal direction. They have similar shooting angles of view in the vertical direction. The horizontal shooting angles of view A1 and A2 have areas that cannot be captured by the user's head. The areas of shooting angles A1 and A2 that are not obstructed by the user's head are defined as the effective angle of view B1 and B2.

[0070] As shown in Figure 5, suppose a user wearing the left earphone 10L and the right earphone 10R faces object 301 and starts calibration. Calibration can be started by the user uttering a trigger word such as "start calibration" or by operating the touch operation part 15 of the earphone 10.

[0071] Figure 6 shows an example of the object recognition range and object recognition range in the video captured by the camera. Figure 6 shows the recognition results of video captured simultaneously by the left camera 16L and the right camera 16R. In Figure 6, the shooting range of the left camera 16L is shifted to the upper right due to misalignment of the left earphone 10L or the shape of the user's ear. The object recognition range L1 and the object recognition range R1 are pre-set.

[0072] In the example shown in Figure 6, the object recognition processing unit 54 determines that the recognized object is an "apple," or that the recognized object includes an "apple." The speech recognition processing unit 52 also shows the object recognition range L2 and object recognition range R2 in the video at the time it detects that the user has spoken "apple." The video selection processing unit 58 calculates the amount of deviation between the object recognition range L2 and the detection target object recognition range L1, and the amount of deviation between the object recognition range R2 and the detection target object recognition range R1. The video selection processing unit 58 then prioritizes the video with the smallest deviation, in other words, the video closer to the reference position, as the video for which object recognition will be performed. In the example shown in Figure 6, the object recognition range R2 in the right video is included in the detection target object recognition range R1, while the object recognition range L2 in the left video is not partially included in the detection target object recognition range L1. Therefore, the right video is selected as the video that is closer to the reference position.

[0073] <Object Detection Method (Image Selection Process)> Next, the image selection process in the object detection system 1 will be explained using Figure 7. Figure 7 is a flowchart showing another example of the flow of the image selection process in the object detection system according to the first embodiment. When the earphones 10 are first used, or each time they are put on, the process shown in the flowchart in Figure 7 is executed as a calibration process. While the process shown in the flowchart in Figure 7 is being executed, the audio acquired by the microphone 17 and the video captured by the camera 16 are transmitted to the information terminal device 40. When calibration is started, the audio output unit 19 of the earphones 10 outputs an audio instruction, such as "Look straight at an easy-to-understand object and say its name."

[0074] The control unit 50, using the object recognition processing unit 54, determines whether the same object is recognized in the left and right images (step S101). The control unit 50, using the object recognition processing unit 54, determines whether the same object is recognized in the left image captured by the left camera 16L and the right image captured by the right camera 16R. If the control unit 50 determines, using the object recognition processing unit 54, that the same object is recognized in the left and right images (step S101; Yes), it proceeds to step S102. If the control unit 50 does not determine, using the object recognition processing unit 54, that the same object is not recognized in the left and right images (step S101; No), it terminates the processing of this flowchart. If, for example, no object is recognized for 30 seconds after the voice instruction to start calibration (S101; No), the processing of this flowchart is terminated.

[0075] If the control unit 50 determines that the same object is recognized from both the left and right images (step S101; Yes), the control unit 50 acquires recognized object information using the information acquisition control unit 56 (step S102). The control unit 50, using the information acquisition control unit 56, sends an instruction to the AI ​​server via the communication control unit 51 to acquire the name of the recognized object along with the left image captured by the left camera 16L and the right image captured by the right camera 16R. The control unit 50, using the information acquisition control unit 56, acquires the name of the recognized object from the AI ​​server via the communication control unit 51. The control unit 50 proceeds to step S103.

[0076] The control unit 50 determines whether or not speech has been detected by the speech recognition processing unit 52 (step S103). If the control unit 50 determines that speech has been detected by the speech recognition processing unit 52 (step S103; Yes), it proceeds to step S104. If the control unit 50 does not determine that speech has been detected by the speech recognition processing unit 52 (step S103; No), it terminates the process of this flowchart. If, for example, no speech of an object name is detected for 30 seconds after the voice instruction to start calibration (S103; No), the process of this flowchart is terminated.

[0077] The control unit 50 recognizes the content of the speech uttered by the speech recognition processing unit 52 (step S104). The control unit 50 recognizes the content of what the user has said using the speech recognition processing unit 52. Since the user speaks the name of the object from which they want to obtain information, the speech recognition processing unit 52 recognizes the name of the object from which the user wants to obtain information. The control unit 50 then proceeds to step S105.

[0078] The control unit 50 determines, using the video selection processing unit 58, whether the object names match (step S105). The control unit 50 determines, using the video selection processing unit 58, whether the name of the object identified from the speech content recognized by the speech recognition processing unit 52 matches the name of the object recognized by the object recognition processing unit 54. If the control unit 50 determines, using the video selection processing unit 58, that the object names match (step S105; Yes), it proceeds to step S106. If the control unit 50 does not determine, using the video selection processing unit 58, that the object names do not match (step S105; No), it terminates the processing of this flowchart.

[0079] The control unit 50, using the video selection processing unit 58, determines whether there is a difference in distance between the object recognition range and the area outside the reference position (step S106). The control unit 50, using the video selection processing unit 58, determines whether there is a difference in distance between the object recognition range of the object recognized in the left video and the reference position, and between the object recognition range of the object recognized in the right video and the reference position. Preferably, the difference in distance is a predetermined or greater difference in distance such that one of the object recognition ranges is not included in the reference position. The video selection processing unit 58 may also determine whether at least a portion of the object recognition range of the object recognized in the left video or the object recognition range of the object recognized in the right video extends beyond the reference position. If the control unit 50 determines, using the video selection processing unit 58, that there is a difference in distance between the object recognition range and the area outside the reference position (step S106; Yes), it proceeds to step S107. If the control unit 50 does not determine, using the video selection processing unit 58, that there is a difference in distance between the object recognition range and the area outside the reference position (step S106; No), it terminates the processing of this flowchart.

[0080] If it is determined that there is a difference in distance between the object recognition range and the reference position (step S106; Yes), the control unit 50 performs priority setting using the video selection processing unit 58 (step S107). The control unit 50, using the video selection processing unit 58, prioritizes the video in which the object recognition range is closer to the reference position in the video, between the right video and the left video, as the video for which object recognition is performed. The control unit 50 performs priority setting.

[0081] The order in which the processes in steps S101 and S102, and steps S103 and S104 are performed is not limited and they may be executed in parallel.

[0082] Steps S101 and S102 may be executed as a single step without being separated.

[0083] <Object Detection Method (Modified Version of Image Selection Process)> Next, using Figures 8 and 9, another example of the image selection process in the object detection system 1 will be described.

[0084] Figure 8 shows another example of the detection target recognition range and object recognition range in the video captured by the camera. The object recognition processing unit 54 determines that the recognized object is an "apple," or that the recognized object includes an "apple." The video selection processing unit 58, from the video at the time it detects that the user has said "apple," prioritizes the video with the higher recognition score when the object recognition processing unit 54 recognizes an "apple" as the video for object recognition.

[0085] In Figure 8, "apple" is recognized in both the left and right images. The recognition score indicating object resemblance in the right image is, for example, 0.95. In the left image, the recognition score indicating object resemblance is, for example, 0.75, due to reasons such as the user's hair obscuring the entire image of the object. In such cases, the image selection processing unit 58 prioritizes the right image for object recognition.

[0086] Through this process, even when a user requests information about an object, if the object cannot be properly recognized due to obstacles such as hair or the camera's orientation, the system can still accurately identify the object for which the user is seeking information.

[0087] Next, the image selection process in the object detection system 1 will be explained using Figure 9. Figure 9 is a flowchart showing another example of the flow of the image selection process in the object detection system according to the first embodiment. The processes in steps S201 to S205 and step S207 are the same as the processes in steps S101 to S105 and step S107 of the flowchart shown in Figure 7.

[0088] The control unit 50, using the video selection processing unit 58, determines whether there is a difference in recognition scores (step S206). A difference in recognition scores is a difference in recognition accuracy between the recognition score of one video and the recognition score of the other video, and it may also be determined whether the difference in recognition scores is above a predetermined level. The control unit 50, using the video selection processing unit 58, determines whether there is a difference in recognition scores from the video at the time the user uttered the object name, when the object recognition processing unit 54 recognized the object. If the control unit 50 determines that there is a difference in recognition scores (step S206; Yes), it proceeds to step S207. If the control unit 50 does not determine that there is a difference in recognition scores (step S206; No), it terminates the processing of this flowchart.

[0089] In the above flow, the priority is determined in step S201 by "recognizing the same object from both the left and right images." However, if an object is recognized from one of the left or right images, but the same object is not recognized from the other, and the name of the object recognized from one image matches the name of the object spoken, then the image from which the object was recognized will be given priority.

[0090] Before executing the processes shown in the flowcharts of Figures 7 and 9, a process is performed to detect whether the left earphone 10L and the right earphone 10R are in the ears. If the left earphone 10L and the right earphone 10R are in the ears, the processes shown in the flowcharts of Figures 7 and 9 may be executed.

[0091] <Object Detection Method (Object Recognition Processing)> Next, the object recognition processing in the object detection system 1 will be explained using Figure 10. Figure 10 is a flowchart showing another example of the processing flow in the object detection system according to the first embodiment. Before the processing shown in the flowchart in Figure 10 is executed, the processing shown in the flowchart in Figure 7 or Figure 9 is executed.

[0092] The control unit 50 determines whether or not a voice command has been acquired by the voice recognition processing unit 52 (step S301). The control unit 50 determines whether or not a trigger word (voice command), which is an utterance that has meaning in questioning object information such as "What is this?", has been recognized from the sound picked up by the microphone 17 placed on the earphone 10 by the voice recognition processing unit 52. If the control unit 50 determines that a voice command has been acquired by the voice recognition processing unit 52 (step S301; Yes), it proceeds to step S302. If the control unit 50 does not determine that a voice command has been acquired by the voice recognition processing unit 52 (step S301; No), it proceeds to step S306.

[0093] If the control unit 50 determines that it has received a voice command (step S301; Yes), the control unit 50 uses the object recognition processing unit 54 to recognize an object from the video captured by the preferred camera 16 (step S302). The control unit 50 uses the object recognition processing unit 54 to send a command to the AI ​​server via the communication control unit 51 to perform object recognition processing along with the video captured by the preferred camera 16, which was selected according to the flowchart in Figure 7 or Figure 9. The control unit 50 uses the object recognition processing unit 54 to obtain the AI ​​server recognition result via the communication control unit 51. The control unit 50 then proceeds to step S303.

[0094] The control unit 50 determines whether or not there is an object recognized by the object recognition processing unit 54 (step S303). If the control unit 50 determines that there is an object recognized by the object recognition processing unit 54 (step S303; Yes), it proceeds to step S304. If the control unit 50 does not determine that there is an object recognized by the object recognition processing unit 54 (step S303; No), it proceeds to step S306.

[0095] If it is determined that there is a recognized object (step S303; Yes), the control unit 50, using the information acquisition control unit 56, acquires information about the detected object as the detected object (step S304). The control unit 50 identifies the recognized object as the detected object using the identification unit 55. The control unit 50, using the information acquisition control unit 56, acquires information about the detected object from the AI ​​server as the recognized object. The control unit 50 proceeds to step S305.

[0096] The control unit 50 outputs audio data for the detected object to the earphone 10 (step S305). The control unit 50 converts the information about the detected object obtained from the AI ​​server into audio data using the speech synthesis processing unit 53. The control unit 50 transmits the converted audio data to the earphone 10 using the communication control unit 51. The control unit 50 proceeds to step S306.

[0097] The control unit 50 determines whether or not to terminate the process (step S306). More specifically, the control unit 50 determines to terminate the process, for example, when an operation to turn off the object recognition function is detected, or when the power of the earphone 10 is turned off. If the control unit 50 determines to terminate the process (step S306; Yes), it terminates the process in this flowchart. If the control unit 50 does not determine to terminate the process (step S306; No), it executes the process in step S301 again.

[0098] In this way, after calibration is complete, the user, while wearing the left earphone 10L and right earphone 10R that constitute the object detection system 1, speaks a trigger word, which causes the system to recognize an object detected from the image of the higher-priority camera 16 among the left camera 16L and right camera 16R, and outputs information about the object as audio data to the earphone 10.

[0099] Information regarding objects recognized within a reference position in the video feed of the high-priority camera 16 may be output as audio data to the earphone 10.

[0100] <Effects> As described above, in this embodiment, if the name of an object identified from the spoken content matches the name of the object recognized by the object recognition processing unit 54, the image on the left or right of the two images, the one in which the range in which the object is recognized is closer to the reference position in the image, can be preferentially used as the image for object recognition. In this embodiment, by selecting the image closer to the reference position, when a user requests information about an object, the object to be detected can be identified from the image of the camera 16 that has an appropriate position for the object being viewed, particularly the reference position, which is the detection target object recognition range. According to this embodiment, misrecognition can be reduced. In this embodiment, when there are multiple objects, the object for which the user wants information can be appropriately identified. In this way, according to this embodiment, the object to be processed can be appropriately identified from the image of the camera 16 attached to the earphone 10.

[0101] In this embodiment, of the left and right images, the image in which the area where an object is recognized is closer to the center of the image can be prioritized as the image for object recognition. By prioritizing the image closer to the center, this embodiment can prioritize the image captured by the camera 16 that is appropriately pointed towards the object for which the user wants to acquire information.

[0102] In this embodiment, if the name of an object identified from the utterance matches the name of an object recognized by the object recognition processing unit 54, the image with the higher recognition score for recognizing the object can be prioritized as the image for object recognition between the left and right images. According to this embodiment, even if the object cannot be properly recognized due to obstructions such as hair or the camera's orientation when the user requests information about an object, the object for which the user desires information can be properly identified.

[0103] In this embodiment, information about an object identified as a target for detection can be acquired, and the acquired information can be notified to the object.

[0104] [Second Embodiment] The object detection system 1A according to this embodiment will be described with reference to Figures 11 to 14. Figure 11 is a block diagram showing an example of the configuration of an information terminal device according to the second embodiment. Figure 12 is a schematic diagram showing the relationship between the camera's shooting angle of view, effective angle of view, and the position of an object. Figure 13 is a diagram showing another example of the detection target object recognition range and object recognition range in the image captured by the camera. Figure 14 is a flowchart showing the processing flow in the object detection system according to the second embodiment. The basic configuration of the object detection system 1A is the same as that of the object detection system 1 of the first embodiment. In the following description, components similar to those of the object detection system 1 are denoted by the same reference numerals or corresponding reference numerals, and their detailed descriptions are omitted. In this embodiment, the control unit 50A is equipped with an adjustment control unit 57A, which is different from the first embodiment.

[0105] When the object detection system 1A finds that the name of an object identified from the spoken content matches the name of an object recognized from the video, it adjusts the object recognition range (reference position), which is the range in which the object to be detected is recognized, based on the object recognition range in the image captured by the camera 16.

[0106] In this embodiment, the camera 16 may be either the left camera 16L or the right camera 16R, or it may be configured to include both the left camera 16L and the right camera 16R.

[0107] When the name of an object identified from the speech content recognized by the speech recognition processing unit 52 matches the name of an object recognized by the object recognition processing unit 54, the adjustment control unit 57A adjusts the detection target object recognition range, which is the range in which the detection target object is recognized, based on the object recognition range in the image captured by the camera 16.

[0108] The adjustment control unit 57A adjusts the detection target object recognition range using the central part of the object recognition range in the image captured by the camera 16 as the reference point.

[0109] The processing of the adjustment control unit 57A will be explained using Figures 12 and 13. Figure 12 is an explanatory diagram showing the case where the camera 16 is the right camera 16R and there is no left camera 16L. The object recognition processing unit 54 determines that the recognized object is an "apple", or determines that the recognized object includes an "apple". In addition, the speech recognition processing unit 52 adjusts the position of the detected object recognition range, centering on the range in the video where an "apple" was recognized at the time it was detected that the user had said "apple".

[0110] In Figure 13, the shooting range of the right camera 16R is shifted downward and to the right due to misalignment of the right earphone 10R or the shape of the user's ear. Therefore, the adjustment control unit 57A shifts the detection target recognition range R1 of the right camera 16R to a position centered on the range where the "apple" was recognized (upper right).

[0111] Figure 13 shows that the object recognition range R1, which was adjusted by the adjustment control unit 57A, was adjusted from its initial value, the object recognition range R1i, so that the object recognition range R2 is in the center of the object recognition range, because the voice recognition processing unit 52 recognized the sound "apple" and the name of the object in the object recognition range R2 in which "apple" was recognized matched.

[0112] Through this process, when a user requests information about an object, the position from which the user views the object is aligned with the position that the camera 16 is facing, particularly within the detection target recognition range. This ensures that the detected object is properly identified. In particular, when multiple objects are present, the object for which the user desires information is properly identified.

[0113] <Object Detection Method (Range Adjustment Process)> Next, the information processing in the object detection system 1A will be explained using Figure 14. Steps S403 to S405 of the flowchart shown in Figure 14 perform the same processing as steps S103 to S105 of the flowchart shown in Figure 7. When the earphones 10 are first used, or each time they are put on, the processing in the flowchart shown in Figure 14 is executed as a calibration process. While the processing in the flowchart shown in Figure 14 is being executed, the audio acquired by the microphone 17 and the video captured by the camera 16 are transmitted to the information terminal device 40. When calibration is started, the audio output unit 19 of the earphones 10 outputs an audio instruction, such as "Look straight at an easy-to-understand object and say its name."

[0114] The control unit 50A determines, using the object recognition processing unit 54, whether or not an object has been recognized from the video (step S401). The control unit 50A determines, using the object recognition processing unit 54, whether or not an object has been recognized from the video captured by the camera 16. If the control unit 50A determines, using the object recognition processing unit 54, that an object has been recognized from the video (step S401; Yes), it proceeds to step S402. If the control unit 50A determines, using the object recognition processing unit 54, that an object has not been recognized from the video (step S401; No), it terminates the processing of this flowchart. If, for example, no object is recognized for 30 seconds after the voice instruction to start calibration (S401; No), the processing of this flowchart is terminated.

[0115] If it is determined that an object has been recognized from the video (step S401; Yes), the control unit 50A acquires recognized object information via the information acquisition control unit 56 (step S402). The control unit 50A, via the information acquisition control unit 56, sends an instruction to the AI ​​server via the communication control unit 51 to acquire information about the object identified as a detection target along with the video captured by the camera 16. The control unit 50A, via the information acquisition control unit 56, acquires information about the object identified as a detection target from the AI ​​server via the communication control unit 51. The control unit 50A proceeds to step S403.

[0116] The control unit 50A adjusts the detection target recognition range with the adjustment control unit 57A (step S406). The control unit 50A adjusts the detection target recognition range with the adjustment control unit 57A using the center of the object recognition range in the image captured by the camera 16 as the reference point.

[0117] The order in which the processes in steps S401 and S402, and the processes in steps S403 and S404 are performed is not limited and they may be executed in parallel.

[0118] Steps S401 and S402 may be executed as a single step without being separated.

[0119] Before executing the process shown in the flowchart of Figure 14, a process may be performed to detect whether the earphone 10 is being worn in the ear.

[0120] <Effects> As described above, in this embodiment, when the name of an object identified from the spoken content matches the name of an object recognized by the object recognition processing unit 54, the range for recognizing the object can be adjusted based on the range in which the object is recognized in the camera's captured image. According to this embodiment, when a user requests information about an object, the position where the user views the object and the position the camera 16 is facing, particularly the position of the object recognition range, can be aligned. According to this embodiment, the object to be detected can be appropriately identified. According to this embodiment, when there are multiple objects, the object for which the user wants information can be appropriately identified. In this way, according to this embodiment, the object to be processed can be appropriately identified from the image of the camera 16 attached to the earphone 10.

[0121] In this embodiment, the central part of the area in the camera's captured image where an object is recognized can be used as a reference point to adjust the range for recognizing the object to be detected.

[0122] [Third Embodiment] The object detection system 1A according to this embodiment will be described with reference to Figures 15 and 16. Figure 15 is a diagram showing the detection target object recognition range and other examples of object recognition range in the image captured by the camera. Figure 16 is a flowchart showing the processing flow in the object detection system according to the third embodiment. In this embodiment, the processing of the adjustment control unit 57A of the control unit 50A differs from that of the second embodiment.

[0123] If the object recognized in the left image is the same object as the object recognized in the left image, and the name of the object identified from the speech content recognized by the speech recognition processing unit 52 matches the name of the object recognized by the object recognition processing unit 54, the adjustment control unit 57A adjusts the range for recognizing the target object based on the object recognition range in the image captured by the camera 16.

[0124] Figure 15 is an explanatory diagram showing the case where camera 16 is equipped with a right camera 16R and a left camera 16L.

[0125] Figure 15 shows that in the right image, when the voice recognition processing unit 52 recognizes the word "apple" and the name of the object in the object recognition range R2 where "apple" was recognized matches, the object recognition range R2 is included in the detection target recognition range R1, and therefore it is determined that no adjustment is needed for the detection target recognition range R1. In other words, the right earphone 10R is properly fitted. Also, in the left image, Figure 15 shows that when the voice recognition processing unit 52 recognizes the word "apple" and the name of the object in the object recognition range L2 where "apple" was recognized matches, the detection target recognition range L1 is adjusted from the initial value of the detection target recognition range L1i to the center of the detection target recognition range. In other words, due to misalignment of the left earphone 10L or the shape of the user's ear, the shooting range of the left camera 16L is shifted downward to the left.

[0126] <Object Detection Method (Range Adjustment Process)> Next, the information processing in the object detection system 1A will be explained using Figure 16. Steps S503 to S504 and S507 of the flowchart shown in Figure 16 perform the same processing as steps S403 to S404 and S406 of the flowchart shown in Figure 14.

[0127] The control unit 50A determines, based on the object recognition processing unit 54, whether or not an object is recognized in the left and right images (step S501). If the control unit 50A determines, based on the object recognition processing unit 54, that an object is recognized in both the left and right images (step S501; Yes), it proceeds to step S502. If the control unit 50A does not determine, based on the object recognition processing unit 54, that an object is recognized in both the left and right images, or if it does not determine that an object is recognized in either the left or right image (step S501; No), it terminates the processing of this flowchart.

[0128] If it is determined that an object has been recognized from both the left and right images (step S501; Yes), the control unit 50A, using the information acquisition control unit 56, acquires recognized object information for the object recognized from the left image and the object recognized from the right image (step S502). The control unit 50A then proceeds to step S503.

[0129] The control unit 50A determines, based on the adjustment control unit 57A, whether the object recognized in the left image and the object recognized in the right image are the same object (step S505). If the adjustment control unit 57A does not determine that the object recognized in the left image and the object recognized in the right image are the same object (step S505; No), it terminates the process of this flowchart. If the adjustment control unit 57A determines that the object recognized in the left image and the object recognized in the right image are the same object (step S505; Yes), it proceeds to step S506.

[0130] The control unit 50A, in accordance with the adjustment control unit 57A, determines whether the name of the object identified from the speech content recognized by the speech recognition processing unit 52 matches the name of the object recognized by the object recognition processing unit 54 from the left and right images (step S506). If the control unit 50A determines, in accordance with the adjustment control unit 57A, that the name of the object identified from the speech content recognized by the speech recognition processing unit 52 matches the name of the object recognized by the object recognition processing unit 54 from the left and right images (step S506; Yes), the process proceeds to step S507. If the control unit 50, in accordance with the adjustment control unit 57A, does not determine that the name of the object identified from the speech content recognized by the speech recognition processing unit 52 matches the name of the object recognized by the object recognition processing unit 54 from the left and right images (step S506; No), the process of this flowchart is terminated.

[0131] <Effects> As described above, in this embodiment, if the object recognized in the left image and the object recognized in the right image are the same object, and the name of the object identified from the speech content matches the name of the object recognized by the object recognition processing unit, the range for recognizing the target object can be adjusted based on the range in which the object is recognized in the image captured by the camera 16.

[0132] [Fourth Embodiment] The object detection system 1A according to this embodiment will be described with reference to Figures 17 and 18. Figure 17 is a flowchart showing the processing flow in the object detection system according to the fourth embodiment. Figure 18 is a flowchart showing the processing flow in the object detection system according to the fourth embodiment. In this embodiment, the processing of the adjustment control unit 57A of the control unit 50A differs from that of the second embodiment.

[0133] If the object recognized in the left image and the object recognized in the right image are the same object, the adjustment control unit 57A adjusts the range for recognizing the object based on the range in which the object is recognized in the image captured by the camera 16.

[0134] Figure 15 shows that the same object, "apple," is recognized in both the left and right images. Therefore, the object recognition range L and the object recognition range R are adjusted based on the object recognition range L2 and the object recognition range R2. The adjustment method can be the same as in the third embodiment.

[0135] <Object Detection Method (Range Adjustment Process)> Next, the object recognition process in the object detection system 1A will be explained using Figure 17. Steps S601, S602, S603, and S604 are the same as steps S501, S502, S505, and S507 in the flowchart shown in Figure 16.

[0136] <Object Detection Method (Object Recognition Processing)> Next, the object recognition processing in the object detection system 1A will be explained using Figure 18. Steps S701, S704 to S706 are the same as steps S301, S304 to S306 in the flowchart shown in Figure 10.

[0137] If the control unit 50A determines that it has received a voice command (step S701; Yes), the control unit 50A uses the object recognition processing unit 54 to recognize an object from the image captured by the camera 16 (step S702). The control unit 50A uses the object recognition processing unit 54 to send a command to the AI ​​server via the communication control unit 51 to perform object recognition processing along with the image captured by the camera 16. At this time, the image used for object recognition processing may be either the right image captured by the right camera 16R, or the left image captured by the left camera 16L, or both. The control unit 50A uses the object recognition processing unit 54 to obtain the AI ​​server recognition result via the communication control unit 51. The control unit 50A then proceeds to step S703.

[0138] The control unit 50A determines, based on the object recognition processing unit 54, whether or not there is an object recognized within the detection target object recognition range (step S703). If the control unit 50A determines, based on the object recognition processing unit 54, that there is an object recognized within the detection target object recognition range (step S703; Yes), the control unit 50A proceeds to step S704. If the control unit 50A does not determine, based on the object recognition processing unit 54, that there is an object recognized within the detection target object recognition range (step S703; No), the control unit 50A proceeds to step S706.

[0139] In this way, after calibration is complete, when the user wears the left earphone 10L and right earphone 10R that constitute the object detection system 1A, they can speak a trigger word to recognize an object detected in the center of the shooting range or object recognition range of the left camera 16L and right camera 16R, and output information about the object as audio data to the earphone 10.

[0140] If the camera 16 is provided only in the left earphone 10L or the right earphone 10R, or if the camera 16 is provided in both the left earphone 10L and the right earphone 10R, and one of the cameras 16 is configured to function as an object recognition camera, steps S702 and S703 in Figure 18 are reinterpreted as, for example, "object recognition from the right image" and "Is there an object recognized within the detection target object recognition range (of the right image)?" and processed accordingly.

[0141] If the left earphone 10L and the right earphone 10R are equipped with cameras 16, steps S702 and S703 in Figure 16 are reinterpreted as, for example, "object recognition from the left and right images" and "are there any (identical) objects recognized within the detection target object recognition range of the right and left images?" and processed accordingly.

[0142] <Effects> As described above, in this embodiment, when the object recognized in the left image and the object recognized in the right image are the same object, the range for recognizing the object can be adjusted based on the range in which the object is recognized in the image captured by the camera 16.

[0143] Each component of the illustrated object detection system is a functional concept and does not necessarily have to be physically configured as shown. In other words, the specific form of each device is not limited to that shown, and all or part of them may be functionally or physically distributed or integrated in any unit depending on the processing load and usage conditions of each device.

[0144] The object detection system is implemented, for example, as software, such as a program loaded into memory. In the above embodiment, these were described as functional blocks implemented through the cooperation of hardware or software. That is, these functional blocks can be implemented in various ways using hardware alone, software alone, or a combination thereof.

[0145] The components described above include those that are easily conceivable by those skilled in the art, and those that are substantially identical. Furthermore, the components described above can be combined as appropriate. In addition, various omissions, substitutions, or modifications of the components are possible without departing from the spirit of the present invention.

[0146] In the above embodiment, the touch operation unit 15 and the operation control unit 32 are not essential components.

[0147] In the above description, the earphone 10 was described as having a left earphone 10L and a right earphone 10R that can communicate with the information terminal device 40, but it is not limited to this. The earphone 10 may also be a master unit in which the left earphone 10L or the right earphone 10R can communicate with the portable electronic device 100, and a slave unit in which the right earphone 10R or the left earphone 10L can communicate with the left earphone 10L or the right earphone 10R.

[0148] In the above example, the control unit 50 of the information terminal device 40 implements the function of identifying the object to be detected, but this is not limited to this. For example, the control unit 30 of the earphone 10 may implement the function of identifying the object to be detected. In this case, the left earphone 10L or the right earphone 10R, which has the control unit 30 that implements the function of identifying the object to be detected, receives the video captured from the other right earphone 10R or left earphone 10L.

[0149] The object detection device and program of this disclosure can be used, for example, in an information terminal device connected to a camera-equipped device such as a camera-equipped earphone.

[0150] 1 Object detection system 10 Earphone (device with camera) 11 Housing 15 Touch operation unit 16 Camera 17 Microphone (sound acquisition unit) 18 Amplifier unit 19 Sound output unit (output unit) 27 Communication unit 28 Battery 29 Power supply unit 30 Control unit 32 Operation control unit 33 Shooting control unit 34 Sound input control unit 35 Sound output control unit (output control unit) 37 Communication control unit 39 Power supply control unit 40 Information terminal device (object detection device) 41 Communication unit 50 Control unit 51 Communication control unit 52 Speech recognition processing unit 53 Speech synthesis processing unit 54 Object recognition processing unit 55 Identification unit 56 Information acquisition control unit 58 Video selection processing unit

Claims

1. An object detection device connected to a camera-equipped device that, when worn on the user's head, is capable of capturing images in front of the user, comprising: an object recognition processing unit that recognizes an object from an image captured by the camera; a speech recognition processing unit that recognizes the content of speech from the user's speech; and an adjustment control unit that, when the name of an object identified from the speech content recognized by the speech recognition processing unit matches the name of an object recognized by the object recognition processing unit, adjusts the range for recognizing the object based on the range in which the object is recognized in the image captured by the camera.

2. The camera-equipped device comprises a left housing equipped with a left camera worn on the user's left ear to capture a left image, and a right housing equipped with a right camera worn on the user's right ear to capture a right image, wherein the adjustment control unit adjusts the range for recognizing the object to be detected based on the range in which the object is recognized in the image captured by the camera when the object recognized in the left image and the object recognized in the right image are the same object, and the name of the object identified from the speech content recognized by the speech recognition processing unit matches the name of the object recognized by the object recognition processing unit.

3. An object detection device connected to a camera-equipped device comprising a left camera worn on the user's left ear to capture a left image, and a right camera worn on the user's right ear to capture a right image, the object detection device comprising: an object recognition processing unit that recognizes an object from the image captured by the camera; and, when the object recognized in the left image and the object recognized in the right image are the same object, an adjustment control unit that adjusts the range for recognizing the object to be detected based on the range in which the object is recognized in the image captured by the camera.

4. The object detection device according to claim 1 or 3, wherein the adjustment control unit adjusts the range for recognizing the object to be detected, using the central part of the range in which the object is recognized in the image captured by the camera as the reference point.

5. The object detection device according to claim 1 or 3, further comprising: an identification unit that identifies an object recognized by the object recognition processing unit as a target object for detection; and an information acquisition control unit that acquires information relating to the target object for detection, wherein the information acquisition control unit acquires information relating to the object identified as a target object for detection and notifies the user of the acquired information.

6. A program that causes an object detection device connected to a camera-equipped device, which is mounted on the user's head and has a camera that can capture images in front of the user, to perform the following processes: a process to recognize an object from the image captured by the camera; a process to acquire the user's spoken voice and recognize the content of the speech; and, if the name of an object identified from the content of the speech recognized from the user's spoken voice matches the name of an object recognized from the image, a process to adjust the range for recognizing the object based on the range in which the object was recognized in the image captured by the camera.

7. A program that causes an object detection device connected to a camera-equipped device comprising a left camera worn on the user's left ear to capture a left image and a right camera worn on the user's right ear to capture a right image to perform the following: a process to recognize an object from the images captured by the cameras; and, if the object recognized in the left image and the object recognized in the right image are the same object, a process to adjust the range for recognizing the object to be detected based on the range in which the object is recognized in the camera's captured image.

Images