Information processing system and information processing method

Abstract

[Problem] To provide assistance to a user in a situation where visual recognition of an object is difficult. [Solution] This information processing system comprises a processing circuit that performs: processing for detecting an object on the basis of sensing data obtained by sensing the surroundings using a sensor unit provided in a device worn by a user; processing for generating auditory information on the basis of information on the detected object; and processing for performing control so as to notify the user of the generated auditory information through auditory stimulation.

Description

Information Processing System and Information Processing Method

[0001] The present disclosure relates to an information processing system and an information processing method.

[0002] In recent years, small devices such as smartphones have become popular.

[0003] In Patent Document 1 below, a data input device of a type worn on a finger is disclosed. The data input device described in Patent Document 1 detects the movement (relative movement direction) of a finger adjacent to the worn finger by a camera provided in the data input device, and assigns it to a general mouse operation, specifically, generates a pointer control signal.

[0004] Japanese Unexamined Patent Application Publication No. 2013-25484

[0005] However, the device described in Patent Document 1 is an operating device, and it is not considered from the viewpoint of a device useful in daily life that recognizes objects around the user and notifies the user.

[0006] Therefore, the present disclosure proposes an information processing system and an information processing method capable of assisting a user in a situation where it is difficult to visually recognize an object.

[0007] According to the present disclosure, there is provided an information processing system including a processing circuit that performs a process of detecting an object based on sensing data sensed by a sensor unit provided in a device worn by a user, a process of generating auditory information based on information of the detected object, and a process of controlling to notify the user of the generated auditory information by an auditory stimulus.

[0008] Further, according to the present disclosure, there is provided an information processing method including a processor detecting an object based on sensing data sensed by a sensor unit provided in a device worn by a user, generating auditory information based on information of the detected object, and controlling to notify the user of the generated auditory information by an auditory stimulus.

[0009] This figure shows the overall configuration of an information processing system 1 according to one embodiment of the present disclosure. This figure illustrates the acquisition of external information using a wearable device 10 according to this embodiment. This block diagram shows an example of the configuration of a wearable device 10 according to this embodiment. This block diagram shows an example of the configuration of an information processing terminal 20 according to this embodiment. This sequence diagram shows an example of the flow of object recognition processing by the information processing system 1 according to this embodiment. This block diagram shows an example of the configuration of a wearable device 10A according to this embodiment. This figure illustrates the identification of an object touched by user U. This figure illustrates the adjustment of the hand position for label recognition according to this embodiment. This figure illustrates the notification of label position according to this embodiment. This sequence diagram shows an example of guidance notification processing according to this embodiment. This figure illustrates the notification of distance to an object according to this embodiment. This sequence diagram shows an example of distance notification processing according to this embodiment. This sequence diagram shows an example of warning notification processing using temperature information according to this embodiment. This sequence diagram shows an example of operation processing of an information processing system 1 with function restriction capabilities according to this embodiment. This block diagram shows an example of the hardware configuration of an information processing device 900 according to one embodiment of the present disclosure.

[0010] Preferred embodiments of this disclosure will be described in detail below with reference to the attached drawings. In this specification and the drawings, components having substantially the same functional configuration are denoted by the same reference numerals, and redundant descriptions will be omitted.

[0011] Furthermore, the explanation will be given in the following order: 1. Configuration 1-1. System configuration 1-2. Configuration of the wearable device 10 1-3. Configuration of the information processing terminal 20 2. Operation processing 3. Specific examples 3-1. Object identification based on captured images 3-2. Guidance of hand position for correct recognition 3-3. Notification of distance to the object 3-4. Hazardous material determination based on temperature information 3-5. Functional limitations 4. Others 5. Hardware configuration 6. Supplementary information

[0012] <1. Configuration> <<1-1. System Configuration>> Figure 1 is a diagram showing the overall configuration of an information processing system 1 according to one embodiment of the present disclosure. The information processing system 1 according to this embodiment includes a wearable device 10, an information processing terminal 20, and a sound output device 30.

[0013] The wearable device 10 is a device that is worn on the user's body. In this embodiment, as an example, a ring-shaped device worn on the user's finger is assumed. However, the wearable device 10 is not limited to this, and may be a bracelet-type or watch-type device worn on the user's arm, or it may be a glasses-type, earring-type, necklace-type, etc. The ring-shaped device may be an annular shape, or it may be an open type shape with no connected rings.

[0014] The wearable device 10 is equipped with an external information acquisition unit 130 (i.e., a sensor unit) that is positioned outward (i.e., the side of the wearable device 10 that contacts the body is designated as the external information acquisition range). The external information acquisition unit 130 acquires surrounding information as external information and transmits the acquired information (i.e., sensed sensing data) to the information processing terminal 20. The external information includes information about the target object. Figure 2 is a diagram illustrating the acquisition of external information using the wearable device 10 according to this embodiment. As shown in Figure 2, when user U holds their hand over the target object 400, information about the target object 400 is acquired by the external information acquisition unit 130 provided on the wearable device 10 attached to the finger. Specifically, for example, visual information of the target object 400, i.e., an image, is acquired.

[0015] The information processing terminal 20 can be implemented as, for example, a smartphone, mobile phone, smartwatch, tablet, PC (Personal Computer), HMD (Head Mounted Display), etc. The information processing terminal 20 detects an object based on external information (sensing data) received from the wearable device 10. Furthermore, the information processing terminal 20 generates sound information (an example of auditory information) to inform the user of the detected object and controls the sound output device 30 to notify the user. Specifically, the information processing terminal 20 transmits the generated sound information to the sound output device 30. In this embodiment, the information processing terminal 20 performs object detection and sound information generation based on external information, but this disclosure is not limited to this. For example, object detection and sound information generation based on external information may be performed by a server on a network.

[0016] The sound output device 30 is a device that notifies the user of information through auditory stimulation. The sound output device 30 may be wireless or wired earphones, headphones, neck speakers, earring (ear cuff) type speakers, etc. The sound output device 30 outputs sound information received from the information processing terminal 20. In this embodiment, as an example, the case in which the sound output device 30 is separate from the information processing terminal 20 is described, but this disclosure is not limited thereto, and the sound output device 30 may be included in the information processing terminal 20.

[0017] As described above, in the information processing system 1 according to this embodiment, when a user holds the wearable device 10, which is attached to their finger or the like, over an object, the wearable device 10 acquires information about the object, and the user can acquire information about the object through auditory stimulation from the sound output device 30.

[0018] Generally, when a visually impaired user wants to recognize objects in their surroundings, they typically do so by feeling around to perceive the shape and texture of the object. However, there are cases where it is difficult to recognize an object by feeling around alone, or the object may be dangerous. Also, when they want to recognize an object in a dark place or a blind spot, they would normally need to touch the object and feel around to recognize it. In contrast, in this embodiment, information about an object can be acquired by holding a wearable device 10, which is attached to a finger or the like, over the object. The user can obtain information about an object without necessarily touching it. Furthermore, since the wearable device 10 used to acquire external information is attached to a finger or the like, it does not occupy the user's hands, which can be inconvenient.

[0019] The outline of the information processing system 1 according to this embodiment has been described above. Next, the configuration of the mounting device 10 and the information processing terminal 20 included in the information processing system 1 according to this embodiment will be described.

[0020] <<1-2. Configuration of the Wearable Device 10>> Figure 3 is a block diagram showing an example of the configuration of the wearable device 10 according to this embodiment. As shown in Figure 3, the wearable device 10 includes a communication unit 110, a control unit 120, an external information acquisition unit 130, a vibration unit 140, and a storage unit 150.

[0021] (Communication Unit 110) The communication unit 110 has a transmitting unit that transmits data to an external device and a receiving unit that receives data from an external device. The communication unit 110 according to this embodiment communicates with an external device or the Internet using, for example, a wired / wireless LAN (Local Area Network), Wi-Fi®, Bluetooth®, a mobile communication network, etc.

[0022] For example, the communication unit 110 can communicate with the information processing terminal 20 and send and receive data. The communication unit 110 transmits external information to the information processing terminal 20 according to the control of the control unit 120.

[0023] (Control Unit 120) The control unit 120 functions as an arithmetic processing unit and control unit, and controls the overall operation of the attached device 10 according to various programs. The control unit 120 is implemented by electronic circuits such as a CPU (Central Processing Unit) or a microprocessor. The control unit 120 may also include a ROM (Read Only Memory) for storing programs and calculation parameters to be used, and a RAM (Random Access Memory) for temporarily storing parameters that change as needed.

[0024] The control unit 120 controls the transmission of data (external information) input from the external information acquisition unit 130 to the information processing terminal 20 via the communication unit 110. The control unit 120 also controls the vibration unit 140 according to the vibration control signal received from the information processing terminal 20.

[0025] (External Information Acquisition Unit 130) The external information acquisition unit 130 acquires information about the surroundings and outputs it to the control unit 120 as external information. The external information acquisition unit 130 is implemented by various sensors. The external information may include information about the target object.

[0026] (Vibrating part 140) The vibrating part 140 is an example of a tactile presentation part that presents tactile stimuli. The vibrating part 140 presents vibration stimuli as tactile stimuli. The specific structure of the vibrating part 140 is not particularly limited, and various structures can be used.

[0027] (Storage Unit 150) The storage unit 150 is implemented by a storage medium that stores programs used in the processing of the control unit 120, calculation parameters, parameters that change as needed, etc.

[0028] Although the configuration of the attachment device 10 has been described in detail above, the configuration of the attachment device 10 according to this disclosure is not limited to the example shown in Figure 3. For example, the attachment device 10 may have a configuration that does not include the vibrating part 140.

[0029] <<1-3. Configuration of the Information Processing Terminal 20>> Figure 4 is a block diagram showing an example of the configuration of the information processing terminal 20 according to this embodiment. As shown in Figure 4, the information processing terminal 20 includes a communication unit 210, an operation display unit 220, a control unit 230, a storage unit 240, an audio input unit 250, and an audio output unit 260.

[0030] (Communication Unit 210) The communication unit 210 has a transmitting unit that transmits data to an external device and a receiving unit that receives data from the external device. The communication unit 210 according to this embodiment may communicate with an external device or the Internet using, for example, a wired or wireless LAN, Wi-Fi, Bluetooth, a mobile communication network, etc.

[0031] For example, the communication unit 210 receives external information from the wearable device 10. The communication unit 210 also transmits vibration control signals to the wearable device 10 and transmits sound information (e.g., voice signals) to the sound output device 30, in accordance with the control unit 230.

[0032] (Operation Display Unit 220) The operation display unit 220 has the function of an operation unit that accepts operation input from the user and the function of a display unit that displays various screens. For example, the operation display unit 220 may be implemented by a touch panel display. Alternatively, the operation unit function and the display unit function may be implemented separately, or an operation unit or display unit may be provided separately from the operation display unit 220. The operation unit may be implemented by, for example, a touch sensor, a switch, a button, etc. The display unit may be implemented by, for example, a display panel such as a liquid crystal display (LCD) or an organic EL (Electro Luminescence) display.

[0033] (Control Unit 230) The control unit 230 functions as an arithmetic processing unit and control unit, and controls the overall operation of the information processing terminal 20 according to various programs. The control unit 230 is implemented by an electronic circuit such as a CPU or microprocessor. The control unit 230 may also include a ROM for storing programs and calculation parameters to be used, and a RAM for temporarily storing parameters that change as needed.

[0034] Furthermore, the control unit 230 also functions as a detection processing unit 231, a recognition processing unit 232, and a notification control unit 233.

[0035] The detection processing unit 231 performs the process of detecting an object based on external information. Specifically, the detection processing unit 231 can perform object detection to detect the position and range of an object. The external information may be, for example, captured images (visible images, infrared images), depth information, and temperature information.

[0036] The recognition processing unit 232 performs processing to recognize detected objects based on external information. Specifically, the recognition processing unit 232 can perform object identification to determine what an object is (type, etc.). The algorithm for the object recognition process is not particularly limited, but for example, deep learning or machine learning may be used. The object recognition process may be performed by analyzing the captured image. The recognition processing unit 232 can also perform processing to recognize characters on the object. For example, the recognition processing unit 232 recognizes characters written on the object. The recognition processing unit 232 can also recognize (determine) whether an object is dangerous or not based on external information. For example, the recognition processing unit 232 may determine an object to be a hazardous material if it is something that may cause injury, such as a needle or a knife, or something that may cause burns due to its high temperature. The recognition processing unit 232 may also determine an object to be a hazardous material if it is something that may cause low-temperature burns if touched directly for a certain period of time. The recognition processing unit 232 may also determine an object to be a hazardous material if it is something that may cause frostbite, such as dry ice. Furthermore, the recognition processing unit 232 may determine whether or not an object should be avoided based on external information, not limited to hazardous materials.

[0037] The notification control unit 233 performs various controls to notify the user of information about the object. Information about the object includes at least one of the following: that the object has been detected (that the presence of some object has been confirmed), the object identification result (what the detected object is and the result of determining its type, or the result of determining if it is a hazardous material, etc.), or the result of recognizing characters on the object.

[0038] More specifically, the notification control unit 233 generates auditory information (e.g., sound information) or tactile information (e.g., vibration information) to notify the user of information about the object through auditory or tactile stimuli. Examples of auditory information include a notification sound indicating that an object has been detected, a warning sound indicating that the object is a hazardous material, a voice message conveying the object identification result, and a voice message reading out text on the object. Examples of vibration information include a vibration control signal indicating that an object has been detected, and a vibration control signal warning that the object is a hazardous material (e.g., a signal that controls the vibration to be stronger and at shorter intervals). The notification control unit 233 then controls the transmission of the auditory information to the sound output device 30 and the vibration information to the wearable device 10.

[0039] As a result, the information processing system 1 according to this embodiment makes it possible to notify the user of the presence of an object, etc., by sound or vibration.

[0040] (Storage Unit 240) The storage unit 240 is implemented by a storage medium that stores programs used in the processing of the control unit 230, calculation parameters, parameters that change as needed, etc.

[0041] (Voice input unit 250 and voice output unit 260) The voice input unit 250 collects the user's voice and outputs the voice data to the control unit 230. The voice output unit 260 reproduces the voice signal according to the control of the control unit 230. The voice input unit 250 may be implemented by a microphone provided on the information processing terminal 20. The voice output unit 260 may be implemented by a speaker, for example.

[0042] In the information processing system 1 according to this embodiment, information about the object is notified to the user by a sound output device 30 separate from the information processing terminal 20. However, this disclosure is not limited to this, and information about the object may also be notified to the user by the sound output device 30 of the information processing terminal 20.

[0043] As described above, the configuration of the information processing terminal 20 has been specifically described. However, the configuration of the information processing terminal 20 according to the present disclosure is not limited to the example shown in FIG. 4. For example, the information processing terminal 20 does not necessarily have all the configurations shown in FIG. 4. Further, the information processing terminal 20 may be realized by a plurality of devices. Also, at least a part of the functions of the information processing terminal 20 may be provided in a server on the network.

[0044] <2. Operational Processing> FIG. 5 is a sequence diagram showing an example of the flow of object recognition processing by the information processing system 1 of the present embodiment.

[0045] As shown in FIG. 5, first, the wearable device 10 acquires external information (step S103) and transmits the external information to the information processing terminal 20 (step S106).

[0046] Next, the information processing terminal 20 analyzes the external information to detect an object (step S109).

[0047] Next, the information processing terminal 20 recognizes the detected object based on the external information (step S112).

[0048] Next, the information processing terminal 20 generates information for notifying the user of the information of the object. Such information is vibration information or sound information.

[0049] Then, the information processing terminal 20 transmits vibration information (for example, a vibration control signal) to the wearable device 10 (step S118), and in the wearable device 10, a vibration output (presentation of a tactile stimulus) for notifying the user of the detection of the object is performed (step S121).

[0050] Also, the information processing terminal 20 transmits sound information (for example, an audio signal) to the sound output device 30 (step S124), and in the sound output device 30, a sound output (presentation of an auditory stimulus) for notifying the user of the detection and recognition results of the object is performed (step S127).

[0051] The flow of object recognition processing according to this embodiment has been described above. Note that the operation process shown in Figure 5 is just one example, and this disclosure is not limited thereto. For example, not all of the processes shown in Figure 5 are performed, and the order of the processes shown in Figure 5 may be different.

[0052] <3. Specific Examples> A more specific example of the configuration of the attachment device 10 will be explained. Figure 6 is a block diagram showing an example of the configuration of the attachment device 10A according to this embodiment.

[0053] As shown in Figure 6, the wearable device 10A includes a communication unit 110, a control unit 120, a camera 130a (an example of an imaging unit), a distance measuring sensor 130b, a temperature sensor 130c, an operation unit 160, an IMU (Inertial Measurement Unit) 162, a vibration unit 140, and a storage unit 150. Configurations that overlap with those described with reference to Figure 3 are omitted from this description.

[0054] Camera 130a, distance sensor 130b, and temperature sensor 130c are specific examples of the external information acquisition unit 130. Camera 130a is an imaging unit that acquires an image of an object as external information. Distance sensor 130b is a distance measuring unit that acquires information indicating the distance to an object as external information. Temperature sensor 130c is a temperature detection unit that acquires information indicating the temperature of an object as external information. The control unit 120 transmits the image, distance information, and temperature information to the information processing terminal 20 as external information.

[0055] The operation unit 160 receives operation input from the user to the wearable device 10. The operation unit 160 is implemented, for example, by a switch or a button. The control unit 120 can, for example, control the power ON / OFF of the wearable device 10 in response to the user's operation.

[0056] The IMU 162 detects information on the three-dimensional inertial motion of the wearable device 10 (three-axis acceleration information and three-axis angular velocity information). The IMU 162 is an example of a detection unit that detects the movement of the user's hand. The control unit 120 may control the power ON / OFF of the wearable device 10 according to the motion information detected by the IMU 162. The control unit 120 may also transmit the motion information detected by the IMU 162 to the information processing terminal 20. The control unit 120 may also switch the recognition mode of the object according to this motion information.

[0057] The configuration of the mounting device 10A described above is just one example, and the mounting device 10A does not have to have all the configurations shown in Figure 6. For example, the mounting device 10A may have at least one of the camera 130a, distance sensor 130b, and temperature sensor 130c as the external information acquisition unit 130. Alternatively, the mounting device 10A may have either the operation unit 160 or the IMU 162.

[0058] The following describes the operation of the information processing system 1 when using the attached device 10A, with several examples.

[0059] <<3-1. Identification of Objects Based on Captured Images>> Figure 7 is a diagram illustrating the identification of an object touched by user U. For example, user U, who has a visual impairment, can determine what an object 400 touched by hand is using the wearable device 10A according to this embodiment, which is attached to user U's finger. Specifically, user U first touches the object 400 with their hand to grasp its presence and location, as shown on the left of Figure 7. Then, as shown on the right of Figure 7, user U removes their hand from the object 400 and holds their hand over the object 400 so that the camera 130a of the wearable device 10A can capture an image of the object 400.

[0060] The mounting device 10A transmits the image of the object 400 captured by the camera 130a to the information processing terminal 20. The mounting device 10A may also be equipped with an illumination unit, such as an LED, near the camera 130a. This allows for imaging with the camera 130a even in dark surroundings.

[0061] The information processing terminal 20 performs image analysis on the captured image to detect the object 400 (object detection) and recognize the object 400 (object identification). In the recognition of the object 400, the type of object 400 is determined. In the example shown in Figure 7, the object 400 is determined to be a "bento box," and a voice message such as "This is a bento box." is played from the sound output device 30. The information processing terminal 20 may further recognize the contents of the bento box through image analysis and play a voice message such as "This is a bento box. It contains pork cutlet, white rice, cherry tomatoes, and potato salad." from the sound output device 30. In addition, the information processing terminal 20 may also recognize text images on the object 400, such as a label attached to the object 400, and play a voice message such as "Pork cutlet bento. 580 yen." from the sound output device 30.

[0062] Furthermore, the information processing terminal 20 may notify the user that the detection and recognition of the object 400 based on the captured image has been successful by vibrating the attached device 10A.

[0063] <<3-2. Guiding the Hand Position for Correct Recognition>> As described above, when recognizing the object 400 based on the image captured by the camera 130a, it is desirable that the object 400 is included in the field of view of the captured image. However, if the label attached to the object 400 is cut off at the edge of the field of view, the recognition of the label (specifically, character recognition) by the information processing terminal 20 may fail.

[0064] Figure 8 is a diagram illustrating the adjustment of the hand position for label recognition according to this embodiment. As shown on the left of Figure 8, when user U removes their hand from the object 400 and holds it over the object 400, and attempts to recognize the object 400 by imaging it with the camera 130a, which is mounted outward on the wearable device 10A, if the label 402 is not fully within the field of view of the camera 130a and is cut off, the information processing terminal 20 cannot recognize it correctly. In this case, as shown on the right of Figure 8, it is desirable to move the hand position so that the label 402 is within the field of view of the camera 130a.

[0065] However, for users with visual impairments, it is difficult to see the position of the label 402 and adjust the position of their hand. Therefore, the information processing system 1 according to this embodiment makes it possible to guide the user's hand position (i.e., the field of view of the camera 130a provided on the wearable device 10A) by sound or vibration. More specifically, the information processing system 1 can guide the user by increasing the sound or vibration as the center position of the field of view approaches the label 402.

[0066] Figure 9 is a diagram illustrating the notification of label position according to this embodiment. First, as shown on the left of Figure 9, if the label 402 is cut off at the edge of the field of view C1, character recognition of the label 402 by the information processing terminal 20 will fail. However, even though character recognition fails, the position of the label 402 (position in the field of view C1) can be determined by image analysis by the information processing terminal 20.

[0067] Next, as shown in the center of Figure 9, when the user moves their hand and the field of view C1 of the camera 130a changes, the information processing terminal 20 controls the sound emitted from the sound output device 30 and the vibrations presented by the wearable device 10A to become stronger as the center position of the field of view C1 approaches the position of the label 402, and to become weaker as it moves away. This allows visually impaired users to be notified of the position of the label 402 and to be guided to adjust their hand position correctly (to move closer to the label 402).

[0068] Then, as shown in Figure 9 (right), when label 402 enters the field of view C1, the information processing terminal 20 successfully recognizes the characters of label 402, and the characters of label 402 are read aloud.

[0069] Figure 10 is a sequence diagram showing an example of guidance notification processing according to this embodiment. As shown in Figure 10, first, the attached device 10A captures an image of the target object with the camera 130a (step S203), and transmits the captured image to the information processing terminal 20 (step S206).

[0070] Next, the information processing terminal 20 analyzes the captured image to detect the target object (step S209).

[0071] Next, the information processing terminal 20 recognizes the detected object based on the captured image (step S212). This recognition also includes character recognition of the label attached to the object.

[0072] Next, the information processing terminal 20 generates guidance notification information corresponding to the positional shift of the label attached to the object within the field of view of the captured image (step S215). The guidance notification information may be vibration information or sound information. For example, the information processing terminal 20 can generate guidance notification information such that the intensity of the vibration or sound increases as the center position of the field of view of the captured image approaches the label, and decreases as it moves away.

[0073] Then, the information processing terminal 20 transmits vibration information (for example, a vibration control signal) to the wearable device 10A (step S218), and the wearable device 10A outputs a vibration (presentation of tactile stimulation) to inform the user of the position of the label (near or far) (step S221).

[0074] Furthermore, the information processing terminal 20 transmits sound information (e.g., an audio signal) to the sound output device 30 (step S224), and the sound output device 30 outputs sound (presents an auditory stimulus) to inform the user of the location of the label (near or far) (step S227).

[0075] The processes described in steps S203 to S227 above are performed continuously and repeatedly. That is, the captured image is continuously transmitted from the attached device 10A to the information processing terminal 20, the information processing terminal 20 analyzes the captured image in real time, continuously generates guidance notification information according to the position of the label within the field of view, and outputs vibration or sound in real time. As a result, the user can move their hand by relying on sound or vibration to bring the field of view of the camera 130a closer to the label (to a position where the label fits within the field of view), so that the information processing terminal 20 can correctly recognize (the character).

[0076] <<3-3. Notification of Distance to Object>> In the information processing system 1 according to this embodiment, the recognition result of the object may be notified before the user U touches the object. If the object is one that would cause injury if touched, the user can avoid touching the object. In this case, the information processing terminal 20 can acquire the distance to the object using the distance measuring sensor 130b provided on the wearable device 10A, and control the output of sound or vibration according to the distance to the object to notify the user U that their hand is approaching the object.

[0077] Figure 11 is a diagram illustrating the notification of distance to an object according to this embodiment. As shown on the left of Figure 11, when user U brings their hand close to the object 450, the distance to the object 450 is sensed by the distance measuring sensor 130b provided on the wearable device 10A. The distance information is transmitted to the information processing terminal 20, which controls the sound output device 30 to become louder and the vibration of the vibration unit 140 provided on the wearable device 10A to become stronger as the user gets closer to the object 450. This notifies the user that their hand is getting closer to the object 450.

[0078] Furthermore, the attached device 10A also transmits the image of the object 450 captured by the camera 130a to the information processing terminal 20. The information processing terminal 20 recognizes the object 450 based on the image and notifies the user U of the recognition result by voice. This allows the user to know, as shown in Figure 11 (right), that the object 450 is, for example, a "thumbtack" or something that could cause injury if touched, before touching it.

[0079] Figure 12 is a sequence diagram showing an example of distance notification processing according to this embodiment. As shown in Figure 12, first, the attached device 10A captures an image of the target object with the camera 130a (step S303), and transmits the captured image to the information processing terminal 20 (step S306).

[0080] Furthermore, the attached device 10A senses the distance to the object using the distance measuring sensor 130b (step S309) and transmits distance information indicating the distance to the object to the information processing terminal 20 (step S312).

[0081] Next, the information processing terminal 20 generates notification information indicating the distance to the object based on the distance information (step S315). For example, the information processing terminal 20 generates notification information that increases the intensity of vibration or sound as the distance to the object decreases.

[0082] Next, the information processing terminal 20 transmits vibration information (for example, a vibration control signal) to the wearable device 10A (step S318), and the wearable device 10A outputs vibrations that become stronger (and more frequently) as it gets closer to the object (step S321).

[0083] Furthermore, the information processing terminal 20 transmits sound information (for example, an audio signal) to the sound output device 30 (step S324), and the sound output device 30 outputs sound that gets louder as it approaches the object (it may be a warning sound) to inform the user of the distance (step S327).

[0084] Meanwhile, the information processing terminal 20 detects and recognizes an object based on the captured image received from the attached device 10A (step S330).

[0085] Next, the information processing terminal 20 generates audio information to notify the recognition result of what the object is (step S333), and transmits the audio information to the sound output device 30 (step S336).

[0086] Then, the sound output device 30 outputs a sound to inform the user what the object is (step S339).

[0087] The processes shown in Figure 12 described above do not necessarily have to be performed in the order shown in Figure 12. For example, steps S303 to S306 and steps S309 to S312 may be performed in parallel or in reverse order. Also, the detection and recognition of the object shown in step S330 may be performed in parallel or in reverse order with the generation of notification information shown in S315.

[0088] <<3-4. Hazardous Material Identification Based on Temperature Information>> Referring to Figures 11 and 12, the above example explained that it is possible to avoid objects that could cause injury if touched, such as thumbtacks. Such objects are examples of hazardous materials, but there are also objects that cannot be determined as hazardous or not by image recognition alone. For example, a heated frying pan is a hazardous material that could cause burns if touched, but it is not possible to determine whether it is hot or not by image recognition alone. Therefore, in this embodiment, the temperature of the object can be sensed by the temperature sensor 130c and appropriate notification can be given to the user.

[0089] Figure 13 is a sequence diagram showing an example of a warning notification process using temperature information according to this embodiment. As shown in Figure 13, first, the attached device 10A captures an image of the target object with the camera 130a (step S403), and transmits the captured image to the information processing terminal 20 (step S406).

[0090] Furthermore, the attached device 10A senses the distance to the object using the distance measuring sensor 130b (step S409) and transmits distance information indicating the distance to the object to the information processing terminal 20 (step S412).

[0091] Furthermore, the attached device 10A senses the temperature of the object using the temperature sensor 130c (step S415) and transmits temperature information indicating the temperature of the object to the information processing terminal 20 (step S418).

[0092] Next, the information processing terminal 20 generates notification information indicating the distance to the object based on the distance information (step S421). The information processing terminal 20 generates notification information that increases the intensity of vibration or sound as the distance to the object decreases.

[0093] Next, the information processing terminal 20 transmits vibration information (for example, a vibration control signal) to the wearable device 10A (step S424), and the wearable device 10A outputs vibrations that become stronger (and more frequently) as it gets closer to the object (step S427).

[0094] Furthermore, the information processing terminal 20 transmits sound information (for example, an audio signal) to the sound output device 30 (step S430), and the sound output device 30 emits sound that becomes louder as the user approaches the object (it may be a warning sound) to inform the user of the distance (step S433). This allows the user to understand the distance to the object and avoid touching it unnecessarily.

[0095] Meanwhile, the information processing terminal 20 detects and recognizes an object based on the captured image received from the attached device 10A (step S436).

[0096] Next, the information processing terminal 20 refers to the temperature information, and if the object is hot (if the object's temperature exceeds a threshold), it generates a warning notification voice information in addition to the recognition result (step S439) and transmits the voice information to the sound output device 30 (step S442). Here, "if the object is hot" is used assuming an object that may cause high-temperature burns, but this embodiment is not limited to this, and the information processing terminal 20 may also generate a warning notification voice information and transmit the voice information to the sound output device 30 if the object is at a temperature that may cause low-temperature burns or frostbite (i.e., if the object's temperature meets the respective conditions (predetermined temperature)).

[0097] The sound output device 30 then outputs a warning sound and an audio message informing the user of what the object is (step S445). For example, the sound output device 30 outputs an audio message such as, "Beep beep beep! It's a hot frying pan!"

[0098] The processes shown in Figure 13 described above do not necessarily have to be performed in the order shown in Figure 13. For example, steps S403 to S406, steps S409 to S412, and steps S415 to S418 may be performed in parallel or in a different order. Also, the detection and recognition of the object shown in step S436 may be performed in parallel or in the reverse order of the notification information generation shown in S421.

[0099] <<3-5. Functional Limitations>> Each device of the information processing system 1 according to this embodiment can be functionally limited as appropriate according to user operation. For example, by allowing the power of the attached device 10A to be switched ON / OFF according to user operation, the battery drain can be reduced. Also, by allowing the recognition mode (object identification, text reading) for recognizing objects in the information processing terminal 20 to be switched according to user operation, the user can obtain the information they desire as appropriate, improving convenience.

[0100] Figure 14 is a sequence diagram showing an example of the operation process of the information processing system 1, which allows for functional restriction according to this embodiment. As shown in Figure 14, first, the power of the wearable device 10A is turned ON in response to user operation (step S503). The user operation may be directed to an operation unit 160 such as a button or switch provided on the wearable device 10A, or it may be a predetermined movement of the finger or hand wearing the wearable device 10A. The wearable device 10A can detect the movement of the finger or hand using the IMU 162.

[0101] Next, the attached device 10A captures an image of the target object with the camera 130a (step S506) and transmits the captured image to the information processing terminal 20 (step S509). The attached device 10A may also transmit distance information to the target object and temperature information of the target object to the information processing terminal 20.

[0102] Next, the information processing terminal 20 analyzes the captured image to detect the target object (step S512).

[0103] Next, the information processing terminal 20 recognizes the detected object based on the captured image (step S515). There are two modes for object recognition: a first mode for identifying the object itself, and a second mode for recognizing characters on the object (such as characters on a label attached to the object). It is assumed that the initial setting is the first mode.

[0104] Next, the information processing terminal 20 transmits a vibration control signal to the wearable device 10A to notify the user that the object has been detected and recognized (step S518), and vibration is output from the wearable device 10A (step S521).

[0105] Next, the information processing terminal 20 generates voice information that notifies the user of the object identification result (i.e., the type of object, etc., what the object is) (step S524), and transmits the voice information to the sound output device 30 (step S527). As a result, the object identification result is output as voice from the sound output device 30 and notified to the user (step S530).

[0106] Next, when the wearable device 10A detects a user switching the recognition mode (step S536), it transmits the operation information to the information processing terminal 20 (step S539). The recognition mode switching operation may be, for example, an operation on the operation unit 160 (such as pressing a mode switching button), or a predetermined gesture made by the hand or fingers detected by the IMU 162.

[0107] Next, the information processing terminal 20 switches the recognition mode (step S542) in accordance with the operation information (recognition mode switching operation information) received from the attached device 10A, and performs recognition in the second mode. Specifically, the information processing terminal 20 recognizes characters on the object (for example, characters on a label attached to the object) based on the captured image, and generates spoken audio information for those characters (step S545).

[0108] Then, the information processing terminal 20 transmits the voice information to the sound output device 30 (step S548), and the sound output device 30 performs voice output (step S551).

[0109] <4. Others> In the embodiments described above, a vibration unit 140 was used as an example of a tactile presentation unit that provides tactile feedback, but this disclosure is not limited thereto. For example, a temperature-sensing presentation unit, an electrical stimulation unit, or an odor-generating unit may be used to notify the user of information about the object. The temperature-sensing presentation unit, electrical stimulation unit, or odor-generating unit may be provided on the wearable device 10 (or wearable device 10A) or the information processing terminal 20, or it may be provided as a separate unit. In addition, information about the object may be notified to the user as taste information using a spoon-shaped device or the like that has a taste presentation function.

[0110] Furthermore, the wearable device 10 (or wearable device 10A) may be equipped with a speaker (audio output unit), and the wearable device 10 (or wearable device 10A) may output sound information to notify the user of information about the object. For example, when the user holds the wearable device 10 over an object, the user is notified by vibration that the detection and recognition of the object has been successful, and in response, the user brings the hand wearing the wearable device 10 to their ear and hears information about the object audibly from the speaker of the wearable device 10.

[0111] <5. Hardware Configuration> An embodiment of the present disclosure has been described above. Next, with reference to Figure 15, an example of a hardware configuration used in the mounting device 10, 10A, or information processing terminal 20 according to an embodiment of the present disclosure will be described.

[0112] Figure 15 is a block diagram showing an example of the hardware configuration of an information processing device 900 according to one embodiment of the present disclosure. The information processing device 900 is an example of a hardware configuration applied to the mounting devices 10, 10A, or information processing terminal 20 according to this embodiment. Note that the information processing device 900 does not necessarily have all of the hardware configurations shown in Figure 15.

[0113] As shown in Figure 15, the information processing device 900 includes a processing circuit 901, a ROM (Read Only Memory) 902, and a RAM (Random Access Memory) 903. The information processing device 900 may also include a host bus 907, a bridge 909, an external bus 911, an interface 913, an input device 915, an output device 917, a storage device 919, a drive 921, a connection port 923, and a communication device 925.

[0114] The processing circuit 901 functions as an arithmetic processing unit and control unit, and controls the overall operation or a part of the operation within the information processing unit 900 according to various programs recorded in the ROM 902, RAM 903, storage device 919, or removable recording medium 927. The ROM 902 stores programs and calculation parameters used by the processing circuit 901. The RAM 903 temporarily stores programs used in the execution of the processing circuit 901 and parameters that change as appropriate during its execution. The processing circuit 901, ROM 902, and RAM 903 are interconnected by a host bus 907, which is composed of an internal bus. Furthermore, the host bus 907 is connected to an external bus 911, such as a PCI (Peripheral Component Interconnect / Interface) bus, via a bridge 909.

[0115] The input device 915 is a device operated by the user, such as a button. The input device 915 may also include a mouse, keyboard, touch panel, switch, and lever. The input device 915 may also include a microphone that detects the user's voice. The input device 915 may be, for example, a remote control device that uses infrared or other radio waves, or an external connection device 929 such as a mobile phone that is compatible with the operation of the information processing device 900. The input device 915 includes an input control circuit that generates an input signal based on information input by the user and outputs it to the processing circuit 901. By operating this input device 915, the user inputs various data to the information processing device 900 or instructs it to perform processing operations.

[0116] The input device 915 may also include an imaging device and sensors. The imaging device is a device that captures real space and generates an image using various components such as an image sensor, such as a CCD (Charge Coupled Device) image sensor or a CMOS (Complementary Metal Oxide Semiconductor) image sensor, and a lens for controlling the imaging of a subject onto the image sensor. The imaging device may capture still images or motion images. The sensors are various types of sensors, such as distance sensors, acceleration sensors, gyro sensors, geomagnetic sensors, vibration sensors, light sensors, and sound sensors. The sensors acquire information about the state of the information processing device 900 itself, such as the orientation of the housing of the information processing device 900, and information about the surrounding environment of the information processing device 900, such as the brightness and noise around the information processing device 900. The sensors may also include a GPS sensor that receives GPS (Global Positioning System) signals and measures the latitude, longitude, and altitude of the device.

[0117] The output device 917 is comprised of a device capable of visually or audibly notifying the user of the acquired information. The output device 917 may be, for example, a display device such as an LCD (Liquid Crystal Display) or an organic EL (Electro-Luminescence) display, or an audio output device such as a speaker or headphones. The output device 917 may also include a PDP (Plasma Display Panel), a projector, a hologram, a printer, etc. The output device 917 outputs the results obtained from the processing of the information processing device 900 as text or images, or as sound such as voice or acoustics. The output device 917 may also include a lighting device that brightens the surroundings.

[0118] The storage device 919 is a data storage device configured as an example of the storage unit of the information processing device 900. The storage device 919 is composed of, for example, a magnetic storage device such as an HDD (Hard Disk Drive), a semiconductor storage device, an optical storage device, or a magneto-optical storage device. This storage device 919 stores programs and various data executed by the processing circuit 901, as well as various data acquired from external sources.

[0119] The drive 921 is a reader / writer for removable recording media 927 such as magnetic disks, optical disks, magneto-optical disks, or semiconductor memory, and is either built into or external to the information processing device 900. The drive 921 reads information recorded on the installed removable recording media 927 and outputs it to the RAM 905. The drive 921 also writes data to the installed removable recording media 927.

[0120] The connection port 923 is a port for directly connecting equipment to the information processing device 900. The connection port 923 may be, for example, a USB (Universal Serial Bus) port, an IEEE 1394 port, or a SCSI (Small Computer System Interface) port. Alternatively, the connection port 923 may be an RS-232C port, an optical audio terminal, or an HDMI (High-Definition Multimedia Interface) port. By connecting an external device 929 to the connection port 923, various types of data can be exchanged between the information processing device 900 and the external device 929.

[0121] The communication device 925 is a communication interface, for example, consisting of a communication device for connecting to an external network 50. The communication device 925 may be, for example, a communication card for wired or wireless LAN (Local Area Network), Bluetooth®, Wi-Fi®, or WUSB (Wireless USB). Alternatively, the communication device 925 may be a router for optical communication, a router for ADSL (Asymmetric Digital Subscriber Line), or a modem for various types of communication. The communication device 925 transmits and receives signals, for example, to the Internet or other communication devices using a predetermined protocol such as TCP / IP. The external network 50 connected to the communication device 925 is a network connected by wire or wireless, for example, the Internet, a home LAN, infrared communication, radio wave communication, or satellite communication.

[0122] For example, when the information processing device 900 functions as a mounting device 10, 10A, or information processing terminal 20 according to the embodiment of this disclosure, the processing circuit 901 of the information processing device 900 functions as a control unit 120 or control unit 230 by executing a program loaded on the RAM 903. The storage device 919 stores the information processing program according to this disclosure and various data stored in the storage unit 150 or storage unit 240.

[0123] The processing circuit 901 reads and executes program data from the storage device 919, but as an alternative, these programs may be obtained from other devices via the external network 50. In other words, the storage device 919 is not limited to being inside the information processing device 900, but may be located outside the information processing device 900.

[0124] The processing circuit 901 is an example of an integrated circuit, and CPU (Central Processing Unit), MPU (Micro Processing Unit), GPU (Graphics Processing Unit), APU (Accelerated Processing Unit), ASIC (Application Specific Integrated Circuit), and FPGA (Field Programmable Gate Array) can all be considered integrated circuits.

[0125] Furthermore, when the information processing device 900 functions as a mountable device 10, 10A, or information processing terminal 20 according to the embodiments of this disclosure, the communication device 925 corresponds to the communication unit 110 or the communication unit 210. Also, the input device 915 and the output device 917 correspond to the operation display unit 220. The input device 915 also corresponds to the external information acquisition unit 130, the camera 130a, the distance measuring sensor 130b, the temperature sensor 130c, the operation unit 160, the IMU 162, or the voice input unit 250. The output device 917 also corresponds to the vibration unit 140 or the voice output unit 260.

[0126] <6. Supplementary Information> Although preferred embodiments of the present disclosure have been described in detail above with reference to the attached drawings, the present technology is not limited to such examples. It is clear that a person with ordinary skill in the art of the present disclosure may conceive of various modifications or alterations within the scope of the technical idea described in the claims, and these will naturally be understood to fall within the technical scope of the present disclosure.

[0127] Furthermore, the components of each illustrated device are functionally conceptual and do not necessarily need to be physically configured as shown. In other words, the specific forms of distribution and integration of each device are not limited to those shown, and all or part of them can be functionally or physically distributed and integrated in any unit according to various loads and usage conditions.

[0128] The information processing system according to this disclosure may consist of a single device consisting of wearable devices 10, 10A, an information processing terminal 20, or a server (not shown). That is, for example, the wearable devices 10, 10A may perform object detection, recognition, and generation of notification information that notifies information about the object. Furthermore, the information processing system according to this disclosure may consist of multiple devices. The multiple devices may be, for example, wearable devices 10, 10A and a server, or wearable devices 10, 10A and a smartphone, mobile phone terminal, tablet terminal, PC, etc. (information processing terminal 20), or wearable devices 10, 10A, an information processing terminal 20 and a server.

[0129] Furthermore, the embodiments and modifications of this disclosure described above can be combined as appropriate in areas where the processing content is not contradictory. Also, the order of each step shown in the sequence diagram or flowchart of this embodiment can be changed as appropriate. For example, each step may be processed chronologically, repeatedly, or partially in parallel.

[0130] Furthermore, one or more computer programs can be created to enable the functions of the mounting device 10, 10A, or the information processing terminal 20 to be performed on the CPU, ROM, RAM, and other hardware built into the mounting device 10, 10A, or the information processing terminal 20. A computer-readable storage medium storing such one or more computer programs is also provided.

[0131] Furthermore, the effects described herein are merely descriptive or illustrative and not limiting. In other words, the technology relating to this disclosure may produce other effects that will be apparent to those skilled in the art from the description herein, in addition to or in lieu of the effects described herein.

[0132] Furthermore, this technology can also be configured as follows: (1) An information processing system comprising a processing circuit that performs: a process of detecting an object based on sensing data obtained by sensing the surroundings with a sensor unit provided on a device worn by a user; a process of generating auditory information based on the information of the detected object; and a process of controlling the generated auditory information to be notified to the user by auditory stimulation. (2) The information processing system according to (1), wherein the information of the object is that the object has been detected, the identification result of the object, or the recognition result of characters on the object. (3) The information processing system according to (2), wherein the processing circuit generates sound information notifying that the object has been detected, voice information informing the identification result of the object, and voice information reading aloud characters on the object as the auditory information, and controls the sound output. (4) The information processing system according to any one of (1) to (3), wherein the sensor unit is an imaging unit, the imaging unit acquires an image as sensing data, and the processing circuit analyzes the image to detect and recognize the object. (5) The information processing system according to (4), wherein the recognition of the object includes the identification of the object and the recognition of characters on the object. (6) The information processing system according to any one of (1) to (5), wherein the sensor unit is provided outward in the device. (7) The information processing system according to (1), wherein the sensor unit is a distance measuring unit that senses the distance to the object, and the information of the object includes distance information. (8) The information processing system according to (7), wherein the processing circuit generates and outputs sound information or tactile information that notifies that it is close to the object based on the distance information. (9) The information processing system according to (1), wherein the sensor unit is a temperature detection unit that senses the temperature of the object, the information of the object includes temperature information, and the processing circuit is controlled to generate and output sound information or tactile information that warns that the object is at a temperature that may cause burns or frostbite based on the temperature information.(10) The information processing system according to (1), wherein the sensor unit is an imaging unit, and the processing circuit controls the presentation of a sound or vibration with a faster pitch the closer the center position of the field of view is to the position of the label, according to the position of the label attached to the object in the field of view of the image of the object. (11) The information processing system according to any one of (1) to (10), wherein the information processing system includes a sound output device, and the processing circuit performs a process of transmitting the sound signal generated as auditory information to the sound output device. (12) The information processing system according to any one of (1) to (11), wherein the device is provided with a tactile presentation unit, and the processing circuit further generates tactile information based on the information of the object and controls the device to perform tactile presentation. (13) The information processing system according to any one of (1) to (12), wherein the recognition of the detected object based on the sensing data includes a first recognition mode for identifying the object and a second recognition mode for recognizing characters on the object, and the processing circuit switches between the first and second recognition modes in response to user operation. (14) An information processing method comprising: a processor detecting an object based on sensing data obtained by sensing the surroundings with a sensor unit provided on a device worn by a user; generating auditory information based on the information of the detected object; and controlling the processor to notify the user of the generated auditory information by auditory stimulation. (15) The information processing method according to (14), wherein the information of the object is that the object has been detected, the result of identifying the object, or the result of recognizing characters on the object. (16) The information processing method according to (15), wherein the processor generates sound information that notifies that the object has been detected, voice information that notifies the result of the identification of the object, and voice information that reads out the characters on the object, and controls the sound output as auditory information.(17) The information processing method according to any one of (14) to (16), wherein the sensor unit is an imaging unit, the imaging unit acquires an image as sensing data, and the processor analyzes the image to detect and recognize the object. (18) The information processing method according to any one of (14) to (17), wherein the sensor unit is provided facing outward in the device.

[0133] 1. Information Processing System 10, 10A Attached Device 110 Communication Unit 120 Control Unit 130 External Information Acquisition Unit 140 Vibration Unit 150 Storage Unit 20 Information Processing Terminal 210 Communication Unit 220 Operation Display Unit 230 Control Unit 231 Detection Processing Unit 232 Recognition Processing Unit 233 Notification Control Unit 240 Storage Unit 30 Sound Output Device

Claims

1. An information processing system comprising a processing circuit that performs the following: a process of detecting an object based on sensing data obtained by sensing the surroundings using a sensor unit provided on a device worn by a user; a process of generating auditory information based on the information of the detected object; and a process of controlling the system to notify the user of the generated auditory information through auditory stimuli.

2. The information processing system according to claim 1, wherein the information of the object is that the object has been detected, the result of identifying the object, or the result of recognizing characters on the object.

3. The information processing system according to claim 2, wherein the processing circuit generates sound information that notifies that the object has been detected, voice information that notifies the identification result of the object, and voice information that reads out the characters on the object, and controls the sound output as auditory information.

4. The information processing system according to claim 1, wherein the sensor unit is an imaging unit, the imaging unit acquires an image as sensing data, and the processing circuit analyzes the image to detect and recognize the object.

5. The information processing system according to claim 4, wherein the recognition of the object includes the identification of the object and the recognition of characters on the object.

6. The information processing system according to claim 1, wherein the sensor portion is provided facing outward in the device.

7. The information processing system according to claim 1, wherein the sensor unit is a distance measuring unit that senses the distance to the object, and the information of the object includes distance information.

8. The information processing system according to claim 7, wherein the processing circuit is controlled to generate and output sound information or tactile information that indicates proximity to the object, based on the distance information.

9. The information processing system according to claim 1, wherein the sensor unit is a temperature detection unit that senses the temperature of the object, the information of the object includes temperature information, and the processing circuit is controlled to generate and output sound information or tactile information that warns that the object is at a temperature that may cause burns or frostbite, based on the temperature information.

10. The information processing system according to claim 1, wherein the sensor unit is an imaging unit, and the processing circuit controls the presentation of a sound or vibration with a faster pitch the closer the center position of the field of view is to the position of the label, according to the position of the label attached to the object in the field of view of the image of the object captured by the object.

11. The information processing system according to claim 1, wherein the information processing system includes a sound output device, and the processing circuit performs a process of transmitting the sound signal generated as auditory information to the sound output device.

12. The information processing system according to claim 1, wherein the device is provided with a tactile presentation unit, and the processing circuit generates further tactile information based on the information of the object and controls the device to provide tactile presentation.

13. The information processing system according to claim 1, wherein the recognition of the detected object based on the sensing data includes a first recognition mode for identifying the object and a second recognition mode for recognizing characters on the object, and the processing circuit switches between the first and second recognition modes in response to user operation.

14. An information processing method comprising: a processor detecting an object based on sensing data obtained by a sensor unit provided on a device worn by a user sensing the surroundings; generating auditory information based on the information of the detected object; and controlling the processor to notify the user of the generated auditory information by auditory stimulation.

15. The information processing method according to claim 14, wherein the information of the object is that the object has been detected, the identification result of the object, or the recognition result of characters on the object.

16. The information processing method according to claim 15, wherein the processor generates sound information that notifies that the object has been detected, voice information that notifies the identification result of the object, and voice information that reads out the characters on the object, and controls the sound output as auditory information.

17. The information processing method according to claim 14, wherein the sensor unit is an imaging unit, the imaging unit acquires an image as sensing data, and the processor analyzes the image to detect and recognize the object.

18. The information processing method according to claim 14, wherein the sensor portion is provided facing outward in the device.

Images