Information processing system and information processing method

Abstract

An information processing system according to the present invention comprises a notification data generation unit. The notification data generation unit selects a notification means for a user on the basis of a direction in which the user is interested, a surrounding environment of the user, and a holding state of a controller by the user.

Description

Information Processing System and Information Processing Method

[0001] The present invention relates to an information processing system and an information processing method.

[0002] It has been proposed to use a moving body such as a drone in various fields such as aerial photography, surveying, inspection, logistics, disaster prevention, security, and pesticide spraying. The moving state and warnings of the moving body are mainly notified using the GUI or voice of the controller.

[0003] Japanese Patent Application Laid-Open No. 2022-012261

[0004] When remotely operating a moving body, the user often not only watches the controller but also the moving body. There is a possibility of overlooking the notification even if it is displayed on the controller. Also, when using the moving body under noisy conditions, the notification may not be heard even if it is made by voice. Although notifications can be made using the LED lights mounted on the moving body, complex notifications cannot be made due to limitations in expression.

[0005] Therefore, the present disclosure proposes an information processing system and an information processing method that can more reliably cause the user to recognize notifications.

[0006] According to the present disclosure, there is provided an information processing system having a notification data generation unit that selects a notification means for the user based on the direction in which the user is interested, the surrounding environment of the user, and the holding state of the controller by the user. Also, according to the present disclosure, there is provided an information processing method in which the information processing of the information processing system is executed by a computer.

[0007] It is a diagram showing an example of a notification mode of notification information. It is a diagram showing an example of a notification priority table. It is a diagram showing an example of a user management table for managing responsiveness. It is a diagram showing an example of the configuration of an information processing system. It is a diagram showing an example of a processing flow in notification control. It is a diagram showing an example of a processing flow in notification control. It is a diagram showing an example of a processing flow in notification control. It is a diagram showing an example of a processing flow in notification control. It is a diagram showing an example of a processing flow in notification control. It is a diagram showing an example of a hardware configuration example of an information processing system.

[0008] Embodiments of the present disclosure will be described in detail below with reference to the drawings. In each of the following embodiments, the same parts will be denoted by the same reference numerals, and redundant descriptions will be omitted.

[0009] The explanation will proceed in the following order: [1. Notification methods for notification information] [2. Notification priority table] [3. User management table] [4. Example configuration of the information processing system] [5. Processing flow] [6. Example hardware configuration]

[0010] [1. Notification Methods for Notification Information] Figure 1 is a diagram showing an example of notification methods for notification information.

[0011] The user US operates the mobile device 10 using the controller 20. The mobile device 10 can be anything that can move autonomously based on user input information, and its shape and structure are not particularly limited. For example, Figure 1 shows a drone as an example of the mobile device 10. The user US can use a commercially available flight controller or a mobile device (smartphone, tablet, notebook PC, etc.) as the controller 20. The mobile device 10 is used for a variety of purposes, such as aerial photography, surveying, inspection, logistics, disaster prevention, security, and pesticide spraying.

[0012] Information detected by the mobile device 10 is output as notification information. This notification information includes information measured by the internal sensors of the mobile device 10, and information showing the results of the analysis of the measured information. The notification information is communicated to the pilot user US in various ways. In the example shown in Figure 1, the notification information is communicated through the UI screen and speaker of the controller 20, or through the LED lights and speaker of the mobile device 10.

[0013] Most notification information is delivered via the controller 20's UI screen or speaker. However, the user US is not always looking at the UI screen. Emergency information may be delivered via the controller 20 or the mobile unit 10's speaker, but the notification sound may not be heard in noisy environments.

[0014] For example, in the upper left example of Figure 1, the user US's attention is focused on the moving object 10 (Situation A). In this case, even if a notification is displayed on the UI screen, the user US may miss it. In the lower left example of Figure 1, the user US's attention is focused on the UI screen (Situation B). In this case, even if a notification is displayed using the LED light on the moving object 10, the user US may miss it. In the right example of Figure 1, the user US is intently watching the moving object 10 in a noisy environment (Situation C). In this case, even if a notification is displayed on the UI screen, the user US may miss it. Furthermore, even if a notification is displayed via speaker, the user US may not be able to hear it due to ambient noise or background noise (environmental sound NS).

[0015] If a single notification method is determined uniformly, notifications may not be properly delivered depending on the surrounding environment and the user's (US's) state. Therefore, this disclosure proposes a method for switching notification methods depending on the situation. Situations that may be affected include the direction in which the user (US) is paying attention, the user's (US's) surrounding environment, and the user's (US's) status of holding the controller 20.

[0016] For example, the direction in which the user US is directing their attention can be determined based on the user US's gaze and the orientation of their face. The surrounding environment includes the brightness around the user US and the level of ambient noise NS. The holding status of the controller 20 refers to how the controller 20 is being held. For example, if the user US is holding the controller 20 in their hand, the holding status is "handheld". If the controller 20 is placed on a table, floor, or ground, the holding status is "floor-mounted".

[0017] [2. Notification Priority Table] Notification methods can be assigned priorities. For example, in bright environments, notifications via the UI screen are effective, but if it's too bright, the UI screen becomes difficult to see. Therefore, it is preferable to use vibration notifications in conjunction with UI notifications. In this case, the display and haptic device can be set as the default notification methods (priority: high).

[0018] When brightness is reduced, glare that makes the UI screen difficult to see is minimized, allowing vibration to be given a lower priority. In this case, the display can be set as the default notification method (high priority), and the haptic device can be set as an additional notification method (low priority). In environments with low ambient noise, audio can be given a higher priority than the UI screen.

[0019] When the device is placed on the floor, notifications may be missed if only the UI screen is used. Therefore, providing audio notifications in addition to the UI screen may reduce the risk of missing notifications. In this case, the display can be set as the default notification method (high priority), and the speaker can be set as an additional notification method (low priority).

[0020] Individuals have varying abilities to recognize notifications. Even when notifications are sent using high-priority notification methods, some users (US) may not respond. In this disclosure, if a user (US) does not respond to a notification, or if the lack of response continues, the notification methods to be used will be expanded to include lower-priority methods.

[0021] The level of responsiveness indicates which notification methods should be used to notify the user (US). Responsiveness is defined as the priority of the lowest-priority notification method to be used. Responsiveness can be set for each user (US). Notifications to the user (US) are carried out using all notification methods assigned a priority equal to or higher than the responsiveness level. The notification methods are configured based on the notification priority table PT shown in Figure 2. Figure 2 is a diagram showing an example of the notification priority table PT.

[0022] The notification priority table PT defines the priority of each notification method used for notifications. In Figure 2, "gaze / face orientation" refers to the direction of the user's gaze or face orientation. "Gaze / face orientation" indicates the direction in which the user's attention is directed. For example, the controller 20 has a built-in camera 42 (see Figure 4) for capturing the user's face. The user's gaze direction and face orientation can be detected by analyzing the images captured by the camera 42.

[0023] "Gaze / Face Direction" being "Controller Side" means that the user US's attention is directed towards the controller 20. "Gaze / Face Direction" being "Aircraft Side" means that the user US's attention is directed towards the mobile object 10. "Gaze / Face Direction" being "Unknown" means that the user US's face is not visible in the captured image, and therefore the user US's gaze direction and face direction cannot be determined.

[0024] "Ambient noise" indicates the level of ambient noise or background noise (ambient noise NS) around the user US. "Brightness" indicates the level of brightness around the user US. The "high" and "low" levels, and the "bright" and "dark" levels can be determined using thresholds. The thresholds can be arbitrarily set by the system developer. "Controller holding status" indicates the status of the controller 20 being held (handheld / floor-standing).

[0025] "Controller GUI" means that the display (UI screen) of controller 20 is used as a notification means. "Controller audio" means that the speaker of controller 20 is used as a notification means. "Controller vibration" means that the vibrator of controller 20 is used as a notification means. "Aircraft LED" means that the LED light of mobile unit 10 is used as a notification means. "Aircraft audio" means that the speaker of mobile unit 10 is used as a notification means.

[0026] The numbers "1" and "2" in the notification method column indicate priority. A lower number indicates higher priority. By default, the notification method with priority "1" is selected. For each notification, the system detects whether or not the user (US) has responded. If there is no response, or if the lack of response continues, the response level is switched to the next lower priority level.

[0027] The notification priority table PT defines a priority for each notification means for each individual situation obtained by combining (i) the direction of interest of the user US, (ii) the surrounding environment of the user US, and (iii) the status of the controller 20 held by the user US. With this configuration, the notification means can be flexibly adjusted for various situations. In the example in Figure 2, two levels of priority, "1" and "2", are shown, but three or more levels of priority may be set.

[0028] For example, the notification priority table PT sets at least one output device on the controller 20 as the highest priority notification method when the user US is directing their attention towards the controller 20. When the user US's gaze is directed towards the controller 20, there is less risk of missing notifications from the controller 20's output devices. Similarly, when the controller 20 is being held, the controller 20's output devices can also be set as the highest priority notification method.

[0029] [3. User Management Table] Figure 3 shows an example of the user management table MT, which manages responsiveness. Responsiveness is determined for each user US. The user management table MT stores the responsiveness of each user US, linked to the user ID. Whether or not a user US has responded to a notification is constantly counted. Responsiveness is updated sequentially based on the count results.

[0030] [4. Example of Information Processing System Configuration] The control of the above-mentioned notifications is carried out by an information processing system 1 as shown in Figure 4. Figure 4 is a diagram showing an example of the configuration of the information processing system 1. The information processing system 1 has a mobile body 10 and a controller 20.

[0031] The mobile unit 10 uses internal sensors to perceive the external environment and collect surrounding information. The mobile unit 10 outputs measurement information from the internal sensors and the results of the analysis of the measurement information as notification information. The notification information may include information indicating an abnormality in the unit.

[0032] The mobile device 10 includes various internal sensors necessary for recognizing the external environment and collecting surrounding information. For example, the internal sensors include a camera, an IMU (Internal Measurement Unit), a barometric pressure sensor, a GPS (Global Positioning System), a geomagnetic sensor, and a ToF (Time of Flight) sensor.

[0033] The controller 20 remotely controls the destination, direction of movement, and speed of the mobile device 10. The controller 20 converts notification information acquired from the mobile device 10 into video signals, audio signals, and haptic signals, and presents them to the user US.

[0034] Figure 4 illustrates the configuration necessary for controlling the output of notification information. For example, the mobile device 10 includes a communication unit 11, a storage unit 12, a notification data generation unit 13, an output control unit 14, an LED light 15, and a speaker 16.

[0035] The memory unit 12 stores notification information generated based on measurement information. The notification data generation unit 13 generates notification data mainly related to information indicating abnormalities in the aircraft (abnormal information). In the example in Figure 4, an LED light 15 and a speaker 16 are shown as notification means. The notification data generation unit 13 generates light emission data for the LED light 15 and audio data for the speaker 16 as notification data.

[0036] The storage unit 12 stores the notification priority table PT and the user management table MT. The notification data generation unit 13 determines the notification means for abnormal information based on the notification priority table PT and the user management table MT. The output control unit 14 notifies the abnormal information from either the LED light 15 or the speaker 16, or both, based on the determination result of the notification means.

[0037] The controller 20 includes a communication unit 21, a storage unit 22, a notification processing unit 23, an input unit 24, and an output unit 25. The input unit 24 has various sensors for detecting the user US's response to notifications. Figure 4 shows, as an example of sensors, a microphone 41, a camera 42, and a touch panel 43.

[0038] For example, the input unit 24 uses detection information from the microphone 41 and the touch panel 43 to detect whether or not the user US has responded to a notification. Based on the image captured by the camera 42, the input unit 24 detects the direction in which the user US is directing their attention at the time of notification. For example, the input unit 24 can acquire the orientation of the user US's face or gaze direction determined from the captured image as the direction in which the user US is directing their attention. With this configuration, an appropriate notification means is selected according to the orientation of the user US's face or gaze direction.

[0039] Although not shown in the diagram, the input unit 24 includes various sensors for detecting the user's surrounding environment and the user's holding status of the controller 20. Examples of such sensors include an ambient light sensor, a microphone, and an IMU.

[0040] For example, the input unit 24 acquires the level of ambient sound detected by the microphone and the ambient brightness detected by the ambient light sensor as information indicating the user's surrounding environment. With this configuration, an appropriate notification means is selected according to the level of ambient sound and ambient brightness. Based on the movement information and posture information of the controller 20 detected by the IMU, the input unit 24 detects the user's holding status of the controller 20 (handheld / placed on the floor). This allows for accurate acquisition of the holding status.

[0041] The function of the input unit 24 may be replaced by a mobile device such as a smartphone. In this case, the controller 20 consists of a controller body that controls the movement of the mobile body 10 and a mobile device that has the function of the input unit 24. The controller body and the mobile device are separable, and they can exchange information through communication.

[0042] The storage unit 22 acquires notification information from the mobile device 10 via the communication unit 21. The storage unit 22 stores the notification priority table PT and the user management table MT. The notification processing unit 23 determines the notification means for the notification information based on the notification priority table PT and the user management table MT.

[0043] The output unit 25 has a plurality of output devices for outputting notification information as notification means. The notification processing unit 23 generates notification data for each notification means based on the notification information. In FIG. 4, as an example of the notification means, a display 51, a speaker 52, and a haptic device 53 are shown. The notification processing unit 23 generates, as the notification data, video data for the display 51, audio data for the speaker 52, and haptic data for the haptic device 53.

[0044] The notification processing unit 23 includes a responsiveness management unit 31, a response control unit 32, a notification data generation unit 33, and an output control unit 34. The notification data generation unit 33 selects a notification means for the user US based on the detection information (the direction in which the user US is interested, the surrounding environment of the user US, and the holding state of the controller 20 by the user US) of the input unit 24. According to this configuration, the notification means can be adaptively switched according to the situation. Therefore, it is possible to more surely make the user recognize the notification under various situations. The correspondence relationship between the detection information and the notification means is defined in the notification priority table PT and the user management table MT.

[0045] The notification data generation unit 33 acquires the notification information from the storage unit 22. The notification data generation unit 33 generates notification data for each device determined as the notification means based on the notification information. The output control unit 34 notifies abnormal information from any one or two or more of the display 51, the speaker 52, and the haptic device 53 based on the determination result of the notification means. For example, the display 51 displays the notification information on the UI screen based on the video data. The speaker 52 presents audio information based on the audio data. The haptic device 53 performs a vibration-based notification based on the haptic data.

[0046] The response control unit 32 monitors the detection status of the input unit 24, and determines that the user US has responded when an appropriate reaction (touch operation, voice operation) is detected within a predetermined time from the notification. For example, when the notification is made by the UI screen, the vibration of the controller 20, and the emission of the LED light 15, a button operation on the "OK button" of the UI screen is detected as a response. When the notification is made by voice, an instruction by voice such as "yes" or "close" by the user US is detected as a response.

[0047] The response degree management unit 31 determines, based on the response history of the user US to the notification, up to which priority notification means should be adopted as the notification means to the user US. The response degree management unit 31 sets the priority of the lowest-level notification means to be adopted as the response degree. According to this configuration, a wider range of notification means is adopted according to the response status of the user US. Therefore, the risk that the user US overlooks the notification is reduced.

[0048] The response degree management unit 31 can update the response degree based on the response status of the user US to the notification. For example, the response degree management unit 31 counts the presence or absence of the response of the user US to the notification. The response degree management unit 31 updates the response degree recorded in the user management table MT based on the count result. According to this configuration, an appropriate notification means is learned based on the response history of the user US. The response degree management unit 31 sets the response degree independently for each user US. Thereby, a notification means considering the individual differences of the user US is adopted.

[0049] For example, the response degree management unit 31 acquires the response degree immediately before the notification from the user management table MT. The response degree management unit 31 decreases the response degree by 1 when a response to the notification is detected, and increases the response degree by 1 when no response is detected. The minimum value of the response degree is set to 1. When a response is detected in the state where the response degree is 1, the response degree is maintained at 1.

[0050] The responsiveness management unit 31 can introduce hysteresis to the count result of whether or not a response was received, based on the most recent response result. Introducing hysteresis to the count result means that the responsiveness level will not change unless the same response result (response received / no response) occurs multiple times in a row. With this configuration, the change in the responsiveness level becomes gradual. Therefore, a notification method that accurately reflects the response status of the user US is selected.

[0051] The system developer can arbitrarily set how many consecutive identical response results are required before the responsiveness level changes. In this disclosure, for example, the responsiveness level does not change unless the identical response result occurs twice in a row.

[0052] For example, if a response is detected, it is denoted as "-1," and if no response is detected, it is denoted as "+1." If user US's response history is "+1," "+1," "-1," and "+1," then in the absence of hysteresis, the difference in the updated responsiveness compared to the default responsiveness will change as follows: "+1," "+2," "+1," and "+2."

[0053] On the other hand, when hysteresis is present, the difference in the updated responsiveness relative to the default responsiveness changes as follows: "+1", "+2", "+2", "+3". If user US's response history is "+1", "+1", "-1", "-1", "+1", the difference in the updated responsiveness relative to the default responsiveness changes as follows: "+1", "+2", "+2", "+1", "+1". It can be seen that adding hysteresis makes the change in responsiveness more gradual.

[0054] The storage unit 22 outputs the user management table MT with the updated responsiveness to the mobile device 10 via the communication unit 21 and the communication unit 11. The mobile device 10 stores the updated user management table MT in the storage unit 12.

[0055] [5. Processing Flow] Figures 5 to 9 show an example of the processing flow in notification control.

[0056] The controller 20 uses the camera 42 to detect the direction in which the user US is paying attention (for example, the direction of the user US's gaze or the orientation of their face) (step S1). The controller 20 uses the microphone 41 to detect ambient sound NS (step S2). The controller 20 uses the ambient light sensor to detect the ambient brightness (step S3). The controller 20 updates the user management table MT to the latest version (step S4).

[0057] The above process is repeated until a notification occurs (Step S5: No). If a notification occurs (Step S5: Yes), the controller 20 monitors the user US's response. First, the controller 20 determines whether it can detect the direction in which the user US is directing their attention (Step S11). For example, if the user US is not visible in the camera 42, or if the user US is visible in the camera 42 but the direction of the user US's gaze or face cannot be determined, it is determined that the direction in which the user US is directing their attention cannot be detected.

[0058] If the direction in which user US is directing their attention can be detected (step S11: Yes), the controller 20 sets the user US's responsiveness level based on the user US's response history (step S12). The controller 20 then sends notifications using all notification means with priority within the range indicated by the responsiveness level (step S13).

[0059] The controller 20 waits for input from the user US. If input is received from the touch panel 43 or microphone 41 within a predetermined time, the controller 20 determines that a response has been received (steps S14 to S16). If a response is received from the user US (step S15: Yes), the controller 20 decreases the responsiveness by 1 (step S18) and terminates the process. If no response is received within the predetermined time (step S15: No), the controller 20 increases the responsiveness by 1 (step S17) and returns to step S12.

[0060] In step S11, if the direction in which the user US is directing its interest cannot be detected (step S11: No), the controller 20 determines the holding status of the controller 20 (step S21).

[0061] If the controller 20 is handheld (step S21: handheld), the notification priority table PT specifies that at least one output device mounted on the controller 20 will be used as one of the notification means. Therefore, the controller 20 decides to use at least one of its output devices as one of the notification means based on the notification priority table PT (step S22). The controller 20 sends a notification from the output device determined to be a notification means (step S23).

[0062] The controller 20 waits for input from the user US (steps S24 to S26). If a response is received from the user US within a predetermined time (step S25: Yes), the controller 20 determines that the notification has been conveyed to the user US and terminates the process.

[0063] If no response is received within a predetermined time (step S25: No), the controller 20 decides to use all output devices mounted on the mobile body 10 and the controller 20 as notification means (step S32). The controller 20 sends notifications from all the determined output devices (step S33).

[0064] The controller 20 waits for input from the user US (steps S34 to S36). If a response is received from the user US within a predetermined time (step S35: Yes), the controller 20 determines that the notification has been conveyed to the user US and terminates processing. If no response is received within a predetermined time (step S35: No), the controller 20 determines that there are no further means of notification and abandons the notification, and terminates processing.

[0065] If the controller 20 is placed on the floor in step S21 (step S21: placed on the floor), the controller 20 determines whether the mobile body 10, the controller itself, and the user US are visible to the camera 42 as identifiable subjects (step S31). This determination process is performed when the functions of the input unit 24, including the camera 42, are replaced by a mobile device separated from the controller body.

[0066] If the aircraft, the controller unit, and the user US are not visible to the camera 42 as identifiable subjects (step S31: No), the mobile device, which is the input unit 24, may be lost. Therefore, the controller 20 decides to use the mobile unit 10 and all output devices mounted on the controller 20 as notification means (step S32), and performs the processing described in steps S33 to S36 above.

[0067] If at least one of the aircraft, the controller unit, and the user US is captured by the camera 42 as an identifiable subject (step S31: Yes), the controller 20 determines an output device closest to the user US as a notification device (step S41). The controller 20 then sends a notification from the determined output device (step S42).

[0068] The controller 20 waits for input from the user US (steps S43 to S45). If a response is received from the user US within a predetermined time (step S44: Yes), the controller 20 determines that the notification has been conveyed to the user US and terminates processing. If no response is received within a predetermined time (step S44: No), the controller 20 decides to use all output devices mounted on the mobile body 10 and the controller 20 as notification means (step S32), and performs the processing described in steps S33 to S36.

[0069] The processing flow described above illustrates an example where the responsiveness score is updated as a discrete value based on the response status to the notification (the count result of whether or not a response was received). However, the responsiveness score does not necessarily have to be a discrete value. For example, the responsiveness score may be calculated as a continuous value by weighting the time to the response or the type of response.

[0070] [6. Hardware Configuration Example] Figure 10 shows an example of the hardware configuration of the information processing system 1.

[0071] The information processing system 1 can be implemented by a computer 1000 as shown in Figure 10. The computer 1000 includes a processing circuit 1100, RAM 1200, ROM 1300, secondary storage device 1400, communication interface 1500, input / output interface 1600, display unit 1700, camera unit 1800, microphone 1900, and speaker 2000. The various parts of the computer 1000 are connected by a bus 1050.

[0072] The processing circuit 1100 operates based on a program stored in the ROM 1300 or secondary storage device 1400, and controls each part. For example, the processing circuit 1100 loads the program stored in the ROM 1300 or secondary storage device 1400 into the RAM 1200 and executes processing corresponding to various programs.

[0073] ROM 1300 stores boot programs such as the BIOS (Basic Input Output System) executed by the processing circuit 1100 when the computer 1000 starts up, as well as programs that depend on the computer 1000's hardware.

[0074] The secondary storage device 1400 is a computer-readable recording medium that non-temporarily records programs executed by the processing circuit 1100 and data used by such programs. Specifically, the secondary storage device 1400 is a recording medium that records programs for each process of the information processing system 1 according to the embodiment of this disclosure, which is an example of program data 1450.

[0075] The communication interface 1500 is an interface for the computer 1000 to connect to the external network 1550. For example, the processing circuit 1100 can receive data from other devices or transmit data it has generated to other devices via the communication interface 1500.

[0076] The input / output interface 1600 is an interface for connecting the input / output device 1650 and the computer 1000. For example, the processing circuit 1100 receives data from input devices such as a microphone 1900 or a touch panel via the input / output interface 1600. The processing circuit 1100 also transmits data to output devices such as a display unit 1700 or a speaker 2000 via the input / output interface 1600. The input / output interface 1600 may also function as a media interface for reading programs recorded on a predetermined recording medium (media). Examples of media include optical recording media such as DVDs (Digital Versatile Discs) and PDs (Phase Change Rewritable Discs), magneto-optical recording media such as MOs (Magneto-Optical Discs), tape media, magnetic recording media, or semiconductor memory.

[0077] The display unit 1700 is an interface for displaying information processed by the computer 1000. The display unit 1700 is, for example, a liquid crystal display or an organic electroluminescent display (OLED display). Alternatively, the display unit 1700 may be a touch panel display device or an image projection device.

[0078] The camera unit 1800 is an interface for the computer 1000 to capture images. The microphone 1900 is an interface for the computer 1000 to capture sound. The speaker 2000 is an interface for the computer 1000 to output processed sound. The various parts of the computer 1000 are connected by the bus 1050. Each interface does not necessarily have to be located inside the computer 1000, but may be located outside the computer 1000 via a network or the like. Furthermore, each part of the computer 1000 may be controlled by a circuit different from the processing circuit 1100. For example, the display unit 1700 may be controlled not by the processing circuit 1100, but by a circuit dedicated to display processing provided within the display unit 1700.

[0079] For example, when the computer 1000 functions as an information processing system 1 according to the embodiment of this disclosure, the processing circuit 1100 of the computer 1000 functions as various detection units (such as the input unit 24) and control units (such as the notification processing unit 23, notification data generation unit 13, and output control unit 14) included in the information processing system 1 by executing a program loaded on the RAM 1200. The secondary storage device 1400 functions as a storage unit 22 as a storage unit 12. The display unit 1700, camera unit 1800, microphone 1900, speaker 2000, and input / output device 1650 function as devices included in the input unit 24 and output unit 25, and as the LED light 15 and speaker 16 of the mobile body 10. The communication interface 1500 functions as a communication unit 11 and a communication unit 21.

[0080] Furthermore, the secondary storage device 1400 stores the information processing program and various data related to this disclosure. The processing circuit 1100 reads the program data 1450 from the secondary storage device 1400 and executes it, but as an alternative, these programs may be obtained from other devices via an external network 1550. In other words, the secondary storage device 1400 is not limited to being inside the computer 1000, but may be located outside the computer 1000. The processing circuit 1100 is an example of an integrated circuit, and CPU, MPU, GPU, APU, ASIC, and FPGA can all be considered integrated circuits.

[0081] Furthermore, the effects described herein are merely illustrative and not limiting, and other effects may also occur.

[0082] [Note] The technology may also be configured as follows: (1) An information processing system having a notification data generation unit that selects a notification means for the user based on the direction the user is interested in, the user's surrounding environment, and the user's holding status of the controller. (2) The information processing system according to (1) above, having a storage unit that stores a notification priority table that defines the priority of each of the notification means, and a responsiveness management unit that determines, based on the user's response history to notifications, which of the priority levels of the notification means should be adopted as the notification means for the user, and sets the priority of the lowest-ranking notification means to be adopted as the responsiveness level. (3) The information processing system according to (2) above, wherein the notification priority table defines the priority for each of the notification means for each situation obtained by combining the direction the user is interested in, the user's surrounding environment, and the user's holding status of the controller. (4) The information processing system according to (2) or (3) above, wherein the responsiveness management unit sets the responsiveness level independently for each user. (5) The information processing system according to (4) above, wherein the responsiveness management unit updates the responsiveness based on the user's response status to the notification. (6) The information processing system according to (5) above, wherein the responsiveness management unit counts whether or not the user has responded to the notification and updates the responsiveness based on the count result. (7) The information processing system according to (6) above, wherein the responsiveness management unit introduces hysteresis to the count result of whether or not a response has been made based on the most recent response result. (8) The information processing system according to any one of (2) to (7) above, wherein the notification priority table sets at least one output device mounted on the controller as the notification means having the highest priority when the direction the user is directing their attention is the controller. (9) The information processing system according to any one of (1) to (8) above, which has an input unit that acquires the direction of the user's face or gaze as the direction the user is directing their attention.(10) The information processing system according to (9) above, wherein the input unit acquires the level of ambient sound and the brightness of the surroundings as information indicating the user's surrounding environment. (11) An information processing method performed by a computer, comprising selecting a notification means for the user based on the direction the user is interested in, the user's surrounding environment, and the user's holding status of the controller. (12) The information processing method according to (11) above, comprising defining the priority of each of the notification means in a notification priority table, determining which of the priority levels of the notification means should be adopted as the notification means for the user based on the user's response history to notifications, and setting the priority of the lowest priority notification means to be adopted as the response level. (13) The information processing method according to (12) above, wherein the notification priority table defines the priority for each of the notification means for each situation obtained by combining the direction the user is interested in, the user's surrounding environment, and the user's holding status of the controller. (14) The information processing method according to (12) or (13) above, wherein the responsiveness level is set independently for each user. (15) The information processing method according to (14) above, wherein the responsiveness level is updated based on the user's response status to the notification. (16) The information processing method according to (15) above, wherein the presence or absence of a response from the user to the notification is counted, and the responsiveness level is updated based on the count result. (17) The information processing method according to (16) above, wherein the count result of the presence or absence of a response is given hysteresis based on the most recent response result. (18) The information processing method according to any one of (12) to (17) above, wherein the notification priority table is set to have the highest priority notification means when the direction of the user's interest is the controller. (19) The information processing method according to any one of (11) to (18) above, which includes obtaining the orientation of the user's face or the direction of their gaze as the direction in which the user is paying attention.(20) The information processing method according to (19) above, which includes acquiring the level of ambient sound and the brightness of the surroundings as information indicating the user's surrounding environment.

[0083] 1 Information Processing System 12 Storage Unit 20 Controller 22 Storage Unit 31 Responsiveness Management Unit 33 Notification Data Generation Unit PT Notification Priority Table US User

Claims

1. An information processing system having a notification data generation unit that selects a notification means to the user based on the direction the user is interested in, the user's surrounding environment, and the user's holding status of the controller.

2. The information processing system according to claim 1, comprising: a storage unit that stores a notification priority table defining the priority of each of the notification means; and a response rate management unit that, based on the user's response history to notifications, determines which of the priority levels of the notification means should be used as notification means to the user, and sets the priority of the lowest priority notification means to be used as the response rate.

3. The information processing system according to claim 2, wherein the notification priority table defines the priority for each notification means for each individual situation obtained by combining the direction in which the user is interested, the user's surrounding environment, and the user's holding status of the controller.

4. The information processing system according to claim 2, wherein the responsiveness management unit independently sets the responsiveness for each user.

5. The information processing system according to claim 4, wherein the responsiveness management unit updates the responsiveness based on the user's response status to the notification.

6. The information processing system according to claim 5, wherein the responsiveness management unit counts whether or not the user has responded to the notification, and updates the responsiveness based on the count result.

7. The information processing system according to claim 6, wherein the responsiveness management unit provides hysteresis to the count result of whether or not a response is present, based on the most recent response result.

8. The information processing system according to claim 2, wherein the notification priority table sets at least one output device mounted on the controller as the notification means having the highest priority when the direction of the user's interest is the controller.

9. The information processing system according to claim 1, further comprising an input unit that acquires the orientation of the user's face or gaze direction as the direction in which the user is directing their interest.

10. The information processing system according to claim 9, wherein the input unit acquires the level of ambient sound and the ambient brightness as information indicating the user's surrounding environment.

11. A computer-based information processing method comprising selecting a means of notifying the user based on the direction in which the user is directing their interest, the user's surrounding environment, and the user's holding status of the controller.

12. The information processing method according to claim 11, comprising: defining the priority of each notification means in a notification priority table; determining which notification means of which priority should be adopted as notification means to the user based on the user's response history to notifications; and setting the priority of the lowest-priority notification means to be adopted as the response level.

13. The information processing method according to claim 12, wherein the notification priority table defines the priority for each notification means for each situation obtained by combining the direction in which the user is interested, the user's surrounding environment, and the user's holding status of the controller.

14. The information processing method according to claim 12, further comprising setting the responsiveness independently for each user.

15. The information processing method according to claim 14, further comprising updating the responsiveness level based on the user's response status to the notification.

16. The information processing method according to claim 15, comprising counting whether or not the user has responded to the notification, and updating the response rate based on the count result.

17. The information processing method according to claim 16, further comprising giving hysteresis to the count result of whether or not a response is present, based on the most recent response result.

18. The information processing method according to claim 12, wherein the notification priority table sets at least one output device mounted on the controller as the notification means having the highest priority when the direction of the user's interest is the controller.

19. The information processing method according to claim 11, comprising obtaining the orientation of the user's face or gaze direction as the direction in which the user is directing their interest.

20. The information processing method according to claim 19, comprising acquiring the level of ambient noise and ambient brightness as information indicating the user's surrounding environment.

Images