Information processing system, information processing method, and program

Abstract

[Problem] To provide an information processing system and the like capable of providing a novel user experience. [Solution] According to one aspect of the present invention, there is provided an information processing system comprising at least one processor, the processor being configured to execute the following steps by reading a program: a first presentation step in which multimodal information is presented to a user located in a closed space, wherein the multimodal information is configured to include a combination of a plurality of pieces of predetermined sensory information, and the sensory information is any one of visual information, auditory information, haptic information, and olfactory information; and a second presentation step in which transient information including at least one piece of sensory information is presented to the user before or after the first presentation step, wherein the transient information is configured such that the number and / or output intensity of the sensory information included in the multimodal information is reduced in comparison with the multimodal information.

Description

Information processing systems, information processing methods, and programs

[0001] This invention relates to an information processing system, an information processing method, and a program.

[0002] For example, Patent Document 1 discloses an environmental control system for a private room that allows for easy stress relief without requiring special knowledge and has a low barrier to system implementation. This environmental control system includes a lighting device installed in a private room that can change the illuminance or color temperature within the private room, a video output device that outputs video within the private room, and a control device equipped with a communication device that can transmit control instructions to the lighting device and the video output device. The control device has a refresh control that, by controlling the lighting device and the video output device, performs lighting relaxation control to decrease the illuminance or color temperature within the private room and video relaxation control to output a first video within the private room, and then performs lighting awakening control to increase the illuminance or color temperature within the private room and video awakening control to output a second video different from the first video within the private room.

[0003] Japanese Patent Publication No. 2020-053346

[0004] However, there is a need to provide a novel user experience that differs from the technology disclosed in Patent Document 1.

[0005] In view of the above circumstances, the present invention aims to provide an information processing system, etc., that can provide a novel user experience.

[0006] According to one aspect of the present invention, an information processing system is provided, comprising at least one processor, the processor configured to perform the following steps by reading a program, wherein in a first presentation step, multimodal information is presented to a user located in a closed space, the multimodal information comprises a predetermined combination of a plurality of sensory information, the sensory information being any of visual information, auditory information, tactile information and olfactory information, and in a second presentation step, before or after the first presentation step, transient information including at least one sensory information is presented to the user, the transient information is configured such that at least one of the number and output intensity of the sensory information included in the multimodal information is reduced compared to the multimodal information.

[0007] This approach makes it possible to provide a novel user experience.

[0008] This is a configuration diagram showing the hardware configuration of the information processing system 1 according to the first embodiment. This is a block diagram showing the functional configuration of the information processing device 2 according to the first embodiment. This is a flowchart outlining the information processing in this embodiment. This is a schematic diagram showing the specific configuration of the information processing system 1. This is an activity diagram showing an overview of the information processing. This is a schematic diagram showing an example of a method for presenting visual information VI. This is a schematic diagram showing an example of a method for presenting multimodal information. This is a schematic diagram showing an example of a method for presenting multimodal information. This is a schematic diagram showing an example of a method for presenting multimodal information. This shows the evaluation results of PRS in this experiment.

[0009] Embodiments of the present invention will be described below with reference to the drawings. The various features shown in the embodiments below can be combined with each other.

[0010] Incidentally, the program for realizing the software appearing in one embodiment may be provided as a computer-readable non-transitor-readable medium, or it may be provided so that it can be downloaded from an external server, or it may be provided so that the program is launched on an external computer and its functions are realized on a client terminal (so-called cloud computing).

[0011] Furthermore, in various information processing according to one embodiment, an input and an output corresponding to the input can be realized. Here, as long as an output is obtained as a result of the input, the form of the information referenced in such information processing (hereinafter referred to as "reference information") is not limited. The reference information may be, for example, rule-based information such as a database, a lookup table, or a predetermined function (including a decision formula such as a regression equation constructed by a statistical method), or a pre-trained model that has learned the correlation between input and output in advance, or a generative AI such as a large-scale language model or visual language model that can output a desired result by inputting a prompt.

[0012] Furthermore, in one embodiment, "part" may include, for example, hardware resources implemented by a circuit in a broad sense, and the information processing of software that can be specifically realized by these hardware resources. Also, in one embodiment, various types of information are handled, and this information can be represented, for example, by the physical values ​​of signal values ​​representing voltage and current, the high or low values ​​of signal values ​​as a set of binary bits composed of 0s or 1s, or by quantum superposition (so-called qubits), and communication and calculations can be performed on a circuit in a broad sense.

[0013] Furthermore, a circuit in a broad sense is a circuit realized by combining at least an appropriate combination of circuits, circuits, processors, and memory. The processor may be a general-purpose processor or a dedicated circuit. In other words, this includes application-specific integrated circuits (ASICs), programmable logic devices (for example, simple programmable logic devices (SPLDs), complex programmable logic devices (CPLDs), and field programmable gate arrays (FPGAs)), etc.

[0014] [Embodiment] 1. Hardware Configuration This section describes the hardware configuration of one embodiment.

[0015] 1.1 Information Processing System 1 Figure 1 is a configuration diagram showing the hardware configuration of the information processing system 1 according to the first embodiment. The information processing system 1 includes, for example, an information processing device 2, an image output device 3, an audio output device 4, a tactile presentation device 5, an olfactory presentation device 6, and a wearable device 7, which are electrically connected. These components will be described further below.

[0016] In this embodiment, the information processing system 1 provides a novel user experience to user U by presenting multimodal information. The multimodal information consists of a combination of multiple sensory information, and the sensory information is any of visual information, auditory information, tactile information, and olfactory information. Furthermore, in this embodiment, before or after the period in which the multimodal information is presented, transitional information is presented to user U in which at least one of the number and output intensity of the sensory information included in the multimodal information is reduced compared to the multimodal information. This reduces the discomfort caused to user U by abrupt environmental changes.

[0017] In one embodiment, the system exemplified by the information processing system 1 consists of one or more devices or components. For example, if it consists only of an information processing device 2, then the information processing system 1 can be the information processing device 2. Alternatively, a combination of the information processing device 2 and at least two of the image output device 3, sound output device 4, tactile presentation device 5, and olfactory presentation device 6 can also constitute the information processing system 1. More specifically, the information processing system 1 may include elements selected from the group consisting of the information processing device 2 and components 3 to 7. Furthermore, multiple information processing devices 2 may be used to correspond to each of the components 3 to 7. Unselected elements may not be included in the information processing system 1, but may be electrically connected to the selected elements as external elements.

[0018] 1.2 Information Processing Device 2 The information processing device 2 comprises a communication unit 21, a storage unit 22, a control unit 23, and an input unit 24. These components are electrically connected within the information processing device 2 via a communication bus 20. Each component will be described further.

[0019] The communication unit 21 preferably uses wired communication methods such as USB, IEEE 1394, Thunderbolt®, and wired LAN network communication, but may also include wireless LAN network communication, mobile communication such as LTE / 5G, and Bluetooth® communication as needed. In other words, it is more preferable to implement it as a collection of these multiple communication methods. That is, the information processing device 2 may communicate various information from the outside via the communication unit 21 and the network. Specifically, the information processing device 2 is connected to the image output device 3, the audio output device 4, the haptic presentation device 5, the olfactory presentation device 6, and the wearable device 7 via the communication unit 21 and each transmission path. Each component 3 to 7 will be described later.

[0020] The storage unit 22 stores various types of information as defined above. This can be implemented, for example, as a storage device such as a solid-state drive (SSD) that stores various programs related to the information processing device 2 executed by the control unit 23, or as a memory such as a random access memory (RAM) that stores temporarily necessary information (arguments, arrays, etc.) related to program calculations. The storage unit 22 stores various programs and variables related to the information processing device 2 executed by the control unit 23.

[0021] The control unit 23 (processor) performs processing and control of the overall operation related to the information processing device 2. The control unit 23 is, for example, a Central Processing Unit (CPU) (not shown). The control unit 23 realizes various functions related to the information processing device 2 by reading predetermined programs stored in the memory unit 22. That is, information processing by software stored in the memory unit 22 is concretely realized by the control unit 23, which is an example of hardware, and can be executed as each functional unit included in the control unit 23. These will be described in more detail in the next section. Note that the control unit 23 is not limited to being a single unit, and may be implemented with multiple control units 23 for each function, or a combination thereof. In particular, the control unit 23 can generate control signals to control the operation of the components 3 to 6 described later. In other words, components 3 to 6 can be controlled by the control unit 23 (processor) of the information processing device 2.

[0022] The input unit 24 may be included in the housing of the information processing device 2, or it may be externally attached. For example, the input unit 24 may be implemented as a touch panel integrated with a display (which may be the display 31 described later). If it is a touch panel, the user can input tap operations, swipe operations, etc. Of course, instead of a touch panel, a switch button, mouse, QWERTY keyboard, microphone for voice input, etc. may be used. In other words, the input unit 24 receives operation input made by the user. This input is transmitted as a command signal to the control unit 23 via the communication bus 20, and the control unit 23 can execute predetermined controls and calculations as needed.

[0023] 1.3 Image Output Device 3 The image output device 3 is a device that presents visual sensory information (hereinafter also referred to as visual information VI) to the user, and may display a graphical user interface (GUI) screen that can be operated by the user. The image output device 3 may be, for example, a stationary display 31 as described later, or a projector 32 that projects onto a screen or the like (see Figure 3). It may also be a head-mounted display (HMD). The HMD can be a device configured for any application, such as VR (Virtual Reality), AR (Augmented Reality), MR (Mixed Reality), etc. If the image output device 3 is a display 31, it is preferable to use different display devices such as CRT displays, liquid crystal displays, organic EL displays, and plasma displays.

[0024] 1.4 Audio Output Device 4 The audio output device 4 is a device that presents auditory sensory information (hereinafter also referred to as auditory information AI) to the user, and may be, for example, headphones 41 described later, or a speaker 42 (see Figure 3). The audio output device 4 outputs sound in a frequency band above the human audible range. As headphones 41, it is preferable to use an open-air type that does not significantly block out external sounds when worn by the user. In particular, when auditory information AI is presented using headphones 41 and a speaker 42 in combination, it is preferable to use an open-air type. On the other hand, when auditory information AI is presented using only headphones 41, it is preferable to use a shielding type with a noise-canceling function to improve immersion. Furthermore, if the audio output device 4 is a speaker 42, the speaker 42 may be connected to the information processing device 2 via an amplifier (not shown), or it may be an active speaker with a built-in amplifier. Furthermore, when a speaker 42 is used, it is preferable to have a surround sound system. Furthermore, the audio output device 4 may be an ultrasonic generator (not shown).

[0025] 1.5 Tactile Presentation Device 5 The tactile presentation device 5 is a device that presents tactile sensory information (hereinafter also referred to as tactile information TI) to the user. Tactile information TI includes contact stimuli such as vibration, warmth, and electrical stimulation from contact-type devices, and non-contact stimuli such as pressure and warmth from substances such as air provided by non-contact-type devices. In this embodiment, the tactile presentation device 5 is, for example, a fan 51 or air conditioner 52 described later that sprays air onto the user U (see Figure 4). The tactile presentation device 5 may also be an ultrasonic generator (not shown). Furthermore, the contact-type tactile presentation device 5 may be provided in the surrounding environment such as an electronic device such as a smart device that the user U grasps, a wearable device, or a seat or table on which the user sits.

[0026] 1.6 Olfactory Presentation Device 6 The olfactory presentation device 6 is a device that presents olfactory sensory information (hereinafter also referred to as olfactory information OI) to the user, and may be, for example, the aroma diffuser 61 described later.

[0027] In summary, the information processing system 1 comprises at least one image output device 3 that provides visual information VI, at least one sound output device 4 that provides auditory information AI, at least one tactile presentation device 5 that provides tactile information TI, and at least one olfactory presentation device 6 that provides olfactory information OI. The processor (e.g., control unit 23) is configured to control at least two of the image output device 3, sound output device 4, tactile presentation device 5, and olfactory presentation device 6 that provide sensory information that acts on different senses of the user U, thereby presenting multimodal information IF2. Multimodal information IF2 will be described in more detail later. According to this embodiment, a novel user experience can be realized as a composite device. Also, as mentioned above, the sound output device 4 or the tactile presentation device 5 may include at least an ultrasonic generator. According to this embodiment, an even more novel user experience can be realized.

[0028] 1.7 Wearable Device 7 The wearable device 7 is worn by user U and is configured to continuously measure user U's biometric information via sensors (not shown). The sensors (not shown) may include temperature sensors, infrared sensors, light sensors, pressure sensors, potential sensors, chemical sensors, etc. The wearable device 7 may also include smart devices such as smartwatches, smart rings, smart glasses, and smart earphones. The wearable device 7 transmits and receives information via wireless communication with the communication unit 21 of the information processing device 2.

[0029] 2. Functional Configuration Next, with reference to Figure 2, the functional configuration of the information processing device 2 of the information processing system 1 will be described. Figure 2 is a block diagram showing the functional configuration of the information processing device 2 according to one embodiment. As shown in Figure 2, the control unit 23 (processor) functions as an acquisition unit 231, a specification unit 232, and a presentation unit 233 by executing various programs stored in the storage unit 22. That is, information processing by software stored in the storage unit 22 is concretely realized by the control unit 23, which is an example of hardware, and can be executed as each functional unit included in the control unit 23. In other words, the information processing system 1 comprises at least one processor (for example, the control unit 23), and the processor functions as the following units by reading a program. From another viewpoint, the program causes the computer to execute each step of the information processing system 1. The information processing method comprises each step of the information processing system 1. Each step includes, for example, an acquisition step, a specification step, and a presentation step.

[0030] The acquisition unit 231 is configured to acquire various types of information as an acquisition step. For example, the acquisition unit 231 acquires user information, including the biometric information of user U. Details will be described later.

[0031] The identification unit 232 is configured to perform a specific step. For example, the identification unit 232 identifies various pieces of information used to present various sensory information to the user U. Further details will be described later.

[0032] The presentation unit 234 is configured to present various sensory information to the user U as a first presentation step, a second presentation step, and a third presentation step. The sensory information includes at least one of visual information VI, auditory information AI, tactile information TI, and olfactory information OI. Further details will be described later.

[0033] Furthermore, "presentation" means making various sensory information accessible to the user, and may include, for example, the control unit 23 generating control signals for controlling the image output device 3, the sound output device 4, the tactile presentation device 5, and the olfactory presentation device 6.

[0034] For example, when controlling the image output device 3, the presentation unit 234 is configured to output display information to the image output device 3. The display information may be the visual information VI itself generated in a form visible to the user, such as a screen, image, icon, text, or the like, or may be, for example, rendering information for causing the image output device 3 to display the visual information VI.

[0035] Further, when controlling the audio output device 4, the control signal may be the auditory information AI itself generated in a form recognizable by the user U, such as a sound waveform, or may be information specifying the volume of a predetermined audio, audio effect, frequency band, digital audio processing, playback speed, and the degree of adjustment of intonation. The audio output from the audio output device 4 includes BGM, environmental sounds, utterance voices of humans and characters, sound effects, and the like.

[0036] Further, when controlling the tactile presentation device 5, the control signal includes information specifying the waveform of vibration and the intensity of electrical stimulation applied by the contact device, as well as the air pressure, air volume, temperature, wind direction and the like of the gas jetted toward the user U by the non-contact device.

[0037] Further, when controlling the olfactory presentation device 6, the control signal includes information instructing the olfactory presentation device 6 what kind of scent to generate and at what density. For example, when the olfactory presentation device 6 is provided with a plurality of cartridges each containing an aromatic substance that presents a predetermined scent, the control signal specifies which cartridge is to spray the aromatic substance into the air and in what amount.

[0038] In the following description, the information presented by each of the devices 3 to 6 that affects different senses of the user may be collectively referred to as sensory information. Furthermore, when different types of stimuli are presented to the user from predetermined sensory information, each may be individually referred to as a sensory stimulus. For example, the image output device 3 may present visual information containing multiple visual sensory stimuli (visual stimuli) to user U. The sound output device 4 may present auditory information containing multiple auditory sensory stimuli (auditory stimuli) to user U. The tactile presentation device 5 may present tactile information containing multiple tactile sensory stimuli (tactile stimuli) to user U. The olfactory presentation device 6 may present olfactory information containing multiple olfactory sensory stimuli (olfactory stimuli) to user U.

[0039] 3. Information Processing 3.1 Overview of Information Processing Figure 3 is a flowchart illustrating the overview of information processing in this embodiment. The dashed lines indicate that at least one of the options is optional. As shown in Figure 3, in the information processing system, the presentation unit presents multimodal information to a user located in a closed space as a first presentation step (step S2). The multimodal information consists of a predetermined combination of multiple sensory information. The sensory information is any of visual information, auditory information, tactile information, and olfactory information. As a second presentation step, the presentation unit presents transitional information containing at least one sensory information to the user before or after the period in which the multimodal information is presented (step S1 or step S3). The transitional information is configured such that at least one of the number and output intensity of the sensory information included in the multimodal information is reduced compared to the multimodal information. This embodiment can provide a novel user experience.

[0040] 3.2 Configuration Example of Information Processing System 1 FIG. 4 is a schematic diagram showing a specific configuration of the information processing system 1. When using the information processing system 1, a user U is assumed to be located in a closed space 11 while wearing a wearable device 7. The closed space 11 is, for example, a small room with white inner walls. In the closed space 11, a display 31 and a projector 32 are disposed as examples of an image output device 3, headphones 41 and a speaker 42 are disposed as examples of an audio output device 4, an electric fan 51 and an air conditioner 52 are disposed as examples of a tactile presentation device 5, and an aroma diffuser 61 is disposed as an example of an olfactory presentation device 6. A visual stimulus VI1 is presented by the display 31, and a visual stimulus VI2 is presented by the projector 32. An auditory stimulus AI1 is presented by the headphones 41, and an auditory stimulus AI2 is presented by the speaker 42.

[0041] Note that these are merely examples, and a plurality of at least some of the constituent elements may be provided. In addition, the specific arrangement of each of the constituent elements 3 to 6 is not particularly limited, and a preferable arrangement may be adopted in accordance with the physical structure of the closed space 11. The user U can experience a new user experience by receiving presentation of multimodal information IF2 in the closed space 11.

[0042] The closed space 11 may be, for example, the interior of a vehicle such as an automobile. In other words, the information processing system 1 is an in-vehicle system, and the closed space 11 is the inside of the vehicle. According to such an aspect, it is possible to contribute to providing a vehicle capable of realizing a new user experience.

[0043] 3.3 Details of Information Processing FIG. 5 is an activity diagram showing an overview of information processing. In a state where the user U is resting in the closed space 11 isolated from the outside world, the wearable device 7 acquires biometric information of the user U (activity A101). The biometric information may include, for example, a sleep score that quantifies the quality and quantity of sleep, blood pressure, pulse, heart rate, blood oxygen concentration, respiratory rate, body temperature, electrocardiogram, brain wave, blood glucose, alcohol, lactic acid, consciousness level, number of steps, walking distance, amount of exercise, a biometric score obtained from at least one of these pieces of information, and the like.

[0044] Next, biometric information is shared from the wearable device 7 to the information processing device 2 via wireless communication (Activity A102). Subsequently, the acquisition unit 231 acquires (receives) the biometric information as user information about the user via the communication unit 21 of the information processing device 2 (Activity A103). Thereafter, even as time passes, the wearable device 7 will continue to acquire the user U's biometric information and share it with the information processing device 2.

[0045] Next, the identification unit 232 identifies the user U's fatigue level based on biometric information (Activity A104). The presentation unit 233 presents the identified user U's fatigue level to the user U by displaying it on the display 31. Subsequently, the user U operates the GUI displayed on the display 31 to select a theme of their choice. The control unit 23 sets the selected theme as theme information IF1 (Activity A105). Note that multiple themes may be prepared in advance, and the presentation unit 233 may present a recommended theme to the user U based on the user U's current fatigue level.

[0046] Next, the identification unit 232, as a first identification step, identifies the priority of sensory stimuli for user U (activity A106). The priority is the order in which user U perceives different types of sensory information as important when viewing predetermined content. The priority may be set commonly for all users, or it may be set individually for each user. Furthermore, the priority for user U may be set in real time based on acquired biometric information. The system may also be configured to accept the specification of sensory information that user U wants to emphasize. In addition, a priority may be set in advance for each set theme information IF1.

[0047] Next, the identification unit 232, as a second identification step, identifies a discrimination threshold (second threshold), which is the output intensity of a sensory stimulus that user U can recognize (activity A107). The discrimination threshold is the output intensity of each sensory information (sensory stimulus) to the extent that user U can recognize that sensory information acting on a predetermined sense is being presented to them. The discrimination threshold may be set in common for all users, or it may be set individually for each user. Alternatively, the discrimination threshold for user U may be set in real time based on acquired biometric information. Alternatively, the discrimination threshold for user U may be set by providing each sensory stimulus with different output intensities in advance and having user U input whether or not they can be discriminated against. Alternatively, a discrimination threshold may be set in advance for each set theme information IF1.

[0048] The discrimination threshold is set individually for each sensory stimulus. As will be described later, if the output intensity of each sensory stimulus is controlled by an individual control unit, it is preferable to set the discrimination threshold for each control unit as well. In other words, multiple discrimination thresholds corresponding to the sensory stimulus may be set for each sensory information. Furthermore, a threshold may also be set for the maximum value of the output intensity of each sensory information (sensory stimulus).

[0049] The identification unit 232 then identifies the control mode for each sensory information (activity A108). In this embodiment, the identification unit 232 identifies the timing for presenting each sensory stimulus to the user U as time progresses, and the variation pattern of the output intensity of each sensory stimulus. In particular, the presentation unit 233, as a second presentation step, presents transient information so as time progresses that it approaches multimodal information. At this time, the presentation unit 233 may vary at least one of the output intensity of the sensory information and the number of sensory stimuli included in the transient information in a manner that corresponds to user information.

[0050] The presentation unit 233 controls at least two of the image output device 3, sound output device 4, tactile presentation device 5, and olfactory presentation device 6 based on the set theme information IF1 and various information identified by the identification unit 232 (Activity A109). For example, each of the devices 3 to 6 is controlled to present each sensory information to the user U based on the control mode of each sensory stimulus identified by the identification unit 232.

[0051] In this embodiment, the presentation unit 233 first presents predetermined sensory information to the user U (activity A110). Then, the presentation unit 233 presents transitional information to the user U (activity A111), and then presents multimodal information IF2 to the user U (activity A112). As a result, transitional information is presented when switching from predetermined sensory information to multimodal information, providing the user U with an experience of fading into multimodal information.

[0052] The presentation unit 233 presents transitional information to user U after multimodal information has been presented (activity A112). Subsequently, the presentation of other sensory information to user U continues (activities A109, A110). This allows user U to experience the fading out of multimodal information by presenting transitional information when switching from multimodal information to predetermined sensory information. Furthermore, while user U is in the closed space 11, the information processing related to activities A109 to A113 may be repeatedly executed.

[0053] Subsequently, if the repeated information processing related to Activities A109 to A113 satisfies the termination condition, the presentation unit 233 terminates the presentation of sensory information. The presentation unit 233 then displays the user U's fatigue level, estimated according to the biometric information, on the display 31 again (Activity A114). This allows the user U to experience the effects of the presented sensory information. The termination condition is not particularly limited and may be, for example, a certain amount of time has elapsed, the fatigue level has fallen below or less than a threshold, the playback of the content set in theme information IF1 has finished, or the user U has performed a predetermined termination operation via the input unit 24.

[0054] The specified sensory information is sensory information different from multimodal information and transitional information, and is, for example, arbitrary content that is configured so as not to have continuity with them in terms of content. The specified sensory information may be, for example, a single piece of sensory information (such as background music or ambient sounds) or a combination of multiple pieces of sensory information (such as a video or a guide screen with feedback such as voice or vibration from a car navigation system).

[0055] Multimodal information is composed of a predetermined combination of multiple sensory information. The sensory information is one of visual, auditory, tactile, and olfactory information. In this embodiment, it is preferable that each of the sensory information is composed of different content. Furthermore, multimodal information is composed of multiple sensory information being presented in synchronous manner. That is, it is preferable that the control information for presenting each sensory stimulus is stored as mutually independent files. For example, the variation patterns of the output intensity of each sensory information with respect to elapsed time are set independently of each other, and these are played back in synchronous time, so that multiple sensory information is presented in synchronous manner and presented to user U as multimodal information. Therefore, it is preferable that this is distinguished from a form in which the control modes of visual and auditory information in the same video file are controlled according to elapsed time.

[0056] Transitional information is configured such that at least one of the number and output intensity of sensory information included in multimodal information is reduced compared to the multimodal information. In this embodiment, the multimodal information and the transitional information may be configured as content of the same nature. For example, when playback of a predetermined content is started, transitional information is configured by starting playback with at least one of the number and output intensity of sensory information presented in synchronization with the content reduced. Similarly, when playback of a predetermined content is finished, transitional information is configured by ending playback with at least one of the number and output intensity of sensory information presented in synchronization with the content reduced. This ensures that the transitional information and the multimodal information have continuity, preventing the user U from experiencing discomfort due to abrupt environmental changes.

[0057] More specifically, the transitional information is configured such that the number of types of sensory information is reduced compared to the sensory information included in the multimodal information, or the number of types of sensory information remains the same, but the output intensity of at least one of them is reduced. If the same type of sensory information in the multimodal information includes multiple different sensory stimuli, the transitional information may be configured to exclude at least one of these sensory stimuli. For example, if visual stimuli are output from the display 31 and the projector 32 in the multimodal information, the transitional information may be configured to output from only one of those visual stimuli. Even in this configuration, the output intensity of the sensory information in the transitional information is reduced compared to the multimodal information. On the other hand, sensory information not included in the multimodal information may be included in the transitional information.

[0058] In other words, in this embodiment, the presentation unit 233 presents multimodal information to a user located in a closed space as a first presentation step. Then, as a second presentation step, the presentation unit 233 presents transitional information including at least one sensory information to the user before or after the period in which the multimodal information is presented.

[0059] Furthermore, as a third presentation step, the presentation unit 233 presents predetermined sensory information to the user that is different from the multimodal information. The presentation unit presents multimodal information after the transitional information has been presented in the second presentation step. The presentation unit presents transitional information after the predetermined sensory information has been presented in the third presentation step.

[0060] This approach can provide a novel user experience, particularly by reducing user discomfort when multimodal information is presented.

[0061] 4. Details of Information Processing This section will explain the details of the information processing outlined in the previous section, using separate diagrams and other visual aids.

[0062] 4.1 Presentation of Multimodal Information The presented visual information VI, auditory information AI, tactile information TI, and olfactory information OI are presented as unified multimodal information IF2 that makes the user U feel a specific theme. More specifically, multimodal information IF2 is information structured to make the user U feel as if they are in a place that matches the theme information IF1. For example, the theme selected by the user U may be something that evokes the image of a place that can actually be visited, such as "forest," "beach," or "countryside," or it may be something that represents something or a concept that humans cannot actually go to in their physical bodies, such as "fire," "seabed," "moon," "space," or "stars." For example, if the set theme is "forest," an image of lush trees is displayed on the display 31 as visual stimulus VI1, and a blurry image of sunlight filtering through trees from the projector 32 is projected onto the inner wall of the enclosed space 11 as visual stimulus VI2. Furthermore, a faint sound of wind is played from the headphones 41 as auditory stimulus AI1, and the sound of birds chirping in the distance is played from the speaker 42 as auditory stimulus AI2. In addition, airflow from the fan 51 is provided as tactile stimulus TI1, warm air at a comfortable temperature range is provided as tactile stimulus TI2 from the air conditioner 52, and the scent of plants is provided as olfactory stimulus OI1 in the enclosed space 11 by the aroma diffuser 61.

[0063] In other words, depending on the theme, control methods for various components 3 to 6 are pre-managed in a table and stored in the memory unit 22. Then, based on the selected theme information IF1, the presentation unit 234 generates control signals for each component 3 to 6 and controls them, thereby presenting multimodal information IF2 that is in line with the theme. With this configuration, it becomes possible to give the user U an illusion of location, and consequently provide a novel user experience.

[0064] Preferably, the presented multimodal information IF2 is configured to change in content or intensity over time. For example, if the set theme is "forest," the wind direction may change, birdsong may be heard, or the lighting from the projector 32 may change as time passes. This configuration allows for the presentation of multimodal information IF2 that keeps the user U engaged, and ultimately provides a novel user experience.

[0065] More preferably, the display unit 234 variably displays multimodal information IF2 according to biological information. For example, the content or intensity may change in response to changes in biological information. For example, if user U is expected to recover from fatigue, the intensity of multimodal information IF2 may be weakened. With this configuration, it is possible to display desirable multimodal information IF2 tailored to user U's physical condition, and thereby provide a novel user experience.

[0066] Figure 6 is a schematic diagram showing an example of a method for presenting visual information VI. As shown in Figure 4, when using the information processing system 1, user U mainly looks directly at the display 31, so visual stimulus VI1 is mainly presented to user U. In addition, it is preferable that visual stimulus VI2 be projected over the entire inner wall of the closed space 11, which is an area not being looked at. In other words, visual information VI includes visual stimulus VI1 (an example of a first visual stimulus) and visual stimulus VI2 (an example of a second visual stimulus). Visual stimulus VI1 is presented in a position that user U looks directly at while in the closed space 11. Visual stimulus VI2 is presented around visual stimulus VI1 so as to cover user U's field of view. With this configuration, contrast can be added to the visual information VI, and consequently, a novel user experience can be provided.

[0067] Although not shown, contrast may also be added to information other than visual information VI. For example, in one embodiment of the information processing system 1, headphones 41 and a speaker 42 are included as the audio output device 4. The headphones 41 can directly deliver the auditory stimulus AI1 to the user U at a position close to their ears. The speaker 42 outputs the auditory stimulus AI2 from a distant position in the enclosed space 11, and can, for example, create an effect where only a faint sound is audible. In other words, the auditory information AI includes auditory stimulus AI1 (an example of a first auditory stimulus) and auditory stimulus AI2 (an example of a second auditory stimulus). Auditory stimulus AI1 is presented at a position close to the user U's ears. Auditory stimulus AI2 is presented in the enclosed space 11 in a way that the user U can perceive it as ambient sound. According to this embodiment, contrast can be added to the auditory information AI, and consequently, a novel user experience can be provided.

[0068] Thus, in this embodiment, multimodal information is composed of a combination of multiple sensory information that acts on different senses of user U, and furthermore, each sensory information may be composed of multiple sensory stimuli. The output intensity of the sensory information may be controlled by controlling the output intensity of at least one sensory stimulus included in each sensory information. That is, when the sensory information includes multiple sensory stimuli, the output intensity of one of the sensory stimuli may be controlled in the manner shown below, or the output intensity of multiple or all of the sensory stimuli may be controlled in the manner shown below.

[0069] The output intensity of visual information can be, for example, luminance (unit: candela per square meter (cd / m2)), luminous intensity (unit: candela (cd)), illuminance (unit: lux (lx)), contrast ratio (luminance ratio between the brightest and darkest parts of the screen; unit: ratio), color intensity (unit: 0 to 255 (in the case of 8-bit representation)), peak luminous intensity (unit: nits (cd / m2)), or gamma correction (unit: gamma value), and the output intensity is controlled using at least one of these as an indicator.

[0070] The output intensity of auditory information can be measured in various ways, such as sound pressure level (in decibels (dB)), power level (in decibel-milliwatts), acoustic intensity (in watts per square meter (W / m2)), loudness (in phons), sound pressure (in pascals (Pa)), or sound power (in watts (W)). At least one of these parameters is used as an indicator to control the output intensity.

[0071] The output intensity of tactile information is controlled using at least one of the following as an indicator when the tactile stimulus is contact-type: for example, vibration frequency (unit: Hertz (Hz)), vibration amplitude (unit: millimeters (mm) or micrometers (μm)), vibration acceleration (unit: m / s²), amount of energy required to generate vibration (unit: watts (W)), force (unit: Newtons (N)), pressure (unit: Pascals (Pa)), displacement (unit: millimeters (mm) or micrometers (μm)), tactile stress (unit: Pascals (Pa)), or tactile electrical stimulation (unit: milliamperes (mA)). When the tactile stimulus is non-contact-type: for example, wind speed (unit: meters per second (m / s)), wind pressure (unit: Pascals (Pa)), wind volume (unit: cubic meters per minute (m³ / min)), wind force (unit: Newtons (N)), or wind temperature (unit: Celsius (°C)).

[0072] The output intensity of olfactory information can be, for example, the concentration of odor components (unit: ppm), the odor index (unit: dimensionless numerical value, calculated based on the number of times the odor is diluted), or the odor intensity (unit: dimensionless numerical value, six-level odor intensity scale (0-5)), and the output intensity is controlled using at least one of these as an indicator.

[0073] Figure 7 is a schematic diagram showing an example of a method for presenting multimodal information. In Figure 7, the pattern of change in the output intensity of each sensory information over time is shown. As a reference example, Figure 7 shows a case in which, after content information IF30 as predetermined sensory information is presented to user U (Activity A110), multimodal information IF10 is presented to user U without transitional information being presented (Activity A112).

[0074] Figure 7 shows an example of a sensory information control method as a reference example. In the control method shown in Figure 7, after content information IF30 constituting a predetermined content is presented to the user U, time t ０In this case, the system switches to multimodal information IF10 without any delay. Consequently, the change in sensory information when multimodal information IF10 is presented becomes abrupt, which may increase the user's tension. This could cause user U to feel uncomfortable.

[0075] Specifically, content information IF30 is composed of visual information VI30 and tactile information TI30. Multimodal information IF10 is composed of visual information VI10, auditory information AI10, tactile information TI10, and olfactory information OI10. Multimodal information IF10 and content information IF30 are different in content, and each sensory information is composed of sensory stimuli to present different content. Content information IF30 is, for example, navigation information, visual information VI30 includes map information displayed on the display 31, and tactile information TI30 includes vibration to notify the user U of predetermined information. Multimodal information IF10 corresponds to the multimodal information IF2 described above. In Figure 7, the output intensity of each sensory information is shown as constant for simplification, but as mentioned above, the content or output intensity of each sensory information may fluctuate according to the elapsed time, and the fluctuation of output intensity according to the content is not shown.

[0076] 4.2 Presentation of Transitional Information Figures 8 and 9 are schematic diagrams illustrating an example of a method for presenting multimodal information. Figures 8 and 9 show the pattern of change in the output intensity of each sensory information over time. Figures 8 and 9 show a case where content information IF31 and IF32, as predetermined sensory information, are presented to user U (Activity A110), then transitional information IF21 and IF22 are presented (Activity A111), and then multimodal information IF11 and IF12 are presented to user U (Activity A112). Note that, as with Figure 7, the output intensity of each sensory information is shown as constant for simplification in Figures 8 and 9, and the variation in output intensity according to the content is not illustrated.

[0077] In Figure 8, content information IF31 is composed of visual information VI31 and tactile information TI31. Multimodal information IF11 is composed of visual information VI11, auditory information AI11, tactile information TI11, and olfactory information OI11. Transitional information IF21 is composed of visual information VI21, auditory information AI21, tactile information TI21, and olfactory information OI21.

[0078] Content information IF31 is navigation information similar to, for example, content information IF30. Multimodal information IF11 corresponds to the multimodal information IF2 described above. Transitional information IF21 is configured to have continuity in content with multimodal information IF11 as described above. Furthermore, transitional information IF21 is configured such that at least one of the number and output intensity of sensory information included in multimodal information IF11 is reduced compared to multimodal information IF11.

[0079] First, the display unit 233 displays time t ０ In this process, the output intensity of each sensory information included in the content information IF31 is controlled to be sufficiently small (set to an output intensity below the discrimination threshold, turn it off, etc.). Then, after a predetermined time has elapsed, time t １ In this step, the presentation unit 233 begins presenting the transitional information IF21. In other words, as a third presentation step, the presentation unit 233 controls the system so that the user cannot perceive predetermined sensory information, and then as a second presentation step, the presentation unit 233 presents the transitional information. This allows the user to distinguish and receive the effects of each piece of content.

[0080] Next, the display unit 233 displays time t １ At this point, the presentation of olfactory information OI21 begins. The presentation unit 233 also starts at time t. ２ At time t, the presentation of tactile information TI21 is initiated. The presentation unit 233 also starts at time t ３ At time t, the presentation of auditory information AI21 is initiated. The presentation unit 233 also starts at time t ４starts presenting the visual information VI21. In other words, as the second presenting step, the presenting unit 233 varies the number of sensory information pieces included in the transition period information so that the closer the elapsed time is to the period in which the multimodal information is presented, the closer the number approaches the number of sensory information pieces included in the multimodal information, and presents the transition period information. Note that the presentation of the transition period information IF21 may be started without sufficiently reducing the output intensity of each piece of sensory information included in the content information IF31, at time t ０ the presentation of the olfactory information OI21 may be started.

[0081] At this time, it is preferable that the intervals until the presentation of each piece of sensory information is started are controlled to exceed a threshold value (first threshold value). That is, the interval (t ２ −t １ ), the interval (t ３ −t ２ ), and the interval (t ４ −t ３ ) are preferably set such that at least one thereof exceeds the threshold value, and particularly preferably all intervals are set to exceed the threshold value. In other words, as the second presenting step, when the number of sensory information pieces included in the transition period information is varied, the presenting unit 233 further varies the number of sensory information pieces included in the transition period information after a time exceeding the first threshold has elapsed, and presents the transition period information.

[0082] Thereby, it is possible to prevent the change in the presented sensory stimulus from becoming steep, and to make it less likely to cause discomfort to the user U. Note that even if predetermined sensory information is combined, if the user U does not perceive it as a steep change in sensory stimulus, it is not necessary to secure an interval exceeding the threshold for the combination of the predetermined sensory stimuli (for example, sensory information with low priority for the user U). Thereby, the time for presenting the transition period information IF21 can be shortened, and the multimodal information IF11 can be presented to the user at an early stage.

[0083] Furthermore, the presentation unit 233 controls the output intensity of each sensory stimulus to increase over time. When the output intensity of each sensory stimulus reaches the output intensity of the corresponding sensory information included in the multimodal information, the presentation unit 233 stops increasing the output intensity and continues outputting the sensory information. When the output intensity of all sensory information reaches the output intensity of the corresponding sensory information included in the multimodal information IF 11, the presentation of the multimodal information IF 11 is started. In other words, as a second presentation step, the presentation unit 233 presents the transitional information by varying the output intensity of at least one of the sensory information included in the transitional information so that the elapsed time approaches the period during which the multimodal information is presented, and the output intensity of at least one of the sensory information included in the transitional information approaches the output intensity of the corresponding sensory information included in the multimodal information.

[0084] The output intensity may be controlled to increase continuously over time, or it may be controlled in steps to increase by a predetermined value after a predetermined time has elapsed. Furthermore, the output intensity is not limited to increasing monotonically; it may also be controlled to temporarily decrease before increasing again.

[0085] Furthermore, in Figure 8, the amount of change in output intensity over time is set in descending order to olfactory information OI21, visual information VI21, tactile information TI21, and auditory information AI21. In this way, it is preferable to control the amount of change in output intensity over time for each sensory information so that it is different. This makes it possible to flexibly control the transient information IF21. In other words, transient information includes multiple sensory information. As a second presentation step, the presentation unit 233 presents the transient information by controlling the amount of change in output intensity over time for each of the multiple sensory information included in the transient information so that it is different for each of the multiple sensory information included in the transient information.

[0086] Furthermore, since the units of output intensity may differ for each sensory information and sensory stimulus as described above, the amount of change in output intensity over time may be defined by the decrease in the percentage of output intensity deficit over time, with the output intensity at the start of presentation of the sensory information being set to 0% and the output intensity of the corresponding sensory information included in the multimodal information IF11 being set to 100%. Alternatively, the amount of change in output intensity over time may be defined as the time from the start of presentation of each sensory information until the output intensity of the corresponding sensory information included in the multimodal information IF11 is reached.

[0087] Furthermore, in order to prevent the changes in the presented sensory stimuli from becoming abrupt, it is preferable to keep the output intensity small when the presentation of sensory information begins, and to also keep the amount of variation in output intensity over time small. From this viewpoint, it is preferable that the output intensity when the presentation of sensory information begins be one-third or less of the output intensity of the corresponding sensory information included in the multimodal information IF11, and more preferably one-fifth or less.

[0088] Furthermore, as a second presentation step, the presentation unit 233 preferably presents the transient information by controlling the output intensity of the sensory information included in the transient information, or the number of sensory information items included in the transient information, based on priority, as the elapsed time approaches the period during which the multimodal information is presented. For example, if the amount of fluctuation in the output intensity of sensory information with high priority for user U is large, the change in stimulus received by the user may become abrupt. Therefore, it is preferable to control the output intensity at the start of presentation, or the amount of fluctuation in output intensity over time, for sensory information with higher priority. Also, from the viewpoint of preventing the presentation time of transient information IF21 from becoming long for high-priority sensory stimuli, the timing of the start of presentation may be controlled to be earlier for sensory information with higher priority. Also, from the viewpoint of presenting high-priority sensory information after user U has adapted to the change in sensory stimulus by being presented with low-priority sensory information, the timing of the start of presentation may be delayed for sensory information with higher priority.

[0089] Furthermore, as a second presentation step, the presentation unit 233 presents the transient information such that the output intensity of the sensory information included in the transient information exceeds the discrimination threshold TH1 (second threshold). This ensures that the effects described above, provided by the transient information IF21, are reliably delivered to the user U.

[0090] Next, with reference to Figure 9, another example of a method for presenting multimodal information will be explained. In Figure 9, the control mode of transient information differs from the multimodal information presentation method shown in Figure 8.

[0091] In Figure 9, content information IF32 is composed of visual information VI32 and tactile information TI32. In Figure 9, multimodal information IF12 is composed of visual information VI12, auditory information AI12, tactile information TI12, and olfactory information OI12. Transitional information IF22 is composed of visual information VI22, auditory information AI22, tactile information TI22, and olfactory information OI22.

[0092] Content information IF32 is navigation information similar to, for example, content information IF30. Multimodal information IF12 corresponds to the aforementioned multimodal information IF2. Transitional information IF22 is configured to have continuity in content with multimodal information IF12 as described above. Furthermore, transitional information IF22 is configured such that at least one of the number and output intensity of sensory information included in multimodal information IF12 is reduced compared to multimodal information IF12.

[0093] The display unit 233 displays time t ０In this configuration, the output of all corresponding sensory information contained in the multimodal information IF12 is initiated without controlling the output intensity of each sensory information contained in the content information IF31 to be sufficiently small. On the other hand, as a second presentation step, the presentation unit 233 presents the transient information by controlling the amount of change in the output intensity of the sensory information contained in the transient information over time to be different for each of the multiple sensory information contained in the transient information. For example, the time from the start of presentation of each sensory information until reaching the output intensity of the corresponding sensory information contained in the multimodal information IF11 is controlled to be different for each other, in ascending order: olfactory information OI22, auditory information AI22, tactile information TI22, and visual information VI22. Even in this configuration, it is prevented that the changes in the presented sensory stimuli become abrupt.

[0094] 5. Modified Example The information processing system 1 according to one embodiment can also be implemented in the following manner.

[0095] In the embodiments described above, the transitional information is presented before the multimodal information (fade-in), but the invention is not limited to this. The transitional information may be presented after the multimodal information is presented (fade-out). In this case, the number of types of sensory information included in the transitional information and / or the output intensity may be reduced so that the transitional information moves away from the multimodal information as time progresses. That is, the number of types of sensory information included in the transitional information and / or the output intensity are controlled so that the transitional information approaches the multimodal information as the period of presentation approaches. This makes it possible to present the transitional information in the same way as the fade-in, even in the case of a fade-out.

[0096] Furthermore, the transitional information presented before and after the presentation of multimodal information may be controlled in different ways. For example, before the presentation of multimodal information, the number of types of sensory information may be controlled to increase over time, while after the presentation of multimodal information, the output intensity of each sensory information may be varied without changing the number of types of sensory information over time.

[0097] In the embodiment described above, the acquisition unit 231 acquires the biometric information of user U as user information, and sensory information is controlled based on the acquired biometric information. However, the invention is not limited to this. User information may also be questionnaires acquired from user U in advance. For example, subjective evaluations of the user's mood and physical condition on that day may be acquired from user U before multimodal information is presented. In addition, the user may be asked to specify the ranking of sensory information that they consider important, or the sensory information they wish to be presented in the multimodal information. Furthermore, instead of user information relating to a specific user, information identified by statistical processing or the like based on various information acquired from multiple users may be acquired as user information. For example, the priority of each sensory stimulus that can be commonly applied to multiple users may be acquired as user information.

[0098] Alternatively, the acquisition unit 231 may be configured to control transient information without acquiring user information. This simplifies the configuration of the information processing device 2.

[0099] In the embodiments described above, the content information and transitional information are exemplified as being composed of multiple types of sensory information, but the invention is not limited to this. The content information and transitional information may be composed of only one type of sensory information. For example, any sensory information may be used for the content information. Furthermore, the transitional information may be composed of fewer types of sensory information and / or with reduced output intensity compared to the multimodal information, and there may be periods composed of only one type of sensory information. Also, for example, if the multimodal information is composed of two types of sensory information, the transitional information may be composed of only one type of sensory information.

[0100] Furthermore, while the embodiment shown in Figure 8 illustrates how both the number of types of sensory information and their output intensity in the transient information vary over time, the system is not limited to this. For example, the output intensity of the sensory information included in the transient information may remain constant, while only the number of types of sensory information is varied. That is, while the timing at which each sensory information is presented is controlled to be different, the output intensity of each sensory stimulus may be controlled to be equivalent to the output intensity of the corresponding sensory information included in the multimodal information at the time the output of the sensory stimulus begins.

[0101] Furthermore, while it was illustrated that the transitional information is the same content as the multimodal information, the transitional information and the multimodal information may also be structured as separate content that user U can recognize as having continuity in their content. For example, different content belonging to the same theme may be used as the transitional information and the multimodal information. Even in such cases, presenting the transitional information can reduce the user's discomfort when the multimodal information is presented.

[0102] The continuity between transient information and multimodal information may be determined based on the similarity of the user experience they provide. Furthermore, the similarity may be determined holistically based on the combination of sensory information contained within each, or it may be determined individually for each piece of sensory information. For example, if a predetermined number or more of the sensory stimuli contained in the sensory information are similar, then the predetermined pieces of sensory information may be judged as similar. If a predetermined number or more of the sensory information are similar, then the transient information and multimodal information may be judged as similar and have continuity. The similarity of each sensory stimulus and piece of sensory information may be determined by a user's sensory evaluation, or it may be determined based on predetermined numerical values.

[0103] Furthermore, the order in which the sensory information included in the transitional information is presented is arbitrary and may be set appropriately based on predetermined criteria. Also, the control methods such as the initial output intensity of the sensory information included in the transitional information, the order in which the output intensity reaches the output intensity of the corresponding sensory information included in the multimodal information, and the relative lengths of the time it takes to reach these levels are also arbitrary and may be set appropriately based on predetermined criteria.

[0104] In one embodiment, it was illustrated that user U is assumed to be resting in a closed space 11, and that a healing effect is provided to the user through multimodal information, but the system is not limited to this. For example, it may be assumed that user U is driving a car in the closed space 11, and the system may be configured to provide the user with enhanced concentration or reduced fatigue through multimodal information.

[0105] Furthermore, user U may eat or drink in the enclosed space 11. In this case, the presentation unit 234 presents multimodal information IF2 to user U who is eating or drinking in the enclosed space 11, thereby causing a change in user U's taste perception. This configuration can cause a change in user U's taste perception and, consequently, provide a novel user experience.

[0106] Biometric information may be acquired by other devices instead of the wearable device 7. For example, other devices may include at least a smartphone, a body composition analyzer, a thermometer, medical equipment such as an MRI, CT scanner, or X-ray machine, a thermograph, a blood pressure monitor, a skin age measuring device, and a blood glucose measuring device. On the other hand, an embodiment that does not acquire biometric information may also be adopted. For example, instead of biometric information from the wearable device 7, the degree of fatigue of user U may be determined from facial expressions. Specifically, the information processing system 1 may be equipped with a camera (not shown), and the user U's face may be captured by the camera. By analyzing the facial expressions included in the captured image data, the degree of fatigue of user U can be determined.

[0107] In one embodiment, a table is used in which the theme information IF1 and the control methods for each component 3 to 6 are related to each other. However, instead of or in addition to the table, reference information such as a trained model that has undergone deep learning or a large-scale language model may be used. For example, when the multimodal information IF2 is made variable over time, the preferences and fatigue recovery tendencies of users U who have used it in the past may be learned, and the system may be designed to perform more favorably each time it is used. Various reference information may be stored in an external memory area of ​​the information processing system 1, and the information processing device 2 may be configured to output the theme information IF1 and the control mode of the transitional information by inputting user information into the reference information.

[0108] In one embodiment, the multimodal information IF2 was presented variably according to biometric information acquired via the wearable device 7. However, the multimodal information IF2 may be presented variably according to specified information desired by the user U, either together with or in place of biometric information. More specifically, the specified information may include information about the state after using the information processing system 1. For example, the specified information may include information about states such as feeling energetic, wanting to calm down, or wanting to relax. The specified information may also be recommended by the information processing system 1, or it may be selected by the user U. In other words, the acquisition unit 231 acquires the specified information designated by the user U as an acquisition step. Then, the presentation unit 233 presents the multimodal information IF2 variably according to the specified information as a first presentation step. With this embodiment, it is possible to present the user U with their preferred multimodal information IF2, and thereby provide a novel user experience.

[0109] When implementing the information processing system 1 as a business, a program may be implemented, an information processing method may be implemented, an information processing device 2 may be implemented, the information processing device 2 and at least two of the components 3 to 6 may be implemented, and a wearable device 7 may also be implemented.

[0110] The information processing device 2 and each of the components 3 to 6 may be remotely connected via a network. In such a case, for example, a user U who wishes to use it personally may purchase each of the components 3 to 6 for a small room in their home and access the information processing device 2, which is being used commercially, to perform the aforementioned information processing (a so-called cloud-type service). When provided as such a service, at least one of the devices included in the information processing system 1 may be installed outside of Japan. For example, even if the information processing device 2 is installed outside of Japan, user U may access the information processing device 2 from within Japan using each of the components 3 to 6 in their own small room. With this configuration, a more interesting experience can be provided to the user through various management forms.

[0111] In this embodiment, the acquisition unit 231, the identification unit 232, and the presentation unit 233 are described as functional units realized by the control unit 23 of the information processing device 2. However, at least a part of these may be implemented as functional units realized by components other than the information processing device 2. Furthermore, the various types of information described in the above example may be stored in a distributed manner not only in the storage unit 22 of the information processing device 2, but also in other external devices. In such cases, distributed ledger management based on blockchain or the like may be implemented. Alternatively, each of the components 3 to 6 may have a processor and be able to communicate with each other, so that information processing can be performed without going through the information processing device 2.

[0112] Furthermore, they may be provided in the following embodiments.

[0113] (1) An information processing system comprising at least one processor, wherein the processor is configured to perform the following steps by reading a program, wherein in a first presentation step, multimodal information is presented to a user located in a closed space, wherein the multimodal information comprises a predetermined combination of a plurality of sensory information, wherein the sensory information is any of visual information, auditory information, tactile information and olfactory information, and in a second presentation step, before or after the first presentation step, transient information including at least one of the sensory information is presented to the user, wherein the transient information is configured such that at least one of the number and output intensity of the sensory information included in the multimodal information is reduced compared to the multimodal information.

[0114] This approach allows for the provision of novel user experiences.

[0115] (2) An information processing system as described in (1) above, wherein in the second presentation step, the output intensity of at least one of the sensory information included in the transient information is varied so that as the elapsed time approaches the first presentation step, it approaches the output intensity of the corresponding sensory information included in the multimodal information, and the transient information is presented.

[0116] This configuration can reduce user discomfort when multimodal information is presented.

[0117] (3) An information processing system as described in (2) above, wherein the transient information includes a plurality of the sensory information, and in the second presentation step, the transient information is presented by controlling the amount of change in the output intensity of the sensory information included in the transient information over time to be different for each of the plurality of sensory information included in the transient information.

[0118] This configuration can reduce user discomfort when multimodal information is presented.

[0119] (4) An information processing system according to any one of (1) to (3) above, wherein in the second presentation step, the number of sensory information included in the transitional information is varied so that as the elapsed time approaches the first presentation step, the number of sensory information included in the multimodal information approaches the number of sensory information included in the multimodal information, and the transitional information is presented.

[0120] This configuration can reduce user discomfort when multimodal information is presented.

[0121] (5) An information processing system as described in (4) above, wherein in the second presentation step, when the number of sensory information included in the transient information is changed, the number of sensory information included in the transient information is changed again after a time has elapsed that exceeds the first threshold, and the transient information is presented.

[0122] This configuration can reduce user discomfort when multimodal information is presented.

[0123] (6) An information processing system according to any one of (1) to (5) above, wherein in a first identification step, the priority of the sensory information to the user is identified, and in a second presentation step, as the elapsed time approaches the first presentation step, the output intensity of the sensory information included in the transitional information, or the number of sensory information included in the transitional information, is controlled based on the priority, and the transitional information is presented.

[0124] This configuration can reduce user discomfort when multimodal information is presented.

[0125] (7) An information processing system according to any one of (1) to (6) above, wherein in the second identification step, a second threshold is identified which is the output intensity of the sensory information that the user can recognize, and in the second presentation step, the transient information is presented such that the output intensity of the sensory information included in the transient information exceeds the second threshold.

[0126] This configuration can reduce user discomfort when multimodal information is presented.

[0127] (8) An information processing system according to any one of (1) to (7) above, wherein in the third presentation step, predetermined sensory information different from the multimodal information is presented to the user, wherein the first presentation step is performed after the second presentation step, the second presentation step is performed after the third presentation step, and the second presentation step is performed after the third presentation step has been controlled so that the predetermined sensory information cannot be recognized by the user.

[0128] This configuration reduces user discomfort when switching between sensory information presented to the user.

[0129] (9) An information processing system according to any one of (1) to (8) above, wherein in the acquisition step, user information relating to the user is acquired; in the second presentation step, the transitional information is presented in such a way that it approaches the multimodal information as time progresses, and in this way, at least one of the output intensity of the sensory information and the number of sensory information included in the transitional information is varied in a manner corresponding to the user information.

[0130] This configuration can reduce user discomfort when multimodal information is presented.

[0131] (10) An information processing system according to any one of (1) to (9) above, wherein each of the sensory information is configured as different content, and the multimodal information is configured by the synchronized presentation of multiple sensory information.

[0132] This approach allows for the provision of novel user experiences.

[0133] (11) An information processing method comprising each step of the information processing system described in any one of (1) to (10) above.

[0134] (12) A program that causes a computer to perform each step of the information processing system described in any one of (1) to (10) above. Of course, this is not limited to this.

[0135] Finally, various embodiments of the present invention have been described, but these are presented as examples and are not intended to limit the scope of the invention. These novel embodiments can be implemented in a variety of other forms, and various omissions, substitutions, and modifications can be made without departing from the spirit of the invention. These embodiments and their variations are included in the scope and spirit of the invention, as well as in the claims and their equivalents.

[0136] In the following, to verify the effectiveness of the information processing method described in the embodiment, the Japanese version of the Perceived Restorative Scale (PRS) was implemented under each of the multimodal information control conditions shown in this embodiment. The PRS is a scale for evaluating the restorative properties of an environment as perceived by humans, and it has a track record of use indoors. Table 2 shows a list of questions for each sub-item of the PRS.

[0137] Figure 10 shows the evaluation results of the PRS in this experiment. In Figure 10, of the two bar graphs corresponding to the sub-items of the PRS, the left bar graph shows the value when multimodal information IF10 was presented in the manner shown in Figure 7, and the right bar graph shows the value when multimodal information IF11 was presented in the manner shown in Figure 8. There were 7 subjects. A paired two-tailed t-test was performed for each sub-item, and the results showed a p-value < 0.05 not only for the three items "escape," "attraction," and "field of view," which are considered particularly important for the recovery characteristics of the environment, but also for "suitability" and "preference," indicating a statistically significant difference. In other words, the environment in which multimodal information IF11 was presented in the manner shown in Figure 8 had higher recovery characteristics than the environment in which multimodal information IF10 was presented in the manner shown in Figure 7.

[0138] 1: Information processing system, 11: Closed space, 2: Information processing device, 20: Communication bus, 21: Communication unit, 22: Memory unit, 23: Control unit, 231: Acquisition unit, 232: Identification unit, 233: Presentation unit, 234: Presentation unit, 24: Input unit, 3: Image output device, 31: Display, 32: Projector, 4: Audio output device, 41: Headphones, 42: Speaker, 5: Tactile presentation device, 51: Fan, 52: Air conditioner 6: Olfactory presentation device, 61: Aroma diffuser, 7: Wearable device, IF10: Multimodal information, IF11: Multimodal information, IF12: Multimodal information, IF21: Transitional information, IF22: Transitional information, IF30: Content information, IF31: Content information, IF32: Content information, AI: Auditory information, OI: Olfactory information, TI: Tactile information, VI: Visual information

Claims

1. An information processing system comprising at least one processor, wherein the processor is configured to perform the following steps by reading a program, wherein in a first presentation step, multimodal information is presented to a user located in a closed space, wherein the multimodal information comprises a predetermined combination of a plurality of sensory information, wherein the sensory information is any of visual information, auditory information, tactile information and olfactory information, and in a second presentation step, before or after the first presentation step, transient information including at least one of the sensory information is presented to the user, wherein the transient information is configured such that at least one of the number and output intensity of the sensory information included in the multimodal information is reduced compared to the multimodal information.

2. An information processing system according to claim 1, wherein in the second presentation step, the output intensity of at least one of the sensory information included in the transient information is varied so that as the elapsed time approaches the first presentation step, the transient information is presented.

3. An information processing system according to claim 2, wherein the transient information includes a plurality of sensory information, and in the second presentation step, the transient information is presented by controlling the amount of change in the output intensity of the sensory information included in the transient information over time to be different for each of the plurality of sensory information included in the transient information.

4. An information processing system according to any one of claims 1 to 3, wherein in the second presentation step, the number of sensory information included in the transitional information is varied so that as the elapsed time approaches the first presentation step, the number of sensory information included in the multimodal information approaches the number of sensory information included in the multimodal information, and the transitional information is presented.

5. An information processing system according to claim 4, wherein in the second presentation step, when the number of sensory information included in the transient information is varied, the number of sensory information included in the transient information is further varied after a time has elapsed that exceeds the first threshold, and the transient information is presented.

6. An information processing system according to any one of claims 1 to 5, wherein in a first identification step, the priority of the sensory information to the user is identified, and in a second presentation step, as the elapsed time approaches the first presentation step, the output intensity of the sensory information included in the transient information, or the number of sensory information included in the transient information, is controlled based on the priority, and the transient information is presented.

7. An information processing system according to any one of claims 1 to 6, wherein in a second identification step, a second threshold is identified which is the output intensity of the sensory information that the user can recognize, and in a second presentation step, the transient information is presented such that the output intensity of the sensory information included in the transient information exceeds the second threshold.

8. An information processing system according to any one of claims 1 to 7, wherein in a third presentation step, predetermined sensory information different from the multimodal information is presented to the user, wherein the first presentation step is performed after the second presentation step, the second presentation step is performed after the third presentation step, and the second presentation step is performed after the third presentation step has been controlled so that the predetermined sensory information cannot be recognized by the user.

9. An information processing system according to any one of claims 1 to 8, wherein in the acquisition step, user information relating to the user is acquired; in the second presentation step, the transitional information is presented such that it approaches the multimodal information as time progresses, and thereafter, at least one of the output intensity of the sensory information and the number of sensory information included in the transitional information is varied in a manner corresponding to the user information.

10. An information processing system according to any one of claims 1 to 9, wherein each of the sensory information is configured as different content, and the multimodal information is configured by the synchronized presentation of a plurality of the sensory information.

11. An information processing method comprising each step of the information processing system described in any one of claims 1 to 10.

12. A program that causes a computer to perform each step of the information processing system described in any one of claims 1 to 10.

Images