Information processing apparatus, information processing method, and program

Abstract

The present disclosure relates to an information processing apparatus, an information processing method, and a program that make it possible to provide a user experience that gives a greater sense of immersion and presence. According to the present invention, a sensor data analysis processing unit performs data analysis processing on a sensor data group that is outputted from a plurality of sensors that are arranged in a prescribed space in which video content is reproduced and acquires sensor data analysis information that includes information obtained by detecting the situation in the space, information obtained by detecting the situation of a user that is viewing the video content, and information obtained by measuring the face and ears of the user, and, on the basis of the sensor data analysis information, a device integrated control unit generates a device control data group for integral control of a plurality of sensation presentation devices that are arranged in the space. The present technology can be applied, for example, to a system that provides an immersive experience in an interior space of a vehicle.

Description

Information processing apparatus, information processing method, and program

[0001] The present disclosure relates to an information processing apparatus, an information processing method, and a program, and particularly relates to an information processing apparatus, an information processing method, and a program that can provide a user experience with higher immersion and presence.

[0002] Conventionally, techniques have been developed to improve the presence of content by controlling various devices such as video, audio, and vibration in a vehicle.

[0003] For example, Patent Document 1 discloses an information processing apparatus that detects a scene where sound is generated from input content, extracts parameters corresponding to the scene, and outputs a signal obtained by enhancing an audio signal with the parameters to a vibration device.

[0004] Japanese Unexamined Patent Publication No. 2023-116109

[0005] However, in the conventional technology, there is no integrated linkage between various devices such as video, audio, and vibration, and the types of sensory stimuli are also limited, so it has not been possible to provide a user experience with higher immersion and presence.

[0006] The present disclosure has been made in view of such circumstances, and aims to enable provision of a user experience with higher immersion and presence.

[0007] An information processing apparatus according to one aspect of the present disclosure includes: a sensor data analysis processing unit that performs data analysis processing on a sensor data group output from a plurality of sensors arranged in a predetermined space where video content is reproduced, and acquires sensor data analysis information including information detecting a state of the space, information detecting a state of a user viewing the video content, and information obtained by measuring the user's face and ears; and an integrated device control unit that generates a device control data group for integrally controlling a plurality of sensory presentation devices arranged in the space based on the sensor data analysis information.

[0008] One aspect of the information processing method or program of this disclosure includes performing data analysis processing on a group of sensor data output from a plurality of sensors arranged in a predetermined space for playing video content, to obtain sensor data analysis information including information detecting the conditions of the space, information detecting the state of a user viewing the video content, and information measuring the user's face and ears, and generating a group of device control data that integrally controls a plurality of sensory presentation devices arranged in the space based on the sensor data analysis information.

[0009] In one aspect of this disclosure, data analysis processing is performed on a group of sensor data output from multiple sensors arranged in a predetermined space for playing video content. Sensor data analysis information is obtained, which includes information detecting the conditions of the space, information detecting the state of the user viewing the video content, and information measuring the user's face and ears. Based on the sensor data analysis information, a group of device control data is generated to integrally control multiple sensory presentation devices arranged in the space.

[0010] This figure shows an example of the arrangement of multiple sensory feedback devices in a vehicle equipped with an immersive experience provision system that applies this technology. This is a block diagram showing an example of the functional configuration of the immersive experience provision system. This is a block diagram showing an example of the configuration of the sensor data analysis processing unit. This is a block diagram showing an example of the configuration of the device integration control unit. This is a block diagram showing an example of the configuration of the device control waveform generation processing unit. This figure illustrates a second application example. This figure illustrates a third application example. This is a flowchart illustrating the immersive experience provision process. This is a flowchart illustrating the sensor data analysis process. This is a block diagram showing an example of the configuration of one embodiment of a computer to which this technology applies.

[0011] The following describes in detail a specific embodiment of this technology, with reference to the drawings.

[0012] <Example Configuration of an Immersive Experience Provisioning System> Referring to Figures 1 to 5, an example configuration of an immersive experience provisioning system to which this technology is applied will be described.

[0013] The immersive experience provision system 11 can provide a more immersive and realistic user experience (hereinafter referred to as "immersive experience"), for example, in the interior space of a vehicle 12. In the example shown in Figure 1, when a user sitting in the rear seat 13 of the vehicle 12 plays video content, the immersive experience provision system 11 can provide an immersive experience through sensory presentation using various sensory presentation devices (for example, presentations that stimulate the user's sight, hearing, touch, and smell).

[0014] Figure 1A shows a plan view of the interior space of a vehicle 12 in which multiple sensory presentation devices of the immersive experience provision system 11 are arranged, and Figure 1B shows a side view of the interior space of a vehicle 12 in which multiple sensory presentation devices are arranged in the same way as in Figure 1A. As shown in Figure 1A, the vehicle 12 is provided with a right rear seat 13R and a left rear seat 13L, but when it is not necessary to distinguish between the right rear seat 13R and the left rear seat 13L, they are simply referred to as rear seats 13. Furthermore, although sensory presentation devices are arranged for each of the rear seats 13R and rear seats 13L, the right sensory presentation device arranged for the right rear seat 13R and the left sensory presentation device arranged for the left rear seat 13L are similarly referred to.

[0015] Figure 1 shows an example of the arrangement of multiple sensory presentation devices placed in a vehicle 12 equipped with an immersive experience provision system 11, including a glasses-free stereoscopic display 21, stereoscopic sound headrest speakers 22, a vibration device 23, a scent presentation device 24, a mist generating device 25, a wind generating device 26, a hot air generating device 27, and an RGB lighting device 28.

[0016] The naked-eye stereoscopic display 21 is positioned in front of the rear seat 13 and stimulates the user's vision by displaying video content in three dimensions. For example, the naked-eye stereoscopic display 21 displays parallax images corresponding to the parallax of each eye to the left and right eyes of the user sitting in the rear seat 13, enabling the user to view three-dimensional images with the naked eye.

[0017] The 3D audio headrest speaker 22 is built into the headrest portion of the rear seat 13 and includes a right speaker, a left speaker, and a circuit board that outputs left and right audio signals to each speaker. The right speaker is positioned to the right ear side when the user is seated in the rear seat 13, and the left speaker is positioned to the left ear side when the user is seated in the rear seat 13, stimulating the user's hearing by outputting audio from video content. For example, the 3D audio headrest speaker 22 makes it possible for the user to hear three-dimensional sound by playing left and right audio according to the user's face position and orientation and the position and shape of the user's ears, based on user face and ear measurement information obtained by measuring the position and orientation of the user's face and the position and shape of the user's ears.

[0018] Furthermore, by positioning the right and left speakers of the 3D sound headrest speaker 22 near the user's right and left ears, respectively, audio interference with other rear seats 13 can be avoided. This allows the 3D sound headrest speaker 22 to adjust the type of sound, volume, timbre, etc., for each user sitting in the rear seat 13.

[0019] The vibration device 23 can provide the user with vibration localization and a sense of three-dimensionality through vibration by linking multiple devices together and controlling the generation of vibrations in an integrated manner (delay control and phase control), allowing the user to experience a rich vibration experience that could not be achieved with a single device. As shown in Figure 1, in this embodiment, the vibration device 23 is composed of a subwoofer 31, a seat belt actuator 32, and a seat vibration actuator 33.

[0020] The subwoofer 31 is fixed to the floor of the vehicle 12 in order to allow the user to experience vibrations from the entire vehicle 12, centered on the floor of the vehicle 12. In the illustrated example, the subwoofer 31 is positioned in front of the user's feet when they are sitting in the rear seat 13 (i.e., on the floor under the front seat), and mainly generates vibrations that are transmitted to the user's feet from a distance in front, stimulating the user's sense of touch. For example, if the subwoofer 31 is positioned at the rear of the vehicle 12, the subwoofer 31 will mainly generate vibrations that are transmitted to the user's feet from a distance at the rear.

[0021] The seat belt actuator 32 is positioned (mounted) on the buckle portion that secures the seat belt of the rear seat 13. For example, the seat belt actuator 32 can generate vibrations that are transmitted to the entire seat belt without phase shift, and mainly generates vibrations on the front side of the user sitting in the rear seat 13, stimulating the user's sense of touch.

[0022] The seat vibration actuator 33 is installed between the seat surface and backrest of the rear seat 13 and is positioned (mounted) to connect to, for example, an ISO FIX fitting, which is an international standard for securing child restraint systems (so-called child seats). For example, the seat vibration actuator 33 can generate vibrations that are transmitted throughout the rear seat 13, mainly generating vibrations on the back side of the user sitting in the rear seat 13 to stimulate the user's sense of touch.

[0023] The scent presentation device 24 is positioned in front of the rear seats 13. In the illustrated example, only one device is positioned in front of the right rear seat 13R and the left rear seat 13L (between the left and right front seats). For example, the scent presentation device 24 stimulates the user's sense of smell by switching between multiple scent types to present the optimal scent at the optimal time to match the scene in the video content. Furthermore, the scent presentation device 24 can instantly switch the scent within the interior space of the vehicle 12 by using a fan to send out volatile fragrance, thereby enhancing the coordination between the scent presented by the scent presentation device 24 and the scenes in the video content and other sensory presentation devices.

[0024] The mist generating devices 25 are positioned on the left and right sides of the headrest portion of the rear seat 13. In the illustrated example, mist generating device 25-1 is positioned on the right side of the headrest portion of the rear seat 13, and mist generating device 25-2 is positioned on the left side of the headrest portion of the rear seat 13. For example, the mist generating device 25 generates water vapor by adding ultrasonic waves to water and sprays mist forward from the left and right sides of the headrest portion to stimulate the user's sense of touch. Furthermore, by using a fan to send out the generated mist, the mist generating device 25 can control the timing and amount of mist to match the scene in the video content.

[0025] The air-generating device 26 and the hot air-generating device 27 are positioned above and in front of the rear seat 13 (near the rear of the front seat and near the ceiling of the vehicle 12). In the illustrated example, the air-generating device 26R and the hot air-generating device 27R are positioned from the front left side as viewed from the right rear seat 13R, towards the vicinity of the face of the user sitting in the right rear seat 13R. Similarly, the air-generating device 26L and the hot air-generating device 27L are positioned from the front right side as viewed from the left rear seat 13L, towards the vicinity of the face of the user sitting in the left rear seat 13L.

[0026] The wind generating device 26 generates wind by rotating a fan. The hot air generating device 27 generates hot air by passing the wind generated by rotating the fan through a heat source. As a result, the wind generating device 26 and the hot air generating device 27 generate wind and hot air with freely controlled temperature, timing, and airflow to stimulate the user's sense of touch.

[0027] The RGB lighting device 28R is positioned above the right side of the right rear seat 13R, and the RGB lighting device 28L is positioned above the left side of the left rear seat 13L. For example, the RGB lighting device 28 changes the lighting color of the interior space of the vehicle 12 by instantly switching between various colors of light emission that match the video content, thereby stimulating the user's vision.

[0028] Note that the arrangement of the naked-eye stereoscopic display 21, stereoscopic sound headrest speaker 22, vibration device 23, scent presentation device 24, mist generating device 25, wind generating device 26, hot air generating device 27, and RGB lighting device 28 shown in Figure 1 is just one example; each sensory presentation device should be placed in a position that makes the sensory experience more effective.

[0029] Furthermore, the immersive experience provision system 11 can provide a more immersive experience to the user by controlling the sensory presentation devices arranged in this manner in conjunction with the video content, thereby stimulating the user's senses of sight, hearing, smell, and touch comprehensively.

[0030] Figure 2 is a block diagram showing an example of the functional configuration of the immersive experience provision system 11.

[0031] As shown in Figure 2, the immersive experience provision system 11 is configured to include a group of sensory presentation devices 41, an operation unit 42, a group of sensor devices 43, and a control processing unit 44.

[0032] As described with reference to Figure 1 above, the sensory presentation device group 41 comprises a naked-eye stereoscopic display 21, a stereoscopic sound headrest speaker 22, a vibration device 23 (subwoofer 31, seat belt actuator 32, and seat vibration actuator 33), a scent presentation device 24, a mist generating device 25, a wind generating device 26, a hot air generating device 27, and an RGB lighting device 28.

[0033] The operation unit 42 specifies the video content to be played when providing an immersive experience using the immersive experience provision system 11. When a user performs an operation to instruct the playback of that video content, the operation unit 42 acquires operation information indicating the user's operation and supplies it to the control processing unit 44.

[0034] The sensor device group 43 comprises a microphone 51, an inertial measurement sensor 52, a vehicle sensor 53, a thermal image sensor 54, a camera 55, an electroencephalogram sensor 56, and a millimeter-wave radar 57.

[0035] The microphone 51 captures sound in the interior space of the vehicle 12 and outputs raw microphone data representing that sound.

[0036] The inertial measurement sensor 52 is a device (6DoF IMU (6 Degree of Freedom Inertial Measurement Unit)) that measures the acceleration and angular velocity of the three axes of the vehicle 12, and outputs raw inertial measurement data showing the acceleration and angular velocity of the three axes of the vehicle 12.

[0037] The vehicle sensor 53 communicates with multiple ECUs (Electronic Control Units) in the vehicle 12 using a CAN (Controller Area Network) to acquire various information about the vehicle 12 and outputs raw vehicle data. For example, the raw vehicle data includes vehicle-to-vehicle communication information such as location information, speed information, and vehicle control information transmitted between vehicles 12, traffic information such as traffic congestion information, road closure information, congestion forecast information, and regulation information, and various other information.

[0038] The thermal image sensor 54 acquires a thermal image of the interior space of the vehicle 12 (including the user sitting in the rear seat 13) using infrared detection elements arranged in an array on the sensor surface, and outputs raw thermal image data showing that thermal image.

[0039] Camera 55 acquires an image of the interior space of the vehicle 12 (including the user sitting in the rear seat 13) and outputs camera RGB data representing that image.

[0040] The electroencephalogram (EEG) sensor 56 measures the electrical signals (EEGs) emitted by the brain of the user sitting in the rear seat 13 and outputs raw EEG data showing those electrical signals.

[0041] The millimeter-wave radar 57 emits millimeter waves into the interior space of the vehicle 12 (including the user sitting in the rear seat 13) and outputs raw millimeter-wave data obtained by measuring the reflected waves.

[0042] Then, the sensor data group obtained by sensing performed by the sensor device group 43 (i.e., raw microphone data, raw inertial measurement data, raw vehicle data, raw thermal image data, raw electroencephalogram data, camera RGB data, and raw millimeter-wave data) is supplied to the sensor data analysis processing unit 65 of the control processing device 44.

[0043] The control processing device 44 is configured to include a video content acquisition unit 61, a video preprocessing unit 62, a video and audio feature analysis unit 63, a scene discrimination unit 64, a sensor data analysis processing unit 65, and a device integrated control unit 66.

[0044] The video content acquisition unit 61 acquires video content specified by a user from a video database (not shown) in accordance with operation information supplied from the operation unit 42. Then, the video content acquisition unit 61 supplies raw video data and raw audio data obtained by playing back the video content to the video preprocessing unit 62 and the device integrated control unit 66.

[0045] The video preprocessing unit 62 performs necessary video preprocessing (for example, processing for reducing resolution, processing for reducing the data amount, etc.) on the raw video data supplied from the video content acquisition unit 61 to increase the processing speed of subsequent processing. Then, the video preprocessing unit 62 supplies the preprocessed raw video data and the raw audio data supplied from the video content acquisition unit 61 to the video and audio feature analysis unit 63 and the scene discrimination unit 64.

[0046] The video and audio feature analysis unit 63 analyzes features of the video according to changes in motion, brightness, color tone, and the like in the video based on the raw video data supplied from the video preprocessing unit 62. The video and audio feature analysis unit 63 also analyzes features of the audio according to the volume of the audio, music components, whether human voices or sound effects are dominant, and the like based on the raw audio data supplied from the video preprocessing unit 62. Then, the video and audio feature analysis unit 63 supplies video analysis information obtained as a result of analyzing the video features and audio analysis information obtained as a result of analyzing the audio features to the device integrated control unit 66.

[0047] The scene discrimination unit 64 discriminates each scene in an order conforming to the time-series data of video and audio, for example, using AI (Artificial Intelligence), based on the raw video data and raw audio data supplied from the video preprocessing unit 62, and acquires a word group matching the content, meaning, and the like of each scene. For example, the scene discrimination unit 64 can acquire a word group such as "smoke is rising", "a dinosaur is approaching from the front", "hot", "dark", and "wind is blowing" based on the raw video data. Furthermore, the scene discrimination unit 64 can acquire a word group such as "music is playing", "an explosion sound is occurring", "silence", and "mysterious sound" based on the raw audio data. Then, the scene discrimination unit 64 supplies detected word information indicating the word group detected for each scene discriminated based on the raw video data and raw audio data to the integrated device control unit 66.

[0048] The sensor data analysis processing unit 65 acquires vehicle situation detection information, user face and ear measurement information, and user state estimation information by performing data analysis processing on a sensor data group supplied from the sensor device group 43, and supplies the acquired information to the integrated device control unit 66. For example, the vehicle situation detection information includes information such as noise level and temperature in the interior space of the vehicle 12, and acceleration and angular velocity representing the motion of the vehicle 12. The user face and ear measurement information includes information such as the position and orientation of the user's face, and the position and shape of the user's ears. The user state estimation information includes information indicating the user's state (concentration level, excitement level, relaxation level, estimated emotion, etc.) estimated from changes in the user's face, facial expression, emotion, heart rate, blood pressure, and the like. The detailed configuration of the sensor data analysis processing unit 65 will be described later with reference to FIG. 3.

[0049] The integrated device control unit 66 can communicate with a server 101 provided on a network via a communication device (not shown). For example, the server 101 accumulates, for each general human type (age, gender, body shape, posture during viewing, etc.), information indicating reactivity that represents how easily a person of that type reacts when receiving a stimulus to the senses (hereinafter referred to as generalized human information).

[0050] For example, the device integration control unit 66 can obtain human generalization information corresponding to a user from the server 101, according to the user type indicated by the result of image recognition processing performed on an image obtained by the camera 55 of a user sitting in the rear seat 13 of the vehicle 12. The device integration control unit 66 can then correct the timing of controlling the sensory presentation device group 41 so that the sensory experience provided by the sensory presentation device group 41 is optimized, based on the responsiveness indicated by the human generalization information. Furthermore, the device integration control unit 66 can estimate the user's responsiveness based on changes in the user's state (level of concentration, level of excitement, level of relaxation, estimated emotion) when the sensory experience provided by the sensory presentation device group 41 is performed, and, if necessary, feed this back to the human generalization information on the server 101.

[0051] The device integration control unit 66 generates a set of device control data for integrally controlling the multiple sensory presentation devices of the sensory presentation device group 41 based on the raw video data and raw audio data supplied from the video content acquisition unit 61, the video analysis information and audio analysis information supplied from the video and audio feature analysis unit 63, the detected word information supplied from the scene discrimination unit 64, and the vehicle status detection information, user face and ear measurement information, and user state estimation information supplied from the sensor data analysis processing unit 65, and supplies this data to the sensory presentation device group 41. For example, the device control data set is data for operating the multiple sensory presentation devices of the sensory presentation device group 41 at appropriate timings and intensities to match the status of the vehicle 12, the user's state, and changes in the video content scene. The detailed configuration of the device integration control unit 66 will be described later with reference to Figures 4 and 5.

[0052] Figure 3 is a block diagram showing an example configuration of the sensor data analysis processing unit 65.

[0053] As shown in Figure 3, the sensor data analysis processing unit 65 is configured to include signal preprocessing units 71 to 77, a noise level calculation unit 78, a vehicle operation calculation unit 79, an in-vehicle temperature calculation unit 80, a user body temperature calculation unit 81, a facial feature point extraction unit 82, an emotion estimation unit 83, a heart rate and blood pressure calculation unit 84, a face and ear measurement unit 85, a face change extraction unit 86, an expression estimation unit 87, and a user state estimation processing unit 88.

[0054] The signal preprocessor 71 applies a bandpass filter to the raw microphone data output from the microphone 51 to obtain an audio signal of the required bandwidth from the raw microphone data and supplies it to the noise level calculation unit 78.

[0055] The signal preprocessing unit 72 performs signal processing such as filtering, offsetting, and drift correction on the raw inertial measurement data output from the inertial measurement sensor 52 to obtain the signal-processed 3-axis acceleration and 3-axis angular velocity, and supplies them to the vehicle motion calculation unit 79.

[0056] The signal preprocessing unit 73 acquires vehicle-to-vehicle communication information and traffic information from the raw vehicle data output from the vehicle sensor 53 and supplies it to the vehicle operation calculation unit 79.

[0057] The signal preprocessing unit 74 applies signal processing such as correction, moving average, and region filtering to the raw thermal image data output from the thermal image sensor 54 to obtain a signal-processed two-dimensional thermal data sequence, which it then supplies to the in-vehicle temperature calculation unit 80 and the user body temperature calculation unit 81.

[0058] The signal preprocessing unit 75 applies signal processing such as scaling and background subtraction to the camera RGB data output from the camera 55 to obtain signal-processed image data (for example, image data that has been reduced to a predetermined size and in which only moving objects have been extracted), and supplies it to the face feature point extraction unit 82.

[0059] The signal preprocessor 76 applies a bandwidth filter to the raw electroencephalogram data output from the electroencephalogram sensor 56 to obtain electroencephalogram signals of the required bandwidth from the raw electroencephalogram data and supplies them to the emotion estimation unit 83.

[0060] The signal preprocessing unit 77 performs signal processing, such as correction and heart rate waveform extraction, on the raw millimeter-wave data output from the millimeter-wave radar 57 to acquire heart rate waveform data that shows the waveform of the user's heart rate, and supplies it to the heart rate and blood pressure calculation unit 84.

[0061] The noise level calculation unit 78 calculates the noise level in the interior space of the vehicle 12 based on the audio signal supplied from the signal preprocessing unit 71 and supplies it to the device integrated control unit 66.

[0062] The vehicle motion calculation unit 79 calculates acceleration and angular velocity representing the operation of the vehicle 12 based on the acceleration and angular velocity of the three axes supplied from the signal preprocessing unit 72, and the inter-vehicle communication information and traffic information supplied from the signal preprocessing unit 73, and supplies these to the device integrated control unit 66.

[0063] The vehicle interior temperature calculation unit 80 calculates the temperature inside the vehicle 12 based on the overall temperature average of the two-dimensional thermal data sequence supplied from the signal preprocessing unit 74, and supplies it to the device integrated control unit 66 and the user state estimation processing unit 88.

[0064] Here, the noise level in the interior space of the vehicle 12 calculated by the noise level calculation unit 78, the acceleration and angular velocity representing the operation of the vehicle 12 calculated by the vehicle operation calculation unit 79, and the temperature in the interior space of the vehicle 12 calculated by the interior temperature calculation unit 80 are collectively referred to as vehicle condition detection information below.

[0065] The user body temperature calculation unit 81 calculates the user's body temperature based on the average temperature of the region corresponding to the user's face from the two-dimensional thermal data sequence supplied from the signal preprocessing unit 74, and supplies it to the user state estimation unit 88.

[0066] The facial feature point extraction unit 82 extracts the user's facial feature points based on the image data supplied from the signal preprocessing unit 75, and obtains a facial mesh in which these feature points are arranged three-dimensionally, as well as facial landmarks indicating the vertices of the facial mesh. The facial feature point extraction unit 82 then supplies the image data, facial mesh, and facial landmarks to the facial and ear measurement unit 85, the facial change extraction unit 86, and the facial expression estimation unit 87.

[0067] The emotion estimation unit 83 estimates the user's emotions based on the electroencephalogram signals supplied from the signal preprocessing unit 76 and supplies emotion information indicating the user's emotions to the user state estimation unit 88.

[0068] The heart rate and blood pressure calculation unit 84 calculates the user's heart rate (bpm) by applying FFT (Fast Fourier Transformation) to the heart rate waveform data supplied from the signal preprocessing unit 77. The heart rate and blood pressure calculation unit 84 also estimates the user's blood pressure change (mmHg) from the time change of heart rate shown in the heart rate waveform data supplied from the signal preprocessing unit 77. The heart rate and blood pressure calculation unit 84 then supplies the user's heart rate and blood pressure change to the user state estimation unit 88.

[0069] The face and ear measurement unit 85 measures the position and orientation of the user's face, the position and shape of the user's ears, etc., based on image data, a face mesh, and face landmarks supplied from the face feature point extraction unit 82. The face and ear measurement unit 85 then supplies user face and ear measurement information, which indicates the position and orientation of the user's face, the position and shape of the user's ears, etc., to the device integrated control unit 66.

[0070] The face change extraction unit 86 extracts changes in the user's face, such as the amount of change in the user's pupil size and the amount of change in skin tone, based on the image data, face mesh, and face landmarks supplied from the face feature point extraction unit 82, and supplies these to the user state estimation processing unit 88.

[0071] The facial expression estimation unit 87 estimates the user's facial expression, such as a smiling face, a crying face, an angry face, or a neutral expression, based on the image data, face mesh, and face landmarks supplied from the face feature point extraction unit 82, and supplies this information to the user state estimation processing unit 88.

[0072] The user state estimation processing unit 88 takes as input the temperature inside the vehicle 12 supplied by the vehicle temperature calculation unit 80, the user's body temperature supplied by the user body temperature calculation unit 81, emotion information supplied by the emotion estimation unit 83, changes in the user's heart rate and blood pressure supplied by the heart rate and blood pressure calculation unit 84, changes in the user's face supplied by the face change extraction unit 86, and the user's facial expression supplied by the expression estimation unit 87, and estimates the user's state (level of concentration, level of excitement, level of relaxation, estimated emotion) according to the user state estimation algorithm. The user state estimation processing unit 88 then supplies the user state estimation information indicating the user's state to the device integration control unit 66.

[0073] Figure 4 is a block diagram showing an example configuration of the device integrated control unit 66.

[0074] As shown in Figure 4, the device integrated control unit 66 is configured to include a device control pattern database 91, a time-series data generation unit 92, a scene feature extraction unit 93, a control pattern processing unit 94, a control correction coefficient database 95, a correction coefficient processing unit 96, and a device control waveform generation processing unit 97.

[0075] The device control pattern database 91 registers control patterns (hereinafter referred to as device control patterns) that serve as criteria for controlling the operation of each sensory presentation device in the sensory presentation device group 41, corresponding to changes in characteristics from one scene to the next. For example, a device control pattern is a waveform (data that changes over time) that includes information such as the intensity and frequency characteristics of each sensory presentation device for the period from one scene to the next, information on the addition of changes in phase and time intervals between each sensory presentation device, and information on the application ratio of video and audio. For example, in response to characteristic changes in a particular scene or story, device control patterns can be created in advance manually or by AI, referencing creators and past movie assets, to determine how to control the sensory presentation device group 41 in a way that is effective in providing a more immersive and realistic user experience, and then registered in the device control pattern database 91.

[0076] For example, in the immersive experience provision system 11, the following are used as device control patterns: scent control pattern, mist reference operation pattern, sound control reference operation pattern, brightness control reference operation pattern, vibration control reference operation pattern, RGB pattern, wind control reference operation pattern, and heat control reference operation pattern. The scent control pattern controls the presentation of scents (type of scent, scent on / off) by the scent presentation device 24. The mist reference operation pattern controls the amount of mist generated by the mist generating device 25. The sound control reference operation pattern controls the volume of sound output by the 3D sound headrest speaker 22. The brightness control reference operation pattern controls the brightness of the RGB lighting device 28 and the glasses-free stereoscopic display 21. The vibration control reference operation pattern controls the magnitude of vibrations generated by the vibration device 23. The RGB pattern controls the color of light emitted by the RGB lighting device 28. The wind control reference operation pattern controls the airflow rate of the wind generated by the wind generating device 26. The heat control reference operation pattern controls the airflow rate of the hot air generated by the hot air generating device 27.

[0077] The time-series data generation unit 92 generates time-series data that describes scenes in video content by chronologically sequencing the word groups indicated by the detected word information supplied from the scene discrimination unit 64, and supplies this data to the scene feature extraction unit 93.

[0078] The scene feature extraction unit 93 extracts features of scenes in the video content based on time-series data describing the scenes of the video content supplied from the time-series data generation unit 92, and supplies time-series data representing the features of the scenes in the video content (for example, items generalized from the horizontal axis of the control example) to the control pattern processing unit 94. For example, the scene feature extraction unit 93 can extract features of each scene by obtaining and abstracting words related to the sensory presentation device group 41 from the time-series data describing the scenes of the video content.

[0079] The control pattern processing unit 94 obtains a group of device control patterns from the device control pattern database 91 that correspond to changes in the scene features represented by time-series data representing the scene features of the video content supplied by the scene feature extraction unit 93. At this time, the control pattern processing unit 94 can refer to human generalization information obtained by communicating with the server 101 on the network and obtain a group of device control patterns according to the type of user sitting in the rear seat 13 of the vehicle 12. The control pattern processing unit 94 then supplies the group of device control patterns (odor control pattern, mist reference operation pattern, sound control reference operation pattern, brightness control reference operation pattern, vibration control reference operation pattern, RGB pattern, wind control reference operation pattern, and heat control reference operation pattern) corresponding to changes in the scene features of the video content to the device control waveform generation processing unit 97.

[0080] Although only one vibration control reference operation pattern is shown in Figure 4, the control pattern processing unit 94 supplies vibration control reference operation patterns corresponding to the number of devices in the vibration device 23 to the device control waveform generation processing unit 97. As shown in Figures 1 and 2 above, if the vibration device 23 includes a subwoofer 31, a seat belt actuator 32, and a seat vibration actuator 33, three vibration control reference operation patterns are generated in the control pattern processing unit 94.

[0081] For example, the control pattern processing unit 94 can generate three vibration control reference operation patterns by adjusting the delay and phase of the vibration control reference operation pattern obtained from the device control pattern database 91, according to the arrangement of the subwoofer 31, the seat belt actuator 32, and the seat vibration actuator 33, and the direction of vibration to be generated in accordance with the video content. As a result, the control pattern processing unit 94 generates a vibration control reference operation pattern for the subwoofer 31, a vibration control reference operation pattern for the seat belt actuator 32, and a vibration control reference operation pattern for the seat vibration actuator 33, and supplies them to the device control waveform generation processing unit 97.

[0082] The control correction coefficient database 95 contains correction coefficients that correct the device control pattern group according to the status of the vehicle 12, the position and orientation of the user's face, the position and shape of the user's ears, the user's state, the user's responsiveness, and so on.

[0083] The correction coefficient processing unit 96 obtains control correction coefficients corresponding to the vehicle status detection information, user face and ear measurement information, and user state estimation information supplied from the sensor data analysis processing unit 65 from the control correction coefficient database 95 and supplies them to the device control waveform generation processing unit 97. At that time, the correction coefficient processing unit 96 can refer to human generalization information obtained by communicating with the server 101 on the network and obtain control correction coefficients according to the type of user sitting in the rear seat 13 of the vehicle 12.

[0084] For example, if the vehicle 12 is in an environment with a lot of vibration, a correction factor is obtained that corrects the vibration intensity to 120%. If the user sitting in the rear seat 13 of the vehicle 12 is elderly, a correction factor is obtained that corrects the volume to 120%. If the user sitting in the rear seat 13 of the vehicle 12 is not relaxed enough, a correction factor is obtained that corrects the intensity of 1 / f fluctuation to 150%. If the vehicle 12 is driving at night, a correction factor is obtained that corrects the volume to 80% and the brightness to 80%.

[0085] The device control waveform generation processing unit 97 receives raw video data and raw audio data supplied from the video content acquisition unit 61, as well as a group of device control patterns supplied from the control pattern processing unit 94, as input. It then generates a group of device control data by performing corrections using video analysis information and audio analysis information supplied from the video and audio feature analysis unit 63, and control correction coefficients supplied from the correction coefficient processing unit 96, and supplies this data to the sensory presentation device group 41. The detailed configuration of the device control waveform generation processing unit 97 will be described later with reference to Figure 5.

[0086] Figure 5 is a block diagram showing an example configuration of the device control waveform generation processing unit 97.

[0087] As shown in Figure 5, the device control waveform generation processing unit 97 is configured to include multipliers 111 to 120, a delay control unit 121 for video synchronization, and a delay control unit 122 for audio synchronization.

[0088] The multiplier 111 performs a calculation that multiplies the thermal control reference operation pattern supplied from the control pattern processing unit 94 by the control correction coefficient supplied from the correction coefficient processing unit 96, as well as the video analysis information and audio analysis information supplied from the video and audio feature analysis unit 63. As a result, the multiplier 111 obtains a thermal fan PWM (Pulse Width Modulation) value that corrects the thermal control reference operation pattern to suit the status of the vehicle 12, the user's state and responsiveness, and the characteristics of the video content, and supplies it to the hot air generating device 27.

[0089] The multiplier 112 performs a calculation in which it multiplies the wind control reference operation pattern supplied from the control pattern processing unit 94 by the control correction coefficient supplied from the correction coefficient processing unit 96, as well as the video analysis information and audio analysis information supplied from the video and audio feature analysis unit 63. As a result, the multiplier 112 obtains a wind fan PWM value that corrects the wind control reference operation pattern to suit the status of the vehicle 12, the user's state and responsiveness, and the characteristics of the video content, and supplies it to the wind generating device 26.

[0090] The multiplier 113 performs a calculation that multiplies the RGB pattern supplied from the control pattern processing unit 94 and the brightness control reference operation pattern supplied from the control pattern processing unit 94 by the control correction coefficient supplied from the correction coefficient processing unit 96, as well as the video analysis information and audio analysis information supplied from the video and audio feature analysis unit 63. As a result, the multiplier 113 obtains LED RGB values ​​that have been corrected to suit the status of the vehicle 12, the user's state and responsiveness, and the characteristics of the video content, and supplies them to the RGB lighting device 28.

[0091] The multiplier 114 performs a calculation in which it multiplies the vibration control reference operation pattern for the subwoofer 31 supplied from the control pattern processing unit 94 by the control correction coefficient supplied from the correction coefficient processing unit 96, as well as the video analysis information and audio analysis information supplied from the video and audio feature analysis unit 63. As a result, the multiplier 114 acquires first vibration data in which the vibration control reference operation pattern for the subwoofer 31 has been corrected to suit the condition of the vehicle 12, the user's state and responsiveness, and the characteristics of the video content (in particular, the direction of the vibration to be generated), and supplies it to the subwoofer 31.

[0092] The multiplier 115 performs a calculation in which it multiplies the vibration control reference operation pattern for the seat belt actuator 32 supplied from the control pattern processing unit 94 by the control correction coefficient supplied from the correction coefficient processing unit 96, as well as the video analysis information and audio analysis information supplied from the video and audio feature analysis unit 63. As a result, the multiplier 115 acquires second vibration data in which the vibration control reference operation pattern for the seat belt actuator 32 has been corrected to suit the condition of the vehicle 12, the user's state and responsiveness, and the characteristics of the video content (in particular, the direction of the vibration to be generated), and supplies it to the seat belt actuator 32.

[0093] The multiplier 116 performs a calculation in which it multiplies the vibration control reference operation pattern for the seat vibration actuator 33 supplied from the control pattern processing unit 94 by the control correction coefficient supplied from the correction coefficient processing unit 96, as well as the video analysis information and audio analysis information supplied from the video and audio feature analysis unit 63. As a result, the multiplier 116 acquires third vibration data in which the vibration control reference operation pattern for the seat vibration actuator 33 has been corrected to suit the condition of the vehicle 12, the user's state and responsiveness, and the characteristics of the video content (in particular, the direction of the vibration to be generated), and supplies it to the seat vibration actuator 33.

[0094] The multiplier 117 performs a calculation in which it multiplies the raw video data supplied from the video content acquisition unit 61 by the brightness control reference operation pattern supplied from the control pattern processing unit 94 and the control correction coefficient supplied from the correction coefficient processing unit 96. As a result, the multiplier 117 corrects the brightness of the video in accordance with changes in the characteristics of the video content, and acquires video data in which the brightness of the video has been corrected to suit the status of the vehicle 12, the user's state and responsiveness, and the characteristics of the video content, and supplies it to the video synchronization delay control unit 121.

[0095] The multiplier 118 performs a calculation in which it multiplies the raw audio data supplied from the video content acquisition unit 61 by the sound control reference operation pattern supplied from the control pattern processing unit 94 and the control correction coefficient supplied from the correction coefficient processing unit 96. As a result, the multiplier 118 corrects the volume of the audio in response to changes in the audio characteristics of the video content, and acquires audio data in which the volume of the audio has been corrected according to the status of the vehicle 12, the user's state and responsiveness, and the characteristics of the video content, as well as the position and orientation of the user's face and the position and shape of the user's ears, and supplies it to the audio synchronization delay control unit 122.

[0096] The multiplier 119 performs a calculation in which it multiplies the mist reference operation pattern supplied from the control pattern processing unit 94 by the control correction coefficient supplied from the correction coefficient processing unit 96. As a result, the multiplier 119 acquires mist data in which the mist reference operation pattern has been corrected to suit the condition of the vehicle 12, as well as the user's condition and responsiveness, and supplies it to the mist generating device 25.

[0097] The multiplier 120 performs a calculation in which it multiplies the odor control pattern supplied from the control pattern processing unit 94 by the control correction coefficient supplied from the correction coefficient processing unit 96. As a result, the multiplier 120 acquires odor data in which the odor control pattern has been corrected to suit the status of the vehicle 12, as well as the user's state and responsiveness, and supplies it to the odor presentation device 24.

[0098] The video synchronization delay control unit 121 delays the video data supplied from the multiplier 117 by a delay time corresponding to the time required for the control processing unit 44 to perform control processing, adjusting the timing of the video display so that it is synchronized with the timing of the sensory presentation device group 41's tactile effects, and then supplies the adjusted video data to the naked-eye stereoscopic display 21.

[0099] The audio synchronization delay control unit 122 delays the audio data supplied from the multiplier 118 by a delay time corresponding to the time required for the control processing unit 44 to perform control processing, adjusting the timing of the audio output so that it is synchronized with the timing of the sensory presentation device group 41's tactile effects, and then supplies the adjusted audio data to the stereophonic headrest speaker 22.

[0100] As described with reference to Figures 4 and 5, the device integrated control unit 66 does not simply control the sensory presentation device group 41 according to relevant words estimated from the video content at any given time. Instead, it can control the sensory presentation device group 41 by interpolating scene changes from scene to scene, and what is likely to be happening in a given scene, in order to make the sensory presentation effect (for example, an effect that stimulates the user's sight, hearing, touch, and smell) more effective. Furthermore, the device integrated control unit 66 can control the sensory presentation device group 41 by correcting the device control pattern group in real time to match the conditions of the vehicle 12, such as noise level, acceleration and angular velocity, and temperature, the user's state, such as concentration level, excitement level, relaxation level, and estimated emotion, and the user's responsiveness based on generalized human information.

[0101] As a result, the device integration control unit 66 can sequentially and comprehensively control the activation timing and intensity of each sensory presentation device in the sensory presentation device group 41 in accordance with the scenes of the video content and changes in the scenes of the video content, thereby enhancing immersion and realism, and improving the sensory effects on the user from each sensory presentation device. Furthermore, the device integration control unit 66 can adjust the delay and phase of the vibration control reference operation pattern supplied to each of the subwoofer 31, seat belt actuator 32, and seat vibration actuator 33 that constitute the vibration device 23, according to their respective arrangements and the direction of vibration generated in accordance with the video content, and can provide a richer vibration experience and sense of realism by integrating and coordinating them with other sensory presentation devices.

[0102] <Examples of Use of the Immersive Experience Provisioning System> The first to third examples of how the immersive experience provisioning system 11 integrates and controls each of the sensory presentation devices of the sensory presentation device group 41, reflecting the continuity of scenes in the video content, will be explained.

[0103] As the first example of application, we will describe the integrated control of a group of sensory presentation devices 41 adapted to video content of an open-air bath.

[0104] For example, with conventional methods, if a fragmented scene of an open-air bath was detected in video content, the sensory presentation device would be controlled individually for each detected object, such as generating hot air based on the assumption that a bath should be warm.

[0105] Therefore, if nudity is shown in the scene leading up to entering the open-air bath, control is activated to generate wind, expressing the coldness of being naked. Next, if hot water or steam is shown in the scene where the character enters the open-air bath, control is activated to generate hot air and mist, expressing the warmth and humidity of the open-air bath. Finally, if a close-up of a person's face is shown in the scene where they are enjoying the scenery from the open-air bath (without hot water or steam being visible), control is activated to stop the hot air and mist.

[0106] In contrast, the immersive experience provision system 11 does not directly control the sensory presentation device group 41 with relevant words estimated from the input video at any given moment, but rather controls the sensory presentation device group 41 by interpolating between scenes that are effective for the user's senses, changes in the characteristics of the scene, and what is likely to be happening in that scene.

[0107] Therefore, in the scenes leading up to entering the open-air bath, control is implemented to generate both wind and vibrations, expressing the feeling of walking barefoot. Next, in the scene where the character enters the open-air bath, the volume of hot air and mist is gradually increased as the character approaches the bath, and the temperature of the hot air is maximized the moment the character enters the bath. Then, in the scene where the character enjoys the scenery in the open-air bath, if a close-up of a person's face is shown, the temperature of the hot air is gradually lowered to maintain a constant temperature, or a fluctuating wind is generated, as this is a continuation from the open-air bath scene.

[0108] Furthermore, the immersive experience provision system 11 does not provide an immersive experience to the user through unidirectional input from video content, but rather optimizes the system for the vehicle 12 and the user's condition by controlling (adjusting the intensity of) the sensory presentation device group 41 using the analysis results of sensor data obtained by sensing the vehicle 12's condition and the user's condition.

[0109] For example, the immersive experience system 11 can sense the user's state and, in a scene of an open-air bath, if the user is not yet relaxed in a calming scene such as "a close-up of a person enjoying the scenery in an open-air bath," it can control the sensory presentation device group 41 to increase the intensity of stimuli related to relaxation (smell, changes in lighting, wind, temperature fluctuations). The immersive experience system 11 can also sense the status of the vehicle 12 and control the sensory presentation device group 41 to increase the volume if there is a lot of noise, or to change the intensity of vibrations if acceleration occurs due to the vehicle 12 stopping, starting, turning left or right.

[0110] Furthermore, the immersive experience provision system 11 can control the sensory presentation device group 41 in real time, taking into account the condition of the vehicle 12 and the user's state, so that it is optimized in accordance with changes in the video content scenes. For example, it can have the effect of suppressing motion sickness in the user caused by a mismatch between the condition of the vehicle 12 and the user's state and the changes in the video content scenes.

[0111] Referring to Figure 6, a second example of application will be described, which is the integrated control of a group of sensory presentation devices 41 adapted to video content in which the protagonist is caught in a vortex of intense explosions on the battlefield.

[0112] For example, in a video content, the first scene might show a distant explosion beginning and the surrounding wind gradually intensifying; the second scene might show the powerful explosion with intense light and flames spreading around; and the third scene might show the aftermath of the explosion, along with smoke and subtle vibrations, highlighting the scattered debris and the terrain beneath the viewer's feet.

[0113] Reflecting the continuity of these scenes, the device integrated control unit 66 gradually increases the amount of vibration and the airflow of wind and hot air over a certain period from scene 1, and then controls the amount of vibration and airflow to be constant until scene 2, thereby allowing the user to experience the initial vibrations of a distant explosion. This initial vibration, combined with the intensity of the visuals, creates an effect that foreshadows a sense of urgency for the user through both visual and tactile senses. Furthermore, between scene 2 and scene 3, the device integrated control unit 66 controls the vibrations to pass sequentially from front to back, giving directionality to the vibrations the user experiences, and by controlling the airflow of wind and hot air to be maximized, the intensity of the emergency evasion can be further enhanced. Furthermore, after Scene 3, the device integrated control unit 66 controls the amount of vibration to gradually reduce it, so that a faint reverberating vibration is transmitted throughout the entire seat, recreating the feeling that the user is still feeling the aftershocks of the explosion. It also controls the amount of wind and hot air to gradually reduce it, providing the user with a stimulus that makes them feel as if they are actually there.

[0114] In this way, the immersive experience provision system 11 can provide a sense of direction and realism through vibrations, wind, and hot air that are synchronized with the sequence of scenes in the video content. In particular, the immersive experience provision system 11 can achieve a three-dimensional representation of the direction of vibrations generated in accordance with the video content, depending on the arrangement of the subwoofer 31, seat belt actuator 32, and seat vibration actuator 33.

[0115] In other words, the immersive experience provision system 11 can adjust the delay and phase of the vibration control reference operation pattern supplied to the subwoofer 31, seat belt actuator 32, and seat vibration actuator 33, respectively, for scenes with three-dimensional movement from front to back. This allows vibrations to be generated in the order of the subwoofer 31, which mainly generates vibrations from the front at a distance; the seat belt actuator 32, which mainly generates vibrations on the front side of the user; and the seat vibration actuator 33, which mainly generates vibrations on the back side of the user. This enables three-dimensional representation through vibration, further emphasizing the three-dimensional representation of video content.

[0116] Referring to Figure 7, a third application example will be described, which is the integrated control of a group of sensory presentation devices 41 adapted to video content depicting a volcanic eruption.

[0117] For example, in video content, Scene 1 shows a volcano that looks like it's about to erupt, Scene 2 shows the tremendous shockwave and the smoke rising violently above the volcano at the moment of the eruption, and Scene 3 shows the gradual subsiding of the shockwave and smoke after the eruption.

[0118] Reflecting the continuity of these scenes, the device integrated control unit 66 gradually increases the temperature and airflow of the hot air and the amount of mist before and after Scene 1, thereby expressing the atmosphere before the volcano erupts. Then, immediately after Scene 2, the device integrated control unit 66 keeps the temperature and airflow of the hot air constant, rapidly increases the amount of mist and light intensity to the maximum, then generates a scent, and subsequently generates vibrations at the maximum amount, thereby expressing the impact of the eruption. After that, the device integrated control unit 66 returns the light intensity to its original level and gradually decreases the amount of vibration and mist. Then, while continuing to decrease the amount of mist, at the timing of Scene 3, the device integrated control unit 66 keeps the gradually decreasing amount of vibration constant and generates wind, thereby allowing the user to experience the aftereffects of the eruption and creating an effect that emphasizes the scent of the eruption.

[0119] As illustrated in the first to third examples of applications described above, the immersive experience provision system 11 can provide a more immersive and realistic user experience by integrating and controlling each sensory presentation device of the sensory presentation device group 41 in accordance with the connections between scenes in the video content.

[0120] <Example of Immersion Experience Provision Processing> Referring to Figures 8 and 9, an example of immersion experience provision processing performed in the immersion experience provision system 11 will be described.

[0121] Figure 8 shows a flowchart illustrating the process of providing an immersive experience.

[0122] For example, when a user operates the control unit 42 to instruct the playback of video content, processing begins, and in step S11, the video content acquisition unit 61 acquires the video content that the user has instructed to play. The video content acquisition unit 61 then supplies the raw video data and raw audio data obtained by playing the video content to the video preprocessing unit 62 and the device integrated control unit 66.

[0123] In step S12, the video preprocessing unit 62 performs video preprocessing on the raw video data supplied from the video content acquisition unit 61 in step S11, and supplies the raw video data and raw audio data to the video and audio feature analysis unit 63 and the scene discrimination unit 64, respectively.

[0124] In step S13, the video and audio feature analysis unit 63 analyzes the video and audio features based on the raw video data and raw audio data supplied from the video preprocessing unit 62 in step S12. The video and audio feature analysis unit 63 then supplies the video analysis information obtained as a result of analyzing the video features and the audio analysis information obtained as a result of analyzing the audio features to the device integrated control unit 66.

[0125] In step S14, the scene determination unit 64 determines each scene in the order according to the time-series data of the video and audio, based on the raw video data and raw audio data supplied from the video preprocessing unit 62 in step S12. The scene determination unit 64 then acquires a group of words that match the content and meaning of each scene and supplies the detected word information representing that group of words to the device integration control unit 66.

[0126] In step S15, the sensor data analysis processing unit 65 performs data analysis on the sensor data group supplied from the sensor device group 43. The data analysis process will be described later with reference to the flowchart in Figure 9.

[0127] In step S16, the device integrated control unit 66 generates a group of device control data based on the raw video data and raw audio data supplied from the video content acquisition unit 61 in step S11, the video analysis information and audio analysis information supplied from the video and audio feature analysis unit 63 in step S13, the detected word information supplied from the scene discrimination unit 64 in step S14, and the vehicle status detection information, user face and ear measurement information, and user state estimation information supplied from the sensor data analysis processing unit 65 in the data analysis processing of step S15, and supplies it to the sensory presentation device group 41.

[0128] As a result, the device integrated control unit 66 can integrally control the multiple sensory presentation devices of the sensory presentation device group 41 so that they operate at appropriate timings and intensities to adapt to the conditions of the vehicle 12, the user's state, and changes in the video content scene. The process then returns to step S11, and the same process is repeated thereafter.

[0129] Figure 9 shows a flowchart illustrating the sensor data analysis process performed in step S15 of Figure 8.

[0130] In step S21, the signal preprocessing units 71 to 77 each perform signal preprocessing on the sensor data to be processed, and each supplies the processed sensor data to the subsequent block.

[0131] In step S22, the noise level calculation unit 78 calculates the noise level in the interior space of the vehicle 12 based on the audio signal supplied from the signal preprocessing unit 71. The vehicle operation calculation unit 79 calculates the acceleration and angular velocity representing the operation of the vehicle 12 based on the three-axis acceleration and three-axis angular velocity supplied from the signal preprocessing unit 72, and the vehicle-to-vehicle communication information and traffic information supplied from the signal preprocessing unit 73. The interior temperature calculation unit 80 calculates the temperature in the interior space of the vehicle 12 based on the overall temperature average of the two-dimensional thermal data sequence supplied from the signal preprocessing unit 74. As a result, vehicle status detection information is supplied to the device integrated control unit 66.

[0132] In step S23, the face and ear measurement unit 85 measures the position and orientation of the user's face, the position and shape of the user's ears, etc., based on the image data, face mesh, and face landmarks extracted by the face feature point extraction unit 82, acquires user face and ear measurement information, and supplies it to the device integrated control unit 66.

[0133] In step S24, the user state estimation processing unit 88 takes as input the temperature inside the vehicle 12 calculated by the vehicle interior temperature calculation unit 80, the user's body temperature calculated by the user body temperature calculation unit 81, the emotion information estimated by the emotion estimation unit 83, the user's heart rate and blood pressure changes calculated by the heart rate and blood pressure calculation unit 84, the changes in the user's face extracted by the face change extraction unit 86, and the user's facial expression estimated by the expression estimation unit 87, and estimates the user's state according to the user state estimation algorithm. The user state estimation processing unit 88 then supplies the user state estimation information indicating the user's state to the device integration control unit 66.

[0134] After that, the sensor data analysis process is completed, and the process proceeds to step S16 in Figure 8.

[0135] By having the immersive experience provision system 11 perform the immersive experience provision processing described above, the system can provide a more immersive and realistic user experience by coordinating multiple sensory presentation devices provided by the sensory presentation device group 41 within the interior space of the vehicle 12, and by using stereoscopic images, stereoscopic sound, mist, wind, sound, vibration, scent, and heat to stimulate the user's sight, hearing, touch, and smell.

[0136] Furthermore, this technology is not limited to application within the interior space of the vehicle 12, but can be applied, for example, to seating spaces in high-speed trains, private rooms in various commercial facilities, and various enclosed or mobile spaces.

[0137] <Description of a computer to which this technology is applied> The series of processes (information processing methods) described above can be executed by hardware or by software. When the series of processes are executed by software, the programs that make up the software are installed on the computer. Here, the term "computer" includes computers built into dedicated hardware, as well as general-purpose personal computers, for example, that can perform various functions by installing various programs.

[0138] Figure 10 is a block diagram showing an example of the hardware configuration of a computer that executes the series of processes described above using a program.

[0139] In a computer, the processing circuit 901, ROM (Read Only Memory) 902, and RAM (Random Access Memory) 903 are interconnected by a bus 904.

[0140] An input / output interface 905 is further connected to the bus 904. An input unit 906, an output unit 907, a storage unit 908, a communication unit 909, and a drive 910 are connected to the input / output interface 905.

[0141] The input unit 906 may include physical or virtual operating means that the user operates to input information, such as a keyboard, mouse, or touch panel, as well as means that the user inputs information through voice, eye gaze, etc. Furthermore, the input unit 906 may include sensors for inputting various physical quantities to the computer. For example, the input unit 906 may include sensors that acquire physical quantities such as light (including infrared light other than visible light) or sound, such as a camera or microphone. Also, for example, the input unit 906 may include sensors that acquire other physical quantities such as temperature, moisture content, acceleration, distance, etc. The output unit 907 may include means that present information to the user by stimulating the user's perception, such as a display, speaker, or haptic device. The storage unit 908 is composed of a hard disk, non-volatile or volatile memory, etc., and stores various types of information (including programs). The communication unit 909 is a network interface, etc., and performs wired or wireless communication with the outside. The drive 910 drives removable media 911 such as a magnetic disk, optical disk, magneto-optical disk, or semiconductor memory.

[0142] The processing circuit 901 includes a processor that executes programs such as a CPU (Central Processing Unit) and a DSP (Digital Signal Processor). The processing circuit 901 (its processor) performs the above-described series of processes by loading the program stored in the storage unit 908 into the RAM 903 via the input / output interface 905 and the bus 904 and executing it. The processing circuit 901 can output the processing results of the series of processes from the output unit 907 via the bus 904 and the input / output interface 905 as needed. The processing circuit 901 can also store the processing results in the storage unit 908 or transmit them from the communication unit 909.

[0143] The program executed by the computer (processing circuit 901) can be provided by recording it on a removable medium 911, such as a package medium. The program can also be provided via wired or wireless transmission media, such as a local area network, the internet, or digital satellite broadcasting.

[0144] In a computer, a program can be installed in the storage unit 908 via the input / output interface 905 by inserting a removable media 911 into the drive 910. Alternatively, a program can be received by the communication unit 909 from another device, such as a server, via a wired or wireless transmission medium, and installed in the storage unit 908. Furthermore, programs can be pre-installed in the ROM 902 or the storage unit 908.

[0145] The programs executed by the computer may be programs that are processed chronologically in the order described herein, or they may be programs that are processed in parallel or at necessary times, such as when a call is made.

[0146] The processes that a computer performs according to a program do not necessarily have to follow the order described in the flowchart. In other words, the processes that a computer performs according to a program include processes that are executed in parallel or individually (e.g., parallel processing and object-based processing).

[0147] The program may be processed by a single computer (processor), or it may be processed in a distributed manner by multiple computers. Furthermore, the program may be transferred to a remote computer and executed there.

[0148] Furthermore, in this specification, a system means a collection of multiple components (devices, modules (parts), etc.), regardless of whether all components are located in the same enclosure or not. Therefore, multiple devices housed in separate enclosures and connected via a network, and a single device in which multiple modules are housed in one enclosure, are both considered systems.

[0149] Furthermore, for example, the configuration described as a single device (or processing unit) may be divided and configured as multiple devices (or processing units). Conversely, the configurations described above as multiple devices (or processing units) may be combined and configured as a single device (or processing unit). It is also possible to add configurations other than those described above to the configuration of each device (or each processing unit). Moreover, if the overall system configuration and operation are substantially the same, a part of the configuration of one device (or processing unit) may be included in the configuration of another device (or other processing unit).

[0150] Furthermore, for example, this technology can be configured as cloud computing, where a single function is shared and processed collaboratively by multiple devices via a network.

[0151] Furthermore, for example, the program described above can be executed on any device. In that case, the device should have the necessary functions (such as functional blocks) and be able to obtain the necessary information.

[0152] Furthermore, each step described in the flowchart above can be executed by a single device or shared among multiple devices. Additionally, if a single step includes multiple processes, these processes can be executed by a single device or shared among multiple devices. In other words, multiple processes within a single step can be executed as multiple steps. Conversely, processes described as multiple steps can be combined and executed as a single step.

[0153] Furthermore, the program executed by the computer may be executed in a chronological order according to the sequence of steps described herein, or it may be executed in parallel or individually at necessary times, such as when a call is made. In other words, as long as no inconsistencies arise, the processing of each step may be executed in an order different from the sequence described above. Moreover, the processing of the steps of this program may be executed in parallel with the processing of other programs, or it may be executed in combination with the processing of other programs.

[0154] Furthermore, the technologies described in this specification can be implemented independently, as long as they do not create a contradiction. Of course, any multiple technologies can also be implemented in combination. For example, some or all of the technologies described in one embodiment can be combined with some or all of the technologies described in another embodiment. In addition, some or all of the above-mentioned technologies can be implemented in combination with other technologies not mentioned above.

[0155] <Examples of Configuration Combinations> The technology can also take the following configurations: (1) An information processing device comprising: a sensor data analysis processing unit that performs data analysis processing on a group of sensor data output from a group of sensors arranged in a predetermined space for playing video content, and acquires sensor data analysis information including information detecting the state of the space, information detecting the state of the user viewing the video content, and information measuring the user's face and ears; and a device integration control unit that generates a group of device control data for integrally controlling a group of sensory presentation devices arranged in the space based on the sensor data analysis information. (2) The information processing device according to (1) above, further comprising a video and audio feature analysis unit that analyzes the video and audio features of the video content and acquires video analysis information and audio analysis information as a result of analyzing the video and audio, wherein the device integration control unit generates the group of device control data based on the video analysis information and audio analysis information as well. (3) The information processing device according to (1) or (2) above, further comprising a scene determination unit that determines each scene of the video content and acquires detection word information indicating a group of words detected for each scene, wherein the device integrated control unit generates the device control data group based on the detection word information as well. (4) The information processing device according to (3) above, further comprising a time-series data generation unit that generates time-series data describing the scenes of the video content by chronologically seriesizing the group of words indicated by the detection word information, and a scene feature extraction unit that extracts features of the scenes of the video content based on the time-series data describing the scenes of the video content and generates time-series data representing the features of the scenes of the video content. (5) The information processing device according to (4) above, further comprising a control pattern processing unit that acquires the control pattern from a control pattern database in which control patterns that serve as a standard for controlling the operation of a plurality of sensory presentation devices in correspondence with the changes in features from one scene to the next, according to time-series data representing the features of the scenes of the video content.(6) The information processing apparatus according to (5) above, wherein the control pattern processing unit adjusts the delay and phase of the control pattern for the plurality of vibration devices according to the arrangement of the plurality of vibration devices and the direction of vibration generated in accordance with the video content when a plurality of vibration devices are arranged in the space as the sensory presentation devices. (7) The information processing apparatus according to (5) or (6) above, wherein the control pattern processing unit refers to information indicating responsiveness representing how easily each type of person reacts when they receive sensory stimuli for each general type of person, and acquires the control pattern according to the type of user. (8) The information processing apparatus according to any one of (5) to (7) above, wherein the device integrated control unit further has a correction coefficient processing unit that acquires the correction coefficient from a control correction coefficient database in which correction coefficients for correcting the control pattern according to the sensor data analysis information are registered, according to the sensor data analysis information. (9) The information processing apparatus according to (8) above, wherein the correction coefficient processing unit refers to information indicating responsiveness representing how easily each type of person reacts when they receive sensory stimuli for each general type of person, and acquires the correction coefficient according to the type of user. (10) The information processing apparatus according to (8) or (9) above, wherein the device integrated control unit further comprises a device control data generation unit that takes the video and audio of the video content and the control pattern as input and performs correction using the video analysis information, the audio analysis information and the correction coefficient to generate the device control data group. (11) An information processing method comprising: performing data analysis processing on a group of sensor data output from a plurality of sensors arranged in a predetermined space for playing video content, and obtaining sensor data analysis information including information detecting the state of the space, information detecting the state of a user viewing the video content, and information measuring the user's face and ears; and generating a group of device control data for integrally controlling a plurality of sensory presentation devices arranged in the space based on the sensor data analysis information.(12) A program for causing the computer of an information processing device to perform data analysis on a group of sensor data output from a plurality of sensors arranged in a predetermined space for playing video content, and to obtain sensor data analysis information including information detecting the state of the space, information detecting the state of the user viewing the video content, and information measuring the user's face and ears; and to generate a group of device control data for integrally controlling a plurality of sensory presentation devices arranged in the space based on the sensor data analysis information.

[0156] It should be noted that this embodiment is not limited to the embodiment described above, and various modifications are possible without departing from the spirit of this disclosure. Furthermore, the effects described herein are merely illustrative and not limiting, and other effects may also exist.

[0157] 11 Immersive experience provision system, 12 Vehicle, 13 Rear seat, 21 Autopsis stereoscopic display, 22 Stereoscopic sound headrest speaker, 23 Vibration device, 24 Odor presentation device, 25 Mist generation device, 26 Wind generation device, 27 Hot air generation device, 28 RGB lighting device, 31 Subwoofer, 32 Seat belt actuator, 33 Seat vibration actuator, 41 Sensory presentation device group, 42 Operation unit, 43 Sensor device group, 44 Control processing unit, 51 Microphone, 52 Inertial measurement sensor, 53 Vehicle sensor, 54 Thermal image sensor, 55 Camera, 56 Electroencephalogram sensor, 57 Millimeter wave radar, 61 Video content acquisition unit, 62 Video preprocessing unit, 63 Video and audio feature analysis unit, 64 Scene discrimination unit, 65 Sensor data analysis processing unit, 66 Device integrated control unit

Claims

1. An information processing device comprising: a sensor data analysis processing unit that performs data analysis processing on a group of sensor data output from a plurality of sensors arranged in a predetermined space for playing video content, and acquires sensor data analysis information including information detecting the conditions of the space, information detecting the state of the user viewing the video content, and information measuring the user's face and ears; and a device integration control unit that generates a group of device control data for integrally controlling a plurality of sensory presentation devices arranged in the space based on the sensor data analysis information.

2. The information processing apparatus according to claim 1, further comprising a video and audio feature analysis unit that analyzes the video and audio features of the video content and acquires video analysis information and audio analysis information as a result of the analysis of the video and audio, wherein the device integrated control unit generates the device control data group based on the video analysis information and the audio analysis information.

3. The information processing apparatus according to claim 2, further comprising a scene discrimination unit that discriminates each scene of the video content and acquires detected word information indicating a group of words detected for each scene, wherein the device integrated control unit generates the device control data group based on the detected word information.

4. The information processing apparatus according to claim 3, wherein the device integrated control unit comprises: a time-series data generation unit that generates time-series data describing scenes of the video content by chronologically sequencing the group of words indicated by the detected word information; and a scene feature extraction unit that extracts features of scenes of the video content based on the time-series data describing scenes of the video content and generates time-series data representing the features of scenes of the video content.

5. The information processing apparatus according to claim 4, wherein the device integration control unit further comprises a control pattern processing unit that acquires a control pattern from a control pattern database, which has registered control patterns that serve as a reference for controlling the operation of a plurality of sensory presentation devices in correspondence with changes in features from one scene to the next, according to time-series data representing the features of the video content scene.

6. The information processing apparatus according to claim 5, wherein the control pattern processing unit adjusts the delay and phase of the control pattern for the plurality of vibration devices according to the arrangement of the plurality of vibration devices and the direction of vibration generated in accordance with the video content, when a plurality of vibration devices are arranged in the space as the sensory presentation device.

7. The information processing apparatus according to claim 5, wherein the control pattern processing unit refers to information indicating responsiveness, which represents how easily a person of each general human type responds when they receive a sensory stimulus, and acquires the control pattern according to the user type.

8. The information processing apparatus according to claim 5, wherein the device integrated control unit further comprises a correction coefficient processing unit that acquires the correction coefficient from a control correction coefficient database, in which correction coefficients for correcting the control pattern according to the sensor data analysis information are registered, in accordance with the sensor data analysis information.

9. The information processing device according to claim 8, wherein the correction coefficient processing unit refers to information indicating responsiveness, which represents how easily a person of each type responds when they receive a sensory stimulus, for each general type of person, and obtains the correction coefficient according to the type of user.

10. The information processing apparatus according to claim 8, wherein the device integrated control unit further comprises a device control data generation unit that receives the video and audio of the video content and the control pattern as inputs and generates the device control data group by performing correction using the video analysis information, the audio analysis information and the correction coefficient.

11. An information processing method comprising: an information processing device performing data analysis on a group of sensor data output from a plurality of sensors arranged in a predetermined space for playing video content, thereby obtaining sensor data analysis information including information detecting the conditions of the space, information detecting the state of a user viewing the video content, and information measuring the user's face and ears; and generating a group of device control data for integrally controlling a plurality of sensory presentation devices arranged in the space based on the sensor data analysis information.

12. A program for causing the computer of an information processing device to perform data analysis on a group of sensor data output from a plurality of sensors arranged in a predetermined space for playing video content, and to obtain sensor data analysis information including information detecting the conditions of the space, information detecting the state of the user viewing the video content, and information measuring the user's face and ears; and to generate a group of device control data for integrally controlling a plurality of sensory presentation devices arranged in the space based on the sensor data analysis information.

Images