Information processing apparatus, information processing method, and program

Abstract

An information processing system including one or more processing circuitry configured to predict a change in a situation around a mobile body based on information indicating a status of the mobile body; obtain real-time information measured while the mobile body is moving; perform processing on audio data of a music piece to be output at the mobile body according to the predicted change in the situation around the mobile body and the real-time information; and cause a transducer at the mobile body to output playback audio generated by performing the processing on the audio data of the music piece according to the predicted change in the situation around the mobile body.

Description

INFORMATION PROCESSING APPARATUS, INFORMATION PROCESSING METHOD, AND PROGRAM

[0001] The present technology particularly relates to an information processing apparatus, an information processing method, and a program that are capable of providing an appropriate music listening experience in response to changes in surrounding situations.

[0002] There is a demand for technology that can improve the user experience while riding. PTL 1 describes a technology for changing the music played in a vehicle depending on the vehicle’s state, which can be acquired by a user terminal such as a smartphone. For example, music playback is controlled according to the area in which the vehicle is traveling, and the driving states such as starting, stopping, turning right, turning left, and the like.

[0003] [PTL 1] WO 2023 / 127422

[0004] Some users perceive the changes in music according to the driving state of the vehicle as monotonous.

[0005] The present technology has been made in view of such circumstances, and makes it possible to provide an appropriate music listening experience in response to changes in surrounding situations.

[0006] An information processing system including one or more processing circuitry configured to predict a change in a situation around a mobile body based on information indicating a status of the mobile body; obtain real-time information measured while the mobile body is moving; perform processing on audio data of a music piece to be output at the mobile body according to the predicted change in the situation around the mobile body and the real-time information; and cause a transducer at the mobile body to output playback audio generated by performing the processing on the audio data of the music piece according to the predicted change in the situation around the mobile body.

[0007] According to the aspect of the present technology, a change in a situation around a mobile body after a certain time is predicted based on position information and information indicating a movement status of the mobile body, and processing according to a prediction result of the change in the situation around the mobile body is performed on audio data of a music piece to be output in the mobile body.

[0008] Fig. 1 is a diagram illustrating an example of playback control of a music piece by an information processing system.Fig. 2 is a diagram illustrating an example of playback control of a music piece.Fig. 3 is a diagram illustrating details of processing.Fig. 4 is a diagram illustrating another example of playback control of the music piece.Fig. 5 is a block diagram illustrating an example of the configuration of the information processing system.Fig. 6 is a flowchart illustrating processing by an information processing unit.Fig. 7 is a diagram illustrating an example of generating a playlist.Fig. 8 is a diagram illustrating another example of generating a playlist.Fig. 9 is a block diagram illustrating another example of the configuration of the information processing system.Fig. 10 is a diagram illustrating a flow of overall processing in the information processing system.Fig. 11 is a block diagram illustrating an example of a detailed configuration of the information processing system.Fig. 12 is a block diagram illustrating another example of the configuration of the information processing system.Fig. 13 is a block diagram illustrating an example of the configuration of a computer.

[0009] Hereinafter, embodiments for implementing the present technology will be described. Description will be made in the following order. 1. Overview of Present Technology 2. Configuration and Operation of Information Processing System 3. Example of Generating Playlist 4. Modification Examples

[0010] Overview of Present Technology Fig. 1 is a diagram illustrating an example of playback control of a music piece by an information processing system according to an embodiment of the present technology.

[0011] An information processing system according to an embodiment of the present technology is provided as an in-vehicle system of a mobile body such as an automobile. The information processing system has a function of generating and playing music to be output to users including a driver and a passenger. Fig. 1 illustrates the state in which a music piece A is being played back. The musical notes illustrated in Fig. 1 indicate that a playback sound of the music piece A is being output inside the vehicle.

[0012] Music generated in an information processing system is composed of a time series of one or more parts characterized by sound elements such as melody, rhythm, harmony, and key. Not only audio data of music pieces created by artists or the like but also audio data of sounds including at least one of various sounds such as human voices, environmental sounds, sound effects, and sounds generated using desk top music (DTM) software is used as the processing target in the information processing system. Information such as visually recognizable images may be output along with the music.

[0013] The following description will mainly focus on the process of playing back audio data of a music piece containing data on a plurality of tracks such as vocals, guitar, bass, and drums.

[0014] In the information processing system, based on the current position and vehicle body information such as the vehicle speed, changes in the scenery around the vehicle after a certain time and a clock time at which the change in scenery will occur are predicted. For example, the information processing system is provided with a navigation function having map information associated with information about the scenery of each position.

[0015] In the example of Fig. 1, the surrounding scenery at clock times after t1 seconds, after t2 seconds, and after t3 seconds are predicted as scenes #1, #2, and #3, respectively. The timing after t1 seconds is the timing when the scenery changes from the current scenery to scene #1, and the timing after t2 seconds is the timing when the scenery changes from scene #1 to scene #2. The timing after t3 seconds is the timing when the scenery changes from scene #2 to scene #3.

[0016] In the information processing system, the playback of the music piece A is controlled such that the playback sound changes in accordance with the timing of each change in the scenery. For example, the playback of the music piece A is controlled by performing processing according to the change in scenery on the audio data of the music piece A, and playing back the audio data generated by the processing.

[0017] Fig. 2 is a diagram illustrating an example of playback control of a music piece.

[0018] In the example of Fig. 2, control is performed such that the music piece composition changes from Verse (A melody) to Bridge (B melody) at the same timing as the change from scene #1 to scene #2. The timing of changes in the music piece composition is adjusted by adding or deleting tracks, changing the rhythm, tempo, and beat, etc.

[0019] In the example of Fig. 2, control is performed such that sound effects are added to the Verse section to add excitement in accordance with the timing of the scenery change. For example, a drum or cymbal track is added to the mix, and a sound effect in which the drum or cymbal sounds gradually increase is added to a partial section of the Verse before it changes to Bridge.

[0020] In this way, in the information processing system, the playback of a music piece is controlled such that the music listened to by the user changes in accordance with the timing of the change in scenery.

[0021] Fig. 3 is a diagram illustrating the details of processing of changing the music piece composition in accordance with changes in the scenery.

[0022] Music has time divisions called bars, and it is natural and pleasant to see developments such as changes in sound occurring at the timing of bar divisions. For example, a major development occurring at the third beat of a certain bar gives an abrupt and unnatural impression.

[0023] For example, when a change in scenery occurs on the third beat of a certain bar, processing is performed to adjust the change in music piece composition such that it occurs at the beginning of the next bar. For example, a drum track is used as a fill-in to create excitement and processing is performed to make it easier to transition to the next development.

[0024] In the example of Fig. 3, the change in scenery occurs at a timing midway through the x+2-th bar, based on the x-th bar constituting the Verse section. In this case, by adding a track such as a drum track, adjustments are made such that the timing of the change in the music piece composition from Verse to Bridge coincides with the beginning of the x+3-th bar. The hatched section in Fig. 3 is a section where a drum track has been added. The drum track is added to the section from the timing when the change in scenery occurs to the beginning of the x+3-th bar.

[0025] Changes in scenery may not always occur at the predicted timing due to an increase in the number of times of stepping on the brake or a decrease in the speed of the vehicle traveling ahead. When the timing at which the change in scenery actually occurs is earlier than the predicted timing of the change in scenery, adjustment is performed such that the music piece composition is changed at the beginning of the next bar by adding a track or the like.

[0026] This process is also performed at another timing when the scenery changes. For example, at the timing when the scenery changes to scene #3, which is a seascape, the playback of the music piece A is controlled such that the tone changes to a tone of an acoustic guitar, the beat changes to a swinging and upbeat beat, and the chord progression changes to a bright chord progression.

[0027] Although not illustrated in Fig. 1, at the timing when the scenery changes to an urban area surrounded by skyscrapers, the playback of the music piece A is controlled such that the tone changes to a tone of a synthesizer, the beat changes to a deep bass beat, and the chord progression changes to a chord progression with less variation.

[0028] Fig. 4 is a diagram illustrating another example of playback control of the music piece.

[0029] In the example of Fig. 4, it is predicted that the vehicle will enter the tunnel at the beginning of the x+3-th bar, based on the x-th bar of the music piece A. At the beginning of the x+3-th bar, the scenery around the vehicle changes from scene #11 before entering the tunnel to scene #12 immediately after entering the tunnel.

[0030] When such a change in scenery is predicted, a cymbal track is added from the x+2-th bar, which is one bar before the change in scenery, and an effect using the sound of the cymbal is given.

[0031] Furthermore, as the music piece A develops in accordance with the change in scenery, for example, processing is performed in which the musical instrument configuration changes from an acoustic piano and acoustic drums to an electronic piano and electronic drums. After the change in scenery occurs, from the x+3-th bar onwards, processing is performed to change the reverberation.

[0032] Table information that associates various types of scenery with processing content, for example, is prepared in the information processing system. The processing content according to the type of predicted scenery is selected, and the selected processing content is applied to the audio data of the music piece A, thereby realizing a change in music according to a change in scenery. The configuration of the information used to select the processing content can be changed in any manner, such as by preparing table information that associates the type of change in scenery with the processing content.

[0033] The processing performed on the audio data includes various processes such as changing the chord progression, rhythm, tempo, and beat, adjusting the volume, and adding tracks. Tracks include not only audio tracks, which are tracks for voices and musical instruments, but also effect tracks such as reverb, delay, and equalizer.

[0034] In this way, in the information processing system, processing is performed on existing music pieces. A music piece can be recomposed through various processes while maintaining the artist’s worldview expressed in the music piece, providing users with an appropriate music listening experience.

[0035] Since the music changes at an appropriate frequency, such as in accordance with the timing of changes in scenery, it is possible to provide the user with a music listening experience that changes appropriately such that the user does not get bored. If the music were to change in response to changes in the vehicle’s speed or steering operation, the music would change more frequently, which would ruin the worldview of the music piece and reduce its quality. On the other hand, if the music changes infrequently, the user will get bored. By changing the music in accordance with the timing of the change in scenery, it is possible to prevent such issues.

[0036] By changing the music in accordance with the changes in the scenery, the user’s movement experience can be matched with the artist’s worldview for the music, preventing the user from feeling uncomfortable. When the timing of the changes in scenery that form the user’s movement experience does not match the timing of the changes in the development of the music that expresses the artist’s worldview, the user may feel uncomfortable or may become bored with the movement due to the continuation of such uncomfortable changes, but this can be prevented.

[0037] Also, changes in scenery do not always occur at the predicted timing due to an increase in the number of times of stepping on the brake or a decrease in the speed of the vehicle traveling ahead. As described with reference to Fig. 3, control is performed to change the music piece composition at the timing of bar divisions, allowing the user to enjoy a natural and pleasant development of the music.

[0038] Music has a mental effect of predicting the development of music while listening to the music. When a change in scenery occurs, the user will be able to predict the development of the music, allowing the user to enjoy the change in scenery and improving the driving experience.

[0039] In this way, the present technology makes it possible to provide an appropriate music listening experience in response to changes in surrounding situations.

[0040] Configuration and Operation of Information Processing System Configuration of Information Processing System Fig. 5 is a block diagram illustrating an example of the configuration of the information processing system.

[0041] As illustrated in Fig. 5, the information processing system 1 includes an information processing unit 11 and an output unit 12. The information processing unit 11 is implemented by executing a predetermined program in an ECU or the like that controls the entire vehicle. An ECU having the information processing unit 11 and a vehicle equipped with the ECU are information processing apparatuses.

[0042] The information processing unit 11 includes a prediction unit 21 and a music generation unit 22. The position information and vehicle body information acquired by other processing units provided in the information processing system 1 are input to the prediction unit 21 as prediction information. The position information indicates the current position of the vehicle in which the information processing system is mounted. The vehicle body information indicates the current speed of the vehicle, etc. The vehicle body information is also input to the music generation unit 22.

[0043] Similarly, in the information processing system 1, each of the sensor data and driving information is acquired. Images captured by the camera, sensor data including data on measurement results from various sensors, and driving information indicating the state of the user’s driving operations such as steering and braking are input to the music generation unit 22 as real-time information.

[0044] The prediction unit 21 predicts a change in the scenery around the vehicle and the clock time when the change in scenery will occur, based on the position information and the vehicle body information. In addition to the current speed of the vehicle, other information about the vehicle body, such as acceleration and average speed over a predetermined time in the past, may be included in the vehicle body information and used to predict changes in scenery, etc. Driving information may be input to the prediction unit 21 and used to predict scenery such as changes in scenery. The prediction by the prediction unit 21 can be made based on various types of information indicating the vehicle’s movement status, including vehicle body information and driving information.

[0045] A plurality of types of scenery are set, such as scenery in which the sea is visible, scenery in which mountains are visible, highway scenery, tunnel scenery, urban area scenery, and rural area scenery. The prediction unit 21 predicts such a change in the type of scenery and the clock time at which the change will occur. For example, the type of scenery at each clock time after a certain time has elapsed is predicted, and the timing at which the type of scenery changes is detected together with the clock time. Changes in the type of scenery, including the time period, such as a sunset or a nighttime view, may be predicted.

[0046] As illustrated in the block of the prediction unit 21, in the example of Fig. 5, it is predicted that the scenery will change at each of the clock times “9:00”, “9:05”, and “9:10”. At the timing of “9:00”, the previous scenery changes to scene #1, and at the timing of “9:05”, scene #1 changes to scene #2. At the timing of “9:10”, scene #2 changes to scene #3. The current clock time is a clock time before “9:00”. Information on the prediction result by the prediction unit 21 is supplied to the music generation unit 22.

[0047] The prediction by the prediction unit 21 may be performed by a predetermined calculation with reference to the map information as described above, or may be performed using a prediction model generated by machine learning. When a prediction model is used, a prediction model that receives the current clock time, position information, and vehicle body information as inputs and outputs information on changes in scenery and information on the clock time at which the change in scenery occurs is prepared in the prediction unit 21.

[0048] The music generation unit 22 sets processing parameters based on the prediction result by the prediction unit 21 and the vehicle body information, sensor data, and driving information acquired by other processing units. The processing parameters are information that defines the content of various processes to be performed on the audio data, such as changing the melody, changing the chord progression, changing the rhythm, tempo, and beat, adjusting the volume, and adding a track. The processing parameters may include parameters that define playback algorithms, such as loop playback or random playback of tracks. Various processing parameters are set such that the music changes in accordance with the changes in the scenery.

[0049] The music generation unit 22 analyzes the images contained in the sensor data and detects (recognizes) the actual scenery that can be seen from the vehicle during traveling. The music generation unit 22 detects a time difference between a predicted timing of a change in scenery and an actual timing of a change in scenery based on the scenery detection result and vehicle body information, driving information, etc., acquired by other processing units. The music generation unit 22 sets processing parameters in accordance with the detected time difference. As described above, when a change in scenery occurs in the middle of a bar, processing parameters are set such that the change in music occurs at the beginning of the bar.

[0050] In the example of Fig. 5, the processing parameters are set such that the music piece composition changes from the previous composition to Verse at the timing of “9:00”. Furthermore, at the timing of “9:02”, the Verse changes to Bridge, and processing parameters are set to add a predetermined sound effect to a partial section of the Verse. At the timing of “9:04”, the Bridge changes to Chorus, and processing parameters are set to add a predetermined sound effect to a partial section of the Bridge.

[0051] For example, the detection of the time difference between the predicted timing of a change in scenery and the actual timing of a change in scenery is repeatedly performed every time real-time information is acquired. The processing parameters set once are repeatedly updated according to the time difference detected as real-time information is acquired.

[0052] The music generation unit 22 reads and acquires audio data to be played back from a DB (not illustrated). The audio data to be played back may be acquired from a server connected via a network such as the Internet. The music generation unit 22 performs processing defined by the processing parameters on the audio data of the music piece to generate audio data to be played back. The music generation unit 22 plays back the generated audio data and outputs the playback sound from the output unit 12, which is a speaker provided at each position inside the vehicle. The playback sound of the audio data may be output from headphones or earphones worn by the user.

[0053] In this way, the music generation unit 22 has at least the function of setting processing parameters, the function of performing processing specified by the processing parameters on the audio data of the music piece to generate audio data to be played back, and the function of playing back the generated audio data.

[0054] Operation of Information Processing Unit The processing of the information processing unit 11 having the above configuration will be described with reference to the flowchart of Fig. 6.

[0055] In step S1, the prediction unit 21 predicts a change in the scenery around the vehicle and the clock time when the change in scenery will occur, based on the position information and the vehicle body information.

[0056] In step S2, the music generation unit 22 sets processing parameters based on the prediction results of the change in scenery. The processing parameters are set based on the actual change situations in the scenery.

[0057] In step S3, the music generation unit 22 performs processing defined by the processing parameters on the audio data of the music piece to generate audio data to be played back.

[0058] In step S4, the music generation unit 22 plays the audio data to be played back, and causes the output unit 12 to output the playback sound.

[0059] The above processing is repeatedly performed, for example, while the vehicle equipped with the information processing system 1 is traveling. This makes it possible to continue providing users with an appropriate music listening experience in response to changes in surrounding situations.

[0060] Example of Generating Playlist Playlist Generation Example A playlist may be generated based on the route to the destination, and the above-described processing may be performed using the music pieces constituting the playlist.

[0061] Fig. 7 is a diagram illustrating an example of generating a playlist.

[0062] In the example of Fig. 7, a theme park is set as the destination. For example, the destination is set in response to a user’s operation using a navigation screen displayed on a display provided in the vehicle.

[0063] A movement route (route) from the current position to the destination is searched for, and changes in the scenery visible from the movement route are predicted. For example, it is predicted that the vehicle will travel through a section with a seascape during the time period “12:00 to 2:15” and will travel through a highway section during the time period “12:15 to 2:30”. It is predicted that the vehicle will arrive near the theme park, which is the destination, at 12:30 and will travel along the surrounding section.

[0064] A music piece A, which is a cheerful and light-hearted music piece, is selected as a music piece that corresponds to the seascape in the time period “12:00 to 2:15,” and a music piece B, which is a music piece with a sense of speed, is selected as a music piece that corresponds to the highway scenery in the time period “12:15 to 2:30”. A music piece C, which is a music piece related to the theme park, is selected as a music piece that corresponds to the scenery around the theme park in the time period after “12:30”. The music piece related to the theme park includes a music piece used within the theme park and a music piece for characters that appear in the theme park.

[0065] In this example, a playlist including music pieces A to C is generated as a playlist according to the movement route from the current position to the destination. While traveling through each section, the above-described processing is performed to change the music in accordance with changes in the scenery using the music pieces A to C.

[0066] Fig. 8 is a diagram illustrating another example of generating a playlist.

[0067] In the example of Fig. 8, a ski resort is set as the destination. Also in this case, a movement route from the current position to the destination is searched for, and changes in the scenery visible from the movement route are predicted. For example, it is predicted that the vehicle will travel through a section with mountain scenery during the time period “8:00 to 8:15” and will travel through a section within a tunnel during the time period “8:15 to 8:30”. It is predicted that the vehicle will arrive near the ski resort, which is the destination, at 8:30 and will travel along the surrounding section.

[0068] A music piece a, which is a thrilling music piece, is selected as a music piece that corresponds to the mountain scenery in the time period “8:00 to 8:15”, and a music piece b, which is a stimulating music piece which does not cause drowsiness, is selected as a music piece that corresponds to the tunnel scenery in the time period “8:15 to 8:30”. A music piece c, which is a music piece related to the ski resort, such as a music piece recommended for the ski resort, is selected as a music piece that corresponds to the scenery around the ski resort in the time period after “8:30”.

[0069] In this example, a playlist including music pieces a to c is generated as a playlist according to the movement route from the current position to the destination. While traveling through each section, the above-described processing is performed to change the music in accordance with changes in the scenery for the music pieces a to c.

[0070] A playlist is prepared according to the movement route from the current position to the destination, and is used as the processing target while traveling along each section, making it possible to generate and play back audio data based on a music piece that corresponds to the changes in scenery along each section. It is possible to provide an appropriate music listening experience in accordance with the changes in the scenery of each section, which is a change in the surrounding situations.

[0071] Configuration of Information Processing System Fig. 9 is a block diagram illustrating another example of the configuration of the information processing system 1. In the configuration illustrated in Fig. 9, the same reference numerals are given to the same configurations as the configurations described with reference to Fig. 5. Duplicate descriptions will be omitted as appropriate.

[0072] The information processing unit 11 illustrated in Fig. 9 includes a playlist generation unit 23 in addition to the prediction unit 21 and the music generation unit 22. The position information and the vehicle body information, which are prediction information, are input to the prediction unit 21. Information about the destination set by the user is also input to the prediction unit 21. Information about the user’s preferences and information about the destination are input to the playlist generation unit 23, and vehicle body information and real-time information, such as sensor data and driving information, are input to the music generation unit 22.

[0073] The prediction unit 21 searches for a movement route from the current position to the destination, predicts changes in scenery of each section on the movement route, and predicts the clock time at which the change in scenery occurs. Information on the prediction result by the prediction unit 21 is supplied to the playlist generation unit 23.

[0074] The playlist generation unit 23 selects music pieces that correspond to the scenery of each section based on the information supplied from the prediction unit 21, and generates a playlist including the selected music pieces. In the example of Fig. 9, a music piece A is selected as the music piece that corresponds to scene #1 in the time period “9:00 to 9:05”, and a music piece B is selected as the music piece that corresponds to scene #2 in the time period “9:05 to 9:10”. Furthermore, a music piece C is selected as a music piece that corresponds to scene #3 in the time period “9:10 to 9:15”.

[0075] When selecting a music piece, information about the user’s preferences and information related to the destination are used as appropriate. The information about the user’s preferences includes, for example, information about the user’s favorite music genres and artists. The selection of a music piece is performed by selecting a music piece in the user’s favorite genre and a music piece by the user’s favorite artist. Furthermore, the selection of a music piece is performed by selecting a music piece related to the destination. Information about the playlist generated by the playlist generation unit 23 is supplied to the music generation unit 22.

[0076] After generating the playlist, the prediction unit 21 repeatedly searches for the movement route again and updates the movement route as appropriate. When the movement route is updated, the changes in scenery in each section on the movement route are re-predicted. When the changes in scenery in each section on the movement route are re-predicted, the playlist generation unit 23 appropriately updates the playlist based on the result of the re-prediction. This makes it possible to generate a playlist according to the changed movement route even when the movement route is changed after the playlist is generated.

[0077] The music generation unit 22 selects the music pieces constituting the playlist generated by the playlist generation unit 23 as music pieces to be played back in sections on the route to the destination. The music piece to be played back is switched according to the section during traveling.

[0078] The playlist generation unit 23 sets processing parameters based on vehicle body information, sensor data, and driving information, and generates audio data to be played back by performing processing defined by the processing parameters on the audio data of the music piece in the section during traveling. The playlist generation unit 23 plays back the generated audio data and causes the output unit 12 to output the playback sound.

[0079] Overall Processing Fig. 10 is a diagram illustrating the flow of overall processing in the information processing system 1, including the generation of a playlist.

[0080] Based on the position information and the vehicle body information, a change in scenery is predicted as indicated by the tip of arrow #101. In the example of Fig. 10, information on a driving course (the route from the current position to the destination) and information on the actual scenery detected by analyzing images captured by an in-vehicle camera are used to predict changes in the scenery.

[0081] The type of scenery in each section is specified as indicated by the tip of arrow #102, and the timing of the change in scenery is predicted as indicated by the tip of arrow #103. As indicated by the tips of arrows #104 and #105, a playlist including music pieces that correspond to the scenery of each section is generated. As indicated by arrows #106 and #107, information about the user’s preferences and destination-related information are used as appropriate to generate the playlist.

[0082] While the vehicle is traveling to the destination, a music piece in the playlist is selected, as indicated by the tip of arrow #108. Instead of the entire music piece, a part of the music piece may be selected and used for playback.

[0083] As indicated by the tip of arrow #109, actual driving situations such as driving operations and waiting at traffic lights are detected based on real-time information, and the time difference between the predicted timing of change in scenery and the actual timing of change in scenery is detected.

[0084] Based on the detected time difference and the user’s operation, the playback timing is adjusted as indicated by the tips of arrows #110 and #111, and processing parameters that define the content such as the mix are set as indicated by the tips of arrows #112 and #113. The audio data of the music pieces in the playlist is processed based on the processing parameters, and as indicated by the tip of arrow #114, a music listening experience suitable for a driving course is provided.

[0085] Configuration of Information Processing System Fig. 11 is a block diagram illustrating an example of a detailed configuration of the information processing system 1. The same reference numerals are given to the same configurations as those described above. Duplicate descriptions will be omitted as appropriate.

[0086] The information processing system 1 is provided in a vehicle 101. In the information processing system 1, not only processing for controlling music playback but also various other processing is performed, such as processing related to driving assistance for the vehicle 101 and processing related to automatic driving.

[0087] The information processing system 1 includes a vehicle control electronic control unit (ECU) 121, a communication unit 122, a map information accumulation unit 123, a position information acquisition unit 124, an external recognition sensor 125, an in-vehicle sensor 126, a vehicle sensor 127, a storage unit 128, a driving assistance / automatic driving control unit 129, a driver monitoring system (DMS) 130, a human machine interface (HMI) 131, and a vehicle control unit 132. These units are communicatively connected to each other via a communication network 141.

[0088] The vehicle control ECU 121 is configured with various processors as processing circuits, such as a central processing unit (CPU) and a micro processing unit (MPU). The vehicle control ECU 121 controls all or some of the functions of the information processing system 1. The information processing unit 11 in Figs. 5 and 9 is implemented in the vehicle control ECU 121.

[0089] The communication unit 122 communicates with various devices inside and outside the vehicle, other vehicles, servers, base stations, and the like. Wireless communication schemes that are used include a 5th generation mobile communication system (5G), a long term evolution (LTE), dedicated short range communications (DSRC), and the like. For example, the communication unit 122 communicates with a server on a network via a base station or an access point.

[0090] The map information accumulation unit 123 accumulates maps acquired from an external source and / or maps created in the vehicle 101. For example, the map information accumulation unit 123 accumulates three-dimensional high-precision maps, global maps covering a wide area, and the like. The maps accumulated in the map information accumulation unit 123 are used as appropriate to predict changes in the scenery.

[0091] The position information acquisition unit 124 receives global navigation satellite system (GNSS) signals from GNSS satellites and acquires position information of the vehicle 101. The acquired position information is supplied to the driving assistance / automatic driving control unit 129. The method is not limited to using GNSS signals, and position information may be acquired using beacons. The position information acquired by the position information acquisition unit 124 is used to predict changes in the scenery.

[0092] The external recognition sensor 125 includes various sensors used to recognize the situation outside the vehicle 101. The external recognition sensor 125 supplies sensor data from each sensor to each unit of the information processing system 1.

[0093] For example, the external recognition sensor 125 includes a camera 151, a radar 152, a Light Detection and Ranging or Laser Imaging Detection and Ranging (LiDAR) 153, and an ultrasonic sensor 154. The number of sensors is not particularly limited as long as they can be installed on the vehicle 101. Furthermore, the type of sensor included in the external recognition sensor 125 is not limited to this example.

[0094] The camera 151 captures an image of the surrounding environment and acquires a captured image that is an RGB image. The imaging method of the camera 151 is not particularly limited. For example, cameras using various imaging methods, such as a time-of-flight (ToF) camera, a stereo camera, a monocular camera, and an infrared camera, which are imaging methods capable of distance measurement, can be applied to the camera 151.

[0095] The external recognition sensor 125 may include an environmental sensor for detecting the environment relative to the vehicle 101. The environmental sensor is a sensor for detecting the environment such as weather, climate, brightness, etc., and can include various sensors such as a raindrop sensor, a fog sensor, a sunlight sensor, a snow sensor, and an illuminance sensor. Furthermore, the external recognition sensor 125 may include a microphone used to detect sounds around the vehicle 101 and the position of sound sources.

[0096] The in-vehicle sensor 126 includes various sensors for detecting information inside the vehicle. The in-vehicle sensor 126 supplies sensor data from each sensor to each unit of the information processing system 1.

[0097] For example, the in-vehicle sensors 126 may include one or more types of sensors among a camera, a radar, an occupancy sensor, a steering wheel sensor, a microphone, and a biological sensor. The biological sensors included in the in-vehicle sensor 126 are provided, for example, on the seats, steering wheel, etc., and detect various types of biological information of occupants such as the driver.

[0098] The vehicle sensor 127 includes various sensors for detecting the traveling situation of the vehicle 101. The vehicle sensor 127 supplies sensor data from each sensor to each unit of the information processing system 1.

[0099] For example, the vehicle sensor 127 includes a speed sensor, an acceleration sensor, an angular velocity sensor (gyro sensor), and an inertial measurement unit (IMU) that integrates these sensors. The vehicle sensor 127 includes a steering angle sensor that detects the steering angle of the steering wheel, a yaw rate sensor, an accelerator sensor that detects the amount of accelerator pedal operation, and a brake sensor that detects the amount of brake pedal operation. For example, the vehicle sensor 127 includes a rotation sensor that detects the number of revolutions of the engine or motor, an air pressure sensor that detects the air pressure of the tires, a slip ratio sensor that detects the slip ratio of the tires, and a wheel speed sensor that detects the rotation speed of the wheels. For example, vehicle body information is acquired based on sensor data acquired by the vehicle sensor 127.

[0100] At least one of the information items acquired by the external recognition sensor 125, the in-vehicle sensor 126, and the vehicle sensor 127 is supplied to the music generation unit 22 as sensor data and used for playback control of a music piece.

[0101] The storage unit 128 includes at least one of a non-volatile storage medium and a volatile storage medium. The storage unit 128 stores programs and data used by each unit of the information processing system 1 in addition to the DB of the music piece.

[0102] The driving assistance / automatic driving control unit 129 controls the driving assistance and automatic driving of the vehicle 101. For example, the driving assistance / automatic driving control unit 129 includes an analysis unit 161, an action planning unit 162, and an operation control unit 163.

[0103] The analysis unit 161 performs analysis processing of the vehicle 101 and the surrounding situation. The analysis unit 161 includes a self-position estimation unit 171, a sensor fusion unit 172, and a recognition unit 173.

[0104] The self-position estimation unit 171 estimates the self-position of the vehicle 101 based on the sensor data from the external recognition sensor 125 and the high-precision map accumulated in the map information accumulation unit 123. For example, the self-position estimation unit 171 generates a local map based on sensor data from the external recognition sensor 125 and estimates the self-position of the vehicle 101 by matching the local map with a high-precision map.

[0105] The local map is, for example, a three-dimensional high-precision map created using technology such as simultaneous localization and mapping (SLAM), an occupancy grid map, or the like. The local map is also used, for example, by the recognition unit 173 in the detection processing and recognition processing of the external situation of the vehicle 101. The self-position estimation unit 171 may estimate the self-position based on the position information acquired by the position information acquisition unit 124 and the sensor data from the vehicle sensor 127.

[0106] The sensor fusion unit 172 performs sensor fusion processing to combine a plurality of different types of sensor data to obtain new information. Methods for combining different types of sensor data include integration, fusion, and federation.

[0107] The recognition unit 173 performs detection processing and recognition processing of the situation outside the vehicle 101 based on information from the external recognition sensor 125, information from the self-position estimation unit 171, information from the sensor fusion unit 172, and the like.

[0108] For example, the recognition unit 173 performs detection processing and recognition processing of objects around the vehicle 101. The detection processing of an object is processing of detecting, for example, the presence or absence, size, shape, position, movement, or the like of an object. The position or the like of an object is represented by, for example, a bounding box. The recognition process of an object is, for example, processing of recognizing attributes such as the type of object, or identifying a specific object.

[0109] In addition, the recognition unit 173 detects objects around the vehicle 101 by performing clustering to classify the point cloud based on sensor data from the radar 152 or the LiDAR 153, etc., into clusters of point groups. This allows the presence or absence, size, shape, and position of objects around the vehicle 101 to be detected.

[0110] The recognition unit 173 detects the movement of objects around the vehicle 101 by performing tracking that follows the movement of the clusters of point groups classified by clustering. This allows the speed and direction of travel (movement vector) of objects around the vehicle 101 to be detected.

[0111] The recognition unit 173 detects or recognizes vehicles, people, bicycles, obstacles, structures, roads, traffic lights, traffic signs, road markings, or the like based on the image data supplied from the camera 151. The recognition unit 173 may also recognize the type of object around the vehicle 101 by performing recognition processing such as semantic segmentation.

[0112] The recognition unit 173 can perform recognition processing of traffic rules around the vehicle 101 based on the map accumulated in the map information accumulation unit 123, the estimation result of the self-position by the self-position estimation unit 171, and the recognition result of objects around the vehicle 101 by the recognition unit 173. Through this processing, the recognition unit 173 can recognize the position and state of traffic lights, the content of traffic signs and road markings, the content of traffic regulations, a lane in which the vehicle can travel, and the like.

[0113] The recognition unit 173 can perform recognition processing of the environment around the vehicle 101. As the surrounding environment to be recognized by the recognition unit 173, weather, temperature, humidity, brightness, road surface conditions, and the like are assumed. Information acquired by processing in the self-position estimation unit 171, the sensor fusion unit 172, and the recognition unit 173, or sensor data used in the processing, is used as ADAS data.

[0114] The action planning unit 162 creates an action plan for the vehicle 101. For example, the action planning unit 162 creates an action plan by performing route planning and route tracking processing. Route planning (global path planning) is processing of planning a rough route from the start to the goal. The action plan created by the action planning unit 162 includes the creation of a movement route from the current position to the destination. The information about the action route created by the action planning unit 162 is used by the information processing unit 11 when generating a playlist, for example.

[0115] The operation control unit 163 controls the operation of the vehicle 101 to realize the action plan created by the action planning unit 162. The operation control unit 163 realizes the function of the ADAS.

[0116] For example, the operation control unit 163 controls a steering control unit 181, a brake control unit 182, and a drive control unit 183 included in the vehicle control unit 132 to perform acceleration / deceleration control and directional control such that the vehicle 101 moves along the calculated trajectory. For example, the operation control unit 163 performs cooperative control for the purpose of realizing ADAS functions such as collision avoidance or impact mitigation, following traveling, maintaining vehicle speed, collision warning for the host vehicle, and lane departure warning for the host vehicle. For example, the operation control unit 163 performs cooperative control for the purpose of automatic driving, which allows the vehicle to travel autonomously without relying on the driver’s operation.

[0117] The DMS 130 performs processing such as authenticating the driver and recognizing the driver’s situation based on sensor data from the in-vehicle sensor 126 and input data input to the HMI 131. As the driver’s situation to be recognized, for example, a physical condition, a degree of alertness, a degree of concentration, a degree of fatigue, a line-of-sight direction, a degree of intoxication, a driving operation, a posture, and the like are assumed.

[0118] The HMI 131 receives inputs of various types of data and instructions, and presents various types of data to the driver or the like. The HMI 131 includes an input device for a person to input data. The HMI 131 generates input signals based on data, instructions, or the like input via an input device, and supplies the signals to each unit of the information processing system 1. The HMI 131 includes operation equipment such as a touch panel, buttons, switches, and levers as input devices.

[0119] The HMI 131 outputs information such as an operation screen, a display of the situation of the vehicle 101, a warning display, and a monitor image showing the situation around the vehicle 101, as visual information. A display device and a projector device are provided as output devices for outputting visual information. The HMI 131 also outputs audio information such as voice guidance, warning sounds, and warning messages as auditory information. An audio speaker and headphones are provided as output devices for outputting auditory information. The output unit 12 is included in the HMI 131.

[0120] The vehicle control unit 132 controls each unit of the vehicle 101. The vehicle control unit 132 includes a steering control unit 181, a brake control unit 182, a drive control unit 183, a body system control unit 184, a light control unit 185, and a horn control unit 186.

[0121] The steering control unit 181 performs detection, control, and the like of a state of a steering system of the vehicle 101. The brake control unit 182 performs detection, control, and the like of a state of a brake system of the vehicle 101. The drive control unit 183 performs detection, control, and the like of a state of a drive system of the vehicle 101. The drive system includes, for example, an accelerator pedal, a driving force generating device, such as an internal combustion engine or a driving motor, for generating a driving force, a driving force transmitting mechanism for transmitting the driving force to the wheels, and the like. The body system control unit 184 performs detection, control, and the like of a state of a body system of the vehicle 101. The body system includes, for example, a keyless entry system, a smart key system, a power window device, a power seat, an air conditioner, an airbag, a seat belt, a shift lever, and the like. The light control unit 185 performs detection, control, and the like of states of various lights of the vehicle 101. The horn control unit 186 performs detection, control, and the like of a state of a car horn of the vehicle 101.

[0122] Information indicating the content of control of each unit of the vehicle control unit 132 is supplied to the music generation unit 22 as driving information and is used for playback control of a music piece.

[0123] Modification Example The prediction of changes in scenery can be made based on various types of information contained in map information, such as road information, building information, topography information, and place name information.

[0124] The present technology can also be applied to controlling the playback of music output in mobile bodies other than cars, such as ships, aircrafts, trains, and bicycles. This can be applied to controlling the playback of music not only when the user is on a mobile body, but also while walking.

[0125] Changes in the scenery around the vehicle are predicted and the prediction results are used to control music playback, but it is also possible to control music playback based on the prediction results of various situation changes. Changes in the surrounding situations other than changes in scenery include changes in the weather, and changes in objects such as buildings, pedestrians, and other traveling vehicles that can be seen in the surroundings. For example, a change in the weather is predicted as a change in the situation around the vehicle, and playback control of music is performed according to the prediction result of the change in the weather.

[0126] Modification Example of Configuration of Information Processing System Fig. 12 is a diagram illustrating another example of the configuration of the information processing system 1.

[0127] As illustrated in Fig. 12, the prediction unit 21 and the music generation unit 22 may be provided in an information processing server 201, which is an information processing apparatus on the Internet. The vehicle 101 is provided with a music playback unit 101A. Various types of information are transmitted and received between the information processing server 201 and the vehicle 101, and the above-mentioned processing is performed.

[0128] For example, the prediction unit 21 of the information processing server 201 predicts changes in the scenery around the vehicle 101 based on the current position and the vehicle body information of the vehicle 101 represented by information transmitted from the vehicle 101. The music generation unit 22 performs processing based on processing parameters set in accordance with changes in scenery or the like, and generates audio data to be played back. The audio data generated by the music generation unit 22 is transmitted to the vehicle 101 and acquired by the music playback unit 101A.

[0129] The music playback unit 101A plays back the audio data transmitted from the information processing server 201 under the control of the information processing server 201, and outputs the playback sound. In this way, even when the prediction unit 21 and the music generation unit 22 are provided in the information processing server 201, which is an external information processing apparatus, playback control of music in the vehicle 101 can be performed.

[0130] About Program The above-described series of processing steps can be executed by hardware or software. When the series of processing steps is executed by the software, a program constituting the software is installed on a computer. Here, the term “computer” includes a computer incorporated in dedicated hardware, and a computer that can execute various functions by installing various programs, such as a computer having a configuration as illustrated in Fig. 13.

[0131] The program executed by the computer (processing circuit 351 (Fig. 13)) can be provided by being recorded on a removable medium 361 such as a package medium, for example. Further, the program can be provided via a wired or wireless transmission medium such as a local area network, the Internet, or digital satellite broadcasting.

[0132] In the computer, the program can be installed in a storage unit 358 via an input / output interface 355 by inserting the removable medium 361 into a drive 360. In addition, the program can be received by a communication unit 359 from another device such as a server via a wired or wireless transmission medium and installed in the storage unit 358. In addition, the program can be installed in advance in a ROM 352 or the storage unit 358.

[0133] The program executed by the computer may be a program in which processing is performed in time series in the order described in the present specification, or may be a program in which processing is performed in parallel or at a necessary timing such as when a call is performed.

[0134] The processing performed by the computer according to the program does not have to be performed in time series in the order described as the flowchart. That is, the processing performed by the computer according to the program includes processing executed in parallel or in an individual manner (for example, parallel processing or processing by an object).

[0135] The program may be processed by one computer (processor) or may be processed in a distributed manner by a plurality of computers. Further, the program may be transferred to a remote computer and executed.

[0136] In this specification, a system refers to one component or a set of a plurality of components (devices, modules (parts), etc.). Thus, one or more components of a computer, for example, a processor alone or a combination of a processor and a memory, is a system. With respect to a set of a plurality of components, it does not matter whether all the components are in the same housing or not. Therefore, the system refers to both a plurality of devices housed in separate housings and connected via a network, and one device in which a plurality of modules are housed in one housing. Furthermore, for example, the entire computer, or a combination of a computer and another device such as a server (not illustrated) is also a system.

[0137] The embodiments of the present technology are not limited to the above-described embodiments, and various modifications can be made without departing from the gist of the present technology.

[0138] Note that the effects described in the present specification are merely examples and are not limited, and other effects may be provided.

[0139] Combination Example of Configurations The present technology may also have the following configurations.

[0140] (1) An information processing apparatus including: a prediction unit that predicts a change in a situation around a mobile body after a certain time based on position information and information indicating a movement status of the mobile body; and a music generation unit that performs processing according to a prediction result of the change in the situation around the mobile body on audio data of a music piece to be output in the mobile body. (2) The information processing apparatus according to (1), in which the prediction unit predicts a change in scenery around the mobile body as the change in the situation around the mobile body. (3) The information processing apparatus according to (2), in which the music generation unit performs processing of changing a playback sound in accordance with a timing of the change in scenery on the audio data of the music piece. (4) The information processing apparatus according to (3), in which the music generation unit performs processing of adjusting a timing of a change in a music piece composition on the audio data of the music piece. (5) The information processing apparatus according to (4), in which the music generation unit performs the processing on audio data of a section of the music piece preceding the timing of the change in the music piece composition. (6) The information processing apparatus according to any one of (3) to (5), in which the music generation unit detects the scenery around the mobile body based on real-time information including sensor data measured while the mobile body is moving, and performs the processing on the audio data of the music piece in accordance with a difference between a timing of a change in the detected scenery and a predicted timing of the change in scenery. (7) The information processing apparatus according to any one of (1) to (6), in which the music generation unit selects the processing according to the prediction result based on information associating a type of the situation around the mobile body with content of processing to be performed on the audio data of the music piece. (8) The information processing apparatus according to any one of (1) to (7), in which the music generation unit plays back audio data generated by performing the processing according to the prediction result, and outputs a playback sound in the mobile body. (9) The information processing apparatus according to any one of (1) to (8), in which the prediction unit predicts a change in a surrounding situation along a movement route of the mobile body to a destination, and the music generation unit generates a playlist including the music pieces each according to a surrounding situation of a section included in the movement route. (10) The information processing apparatus according to (9), in which the music generation unit switches the music piece that is a target of the processing according to the prediction result in accordance with a section in which the mobile body is moving. (11) The information processing apparatus according to (9) or (10), in which the music generation unit selects a music piece related to the destination as the music piece according to the surrounding situation of the section around the destination. (12) The information processing apparatus according to any one of (9) to (11), in which the prediction unit re-predicts the movement route based on the position information and the information indicating the movement status of the mobile body, and the music generation unit updates the playlist based on the re-predicted movement route. (13) An information processing method including: predicting, by an information processing apparatus, a change in a situation around a mobile body after a certain time based on position information and information indicating a movement status of the mobile body; and performing, by the information processing apparatus, processing according to a prediction result of the change in the situation around the mobile body on audio data of a music piece to be output in the mobile body. (14) A program causing a computer to execute processing including: predicting a change in a situation around a mobile body after a certain time based on position information and information indicating a movement status of the mobile body; and performing processing according to a prediction result of the change in the situation around the mobile body on audio data of a music piece to be output in the mobile body.

[0141] 1 Information processing system 11 Information processing apparatus 21 Prediction unit 22 Music generation unit 23 Playlist generation unit

Claims

An information processing system comprising:one or more processing circuitry configured topredict a change in a situation around a mobile body based on information indicating a status of the mobile body;obtain real-time information measured while the mobile body is moving;perform processing on audio data of a music piece to be output at the mobile body according to the predicted change in the situation around the mobile body and the real-time information; andcause a transducer at the mobile body to output playback audio generated by performing the processing on the audio data of the music piece according to the predicted change in the situation around the mobile body.The information processing system according to claim 1, wherein the predicted change in the situation around the mobile body includes a change in scenery around the mobile body.The information processing system according to claim 2, wherein the processing includes changing a playback sound of the audio data of the music piece in accordance with a timing of the change in scenery.The information processing system according to claim 3, wherein the processing includes adjusting a timing of a change in a music piece composition of the audio data of the music piece.The information processing system according to claim 4, wherein the one or more processing circuitry is configured to perform the processing on audio data of a section of the music piece preceding the timing of the change in the music piece composition.The information processing system according to claim 3, whereinthe real-time information includes sensor data measured while the mobile body is moving, andthe one or more processing circuitry is configured todetect the scenery around the mobile body based on the sensor data measured while the mobile body is moving, andperform the processing on the audio data of the music piece in accordance with a difference between a timing of a change in the detected scenery and a predicted timing of the change in scenery.The information processing system according to claim 1, wherein the one or more processing circuitry is configured to select the processing according to the predicted change in the situation around the mobile body based on information associating a type of the situation around the mobile body with content of processing to be performed on the audio data of the music piece.The information processing system according to claim 1, wherein the one or more processing circuitry is configured to:predict a plurality of changes in the situation around the mobile body along a movement route of the mobile body to a destination, andgenerate a playlist including music pieces each according to one of the plurality of changes in the situation around the mobile body included in a section of the movement route.The information processing system according to claim 8, wherein the one or more processing circuitry is configured to switch one of the music pieces that is a target of the processing according to the predicted change in the situation around the mobile body in accordance with the section in which the mobile body is moving.The information processing system according to claim 8, wherein the processing circuitry is configured to select a music piece related to the destination as the one of the music pieces for the section around the destination.The information processing system according to claim 8, wherein the one or more processing circuitry is configured to:re-predict the the information indicating the status of the mobile body, andupdate the playlist based on the re-predicted movement route.The information processing system according to claim 1, wherein the information indicating the status of the mobile body includes position information of the mobile body and information indicating a movement status of the mobile body.The information processing system according to claim 1, wherein the mobile body is an automobile.The information processing system according to claim 1, wherein at least one of the one or more processing circuitry is disposed within an automobile.The information processing system according to claim 13, wherein at least one of the one or more processing circuitry is disposed within the automobile.The information processing system according to claim 15, wherein the at least one of the one or more processing circuitry that is disposed within the automobile is configured to perform processing related to at least one of driving assistance or automated driving.The information processing system according to claim 1, wherein the transducer is disposed within one of a speaker, headphones, or earphones.The information processing system according to claim 1, wherein the mobile body is one of a ship, an aircraft, a train, and a bicycle.An information processing method comprising:predicting, by one or more processing circuitry, a change in a situation around a mobile body based on information indicating a status of the mobile body;obtaining, by the one or more processing circuitry, real-time information measured while the mobile body is moving;performing, by the one or more processing circuitry, processing on audio data of a music piece to be output at the mobile body according to the predicted change in the situation around the mobile body and the real-time information; andcausing, by the one or more processing circuitry, a transducer at the mobile body to output playback audio generated by performing the processing on the audio data of the music piece according to the predicted change in the situation around the mobile body.A non-transitory computer readable medium storing a program, which when executed causes a computer to perform a method, the method comprising:predicting a change in a situation around a mobile body based on information indicating a status of the mobile body;obtaining real-time information measured while the mobile body is moving;performing processing on audio data of a music piece to be output at the mobile body according to the predicted change in the situation around the mobile body and the real-time information; andcausing a transducer at the mobile body to output playback audio generated by performing the processing on the audio data of the music piece according to the predicted change in the situation around the mobile body.

Images