Control device, control method, computer program product, storage medium, and computer program
The control device synchronizes motion graphics with music playback on vehicle displays, enhancing user experience by adapting to vehicle and environmental states, addressing the limitations of existing systems.
Patent Information
- Application Number
- PCT/JP2026/018123
- Authority / Receiving Office
- WO · WO
- Patent Type
- Applications
- Current Assignee / Owner
- Priority Date
- 2025-05-13
- Filing Date
- 2026-05-13
- Publication Date
- 2026-11-19
AI Technical Summary
Existing music playback systems in vehicles do not effectively display motion graphics synchronized with music playback, limiting user engagement and enjoyment.
A control device that acquires lyric data and synchronizes motion graphics with music playback on a vehicle display, allowing display on interior and exterior surfaces, with adaptive changes based on vehicle and environmental states detected by sensors.
Enhances user experience by providing synchronized motion graphics that adapt to vehicle conditions, increasing engagement and enjoyment for both occupants and external viewers.
Smart Images

Figure JP2026018123_19112026_PF_FP_ABST
Abstract
Description
Control device, control method, computer program product, storage medium, and computer program Cross-Reference to Related Applications
[0001] This international application claims priority based on Japanese Patent Application No. 2025-080660 filed with the Japan Patent Office on May 13, 2025, and the entire content of Japanese Patent Application No. 2025-080660 is incorporated into this international application by reference.
[0002] The present disclosure relates to a control device.
[0003] There is known a music playback system that displays motion graphics including lyrics in synchronization with the playback of a music piece. For example, Patent Document 1 discloses a display control system that acquires genre data of a music piece from a server, and causes a display device to display lyrics via motion graphics corresponding to the genre of the music piece.
[0004] Japanese Patent No. 6978028
[0005] The present inventors are considering applying the above-described technology related to the display of motion graphics to an in-vehicle display. One aspect of the present disclosure desirably provides a new technology that applies the technology related to the display of motion graphics including lyrics to a vehicle.
[0006] One aspect of the present disclosure is a control device comprising at least one processor and at least one memory. The memory stores instructions that cause the at least one processor to execute processing. The processing includes acquiring lyric data of a music piece played by an audio device, and causing a display device mounted on the vehicle to display motion graphics including the lyrics of the music piece in synchronization with the progress of the music piece played by the audio device toward the interior of the vehicle cabin, based on the lyric data. The display device is configured to display the motion graphics toward at least one of the interior of the vehicle cabin and the outside of the vehicle.
[0007] With this configuration, motion graphics including lyrics can be displayed towards at least one of the vehicle's interior and exterior, in sync with the progression of music played by the sound system. This allows for a more enjoyable experience for users inside and around the vehicle.
[0008] In one aspect of this disclosure, the display device may be configured to display motion graphics toward the outside of the vehicle.
[0009] With this configuration, motion graphics including lyrics can be displayed to the outside of the vehicle in sync with the progression of music played by the sound system. This allows for a more engaging experience for users around the vehicle.
[0010] In one aspect of this disclosure, the display device may include at least one screen. The at least one screen may be shaped to conform to the shape of a vehicle part. Lyrics in motion graphics may be displayed on the at least one screen in a layout based on the shape of the at least one screen.
[0011] With this configuration, lyrics in motion graphics can be displayed in a layout that conforms to the shape of the vehicle's parts.
[0012] In one aspect of this disclosure, at least a portion of the lyrics in the motion graphics may be displayed on at least one screen so as to align with the boundary of at least one screen.
[0013] With this configuration, lyrics in motion graphics can be displayed in a layout that follows the boundaries of the vehicle's parts.
[0014] In one aspect of this disclosure, the process may include changing the lyrics included in motion graphics displayed on a display device, provided that at least one of one of a predetermined state relating to the vehicle is detected. The predetermined state is one or more states including at least one of the state of the vehicle and the state of the area surrounding the vehicle, and may be detected based on input signals from at least one sensor mounted on the vehicle.
[0015] With this configuration, expressions corresponding to the vehicle's state can be reflected in the lyrics of the motion graphics and displayed on the display device.
[0016] In one aspect of this disclosure, the motion graphics may further include a background. The process may include changing at least one of the lyrics and background included in the motion graphics displayed on the display device, provided that at least one of one of a predetermined state relating to the vehicle is detected. The predetermined state is one or more states including at least one of the state of the vehicle and the state of the area surrounding the vehicle, and may be detected based on input signals from at least one sensor mounted on the vehicle.
[0017] With this configuration, expressions corresponding to the vehicle's state can be reflected in at least one of the lyrics and background in the motion graphics and displayed on the display device.
[0018] In one aspect of this disclosure, the change may include selecting one change pattern from a plurality of change patterns relating to motion graphics based on at least one detected state, and changing at least one of the lyrics and the background based on that change pattern.
[0019] With this configuration, motion graphics that reflect the vehicle's state can be displayed on the display device.
[0020] In one aspect of this disclosure, the change may include selecting one change pattern from a plurality of change patterns based on the priority set for each of the first and second states, if both the first and second states are detected within a predetermined period. At least one sensor may include a first sensor and a second sensor. The first state may be one or more predetermined states detected based on an input signal from the first sensor. The second state may be one or more predetermined states detected based on an input signal from the second sensor.
[0021] With this configuration, if multiple states are detected within a predetermined period, it is possible to select a change pattern corresponding to the state with the highest priority.
[0022] In one aspect of this disclosure, the change may include changing at least one of the lyrics and background included in motion graphics displayed on a display device based on the specific condition, provided that the specific condition is detected. At least one sensor may include a camera configured to capture images of the area around the vehicle. The specific condition may be one or more predetermined conditions of the area around the vehicle detected based on an image or video represented by an input signal from the camera.
[0023] With this configuration, motion graphics can be changed based on images or videos captured by the camera.
[0024] One aspect of this disclosure is a control method performed by a computer, which includes acquiring lyric data of a musical piece played by an audio device, and, based on the lyric data, causing motion graphics including the lyrics of the musical piece to be displayed on a display device mounted in the vehicle in accordance with the progress of the musical piece played by the audio device toward the interior of the vehicle. The display device is configured to display the motion graphics toward at least one of the interior and exterior of the vehicle.
[0025] This control method produces the same effect as the control device described above.
[0026] One aspect of this disclosure may provide a computer program product comprising a computer program that includes instructions for causing a computer to execute the control method described above.
[0027] One aspect of this disclosure may provide a non-temporary computer-readable storage medium that stores instructions for causing a computer to execute the control method described above.
[0028] One aspect of this disclosure may provide a computer program that includes instructions for causing a computer to perform the control method described above.
[0029] This is a block diagram showing the configuration of the control device. This is a diagram showing the configuration of motion graphics data. This is a diagram showing an example of lyrics for motion graphics displayed on a screen shaped to follow the shape of the vehicle's windows. Figure 4A shows an example of lyrics for motion graphics displayed on a screen shaped to follow the shape of the vehicle's hood. Figure 4B shows an example of lyrics for motion graphics displayed on a screen shaped to follow the shape of the vehicle's fenders, doors, side sills, etc. This is a diagram showing the configuration of lyrics data. This is a flowchart showing the playback display process. This is a flowchart showing the playback plan creation process. This is a diagram showing an example of motion graphics before and after a change pattern. This is a table that associates each sensor included in the sensor group with the state detected as a predetermined state and the change pattern corresponding to the detected state.
[0030] Hereinafter, exemplary embodiments of the present disclosure will be described with reference to the drawings. [1. First Embodiment] [1-1. Configuration] [1-1-1. Overall Configuration] The music playback system 1 of this embodiment shown in Figure 1 comprises a control device 10, a vehicle 2, a sensor group 30, a speaker 51, a display 52, and a server group 7. The control device 10, the sensor group 30, the speaker 51, and the display 52 are all mounted and used in the vehicle 2.
[0031] The control device 10, sensor group 30, speaker 51, and display 52 may each be built into the vehicle 2, or some or all of them may be attached externally to the vehicle 2. The speaker 51 and display 52 may be provided in an information terminal used by a user riding in the vehicle 2. Examples of information terminals include smartphones and tablet devices.
[0032] The music playback system 1 is configured to play music data D1 through a speaker 51 using a control device 10, and to display motion graphics, including lyrics and background, on a display 52 in accordance with the progress of the music. The control device 10 determines the display format of these motion graphics based on lyrics data DL, song structure data DS, and genre data DG acquired from the server group 7.
[0033] The control device 10 comprises a processor 11, a memory 12, a storage 13, a user interface 14, and a communication interface 15. The processor 11 is configured to perform various functions of the control device 10 by executing one or more instructions contained in a computer program stored in the memory 12 and / or the storage 13.
[0034] Memory 12 is configured to store computer programs. Memory 12 is used as a working area when processing is executed by the processor 11. Examples of memory 12 include RAM (Random Access Memory), ROM (Read Only Memory), and flash memory.
[0035] Storage 13 holds computer programs and data used when executing processes that follow the computer programs. Examples of storage 13 include eMMC (embedded Multi Media Card), UFS (Universal Flash Storage), SSD (Solid State Drive), HDD (Hard Disk Drive), etc.
[0036] The data held in storage 13 includes motion graphics data D2 (see Figure 2) for each genre of music, and change pattern sets. Each of the motion graphics data D2 defines motion graphics for text images and background images to be displayed on the display 52 when music is played. The change pattern sets define one or more change patterns that change at least one of the text images and background images in the motion graphics data D2. Details of the motion graphics data D2 and change patterns will be described later.
[0037] The user interface 14 is a general term for interfaces that accept various input operations from the user and interfaces that output various information to the user. Examples of user interfaces 14 include touch panels, physical buttons, monitors, etc.
[0038] The communication interface 15 is an interface configured to communicate various data with devices inside and outside the vehicle 2 according to a predetermined standard. Communication with devices mounted on the vehicle 2 can be achieved, for example, by in-vehicle network communication such as CAN (Controller Area Network) or in-vehicle Ethernet, short-range wireless communication such as Bluetooth or Wi-Fi, long-range wireless communication such as LTE or 5G, or wired communication such as USB. Ethernet, Bluetooth, and Wi-Fi are all registered trademarks.
[0039] The sensor group 30 is at least one sensor mounted on the vehicle 2. The sensor group 30 continuously acquires various information representing the state of the vehicle 2 and transmits the acquired information to the control device 10 as an input signal. The state of the vehicle 2 may be the state of the surroundings of the vehicle 2, the state of the vehicle 2 itself, the driving state of the vehicle 2, etc. For example, the state of the vehicle 2 may be "the vehicle 2 is driving through a forest," "the temperature inside or outside the vehicle 2 is 30°C or higher," or "the vehicle 2 has passed over a step."
[0040] In the present embodiment, the sensor group 30 includes a camera 31, a LiDAR 32, an illuminance sensor 33, a temperature sensor 34, a vibration sensor 35, and a GPS receiver 36. The camera 31 is configured to capture images of objects around the vehicle 2.
[0041] The LiDAR 32 (Light Detection and Ranging) is configured to measure the distance to an object present in a predetermined area in front of the vehicle 2. The illuminance sensor 33 is configured to measure the illuminance around the vehicle 2.
[0042] The temperature sensor 34 is configured to measure the temperature inside and outside the vehicle 2. The vibration sensor 35 is configured to measure the magnitude of vibration generated in the vehicle 2. The GPS (Global Positioning System) receiver 36 is configured to measure the current position of the vehicle 2 by receiving GPS signals transmitted from GPS satellites.
[0043] Communication between the control device 10 and the sensor group 30 is implemented via an in-vehicle network. The control device 10 is configured by the communication interface 15 to acquire information captured or measured by the sensor group 30 via the in-vehicle network.
[0044] Communication between the control device 10 and the server group 7 is implemented via a wide area network. The wide area network is, for example, the Internet. The control device 10 is configured by the communication interface 15 to acquire music data D1, lyrics data DL, music structure data DS, and genre data DG via the wide area network.
[0045] The speaker 51 outputs various types of audio to the user. For example, the speaker 51 includes a plurality of speaker units and amplifiers corresponding to a plurality of different sound ranges. The speaker 51 converts an audio signal input from the control device 10 into sound and outputs the sound.
[0046] The display 52 includes at least one screen 52a. The display 52 displays various types of information to the user on the at least one screen 52a. For example, during music playback, the display 52 receives a video signal representing motion graphics synchronized with the playback of the music from the control device 10, converts the video signal into a video, and displays the motion graphics on the screen 52a. In the present disclosure, displaying on the screen 52a of the display 52 is also simply referred to as "displaying on the display 52".
[0047] Examples of the display 52 include a liquid crystal display, an organic EL display, a micro LED display, and the like. The display 52 may be a transparent display. The display 52 may be configured to project the screen 52a onto a part of the vehicle 2 (for example, a window glass) like a head-up display.
[0048] The display 52 may be used as a single display that displays one piece of motion graphics on one screen 52a. The display 52 may be used as a multi-display that displays one piece of motion graphics across a plurality of screens 52a.
[0049] The shape of the screen 52a is not particularly limited. For example, the screen 52a may have a quadrilateral shape. The screen 52a may have a shape that follows the shape of a component of the vehicle 2. For example, as shown in FIG. 3, the screen 52a may have a shape that follows the shape of the window glass of the vehicle 2. Alternatively, as shown in FIG. 4A, the screen 52a may have a shape that follows the shape of the hood of the vehicle 2. Alternatively, as shown in FIG. 4B, the screen 52a may have a shape that follows the shape of the fender, door, side sill, or the like of the vehicle 2.
[0050] The motion graphics displayed on the display 52 include changes in character images and changes in background images. The background image is located behind the character image, and decorates the character image around the character image.
[0051] The speaker 51 and the display 52 may be built into the vehicle 2 or attached to the vehicle 2 externally. The speaker 51 is configured to output various sounds toward at least one of the interior and exterior of the vehicle 2. The display 52 is configured to display motion graphics toward at least one of the interior and exterior of the vehicle 2.
[0052] For example, speaker 51 may be installed inside the vehicle 2. Display 52 may be installed inside the vehicle 2. A user riding in vehicle 2 can view the sound output from speaker 51 and the image displayed on display 52.
[0053] Alternatively, the speaker 51 may be mounted on the outside of the vehicle 2. The display 52 may also be mounted on the outside of the vehicle 2. Users near the vehicle 2 can view the sound output from the speaker 51 and the images displayed on the display 52.
[0054] The screen 52a of the display 52 may be transparent. For example, as shown in Figure 3, the screen 52a may be placed on the window glass of the vehicle 2. In this case, the motion graphics are displayed on the window glass of the vehicle 2. The screen 52a may display the motion graphics either facing the interior of the vehicle 2 or facing the exterior of the vehicle 2.
[0055] The control device 10 transmits audio signals to the speaker 51 and video signals to the display 52 via the communication interface 15. The server group 7 includes one or more servers. Each server is configured to hold various types of data.
[0056] The various data held by server group 7 include song data D1, lyrics data DL corresponding to song data D1, song structure data DS, and genre data DG. Details of the various data will be described later. Server group 7 is configured to transmit the various data requested by control device 10 to control device 10 in accordance with the request signal from control device 10.
[0057] The various types of data held in server group 7 may be held on the same server as other data, or on different servers. For example, in this embodiment, song data D1, lyrics data DL, and song structure data DS are held on different servers, but some or all of the data of song data D1, lyrics data DL, and song structure data DS may be held on a single server. Song structure data DS and genre data DG are held on the same server, but may be held on different servers.
[0058] [1-1-2. Composition of Various Data] The following describes the lyrics data DL, song structure data DS, genre data DG, song data D1, and motion graphics data D2.
[0059] As shown in Figure 5, the lyrics data download is structured so that the lyrics of the song are written in a way that they are separated into at least one line at a time. The lyrics data download is structured so that at multiple points in the lyrics of the song, a time code TC representing the start time of singing at that point is embedded in a predetermined description format.
[0060] The timecode TC corresponds to the playback position or playback time at the above point, with the beginning of the music being played as the origin. Figure 5 conceptually shows the structure of lyrics data DL with timecode TC attached. In the example shown in Figure 5, the lyrics data DL has a timecode TC at the beginning of each line of lyrics, and also has a timecode TC at the middle of each line.
[0061] Song structure data DS is data based on the results of an automated analysis that understands the repeat structure and modulation of a song, analyzing the musical elements it possesses. For example, song structure data DS for pop music includes information that can identify sections of the song such as "intro," "verse," "pre-chorus," "chorus," "bridge," "outro," and "interlude." "Verse," "pre-chorus," "chorus," and "bridge" are terms used in the Japanese market. As indicated in parentheses, "verse," "pre-chorus," "chorus," and "bridge" correspond to "verse," "pre-chorus," "chorus," and "bridge," respectively.
[0062] The song structure data DS includes information that allows for the identification of sections that make up a song, such as sections that are defined by points where the musical style changes, sections that indicate repetition, and sections that correspond to the chorus or other climactic parts.
[0063] Genre data DG represents the genre of the song. The song genre (hereinafter also referred to as song genre) can be understood as a classification of songs such as pop, rock, and reggae, not based on the lyrics. In this embodiment, genre data DG is stored in the server group 7 together with the song structure data DS analyzed as described above.
[0064] The music data D1 includes sound information for playing the music. When playback processing is performed by the processor 11, the music data D1 is played back through the speaker 51. Playback processing may be a process in which the control device 10 decodes the music data D1 into a form in which the music can be output from the speaker 51. In this embodiment, as playback processing, the processor 11 performs processing to realize streaming playback of the music data D1. However, playback processing is not limited to streaming playback. For example, playback processing may be a process of offline playback of music data D1 stored in the storage 13.
[0065] The music data D1 is structured with metadata attached to the music data itself. The metadata contains song information such as the song title, album name, and singer (artist) name. The metadata is used to obtain the lyrics data DL, song structure data DS, and genre data DG corresponding to the music data D1 to be played from the server group 7.
[0066] As shown in Figure 2, the motion graphics data D2 includes multiple component data D21 and multiple condition data D22 for each section. The multiple component data D21 are used alternately. Each of the multiple condition data D22 is data that defines the component data D21 to be used in the corresponding section.
[0067] Each component data D21 includes an identification code for the component data D21 (ID_i, where i is an integer between 1 and N, and N is an integer greater than or equal to 1), as well as text motion definition data D211 and background motion definition data D212.
[0068] Text motion definition data D211 defines the motion of the character image to be displayed on the display 52. For example, text motion definition data D211 defines the font type, font size, movement pattern, speed, etc. of the character image. It should be understood that text motion definition data D211 does not define the character itself. The characters displayed in motion graphics are lyrics and other characters related to the song.
[0069] The background motion definition data D212 defines the motion of the background image to be displayed on the display 52. For example, the background motion definition data D212 defines the type of background image, the motion pattern, the speed, etc.
[0070] A group of component data D21 may be understood to include component data D21 that defines motion graphics consisting only of a background image. Component data D21 may not have text motion definition data D211, or it may formally have text motion definition data D211, but the text motion definition data D211 may be substantially empty.
[0071] Each of the condition data D22 shown in Figure 2 defines the component data D21 to be used in the corresponding section, for each type of lyric content. Hereafter, the type of lyric content will simply be referred to as lyric classification.
[0072] As shown in the figure, the condition data D22 has, for each lyric classification, a classification code D221 which is the identification code for the lyric classification, and an identification code (ID_i) for the feature data D222 and the component data D21. The feature data D222 represents the characteristics of the lyric content. The identification code (ID_i) for the component data D21 represents the component data D21 that should be used to display motion graphics containing lyrics belonging to the lyric classification corresponding to the classification code D221.
[0073] Feature data D222 may be data that represents the characteristics of the lyrics content by a keyword group containing one or more keywords. In this case, lyrics that contain one or more words belonging to the keyword group indicated by feature data D222 can be determined to belong to the lyric classification corresponding to this feature data D222.
[0074] Feature data D222 may be data that represents the characteristics of the lyrics content through the distribution of keywords. In this case, lyrics having a word distribution that matches the keyword distribution shown in feature data D222 can be determined to belong to the lyric classification corresponding to this feature data D222.
[0075] Conditional data D22 may be understood to contain information defining component data D21 to be used when lyrics are not present. That is, conditional data D22 may be configured to have an identification code for component data D21 to be used when lyrics are not present, associated with a classification code D221 for the lyrics classification "no lyrics".
[0076] Conditional data D22 may be understood to contain information defining component data D21 to be used when the lyrics content is unknown (in other words, when it is difficult to appropriately classify the lyrics content). That is, conditional data D22 may be configured to have an identification code for component data D21 to be used when the lyrics content is unknown, associated with a classification code D221 for the lyrics classification of "other".
[0077] [1-2. Processing] The series of processes performed by the processor 11 will be explained using the flowcharts in Figures 6 and 7.
[0078] When the processor 11 receives a playback command from the user through the input interface 50, it starts the playback display process shown in Figure 6. First, in S10, the processor 11 executes the playback plan creation process shown in Figure 7.
[0079] In the flowchart shown in Figure 7, first, at S100, the processor 11 obtains the music data D1 of the music to be played and displayed (hereinafter also referred to as the music to be played) from the server group 7.
[0080] Next, in S105, the processor 11 uses the metadata of the music data D1 to obtain the lyrics data DL, song structure data DS, and genre data DG corresponding to the music data D1 of the song to be played from the server group 7.
[0081] Next, in S110, the processor 11 refers to the acquired song genre data DG to determine the song genre of the song to be played. The processor 11 selects from among the multiple motion graphics data D2s the motion graphics data D2 corresponding to the song genre of the song to be played as the reference motion graphics data D2.
[0082] After selecting the motion graphics data D2, the processor 11 repeatedly executes the processes S120-S135 to refer to the condition data D22 of the referenced motion graphics data D2 for each section of the song identified from the acquired song structure data DS, and selects the component data D21 to be used for displaying the motion graphics from among the multiple component data D21 contained in the motion graphics data D2. Specifically, for each section, the component data D21 of the lyric classification corresponding to the lyrics content of that section is selected as the component data D21 to be used.
[0083] Next, in S115, the processor 11 obtains a set of change patterns from the storage 13 that defines one or more change patterns. The method for obtaining the set of change patterns is not particularly limited. For example, the processor 11 may obtain the set of change patterns based on the song genre determined in S110, based on the motion graphics data D2 selected in S110, or randomly.
[0084] The change pattern is a method of changing at least one of the character image and background image included in the motion graphics data D2 displayed on the display 52 based on a predetermined pattern. Specifically, the change pattern is a method of changing at least one of the motion of the character image defined by the text motion definition data D211 and the motion of the background image defined by the background motion definition data D212 in the motion graphics data D2.
[0085] For example, the change pattern may be a method of changing the font size, a method of changing the font color, or a method of moving the character image along a predetermined trajectory in the text motion definition data D211.
[0086] Alternatively, the change pattern may be a method of changing the color, type, movement, etc., of the background image in the background motion definition data D212. As an example, the change pattern shown in Figure 8 includes a method of significantly changing the font size of the text image, a method of changing the display position of the text image, and a method of changing the color and type of the background image, among the motion graphics data D2 displayed on the screen 52a of the display 52.
[0087] Next, in S120, the processor 11 selects one of several sections included in the song as the section to be processed. Then, in S125, the processor 11 extracts the lyrics portion of the section to be processed from the downloaded lyrics data.
[0088] Next, in S130, the processor 11 analyzes the extracted lyrics and selects the component data D21 corresponding to the lyrics classification of the section to be processed as the component data D21 to be used in the section to be processed.
[0089] For example, the processor 11 can analyze the extracted lyrics and generate feature data D222 that represents the characteristics of the lyrics. The process of analyzing the lyrics may involve extracting one or more keywords from the lyrics. The feature data D222 may be data representing a list or distribution of the extracted keywords.
[0090] The processor 11 compares the feature data D222 generated from the lyrics portion of the section to be processed with the feature data D222 of each lyric classification indicated by the condition data D22 of the section to be processed, and determines that the lyric classification of the feature data D222 that best matches the features is the lyric classification of the section to be processed. The processor 11 can select the component data D21 associated with that lyric classification as the component data D21 to be used.
[0091] For example, the processor 11 can determine that the lyric classification associated with feature data D222 having keywords extracted from the lyrics portion of the section to be processed is the lyric classification of the section to be processed. By assigning keywords so as not to overlap between lyric classifications, it is possible to identify one lyric classification from the keywords extracted from the lyrics portion of the section to be processed. Alternatively, the processor 11 can determine that the lyric classification associated with feature data D222 having a keyword distribution that matches the distribution of keywords extracted from the lyrics portion of the section to be processed is the lyric classification of the section to be processed.
[0092] If the lyric classification of the section to be processed is unknown, the processor 11 can select component data D21 for "unknown" lyric classification as the component data D21 to be used for that section. If the section to be processed does not have lyrics, the processor 11 can select component data D21 for "no lyrics" lyric classification.
[0093] Next, in S135, the processor 11 determines whether or not it has performed the processing in S125-S130 for all sections included in the music. If the processor 11 determines in S135 that it has not performed the processing in S125-S130 for all sections (S135: NO), it returns to S120, selects one of the unprocessed sections as the section to be processed, and performs the processing from S125 onwards.
[0094] On the other hand, if the processor 11 determines in S135 that it has performed the processing in S125-S130 for all sections (S135: YES), it proceeds to S140. In S140, the processor 11 creates a playback plan to display motion graphics of character images (lyrics) and background images on the display 52 at each playback position of the song, based on the lyrics data DL, song structure data DS, and component data D21 for each section selected in S130.
[0095] In other words, the processor 11 creates a playback plan to display motion graphics of character images (lyrics) according to the text motion definition data D211 contained in the component data D21 of the corresponding section, and motion graphics of background images according to the background motion definition data D212 contained in the same component data D21, on the display 52 in accordance with the progress of the music (changes in playback position).
[0096] At this time, the processor 11 creates a playback plan so that, according to the timecode TC included in the lyric data DL, the motion graphics of the corresponding lyric portion are displayed on the display 52 at each playback position of the song corresponding to each timecode TC.
[0097] Subsequently, the processor 11 completes the playback plan creation process shown in Figure 7. Next, returning to Figure 6, the processor 11 starts playing the music and displaying motion graphics in S20. The music is played at least towards the interior of the vehicle 2.
[0098] Specifically, the processor 11 controls the video signal transmitted to the display 52 so that motion graphics corresponding to the sound output from the speaker 51 are displayed on the display 52, based on the playback plan and the playback position of the music. In this way, the processor 11 displays motion graphics of lyrics, text images that match the content of the lyrics, and background images on the display 52 in accordance with the progress of the music being played towards the interior of the vehicle 2.
[0099] Motion graphics that are appropriate for the lyrics may be motion graphics that correspond to the emotions, situations, and environments associated with the lyrics. The user's impression of motion graphics of text images changes depending on the font type, size, movement, speed of position change, speed of shape change, etc. Font types include rounded fonts that give a gentle impression, angular fonts that give a harsh impression, feminine fonts, and masculine fonts.
[0100] The motion graphics for the background image could be, for example, motion graphics of a simple geometric shape. In this case, the user's impression of the motion graphics for the background image will vary depending on, for example, the shape, size, movement, position change speed, shape change speed, etc. From this perspective, the motion graphics data D2 can be created, for example, manually.
[0101] For example, at the point in which a song with sad lyrics is played, motion graphics of background images that resemble or evoke "tears" or "rain" may be displayed on display 52. For example, at the point in which a song with lyrics about autumn is played, motion graphics of background images that evoke falling leaves may be displayed. At the point in which a song with lyrics about winter is played, motion graphics of background images that evoke falling snow may be displayed. At the point in which a song with lyrics about anger or shock is played, motion graphics of background images that evoke lightning may be displayed. At the point in which a song with lyrics about love is played, motion graphics of heart-shaped background images may be displayed.
[0102] Motion graphics of text images using fonts that match the impression of the lyrics may be displayed on a background image motion graphic that matches the impression of the lyrics. As mentioned above, the speed and movement of the text images also give different impressions to the user depending on the differences. Examples of movement patterns include curved movements that make the text images appear to float, and linear movements that make the text images appear to rain down. Motion graphics are displayed in a way that matches the impression of the lyrics in terms of the speed and movement patterns of the text images. Motion graphics of text images may include the movement, rotation, and font size changes of one or more text images.
[0103] The processor 11 may control the display of the lyrics in the motion graphics in a layout based on the shape of the screen 52a of the display 52. The processor 11 may also control the display of a portion of the lyrics in the motion graphics so as to follow the boundary of the screen 52a of the display 52.
[0104] For example, in Figure 3, screen 52a is shaped to conform to the window glass of vehicle 2. The processor 11 may control the display to change the size and shape of the lyrics in the motion graphics to match the shape of screen 52a.
[0105] Alternatively, in Figure 4A, screen 52a is shaped to follow the contours of the vehicle 2's hood. The processor 11 may control the display of the lyrics in the motion graphics to change their size and shape to match the shape of screen 52a. In the example in Figure 4A, the processor 11 displays the lyrics in the motion graphics along the boundaries of screen 52a, which is shaped to match the contours and steps of the hood.
[0106] Alternatively, in Figure 4B, screen 52a is shaped to conform to the front fender, front door, rear door, and side sill of vehicle 2. The processor 11 may control the size and shape of the lyrics in the motion graphics to change and display them in accordance with the shape of screen 52a. In the example in Figure 4B, the processor 11 displays the lyrics in the motion graphics along the boundary of screen 52a, which is shaped to conform to the contours and steps of the front fender, front door, rear door, and side sill.
[0107] Next, in S30, the processor 11 performs detection processing to detect a predetermined state relating to the vehicle 2 based on the input signals from the sensor group 30. Specifically, the processor 11 determines that a predetermined state has been detected when the information represented by the input signals from the sensor group 30 is that of a predetermined state. The processor 11 determines that a predetermined state has not been detected when the information represented by the input signals from the sensor group 30 is not that of a predetermined state.
[0108] A predetermined state is one or more states that include at least one of the state of vehicle 2 and the state of the surroundings of vehicle 2. The predetermined states are set in advance to correspond to each change pattern. For example, predetermined states include states such as "vehicle 2 is driving through a forest," "the temperature inside or outside vehicle 2 is 30°C or higher," and "vehicle 2 has passed over a bump."
[0109] Figure 9 shows, as an example, a correspondence table that associates each sensor included in the sensor group 30 with a detected state, which is a predetermined state detected based on the input signal from each sensor, and a change pattern corresponding to each detected state. The processor 11 determines whether or not any of the states in the correspondence table has been detected based on the input signal from the sensor group 30.
[0110] If the processor 11 determines in S30 that a predetermined state has not been detected (S30: NO), it proceeds to S60. On the other hand, if the processor 11 determines in S30 that a predetermined state has been detected (S30: YES), it proceeds to S40 and selects a change pattern based on the detected state among the predetermined states. The storage 13 holds a correspondence table that associates the detected state with the change pattern. The processor 11 selects a change pattern according to this correspondence table.
[0111] (Pattern A) If it is detected that the vehicle 2 is driving through a forest based on the input signal from the camera 31, the processor 11 selects Pattern A as the change pattern, which changes the background image to a green color that represents a forest.
[0112] (Pattern B) If the input signal from camera 31 detects that vehicle 2 is driving through a city of buildings, the processor 11 selects Pattern B as the change pattern, which changes the background image to a gray color that represents a concrete jungle.
[0113] (Pattern C) If, based on the input signal from LiDAR 32, it is detected that the distance between vehicle 2 and another vehicle traveling in front of it is short (for example, 50 meters or less), the processor 11 selects Pattern C as the change pattern, which increases the font size of the character image representing the lyrics.
[0114] (Pattern D) If the processor 11 detects that the area around the vehicle 2 is dark (for example, 30 lux or less) based on the input signal from the illuminance sensor 33, it selects Pattern D as the change pattern, which changes the background image to a dark color that represents darkness.
[0115] (Pattern E) If, based on the input signal from the temperature sensor 34, it is detected that the temperature inside or outside the vehicle 2 is high (for example, 30°C or higher), the processor 11 selects Pattern E as the change pattern, which changes the lyrics text image and background image to warm colors that evoke the image of scorching heat.
[0116] (Pattern F) If the input signal from the vibration sensor 35 detects that the vehicle 2 has passed over a step, the processor 11 selects Pattern F as the change pattern, which shakes the lyrics character image vertically once.
[0117] (Pattern G) If the input signal from the GPS receiver 36 detects that the vehicle 2 is traveling in a coastal area, the processor 11 selects Pattern G as the change pattern, which changes the background image to a blue color that evokes the sea.
[0118] In the detection process of S30, the processor 11 can perform, for example, the processing necessary for detecting states related to the selection of patterns A-G described above. If multiple predetermined states are detected in S30, the processor 11 selects one from multiple change patterns corresponding to each state. However, multiple change patterns that do not conflict with each other (for example, pattern A and pattern C) may be selected.
[0119] The method for selecting a change pattern is not particularly limited. For example, the processor 11 may select a change pattern randomly, or it may select one based on a predetermined priority for each change pattern, each state, or each sensor. As an example, if pattern A in Figure 9 is assigned a higher priority than pattern E, and states corresponding to both pattern A and pattern E are detected within a predetermined period (e.g., within 1 second), the processor 11 may select the change pattern corresponding to pattern A, which has the higher priority.
[0120] Next, in S50, the processor 11 changes the currently displayed motion graphics based on the selected change pattern. As a result of this process, at least one of the character image and background image in the motion graphics displayed on the display 52 changes based on the change pattern. After that, the processor 11 proceeds to S60.
[0121] The timing for canceling a change based on a change pattern is not particularly limited; for example, it may be after a certain period of time has elapsed, at the end of the song, or when a change based on another change pattern occurs. A change based on a change pattern may be overwritten by another change pattern.
[0122] Next, in S60, the processor 11 determines whether or not the playback of the music has finished. For example, the processor 11 can determine this by whether or not the playback position of the music has reached the end of the music.
[0123] If processor 11 determines in S60 that music playback has not finished (S60: NO), it returns to S30. On the other hand, if processor 11 determines in S60 that music playback has finished (S60: YES), it terminates the process shown in Figure 6.
[0124] In this way, the processor 11 displays motion graphics, including character images and background images, on the display 52 in accordance with the progress of the music (i.e., based on the playback position of the music) while the music is playing. The characters displayed in the motion graphics are lyrics and other characters related to the music. Furthermore, while the music is playing, if a predetermined state is detected based on input signals from the sensor group 30, the processor 11 changes at least one of the character images and background images included in the motion graphics displayed on the display 52 based on the detected state.
[0125] [1-3. Effects] According to the embodiments described above, the following actions and effects can be obtained. (1a) The processor 11 creates a playback plan by switching the component data D21 to be used for each section of the music identified from the song structure data DS, according to the lyrics content of that section. Based on this playback plan, the processor 11 displays motion graphics on the display 52 based on the changes in the structure and lyrics content of the music being played. The display 52 is configured to display motion graphics toward at least one of the interior and exterior of the vehicle 2.
[0126] Through this process, the control device 10 can display motion graphics on the display 52 that reflect the progression of the music, thereby providing greater enjoyment to users inside or around the vehicle 2.
[0127] (1b) The screen 52a may be shaped to conform to the shape of the parts of the vehicle 2. The processor 11 may control the display of the lyrics in the motion graphics in a layout based on the shape of the screen 52a of the display 52. The processor 11 may control the display of a portion of the lyrics in the motion graphics to conform to the boundary of the screen 52a of the display 52.
[0128] With this processing, the lyrics in the motion graphics are displayed on the display 52 in a layout that conforms to the shape of the vehicle 2's components. This allows for greater enjoyment for users inside or around the vehicle 2.
[0129] (1c) During music playback, if a predetermined state is detected based on input signals from the sensor group 30, the processor 11 changes at least one of the character image and background image included in the motion graphics displayed on the display 52 based on the detected state.
[0130] Through this process, the control device 10 can display motion graphics on the display 52 that reflect the state of the vehicle 2, thereby providing greater enjoyment to users inside or around the vehicle 2.
[0131] (1d) In S40, the processor 11 selects a change pattern based on the detected state. The processor 11 may select one change pattern from a plurality of change patterns. Through this process, the motion graphics displayed on the display 52 change based on the detected state. As a result, motion graphics that reflect the state of the vehicle 2 can be displayed on the display 52, which can further entertain the user.
[0132] (1e) If the processor 11 has detected multiple predetermined states in S30, it selects one of multiple change patterns corresponding to each of the multiple predetermined states. The processor 11 may make the selection based on a predetermined priority for each change pattern, each state, or each sensor. This process makes it possible to select a more appropriate change pattern.
[0133] [1-4. Correspondence between terms] In the above embodiment, speaker 51 corresponds to an example of an acoustic device, display 52 corresponds to an example of a display device, sensor group 30 corresponds to an example of at least one sensor, and character image and background image in motion graphics data D2 correspond to an example of lyrics and background included in motion graphics. The state corresponding to pattern A in Figure 9 corresponds to an example of a first state, the state corresponding to pattern E corresponds to an example of a second state, camera 31 corresponds to an example of a first sensor, and temperature sensor 34 corresponds to an example of a second sensor.
[0134] The process in S105 executed by the processor 11 is an example of a process executed as an acquisition unit, the process in S20 is an example of a process executed as a display unit, and the process in S50 is an example of a process executed as a change unit.
[0135] [2. Other Embodiments] Although embodiments of the present disclosure have been described above, it goes without saying that the present disclosure is not limited to the embodiments described above and can take various forms.
[0136] (2a) In the above embodiment, some or all of the various information held in the storage 13 may be held in a device other than the storage 13. The control device 10 may acquire the various information from a device other than the storage 13 instead of acquiring it from the storage 13. For example, the control device 10 may acquire the various information from the memory 12, or from outside the control device 10 through the communication interface 15.
[0137] (2b) In the above embodiment, the sensor group 30 includes a camera 31, a LiDAR 32, an illuminance sensor 33, a temperature sensor 34, a vibration sensor 35, and a GPS receiver 36. However, the sensors included in the sensor group 30 are not limited to these. The sensor group 30 does not have to include all of the sensors in the above embodiment. The sensor group 30 may include sensors other than those in the above embodiment. For example, the sensor group 30 may include a speed sensor, an acceleration sensor, a steering angle sensor, a turn signal, a seating sensor, and the like.
[0138] The change pattern is not limited to the example shown in Figure 9, and may also be a method in which at least one of the character image and background image included in the motion graphics data D2 displayed on the display 52 is changed to any manner, corresponding to a predetermined state detected based on input signals from any sensor included in the sensor group 30.
[0139] (2c) In the above embodiment, the control device 10 acquires various data from the server group 7 through the communication interface 15. The various data include song data D1, lyrics data DL, song structure data DS, and genre data DG. However, the control device 10 may acquire some or all of the various data from sources other than the server group 7.
[0140] For example, the control device 10 may acquire various data that is pre-stored in the storage 13. The control device 10 may also acquire various data from a communication device such as a smartphone or tablet terminal owned by a user riding in the vehicle 2 via the communication interface 15.
[0141] For example, the user may transmit a playback signal for playing music data D1 to the control device 10 using wired or wireless communication from their communication device, causing the control device 10 to perform playback and display processing of the music. Through this process, the user can operate their communication device to play the music and, through the control device 10, view the audio of the music output from the speaker 51 and motion graphics displaying the lyrics of the music on the display 52.
[0142] (2d) In the above embodiment, the motion graphics displayed on the display 52 include lyrics and a background. However, the motion graphics displayed on the display 52 do not necessarily have to include a background.
[0143] (2e) In the above embodiment, the music is played at least towards the interior of the vehicle 2. However, the music may be played towards the outside of the vehicle 2 instead of, or in addition to, the interior of the vehicle 2.
[0144] (2f) In the process shown in Figure 6, the processor 11 starts playing the music in S20 and starts displaying the motion graphics, and then repeats the processes from S30 to S60. However, the processor 11 does not have to perform the processes from S30 to S50.
[0145] (2g) Multiple functions of one component in the above embodiment may be realized by multiple components, or one function of one component may be realized by multiple components. Multiple functions of multiple components may be realized by one component, or one function realized by multiple components may be realized by one component. Some of the configurations of the above embodiment may be omitted. At least some of the configurations of the above embodiment may be added to or replaced with the configurations of other above embodiments.
[0146] (2h) The present disclosure can be implemented in various forms other than the control device described above. For example, it can be implemented in the form of a system in which the control device is a component, a computer program for causing a computer to function as the control device, a non-transitional physical recording medium such as a semiconductor memory on which the computer program is recorded, a control method, and so on.
[0147] 1...Music playback system, 2...Vehicle, 10...Control device, 11...Processor, 12...Memory, 13...Storage, 14...User interface, 15...Communication interface, 30...Sensor group, 50...Input interface, 51...Speaker, 52...Display, D1...Music data, D2...Motion graphics data, DL...Lyrics data.
Claims
1. A control device comprising: at least one processor; and at least one memory for storing instructions for causing the at least one processor to perform processing, wherein the processing includes: acquiring lyric data of a song to be played by an audio device; and, based on the lyric data, causing motion graphics including the lyrics of the song to be displayed on a display device mounted on the vehicle in accordance with the progress of the song to be played by the audio device toward the interior of the vehicle, wherein the display device is configured to display the motion graphics toward at least one of the interior and exterior of the vehicle.
2. A control device according to claim 1, wherein the display device is configured to display the motion graphics toward the outside of the vehicle.
3. A control device according to claim 1 or claim 2, wherein the display device comprises at least one screen, the at least one screen is shaped in accordance with the shape of the vehicle's parts, and the lyrics in the motion graphics are displayed on the at least one screen in a layout based on the shape of the at least one screen.
4. A control device according to claim 3, wherein at least a portion of the lyrics in the motion graphics is displayed on the at least one screen along the boundary of the at least one screen.
5. A control device according to any one of claims 1 to 4, wherein the process includes changing the lyrics included in the motion graphics displayed on the display device, on the condition that at least one of one of one predetermined states relating to the vehicle is detected, the one or more predetermined states being one or more states including at least one of the state of the vehicle and the state of the area surrounding the vehicle, and detected based on an input signal from at least one sensor mounted on the vehicle.
6. A control device according to any one of claims 1 to 4, wherein the motion graphics further include a background, and the process includes changing at least one of the lyrics and the background included in the motion graphics displayed on the display device, provided that at least one of one of one of a predetermined state relating to the vehicle is detected, the one or more predetermined states being one or more states including at least one of the state of the vehicle and the state of the area surrounding the vehicle, and detected based on an input signal from at least one sensor mounted on the vehicle.
7. A control device according to claim 6, wherein the act of changing includes selecting one change pattern from a plurality of change patterns relating to the motion graphics based on the detected at least one state, and changing at least one of the lyrics and the background based on the one change pattern.
8. A control device according to claim 7, wherein the change includes, if both the first state and the second state are detected within a predetermined period, selecting one change pattern from a plurality of change patterns based on a priority set for each of the first state and the second state, the at least one sensor includes a first sensor and a second sensor, the first state is a state detected based on an input signal from the first sensor among one or more predetermined states, and the second state is a state detected based on an input signal from the second sensor among one or more predetermined states.
9. A control device according to any one of claims 6 to 8, wherein the changing includes, on the condition that a specific state is detected, changing at least one of the lyrics and the background included in the motion graphics displayed on the display device based on the specific state, wherein the at least one sensor includes a camera configured to photograph the area around the vehicle, and the specific state is one or more predetermined states of the area around the vehicle detected based on an image or video represented by an input signal from the camera.
10. A control method performed by a computer, comprising: acquiring lyric data of a song to be played by an audio device; and, based on the lyric data, causing motion graphics including the lyrics of the song to be displayed on a display device mounted on the vehicle in accordance with the progress of the song to be played by the audio device toward the interior of the vehicle, wherein the display device is configured to display the motion graphics toward at least one of the interior and exterior of the vehicle.
11. A computer program product comprising a computer program that includes instructions for causing a computer to execute the control method described in claim 10.
12. A non-temporary computer-readable storage medium storing instructions for causing a computer to execute the control method described in claim 10.
13. A computer program that includes instructions for causing a computer to execute the control method described in claim 10.
