Motion primitive framework for multimedia integration in vehicle suspensions

Abstract

Aspects disclosed herein relate to methods for operating vehicle systems. In some embodiments, the method comprises playing a video including a motion track inside the vehicle, providing a command signal based on information in the motion track to a microprocessor-based controller of an actuator of the active suspension system, and, in response to the command signal, moving a portion of the vehicle with the actuator synchronously with the series of video images. Aspects disclosed herein also relate to methods of adding a motion track to a video recording.

Description

L0710.70117WQ00MOTION PRIMITIVE FRAMEWORK FOR MULTIMEDIA INTEGRATION IN VEHICLE SUSPENSIONSCROSS REFERENCE TO RELATED APPLICATIONS

[0001] This application claims priority under 35 U.S.C. § 119(e) of US Provisional Application Serial No. 63 / 753,912, filed February 4, 2025, the disclosure of which is incorporated by reference in its entirety.BACKGROUND

[0002] Typically, vehicle suspension systems are designed to enhance ride comfort and vehicle handling by mitigating road-induced vibrations. While advancements in active suspension systems have enabled real-time adjustments for improved handling and stability, their applications remain largely limited to road surface compensation.SUMMARY

[0003] According to aspects of the disclosure, there is provided a method for operating a vehicle that is equipped with an active suspension system and a video display unit, where the method includes playing, in the vehicle, a video recording with a motion track, where the motion track includes information characterizing one or more preselected motion primitives associated with one or more video features; producing a video signal; providing the video signal to the video display unit to produce a series of video images; producing a command signal based on the information; providing the command signal to a microprocessor-based controller of an actuator of the active suspension system; and in response to the command signal, moving a portion of the vehicle with the actuator synchronously with the series of video images. In some embodiments, the video recording is streamed from a streaming service or was previously stored on-board the vehicle. In some embodiments the portion of the vehicle is a vehicle body that includes a compartment for occupants. In some embodiments the vehicle is stopped or parked. In some embodiments the video recording also includes a soundtrack, where the motion track also includes motion primitives associated with one or more audio features contained in the soundtrack.- 1 - #14861420vlL0710.70117WQ00

[0004] According to aspects of the disclosure, there is provided a method for adding a motion track to a video recording for viewing in a vehicle with an active suspension system, where the method includes replaying the video recording; selecting at least one feature of the video recording; matching at least one motion primitive from a motion primitive library with the selected feature, where the at least one motion primitive is configured to produce a motion that is consistent with the at least one feature of the video recording; and incorporating the motion primitive in the motion track, such that the motion track is configured to operate the active suspension system of the vehicle to create an immersive environment for viewing the video in the vehicle, where a viewer receives tactile stimuli that are consistent with the visual and / or audio stimuli provided by observing the video recording. In some embodiments the selecting of the feature is performed automatically and / or manually. In some embodiments the matching is performed automatically and or manually. In some embodiments a user interface is provided for adjusting one or more parameters associated with the motion track while observing the video in the vehicle. In some embodiments the motion primitive library is developed automatically and / or manually.

[0005] It should be appreciated that the foregoing concepts, and additional concepts discussed below, may be arranged in any suitable combination, as the present disclosure is not limited in this respect. Further, other advantages and novel features of the present disclosure will become apparent from the following detailed description of various nonlimiting embodiments when considered in conjunction with the accompanying figures.BRIEF DESCRIPTION OF FIGURES

[0006] The accompanying drawings are not intended to be drawn to scale. In the drawings, each identical or nearly identical component that is illustrated in various figures may be represented by a like numeral. For purposes of clarity, not every component may be labeled in every drawing. In the drawings:

[0007] Fig. 1 illustrates an architecture of the motion primitive library for vehicle active suspension systems;

[0008] Fig. 2 illustrates an example of creating a motion track using the motion library;

[0009] Fig. 3A illustrates a scenario where video and audio linked to events occurring outside of a vehicle are processed and associated with motion primitives in real-time;- 2 - #14861420vlL0710.70117WQ00

[0010] Fig. 3B shows a block diagram of the signal flow in the scenario illustrated in Fig. 3A.

[0011] Fig. 4 illustrates the detection of motion events from video and audio from inside the vehicle;

[0012] Fig. 5 illustrates how a user in the vehicle may adjust the motion;

[0013] Fig. 6 shows a block diagram for post-processing and customization for an entry from the motion libraryDETAILED DESCRIPTION

[0014] This disclosure describes systems and methods for implementing a framework for associating video, sound effects, and / or music with a library of motion primitives that are configured to activate a vehicular active suspension system. In some embodiments, the library architecture enables the creation, storage, and playback of a motion track associated with a prerecorded video and / or audio recording that can be used to generate motion, in a manner that enhances the experience of watching the video (e.g. movie or transition clips of intro or transitional cutscenes of video game) and, listening to the associated sound effects, and / or listening to music, while in a vehicle equipped with active suspension. Such a system allows for pre-generated motion tracks, created manually or automatically, that correspond to previously recorded video and / or audio.

[0015] Alternatively or additionally, in some embodiments, the system may also be used to generate a motion track by processing, in real-time, video and / or audio signals with e.g. a camera and a microphone (e.g. inside or outside of a vehicle), and mapping the recorded video and audio signals, to corresponding motion primitives. In such a system, the motion track may be implemented immediately to create motion in the vehicle body with an active suspension system or stored for later replay.

[0016] In some embodiments, users may be offered access to tuning parameters, including options to switch between different motion styles and adjust parameters such as force magnitude or displacement, intensity, or rate of change of the magnitude of the force or displacement. The system may be used to enhance the in-vehicle experience by implementing active suspension systems to deliver dynamic and immersive motion effects in a wide range of scenarios. As used herein, the phrase “an immersive environment for viewing a video” refers to an environment where the viewer receives tactile stimuli that are consistent,- 3 - #14861420vlappropriate, and synchronized with the visual and audio stimuli contained in the video. This creates a cohesive and engaging experience allowing the viewer to feel more connected to the content of the video.

[0017] Fig. 1 shows an example of a architecture of a library of motion primitives. Library 101 is a collection of predefined motion primitives. For example, 102 is an entry in the library. Each entry may contain two sets of information. For example, entry 103 may contain the basic shape of the motion. An entry in the library may include, e.g. four signals, representing the command to be sent to each suspension microprocessor-based controller of an active suspension actuator, 115-118 to execute. The command may be e.g. a force command, a position command, a velocity command, an acceleration command, or other executable commands for the actuators. Alternatively or additionally to providing the commands as signals, commands may be comprised of functions, such as 111-114, that are configured to generate the appropriate command signals. Each entry from the motion library may be associated with metadata (104 and 106), which may include tunable parameters. For example, each force range may have a parameter of magnitude, which may be determined by the physical range of the actuators for a particular vehicle and may also be determined by the preferences of users.

[0018] Fig. 2 illustrates a user interface for editing the metadata associated with various entries in the motion library. It may be used or reused to adjust a motion track. In Fig. 2, 201 is an area of control widgets, such as push buttons and dropdown menus. In this area, users may select a video or audio file, or content from a streaming platform, such as YouTube. There may be a push button; by clicking on it, an automated motion track generation algorithm may be executed in the background to select appropriate motion primitives and place them on motion track timeline 211. Area 202 is a video / audio player. Once a media file is loaded, the images, if available, may show in 202, and the soundtracks from the audio, if available, may show in 210. The playing progress of the media is indicated and may be controlled by moving the indicator 204, or the vertical bar 205. Note that 204 and 205 may be linked, and they indicate the same time step within the media. In 211, users may edit the motion track manually. The motion track may have overlapped with motion primitives. For example, motion primitive 206 and motion primitive 207 are overlapped in time. There are multiple ways to handle overlaps when playing the motion track in the suspension actuators. For example, if the motion primitives define force command. They in the overlapped time - 4 - #14861420vlL0710.70117WQ00interval, the forces can be added together. If the motion primitive defines references such as actuator positions, then one motion primitive may override others overlapping with it. Once a motion primitive placed on the motion track is selected by the user (for example, 209 is selected), the tunable parameters associated with the motion will appear in area 208 for user to edit. In particular, the user can modify the start and / or end time of the motion primitive on the timeline. For convenience, the user may use the control button 212 in the video / audio player to play and pause the media content. While the media content is being played, the indicator 204 and the vertical bar 205 may move automatically. When 205 reaches the next motion primitive, on the motion track, the animation box 203 may illustrate how the car would move in response to this motion primitive, until 205 reaches the end of the motion primitive.

[0019] Fig. 3A illustrates a scenario in which a motion primitive library may be used in realtime. Vehicle 305 is parked or stopped and one or more persons 301 may be proximate to the car. Camera 302 and / or microphone 303 may be used to capture the surrounding images and / or audio, respectively. Sensors such as camera 302 and / or microphone 303 may be incorporated in the vehicle or alternatively be provided with a portable device, such as a laptop or a smartphone. In the embodiment illustrated in Fig. 3A, signals from these sensors are received by computing device 306 in the vehicle via cables, or wireless connections (WiFi, Bluetooth, etc.). The computing device 306 may communicate directly with the microprocessor based suspension controllers to send control commands to and / or receive sensor data from the actuators (for example, 304 the front left actuator, and 307 the rear left actuator). A person 301 may interact with the vehicle by changing body pose, such as raising the arms or bow. At the same time, the person may also interact with the vehicle by talking or making certain sounds that is captured by the microphone 303. As a result, the computing device 306 working in conjunction with the vehicle’s suspension system may use motion primitives from a library to induce a motion in a portion of the vehicle (e.g. vehicle body, vehicle wheel) in response to gestures and / or voice commands from person 301, in real-time. Alternatively or additionally, the computing device 306 may work with other components in the vehicle to provide other responses (e.g. flash lights, honk the horn, etc.).

[0020] Fig. 3B shows a block diagram of the signal flow in the scenario illustrated in Fig. 3A. The output of the camera, i.e. video 308, and the output of the microphone, i.e. audio signal 309. These outputs are processed together in 310. However, it should be noted that - 5 - #14861420vlL0710.70117WQ00either audio or video inputs may be used, as the disclosure is not so limited. The output of 310 is then provided to a detection module 311, which can detect the features or attributes present in the captured video and / or the audio. For example, the audio signal converted to images of a spectrogram of the audio signal during a predetermined time window. The spectrogram is then provided to a pretrained deep learning model, such as YamNet, to identify audio features. The camera image may be resized and processed by pretrained human pose estimation networks, such as PoseNet, OpenPose, or Mask R-CNN. These networks may be used to extract body landmarks, including e.g. the nose, eyes, hands, and fingers, from the images. The detector module (311) is designed to output a vector of scores corresponding to predefined features. The detected attributes are then passed to a motion selection module 312. This module may contain a classifier that finds the most suitable motion primitive from the motion library based on the features detected. The selected motion primitive is then sent to a postprocessing module 314 to assign the appropriate values to the appropriate tunable parameters. For example, actuators in a given vehicle may have different ranges of force that they can produce. Hence, the magnitude of the motion primitive may be modified accordingly. The motion primitive may also need to be filtered to avoid making the vehicle move mostly at its body resonant frequency (for example, a notch filter may be applied to attenuate the 1Hz content of the motion primitive, if the car’s body resonant frequency happens to be at 1Hz). After postprocessing 314, the control command 315 may be sent to the actuators to be implemented.

[0021] Fig. 4 illustrates a vehicle 401 and a person 405, inside the vehicle, interacting with the vehicle with voice commands, or by making facial expressions, hand poses or gestures or changing body pose. One or more cameras 407 and / or one or more microphones 406 may be used to record these interactions.

[0022] Fig. 5 illustrates various options of in-vehicle controls of the motions while the motions are being played. Note that this drawing is a view from inside of the vehicle facing windshield 501. The steering wheel is shown as 503. There may be a digital screen 502, and an area 506 with physical controls, such as knobs and buttons. By using interface 502, the user may change the magnitude of the force by changing the bar 502 to reach maximum magnitude of the motion track or 0 magnitude (i.e., no motion). The user may also select from a few assorted styles of motion tracks to meet the different preferences of different users. If 502 is a touch screen, the user may directly drag the magnitude bar 504 and select - 6 - #14861420vldifferent motion tracks by pressing the buttons on the screen (e.g., 507). Alternatively, the user may use a physical controller, such as a knob 505 to tune down or tune up the magnitude of the motion. Note that 505 may be the sound volume control knob and adjust the magnitude of the motion and the volume of the sound simultaneously.

[0023] Fig. 6 illustrates the post-processing of a selected Motion Library Entry 601 by, for example, a user. The tunable parameters may be assigned appropriate values in block 602. This process may take into account the physical limitations of the vehicle, and the preference of the editor / users. They may be generated automatically and / or adjusted manually. In this embodiment, once the tunable parameters are determined, the motion primitive entry becomes an actual executable command. However, before sending it to the controller, a filtering process 604 may be implemented. For example, a notch filter may be applied to attenuate the 1Hz content of the motion primitive, if the car’s body resonant frequency happens to be approximately at or near 1Hz resulting in a smoother vehicle response. The filtered control command 605 may then be sent to the suspension actuators 606. As illustrated in Fig. 5, the user may have access to control the magnitude of the motion. For example, the user may want to reduce the magnitude of the motion by tuning a knob. This feedback signal 607 is sent back to modify the filtered commands.

[0024] While the present teachings have been described in conjunction with various embodiments and examples, it is not intended that the present teachings be limited to such embodiments or examples. On the contrary, the present teachings encompass various alternatives, modifications, and equivalents, as will be appreciated by those of skill in the art. Accordingly, the foregoing description and drawings are by way of example only.

[0025] The above-described embodiments of the technology described herein may be implemented in any of numerous ways. For example, the embodiments may be implemented using hardware, software or a combination thereof. When implemented in software, the software code may be executed on any suitable processor or collection of processors, whether provided in a single computing device or distributed among multiple computing devices. Such processors may be implemented as integrated circuits, with one or more processors in an integrated circuit component, including commercially available integrated circuit components known in the art by names such as CPU chips, GPU chips, microprocessor, microcontroller, or co-processor. Alternatively, a processor may be implemented in custom circuitry, such as an ASIC, or semicustom circuitry resulting from configuring a- 7 - #14861420vlprogrammable logic device. As yet a further alternative, a processor may be a portion of a larger circuit or semiconductor device, whether commercially available, semi-custom or custom. As a specific example, some commercially available microprocessors have multiple cores such that one or a subset of those cores may constitute a processor. Though, a processor may be implemented using circuitry in any suitable format. It should also be understood that any reference to a controller in the current disclosure may be understood to reference the use of one or more processors configured to implement the one or more methods disclosed herein.

[0026] Further, it should be appreciated that a computing device including one or more processors may be embodied in any of a number of forms, such as a rack-mounted computer, a desktop computer, a laptop computer, or a tablet computer. Additionally, a computing device may be embedded in a device not generally regarded as a computing device but with suitable processing capabilities, including a Personal Digital Assistant (PDA), a smart phone, tablet, or any other suitable portable or fixed electronic device.

[0027] Also, a computing device may have one or more input and output devices. These devices may be used, among other things, to present a user interface. Examples of output devices that may be used to provide a user interface include display screens for visual presentation of output and speakers or other sound generating devices for audible presentation of output. Examples of input devices that may be used for a user interface include keyboards, individual buttons, and pointing devices, such as mice, touch pads, and digitizing tablets. As another example, a computing device may receive input information through speech recognition or in other audible format.

[0028] Such computing devices may be interconnected by one or more networks in any suitable form, including as a local area network or a wide area network, such as an enterprise network or the Internet. Such networks may be based on any suitable technology and may operate according to any suitable protocol and may include wireless networks, wired networks or fiber optic networks. Also, the various methods or processes outlined herein may be coded as software that is executable on one or more processors that employ any one of a variety of operating systems or platforms. These methods may be embodied as processor executable instructions stored on associated non-transitory computer readable media that when executed by the one or more processors perform any of the methods disclosed herein. Additionally, such software may be written using any of a number of suitable programming - 8 - #14861420vllanguages and / or programming or scripting tools, and also may be compiled as executable machine language code or intermediate code that is executed on a framework or virtual machine.

[0029] In this respect, the embodiments described herein may be embodied as a computer readable storage medium (or multiple computer readable media) (e.g., a computer memory, one or more floppy discs, compact discs (CD), optical discs, digital video disks (DVD), magnetic tapes, flash memories, RAM, ROM, EEPROM, circuit configurations in Field Programmable Gate Arrays or other semiconductor devices, or other tangible computer storage medium) encoded with one or more programs that, when executed on one or more computers or other processors, perform methods that implement the various embodiments discussed above. As is apparent from the foregoing examples, a computer readable storage medium may retain information for a sufficient time to provide computer-executable instructions in a non-transitory form. Such a computer readable storage medium or media may be transportable, such that the program or programs stored thereon may be loaded onto one or more different computing devices or other processors to implement various aspects of the present disclosure as discussed above. As used herein, the term "computer-readable storage medium" encompasses only a non- transitory computer-readable medium that may be considered to be a manufacture (i.e., article of manufacture) or a machine. Alternatively or additionally, the disclosure may be embodied as a computer readable medium other than a computer-readable storage medium, such as a propagating signal.

[0030] The terms “program” or “software” are used herein in a generic sense to refer to any type of computer code or set of computer-executable instructions that may be employed to program a computing device or other processor to implement various aspects of the present disclosure as discussed above. Additionally, it should be appreciated that according to one aspect of this embodiment, one or more computer programs that when executed perform methods of the present disclosure need not reside on a single computing device or processor, but may be distributed in a modular fashion amongst a number of different computers or processors to implement various aspects of the present disclosure.

[0031] Computer-executable instructions may be in many forms, such as program modules, executed by one or more computers or other devices. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform particular tasks or- 9 - #14861420vlL0710.70117WQ00implement particular abstract data types. Typically, the functionality of the program modules may be combined or distributed as desired in various embodiments.

[0032] The embodiments described herein may be embodied as a method, of which an example has been provided. The acts performed as part of the method may be ordered in any suitable way. Accordingly, embodiments may be constructed in which acts are performed in an order different than illustrated, which may include performing some acts simultaneously, even though shown as sequential acts in illustrative embodiments.

[0033] Further, some actions are described as taken by a “user.” It should be appreciated that a “user” need not be a single individual, and that in some embodiments, actions attributable to a “user” may be performed by a team of individuals and / or an individual in combination with computer-assisted tools or other mechanisms.

[0034] While the present teachings have been described in conjunction with various embodiments and examples, it is not intended that the present teachings be limited to such embodiments or examples. On the contrary, the present teachings encompass various alternatives, modifications, and equivalents, as will be appreciated by those of skill in the art. Accordingly, the foregoing description and drawings are by way of example only.- 10 - #14861420vl

Claims

CLAIMS1. A method of operating a vehicle that is equipped with an active suspension system and a video display unit, the method comprising:in the vehicle, playing a video recording that includes a motion track, wherein the motion track includes information characterizing one or more preselected motion primitives associated with one or more video features;producing a video signal;providing the video signal to the video display unit to produce a series of video images;producing a command signal based on the information;providing the command signal to a microprocessor-based controller of an actuator of the active suspension system; andin response to the command signal, moving a portion of the vehicle with the actuator synchronously with the series of video images.

2. The method of claim 1, wherein the video recording is streamed from a streaming service.

3. The method of claim 1, wherein the video recording was previously stored on-board the vehicle.

4. A method as in any one of claims 1-3, wherein the portion of the vehicle is a vehicle body that includes a compartment for occupants.

5. A method as in any one of claims 1-4, wherein the vehicle is stopped or parked.

6. A method as in any one of claims 1-5 wherein the video recording also includes a soundtrack, and wherein the motion track also includes motion primitives associated with one or more audio features contained in the soundtrack.- 11 - #14861420vl7. A method for adding a motion track to a video recording for viewing in a vehicle with an active suspension system, the method comprising:replaying the video recording;selecting at least one feature of the video recording;matching at least one motion primitive from a motion primitive library with the selected feature, wherein the at least one motion primitive is configured to produce a motion that is consistent with the at least one feature of the video recording; and incorporating the motion primitive in the motion track, wherein the motion track is configured to operate the active suspension system of the vehicle to create an immersive environment for viewing the video in the vehicle, wherein a viewer receives tactile stimuli that are consistent with visual and / or audio stimuli provided when observing the video recording.

8. The method of claim 7, wherein selecting the feature is performed automatically.

9. The method of claim 7, wherein selecting the feature is at least partially performed manually.

10. The method of claim 7, wherein the matching is performed automatically.

11. The method of claim 7, wherein the matching is at least partially performed manually.

12. The method of claim 7, further comprising a user interface for adjusting one or more parameters associated with the motion track while observing the video in the vehicle.

13. The method of claim 7, wherein the motion primitive library is developed automatically.

14. The method of claim 7, wherein the motion primitive library is at least partially developed manually.- 12 - #14861420vl

Images