In-Cabin Object Querying Using Multi-Modal Caption History
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Traditional single-modal models struggle to effectively manage and monitor in-cabin objects in vehicles due to their limitations in processing diverse data types, leading to inefficiencies and safety concerns.
Innovation Solution
Implementing a large multi-modal model (LMM) that integrates visual, audio, and textual data to detect, recognize, and describe in-cabin objects, enabling continuous object tracking and personalized responses through continuous learning.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If traditional single-modal models are used to process in-cabin data, then the system complexity is low, but the ability to effectively manage and monitor diverse in-cabin objects is insufficient
Solution Approach 1:
The patent combines multiple single-modal models (visual, audio, textual) into a unified large multi-modal model that can process diverse data types simultaneously. This merging approach enables the system to effectively manage and monitor various in-cabin objects by integrating different data modalities, resolving the contradiction between processing versatility and system complexity.
Solution Approach 2:
The large multi-modal model is designed to perform multiple functions across different data modalities (visual, audio, textual) within a single system. This universal model can detect, recognize, and respond to various types of in-cabin objects and situations, providing adaptability without requiring separate specialized systems for each data type.
2Measurement precision
If continuous video capture and object tracking are implemented, then object monitoring accuracy is improved, but energy consumption increases
Solution Approach 1:
The system performs preliminary actions by continuously capturing video frames and generating captions for all detected objects, storing this information in advance. When a user queries about a specific object, the system can quickly retrieve pre-processed information rather than analyzing current video data from scratch, thus improving monitoring accuracy while reducing real-time energy consumption.
Solution Approach 2:
The system applies partial action by focusing computational resources on generating detailed captions and tracking only for objects that are relevant to user queries or safety concerns. Not all objects receive equal levels of processing intensity, allowing the system to maintain good monitoring accuracy while conserving energy on less critical objects.
3Adaptability or versatility
If the system infers object locations from historical captions, then it can provide information about objects outside camera view, but inference accuracy may decrease
Solution Approach 1:
The system uses historical captions as an intermediary data source to infer current object locations when objects are outside direct camera view. The captions serve as a mediator between past visual data and current query responses, enabling the system to provide information about objects not currently visible while managing the inherent accuracy trade-off through probabilistic inference.
Data Source
AI summary
System and method for responding to queries about objects in a cabin of a vehicle. The system detects a trigger that causes an in-cabin camera to capture video of the cabin, and the system generates a history of captions for at least selected frames of the video by a large multi-modal model (LMM). The system converts a spoken query received by a microphone to a text-based prompt and generates, by the LMM, a response to the prompt based on the history of captions. The response is converted to speech that is output to a speaker.