Adaptive 360° Wallpapers Using Multi-Sensory Prompt Refinement
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current generative AI models in virtual reality and augmented reality do not incorporate device sensors as inputs, leading to static and non-adaptive digital experiences that fail to dynamically adjust to user emotions and environmental contexts, resulting in decreased user engagement.
Innovation Solution
A system that generates real-time adaptive 360° wallpapers using multi-sensory data by integrating an encoder layer for multimodal input, a machine learning layer for feature generation, and an on-device LLM for prompt refinement, leveraging sensors to create immersive and personalized digital environments.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If generative AI models do not incorporate device sensors as inputs, then the system complexity remains low, but the adaptability and personalization of digital experiences deteriorate
Solution Approach 1:
The system is divided into distinct functional modules: sensor data acquisition module, encoder layer for multimodal input processing, machine learning layer for feature generation, and on-device LLM for prompt refinement. This segmentation allows each component to handle specific tasks independently, managing overall system complexity while achieving high adaptability through coordinated operation of specialized subsystems.
Solution Approach 2:
The patent implements a nested architecture where the encoder layer is embedded within the machine learning layer, which in turn is integrated with the on-device LLM. This nested structure allows hierarchical processing of sensor data, with each layer building upon and refining the output of the previous layer, thereby managing complexity through organized nesting while maintaining high adaptability.
2Adaptability or versatility
If generative AI models use static inputs without sensor data, then the processing speed remains high, but the personalization and user engagement deteriorate
Solution Approach 1:
The system performs preliminary encoding of sensor data into standardized feature representations before passing them to the machine learning layer. This pre-processing step organizes raw sensor inputs into structured formats that can be quickly processed by subsequent layers, maintaining high processing speed while enabling detailed personalization through comprehensive sensor data analysis.
Solution Approach 2:
The on-device LLM autonomously refines prompts using generated features without requiring external server intervention. This self-service capability allows the system to personalize content in real-time based on sensor data while maintaining high processing speed by eliminating network latency and external processing bottlenecks.
3Adaptability or versatility
If multi-sensory data is incorporated in real-time, then the user engagement improves, but the energy consumption increases
Solution Approach 1:
The system selectively processes sensor data based on relevance and current context, rather than continuously processing all available sensor inputs at full resolution. The encoder layer and machine learning layer dynamically adjust the level of processing detail, maintaining high user engagement through relevant personalization while reducing energy consumption by avoiding unnecessary full-scale processing of all sensor streams.
Data Source
AI summary
A method includes obtaining a multimodal input using at least one sensor and converting the multimodal input into encoded features. The method also includes adjusting the encoded features to produce machine learning (ML) outputs based on user profiles, web browser data, and behavior patterns using an ML model. The method further includes generating text prompts based on the ML outputs using an on-device large language model (LLM) and generating a 360° multimodal wallpaper based on the text prompts from the on-device LLM using one or more generative models. The method also includes refining the text prompts from the on-device LLM based on ongoing sensor data and user interaction using a feedback loop between an output of the one or more generative models and the on-device LLM.