Generative AI Streaming With Retrieval for Structured Responses
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing generative AI systems face challenges in generating responses quickly due to increasing model size, leading to long wait times and errors in outputting data in specific formats, and are limited in making item recommendations based on current availability or post-training additions.
Innovation Solution
The system employs a hybrid approach combining generative models with retrieval systems, streaming output in intermediate tokens, and using data objects to augment responses, enabling faster and error-reduced generation of structured data.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If the size of generative models is increased to improve response quality and capabilities, then the model's understanding and generation capabilities are enhanced, but the time required to generate tokens increases significantly
Solution Approach 1:
The patent segments the response generation process into multiple parallel operations: (1) generating a structured outline with headings and subheadings, (2) retrieving relevant data objects from databases, (3) generating content for each section, and (4) assembling the final response. This segmentation allows different components to be processed in parallel, reducing overall generation time while maintaining model quality.
Solution Approach 2:
The patent performs preliminary actions by first generating the structural framework (headings, subheadings, and outline) before generating the actual content. It also pre-retrieves data objects from databases during the outline generation phase. This preliminary structuring enables faster content generation in subsequent steps, as the model only needs to fill in predetermined sections rather than generate entire responses from scratch.
2Ease of operation
If the generative model is trained to output data in a strict data format such as markup format, then the output can be directly presented in a user interface, but syntax errors occur that prevent the user interface from presenting the output
Solution Approach 1:
The patent segments the formatting task into two distinct phases: (1) generating the structural framework with proper markup tags during outline generation, and (2) populating content within the already-formed structure. This separation ensures that the syntactic structure is established correctly first, reducing the likelihood of syntax errors, while content generation occurs within the safe boundaries of the pre-formed structure.
Solution Approach 2:
The patent introduces an intermediary structured outline format that acts as a bridge between the generative model's free-form text generation and the user interface's requirement for strict markup format. The outline serves as an intermediate representation with predetermined markup structure, allowing the model to generate content without directly manipulating the final markup syntax, thereby reducing syntax errors.
3Reliability
If the entire response is generated before being returned to the user interface, then the output is complete and ready for presentation, but the user must wait for the entire generation process to complete
Solution Approach 1:
The patent performs preliminary generation of the response structure (headings, subheadings, and outline) and retrieves data objects before the user interface needs to display content. This preliminary structuring allows the system to begin presenting content to the user interface while still generating the remaining content in the background, effectively overlapping computation and presentation phases.
Solution Approach 2:
The patent segments the response into multiple independent sections (headings, subheadings, content blocks) that can be generated and transmitted to the user interface in stages. The structured outline and retrieved data objects can be prepared and partially presented before the full content generation completes, enabling progressive disclosure of information to the user.
4Device complexity
If a single generative model is used to handle all tasks including item recommendations, then the system is simple to implement, but the model cannot provide recommendations based on current availability or post-training additions
Solution Approach 1:
The patent merges the generative model with external retrieval systems and databases to create a hybrid system. The generative model handles high-level reasoning and content generation, while separate retrieval components handle real-time data fetching for item recommendations. This combination maintains relative system simplicity while enabling real-time adaptability through the integrated retrieval architecture.
Solution Approach 2:
The patent introduces retrieval systems and data objects as intermediaries between the generative model and real-time data sources. These intermediaries handle the complex task of querying current availability and post-training additions from external databases, allowing the generative model to focus on content generation while accessing up-to-date information through the intermediary retrieval layer.
Data Source
AI summary
An artificial intelligence system includes one or more generative artificial intelligence models (“generative model”) and a retrieval system. The artificial intelligence (“AI”) system receives a natural language input query. The retrieval system retrieves data objects associated with the input query. A generative model streams content in response to input query. The AI system formats output from the generative model and the data objects into formatted data. A user interface outputs the formatted data.