Iterative Data Processing Engine Using EM Models for LLM Cost Control
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional data intelligence systems lack comprehensive logic and infrastructure for iterative data processing, leading to inefficiencies, scalability issues, reduced accuracy, and computational challenges when handling large datasets, particularly with large language models (LLMs).
Innovation Solution
An iterative data processing optimization engine employing an Expectation Maximization approach with an expectation step model (e.g., LLM) and a maximization step model (e.g., LightGBM or XGBoost) iteratively refines data processing, using probe questions and metadata to filter and score data, enabling efficient and precise analysis.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If large language models process large datasets in their entirety, then comprehensive analysis is achieved, but computational cost and processing time become prohibitive
Solution Approach 1:
The patent segments the large dataset into smaller subsets and processes them iteratively through multiple passes. The LLM analyzes a first subset to generate initial insights, then a second subset to refine those insights, and continues this process across multiple iterations. This segmentation allows comprehensive analysis of the entire dataset while maintaining manageable computational costs at each step.
Solution Approach 2:
The patent applies preliminary filtering and processing steps before the LLM analyzes the data. Metadata is extracted and processed in advance, and data is prepped and structured before being presented to the LLM. This preliminary action reduces the complexity and volume of data the LLM must process, thereby reducing computational cost while preserving analysis accuracy.
2Reliability
If iterative processing is implemented to improve accuracy and scalability, then data analysis quality enhances, but system complexity increases
Solution Approach 1:
The patent implements a feedback mechanism where the LLM's analysis of each data subset feeds into the next iteration. The insights generated from processing the first subset inform the analysis of the second subset, and this feedback loop continues across multiple iterations. This feedback approach improves data analysis accuracy by allowing the system to refine its understanding progressively, while the structured nature of the feedback mechanism helps manage system complexity.
3Loss of information
If comprehensive data processing is performed without iterative approaches, then complete information is captured, but processing time and computational resources are excessive
Solution Approach 1:
The patent employs periodic action by processing data in repeated cycles or iterations. Each iteration processes a different subset of the data or reprocesses data with refined parameters. This periodic approach ensures that complete information is captured across all iterations while significantly reducing the processing time compared to attempting to analyze the entire dataset in a single pass.
Data Source
AI summary
Methods, systems, and computer storage media for providing iterative data processing optimization using an iterative data processing optimization engine in a data intelligence system are described. Iterative data processing refers to handling data where the processing steps are repeated multiple times, across multiple views or modalities, to train machine learning models, filter and score data or generate output. The iterative data processing optimization engine employs expectation step machine learning models that are simple but with fast language models to efficiently and effectively probe and analyze data, while iteratively refining maximization step machine learning models that are optimized and fast to approximate the probing mechanism of the expectation step machine learning models more efficiently, for example, using metadata, external information, and compressed representation. The iterative data processing optimization engine can operate based on an agentic framework using lightweight artificial intelligence (AI) agents to perform model fitting, featurization, and report generation autonomously.