Dynamic Workflow Agent for Multi-Document Information Extraction
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing techniques struggle to efficiently extract accurate insights from large collections of unstructured and varied documents, due to challenges in analyzing tabular data, graphs, and traditional text, and standalone prompt-based language models often fail to accurately retrieve information.
Innovation Solution
A system that processes queries by dynamically generating a workflow with multiple states, selecting appropriate sub-agents for each state, and executing them to generate responsive output. This system uses a large language model (LLM) to determine the states and sub-agents needed for the query, and to validate responses.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If traditional techniques are used to extract information from documents, then the process is simpler, but accuracy and efficiency deteriorate due to inability to handle unstructured and varied document formats
Solution Approach 1:
The system segments the document collection into individual documents and processes each document through a structured workflow with multiple states (retrieve, load and convert, extract and analyze, document search, identify, validate, compile). Each state handles specific aspects of information extraction, allowing the system to manage complexity through modular processing while maintaining high accuracy across diverse document formats
Solution Approach 2:
The system introduces an LLM-based intermediary that dynamically generates prompts and orchestrates the workflow between the query and document processing. This intermediary translates user queries into structured processing steps and coordinates the extraction pipeline, enabling accurate handling of unstructured data without requiring complex manual configuration
2Measurement precision
If standalone prompt-based language models are used, then the system is simpler, but information retrieval accuracy deteriorates due to model overload from too much information
Solution Approach 1:
The processing pipeline is divided into distinct states where information is progressively extracted and refined. Each state processes a specific aspect (retrieval, conversion, extraction, analysis, validation), preventing model overload by presenting manageable information chunks to the LLM at each step rather than overwhelming it with all document content simultaneously
Solution Approach 2:
The system performs preliminary actions before the main LLM processing: documents are retrieved, loaded, and converted to appropriate formats in advance. This preparation reduces the complexity of the main extraction task by pre-organizing data, so the LLM receives pre-processed information that requires less intensive processing
3Productivity
If manual analysis is used, then flexibility is higher, but productivity deteriorates due to time-consuming processes that are difficult to implement at scale
Solution Approach 1:
The system implements self-service through automated workflow generation and execution. The LLM dynamically creates processing workflows based on query requirements and automatically executes the multi-state extraction pipeline without manual intervention. This automation maintains flexibility by adapting to different query types while achieving high throughput by processing multiple documents simultaneously through the structured pipeline
4Adaptability or versatility
If specialized models are trained for a given domain, then accuracy improves, but adaptability deteriorates when minor changes occur requiring extensive retraining
Solution Approach 1:
The system uses dynamic prompt generation where the LLM creates customized processing instructions based on the specific query and document characteristics. This dynamic adaptation allows the system to maintain high accuracy across different domains without retraining, as the prompts are generated on-demand to match the specific extraction task requirements rather than relying on static pre-trained domain models
Data Source
Figure 1
Figure 2
Figure 3A
AI summary
A system is provided for processing user queries by using an automated agent and a workflow. The system comprises reusable components that include states, tools, and/or data sources. Based on analysis of a query's content and goals, the system generates a workflow comprising a sequence of states, each state optimized for a subtask and dynamically bound to a selected tool(s) for that specific query. The workflow can provide a structured high-level control, while allowing for flexible selection of the tool(s) for each state of the workflow for that given query. The system produces a result using the structured workflow and selected tools, answering a user's original query.