Natural Language Query Interface via LLM and UI Metadata
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Modern enterprise computing systems face challenges in accessing and retrieving insights from vast amounts of data due to complex database queries and numerous applications with overlapping functions, leading to a steep learning curve and unreachable data.
Innovation Solution
The implementation of intelligent natural language queries using large language models and user interface element metadata allows for the processing of user queries by generating a graph representation of user interface elements, identifying matching elements, and navigating to relevant pages to extract answers.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of operation
If traditional database query interfaces and application navigation are used, then data access capability is maintained, but user operation complexity increases due to learning curves and overlapping application functions
Solution Approach 1:
The patent introduces a large language model as an intermediary between the user and the enterprise computing system. The LLM translates natural language queries into structured database queries and navigational commands, eliminating the need for users to directly interact with complex database interfaces or navigate through multiple applications. This mediator handles the complexity of data access and application navigation while presenting a simple conversational interface to users.
Solution Approach 2:
The patent replaces traditional mechanical interaction methods (clicking through menus, navigating application interfaces, writing SQL queries) with a natural language processing system. Instead of requiring users to manually navigate through complex application hierarchies or write technical queries, the system uses LLM-based natural language understanding to automatically generate the necessary database queries and navigation sequences, substituting mechanical interface interactions with intelligent language processing.
2Loss of information
If comprehensive database queries are implemented to access all available data, then data completeness is improved, but query processing time increases due to complex query structures
Solution Approach 1:
The patent segments complex database queries into smaller, manageable components generated by the large language model. Instead of constructing and executing single complex queries that scan vast amounts of data, the LLM breaks down the query intent into multiple focused database operations, filtering and retrieving only the specific data needed to answer the user's question. This segmentation reduces the amount of data processed and improves query execution time while maintaining data completeness.
Solution Approach 2:
The system performs preliminary actions by using the large language model to analyze and structure the query intent before actual database execution. The LLM pre-processes the natural language input, identifies relevant data sources, formulates optimized query structures, and prepares navigation paths in advance. This preliminary structuring of queries ensures that when database operations are executed, they are already optimized for efficiency, reducing processing time while accessing comprehensive data.
Data Source
Figure 1
Figure 2
Figure 3
AI summary
Natural language queries can be answered by using a large language model to find an appropriate user interface element appearing in an application. User interface element metadata can be incorporated when choosing the user interface element. Browser automation can then navigate to a page in the application on which the interface element appears, extract answer data, and then present the answer data as an answer to the natural language query. Input values can be supported, and a large language model can select a matching input value based on semantic matching, even if the natural language query does not have an exactly matching input value. Additional features such as pre-calculating embeddings, pre-determining candidate input values, and the like can be supported. The technologies can provide natural language access to web applications that can result in immediate access for new users and faster query execution by experienced users.