Natural Language Control of Multidimensional Data Visualizations
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing data visualization tools lack the ability to efficiently and intuitively manipulate and visualize multidimensional data using natural language inputs, limiting user interaction and functionality.
Innovation Solution
A computer-implemented method that utilizes a large language model (LLM) to generate structured objects from natural language requests, allowing for the dynamic manipulation and visualization of multidimensional data in user interfaces, including filtering, drilling down/up, and modifying graphical characteristics based on user commands.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of operation
If traditional data visualization tools are used, then data can be displayed and manipulated through conventional interfaces, but user interaction is limited and less intuitive
Solution Approach 1:
The patent replaces traditional mechanical interface interactions (clicking, selecting, navigating menus) with natural language processing. Users communicate with the system through conversational prompts, and the LLM translates these into appropriate visualization actions, eliminating the need for learning complex interface mechanics while expanding functional capabilities.
Solution Approach 2:
The LLM-based system provides universal control over multiple visualization functions through a single interface. One natural language prompt can trigger filtering, sorting, drilling down, drilling up, or modification of graphical characteristics, making the system adaptable to various data manipulation tasks without requiring separate interfaces for each function.
2Ease of operation
If natural language processing is introduced to enable intuitive data manipulation, then ease of operation improves, but system complexity increases
Solution Approach 1:
The LLM serves as an intermediary layer between the user's natural language requests and the underlying data visualization system. This mediator translates human-friendly commands into structured actions that the visualization engine can execute, abstracting away the complexity of the original system while maintaining intuitive operation.
Solution Approach 2:
The system creates a virtual representation of the data structure and visualization capabilities within the LLM's understanding. The LLM learns the schema of the data and the available visualization operations, creating an internal model that allows it to generate appropriate commands without requiring the user to understand the actual system architecture.
3Adaptability or versatility
If LLM is used to generate structured objects from natural language, then adaptability and functionality improve, but processing time and computational resources increase
Solution Approach 1:
The system performs preliminary processing by having the LLM generate complete structured objects (JSON schemas) in advance that define the desired visualization state. These pre-generated structures can then be directly executed by the visualization system without requiring additional interpretation or processing steps, reducing overall response time.
Solution Approach 2:
The patent optimizes the LLM's output format by changing the parameter of response generation to produce structured JSON objects directly. This parameter change in the LLM's output mode enables more efficient parsing and execution by the visualization system, reducing the time needed to translate natural language into actionable commands.
Data Source
AI summary
Systems, methods, and computer-readable media are provided for triggering functionality on data to be generated in a user interface and/or data shown or visualized in a user interface based on a natural language request that references actions to be performed and data items to use in performing the actions. The user interface actions are triggered based on a structured object generated by a large language model (LLM), which may then be processed, validated, and used to carry out the actions. The LLM may be instructed to use control(s) of a displayed representation of a set of data, and the structured object generated by the LLM may cause updating, on the user interface, the displayed representation to reflect change(s) requested (e.g., to adjust filters, change a visualization or view, or zoom in or out on a set of multidimensional data). The control(s) may be selected from among representation transformation action(s) that are also available to be performed against the displayed representation via direct user input.