Induced Dynamic Grammars for Accurate API Call Decoding
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing language models (LMs) struggle to accurately and quickly call APIs and programs during interactions, leading to inefficiencies in their ability to engage with humans effectively.
Innovation Solution
The system generates dynamic context-free grammars that specify constraints for decoding systems, allowing for automated agents to construct grammars from API specifications, business logic, and conversation history, ensuring accurate and efficient API calls.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If language models generate text token by token without constraints, then they maintain flexibility in generation, but they fail to accurately and quickly call APIs and programs
Solution Approach 1:
The patent applies dynamics by making the grammar structure adaptive and changeable during the generation process. The decoding system dynamically adjusts the grammar based on the current generation state, allowing it to switch between free-form text generation and constrained API call generation as needed, thereby maintaining both flexibility and accuracy.
Solution Approach 2:
The patent changes the parameter of grammar constraints during generation. By modifying the grammar structure based on the generation context, the system can enforce specific constraints when API calls are needed while maintaining flexibility for regular text generation, resolving the contradiction between reliability and complexity.
2Reliability
If static grammars are used for constrained decoding, then API call accuracy improves, but the system cannot account for request-specific details
Solution Approach 1:
The patent transitions from static grammars to dynamic grammars that can adapt during the generation process. The grammar structure is modified based on the current generation state and input characteristics, allowing the system to maintain accuracy for API calls while simultaneously adapting to request-specific requirements.
Solution Approach 2:
The patent performs preliminary analysis of the input to determine what constraints are needed before generation begins. By pre-processing the input to identify API call requirements and request-specific details, the system can configure the appropriate grammar constraints in advance, ensuring both accuracy and adaptability.
3Measurement precision
If dynamic grammars are constructed for each request, then request-specific accuracy improves, but processing time increases
Solution Approach 1:
The patent performs preliminary analysis of the input to identify whether API call constraints are needed before the generation process begins. This early detection allows the system to only construct constrained grammars when necessary, avoiding the time cost of building dynamic grammars for every request while maintaining high accuracy when constraints are required.
Solution Approach 2:
The patent applies partial action by constructing full dynamic grammars only when API call accuracy is critical, rather than for every single request. The system selectively applies the computationally intensive grammar construction approach only where needed, balancing precision with processing time efficiency.
4Speed
If language models are made faster for real-time interaction, then responsiveness improves, but accuracy of program calls deteriorates
Solution Approach 1:
The patent performs preliminary detection of API call requirements before the main generation process. By identifying when constrained generation is needed in advance, the system can prepare appropriate grammars efficiently and execute the generation process at optimal speed without sacrificing accuracy through repeated iterations.
Data Source
AI summary
The present technology, roughly described, generates grammars, such as context free grammars, dynamically and by induction, that can specify constraints for a decoding system. The grammar can be created to support an automated agent involved in an interaction (e.g., a conversation) with a client. In some instances, grammars can be generated dynamically based on copying portions of an interaction history and based on business logic rules. Grammars can also be induced automatically from API specifications and collections of strings. The newly generated context free grammars can be used by a language model to perform constrained decoding.