Symbolic Property Mining for One-Shot Code Generation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current methods for selecting relevant examples for one-shot and few-shot learning in large language models (LLMs) are unsupervised and rely on assumptions about similarity or probabilistic approaches, making it unclear which parts of the utterance and code are causing high similarities or probabilities, and how many examples to provide for improved code generation.
Innovation Solution
A system that uses a latent property model to analyze embeddings and delta values to determine the relevance of program code examples, training a property recognition model to identify symbolic properties that improve LLM performance, and a symbolic property mining pipeline to select relevant examples from a knowledge base.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Extent of automation
If unsupervised similarity-based approaches are used to select examples, then example selection can be performed automatically, but it is unclear which parts of the utterance and code are causing high similarities
Solution Approach 1:
The patent introduces an intermediary explanation mechanism that bridges the gap between automated similarity-based example selection and interpretability. This intermediary component analyzes the embedding space and identifies which specific parts of the utterance and code contribute to high similarity scores, providing human-understandable explanations for the automated selection process without sacrificing automation efficiency.
2Extent of automation
If probabilistic approaches are used to select examples, then selection can be performed without manual supervision, but token probabilities are often unavailable
Solution Approach 1:
The patent employs a pragmatic approach by using approximate or surrogate probability estimates when exact token probabilities are unavailable. Instead of requiring precise probability values, the system uses readily available embedding similarities and heuristic scoring mechanisms that provide sufficient reliability for example selection without the computational overhead of obtaining exact probability estimates from the language model.
3Manufacturing precision
If more examples are provided to the LLM, then code generation accuracy may improve, but it is unknown how many examples to pick
Solution Approach 1:
The patent implements a feedback-driven example selection process where the system evaluates the performance impact of selected examples and uses this feedback to refine future selections. By monitoring whether provided examples actually improve code generation accuracy and using this information to adjust selection criteria, the system dynamically determines the optimal number of examples to provide without requiring complex manual configuration or exhaustive search.
Data Source
AI summary
A data processing system implements extracting symbolic property information from a training dataset by analyzing the training dataset with a symbolic property mining pipeline to extract properties of program code from one-shot program code examples, the symbolic property information indicative of types of properties of the one-shot program code examples determined to improve program code output by a large language model (LLM) in response to natural language utterances; and training a property recognition model to recognize symbolic properties associated with a natural language utterance using the training dataset and the symbolic property information, the property recognition model being configured to analyze the natural language utterance and to output the symbolic properties of program code.