Directed Graph Feature Extraction from Execution Traces
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Feature engineering in machine learning is difficult and expensive due to the lack of domain knowledge among data scientists, particularly in extracting relationships between columns and table transformations from execution traces.
Innovation Solution
A computer-implemented method to create and reduce a directed graph from execution traces by classifying invocations as call-with-object or call-with-column, building a graph that indicates data flow, and performing automated graph reduction to eliminate intermediate nodes, thereby suggesting new feature creations.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If feature engineering is performed manually using domain knowledge, then feature quality and machine learning performance are improved, but the process becomes expensive and time-consuming
Solution Approach 1:
The system performs self-service by automatically extracting feature engineering knowledge from execution traces without requiring manual domain expertise. The automated graph reduction process identifies relationships between columns and table transformations, generating feature suggestions that would otherwise require expensive manual analysis by domain experts.
Solution Approach 2:
Execution traces serve as an intermediary that captures domain knowledge implicitly embedded in data processing operations. By analyzing these traces and building dependency graphs, the system extracts feature engineering patterns without requiring direct human intervention, thus bridging the gap between manual expertise and automated processing.
2Productivity
If feature engineering is automated without domain knowledge, then the process becomes faster and cheaper, but feature quality deteriorates
Solution Approach 1:
Execution traces act as an intermediary that preserves domain knowledge implicitly contained in actual data processing operations. By mining these traces, the system captures expert-like reasoning patterns automatically, maintaining feature quality while achieving scalability and speed.
Solution Approach 2:
The system performs preliminary analysis of execution traces to extract and store feature engineering patterns in advance. This pre-computed knowledge is then reused for generating feature suggestions, eliminating the need for repeated manual analysis while maintaining consistent quality across different feature engineering tasks.
3Measurement precision
If detailed execution traces are analyzed to extract feature knowledge, then feature engineering accuracy is improved, but data processing complexity increases
Solution Approach 1:
The system extracts only the essential feature engineering knowledge from detailed execution traces by building dependency graphs that focus specifically on relationships between columns and table transformations. This selective extraction maintains high accuracy while reducing processing complexity by filtering out irrelevant trace details.
Solution Approach 2:
The automated graph reduction process extracts only the critical intermediate nodes that represent meaningful feature transformations. By eliminating redundant nodes while preserving essential relationships, the system maintains feature extraction accuracy while significantly simplifying the graph structure for easier processing.
Data Source
AI summary
A method is provided for creating a directed graph. The method obtains an execution trace having a sequence of invocations of table transformations. Each invocation is classified as a call-with-object invocation or call-with-column invocation. The call-with-object invocation indicates that an invoked function has object IDs as at least one of input and output information. The call-with-column invocation indicates that the invoked function has column names as the at least one of input and output information. The method processes the trace to build the graph. Each node indicates an object or a column and each edge indicates data flow. The method performs, in response to all invocations being processed, an automated graph reduction process by eliminating given nodes indicating a respective object such that that every two nodes that connect with each other, via a respective one of the given nodes as an intermediate node, can keep the connection therebetween.