Click-to-Script Mapping for Optimized Big-Data Execution Plans
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Developers face challenges in debugging and optimizing big-data job scripts due to the unintuitive transformations made by SCOPE's query optimizer, making it difficult to identify and troubleshoot issues in optimized execution plans.
Innovation Solution
A cloud-based service that generates tokens for each operation in a big-data job script, allowing developers to visualize an optimized execution plan and click back to the original script operations, facilitating quicker troubleshooting.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If the SCOPE optimizer transforms a job script into an optimized query through substitutions, explorations and transformations, then execution efficiency is improved, but the resulting plan becomes difficult for users to comprehend and troubleshoot
Solution Approach 1:
The patent introduces an intermediary mapping system that connects optimized execution plan operations back to original script lines. Token markers are embedded in the optimized plan to serve as intermediaries, allowing users to trace back from optimized operations to source code without seeing the transformation details. This resolves the contradiction by maintaining execution efficiency while providing a bridge for user comprehension.
Solution Approach 2:
The patent creates a copied representation of the execution plan that preserves traceability to original script lines. Instead of showing users the transformed optimized plan directly, the system generates a copied view with embedded metadata links that map each optimized operation back to its source. This allows users to work with the optimized plan while maintaining connection to original code for troubleshooting.
2Productivity
If the optimizer applies transformations such as replicating and applying filters earlier and removing redundancy, then query performance is improved, but it becomes unintuitive to reverse-engineer from a SCOPE runtime stage to a script line
Solution Approach 1:
The patent applies preliminary action by embedding token markers and metadata during the optimization transformation process itself, before the optimized plan is executed. These markers are inserted into the optimized execution plan to record the mapping relationship between optimized operations and original script lines. This preliminary tagging makes reverse-engineering intuitive without affecting the performance benefits of the optimizations.
Solution Approach 2:
The patent implements feedback by providing users with clickable links from optimized operations back to source code. When users interact with the optimized execution plan, the system provides feedback in the form of traceable connections to original script lines, enabling intuitive reverse-engineering despite the transformations applied.
3Reliability
If developers spend time investigating and editing scripts to address performance issues, then job execution problems are resolved, but the process is time-consuming due to the inability to quickly locate blaming statements
Solution Approach 1:
The patent uses token markers as intermediaries that directly connect optimized execution operations to source code locations. When performance issues occur, developers can click on problematic optimized operations to instantly jump to the corresponding script lines, eliminating time-consuming manual investigation while maintaining reliable problem resolution.
Data Source
AI summary
A click-to-script service enables developers of big-data job scripts to quickly see the underlying script operations from optimized execution plans. Once a big-data job is received, the disclosed examples compile it and generate tokens that are associated with each operation of the big-data job. These tokens include may include the file name of the job, the line number of the operation, and/or an Abstract Syntax Tree (AST) node for the given operations. An original execution plan is optimized into an optimized execution plan, and the tokens for the original operations of the job script are assigned to the optimized operations of the optimized execution plan. The optimized execution plan is graphically displayed in an interactive manner such that users may view the optimized execution plan and click on its optimized operations to find the original operations of the job script.