Script Operation Duplication Detection and Reuse
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing systems face challenges in efficiently reusing data across multiple users and operations, as regenerating results for analytics on large data sources is costly in terms of time and storage, and manual determination of data reuse is required.
Innovation Solution
A system and method that parse operations into operation names and input data sets, determine if operations have been previously executed, link outputs to stored outputs, and execute a revised script to reduce redundant operations and data processing.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If operations are re-executed on large data sources, then up-to-date results are obtained, but time and storage costs increase significantly
Solution Approach 1:
The system pre-computes and stores results of operations on large data sources in a results database. When the same operation is requested again, the stored results are retrieved and reused, eliminating the need to re-execute the operation. This preliminary action of storing results resolves the contradiction by providing fresh results without the time cost of re-processing.
Solution Approach 2:
The system creates copies of operation results and stores them in a results database for future reuse. Instead of regenerating results, the system copies and retrieves previously computed outputs, matching the same operation name and input data set description. This copying approach maintains data reliability while avoiding redundant processing time.
2Reliability
If operations are re-executed on large data sources, then current results are obtained, but storage costs increase
Solution Approach 1:
The system creates copies of operation results and stores them in a results database for future reuse. Instead of regenerating results, the system copies and retrieves previously computed outputs, matching the same operation name and input data set description. This copying approach maintains data reliability while avoiding redundant processing time.
Solution Approach 2:
The system changes the parameter of result storage by implementing a caching mechanism that stores operation results with their associated metadata (operation name, input data set description). This parameter change allows the system to retrieve results efficiently without full re-computation, reducing both storage costs and energy consumption while maintaining data freshness.
3Adaptability or versatility
If manual determination of data reuse is implemented, then data reuse between users is possible, but system complexity increases
Solution Approach 1:
The system implements self-service automation where the operation execution module automatically determines whether to reuse data by querying the results database with the operation name and input data set description. The system autonomously retrieves stored results when matches are found, eliminating the need for manual user determination. This self-service approach increases adaptability while reducing system complexity by automating the data reuse decision process.
Solution Approach 2:
The system implements a feedback mechanism where the operation execution module continuously queries the results database to check for existing results matching the current operation parameters. This feedback loop automatically determines data reuse opportunities without manual intervention, enhancing versatility while managing complexity through systematic automated checking and result retrieval.
4Productivity
If redundant operations are eliminated through result linking, then processing efficiency improves, but system complexity increases
Solution Approach 1:
The system pre-computes and stores results of operations on large data sources in a results database. When the same operation is requested again, the stored results are retrieved and reused, eliminating the need to re-execute the operation. This preliminary action of storing results resolves the contradiction by providing fresh results without the time cost of re-processing.
Solution Approach 2:
The system introduces a results database as an intermediary between operation execution and result delivery. This intermediary stores operation results and enables automatic matching based on operation name and input data set description. The intermediary layer simplifies the overall architecture by centralizing result storage and retrieval logic, improving productivity while managing complexity through a dedicated mediation layer.
Data Source
AI summary
The present invention generally relates to systems and methods for executing scripts (a sequence of declarative operations) on large data sets. Some implementations store descriptions of previously-executed operations and associated input and output data sets. When executing similar operations on the same, a subset of, a superset of, or any fragment of data subsequently, some implementations detect duplication of operations and access previously-stored output data sets in order to re-use data and reduce the amount of execution, thus avoiding time-consuming duplicative computations.