Scripting Core Commands for Parallel Data Processing
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Generating computing instructions for parallel-processing can be complex, especially for programmers needing to perform simple tasks on large datasets, limiting their ability to leverage execution efficiencies.
Innovation Solution
Providing scripting core commands optimized for parallel-processing that can be accessed via simplified programming techniques, such as script-based languages, which include operations like data partitioning, aggregation, and joining, allowing script authors to generate executable instructions for distributing functions across multiple processes.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If parallel-processing algorithms are used to efficiently process large quantities of data, then processing speed and productivity are improved, but the complexity of generating computing instructions increases significantly
Solution Approach 1:
The patent introduces an intermediary system comprising an interpreter and a library of pre-compiled parallel processing algorithms that mediates between the simple scripting language and the complex parallel processing execution environment. The interpreter translates high-level script commands into executable instructions, while the library provides ready-to-use parallel processing routines, thereby shielding users from complexity while maintaining high productivity
Solution Approach 2:
The patent segments the complex parallel processing system into distinct modular components: a scripting layer for user input, an interpretation layer for translation, a library layer for algorithm selection, and an execution layer for parallel processing. This segmentation allows each component to handle specific tasks independently, reducing overall system complexity while maintaining processing efficiency
2Ease of operation
If traditional programming methods are used to perform simple tasks on large datasets, then ease of operation is maintained, but processing time and productivity are significantly reduced
Solution Approach 1:
The patent creates simplified copies of parallel processing capabilities through a scripting language that mirrors traditional programming syntax. The scripting layer provides a copy of the familiar programming interface while the underlying implementation uses optimized parallel algorithms, allowing users to maintain simple coding practices while achieving high processing speeds through the copied interface
3Adaptability or versatility
If generic fault-tolerant storage systems are used, then adaptability and versatility are improved, but the ability to optimize for specific processing tasks is reduced
Solution Approach 1:
The patent implements a dynamic system where the library of parallel processing algorithms can be selectively invoked based on the specific task requirements. The interpreter dynamically selects appropriate algorithms from the library based on the script commands, allowing the system to adapt its processing strategy to match the specific task while maintaining a generic storage infrastructure
Data Source
AI summary
Scripting core commands and aggregations of such commands are provided to script authors to enable them to generate scripts that can be parallel-processed without requiring the author to be aware of parallel-processing techniques. The scripting core commands and aggregations abstract mechanisms that can be executed in parallel, enabling the script author to focus on higher-level concepts. The scripting core commands provided include commands for applying a function in parallel and distributing and joining data in parallel. For added flexibility, one or more scripting core commands can utilize functions written in a different programming language and referenced appropriately in code blocks.