Parallel Data Model Creation in Cloud Platforms
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The transfer of large data files in cloud computing environments is time-consuming and error-prone, particularly when transmitting several gigabytes of data, which can lead to long processing and transmission times and increased risk of failures.
Innovation Solution
The system employs parallelization by dividing data into multiple messages, processing user-defined models and data tables separately and independently using vertical and horizontal parallelizers, allowing concurrent transmission and processing of dimension clusters and data chunks, thereby optimizing message size and reducing the risk of failure.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If large data files are transmitted in a single message, then data completeness is ensured, but transmission time and error risk increase significantly
Solution Approach 1:
The patent divides large data files into multiple smaller messages for parallel transmission. Each message contains a portion of the data along with metadata including message ID, total message count, and checksum. This segmentation reduces individual message transmission time and allows parallel processing while maintaining data completeness through reassembly using the metadata information.
2Productivity
If data is divided into multiple messages for parallel transmission, then transmission time is reduced, but system complexity increases
Solution Approach 1:
The system performs preliminary actions by pre-calculating and attaching metadata (message ID, total count, checksum) to each message during the segmentation phase. This preliminary organization of data and metadata simplifies subsequent parallel processing and reassembly operations, reducing the complexity burden during high-speed transmission and processing phases.
Solution Approach 2:
The patent implements feedback mechanisms through checksum validation in each message and acknowledgment protocols. The receiving system verifies message integrity using embedded checksums and reports status back to the transmitting system, enabling automatic error detection and recovery. This feedback loop manages complexity by providing structured verification rather than requiring complex manual coordination.
3Measurement precision
If all data is processed sequentially, then processing accuracy is maintained, but processing time becomes unacceptably long
Solution Approach 1:
The patent segments data processing into independent parallel tasks, each handling a specific message or data chunk. Multiple processor cores can simultaneously execute different segmentation tasks while maintaining data integrity through the metadata structure. This enables accurate parallel processing where each segment is processed independently with the same level of precision as sequential processing would provide.
Solution Approach 2:
The patent merges results from multiple parallel processing operations into a unified output. After parallel processors complete their respective tasks on different data segments, the system combines the results using the message metadata (message IDs, checksums) to ensure correct reassembly and validation. This merging process maintains the accuracy of individual parallel operations while achieving the speed benefits of parallelism.
Data Source
AI summary
A system, method, and computer-readable medium to receive at least one data table relating to a data set; receive a plurality of dimensions; combine, by a first parallelization process, the plurality of dimensions into a plurality of different clusters, each cluster being a grouping of different dimensions; transmit each of the plurality of different clusters to a cloud platform; divide, by a second parallelization process, the at least one data table into a plurality of different chunks of data; and transmit each of the plurality of different chunks of data to the cloud platform, in parallel to and independent of the transmission of the plurality of different clusters the cloud platform, the cloud platform to generate a data model based on a combination of the transmitted plurality of different clusters and the transmitted plurality of different chunks of data.