Multi-dimensional Dataset Wrangling via Parallel Sub-table Segmentation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing data wrangling processes for OLAP cubes are inefficient, particularly in terms of processing time and memory utilization, especially as the number of columns to be processed increases and operations become more complex.
Innovation Solution
The method involves dividing a multi-dimensional dataset into sub-tables based on operation types and performing data wrangling operations in parallel on these sub-tables, allowing only changed columns/rows to be processed, thereby reducing processing time and memory usage.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If data wrangling operations are performed sequentially on entire tables, then processing is straightforward and simple to implement, but processing time increases significantly and memory utilization becomes excessive
Solution Approach 1:
The patent divides a large multi-dimensional dataset into multiple smaller sub-tables based on operation types. Each sub-table contains only the data relevant to specific operations, enabling parallel processing of multiple sub-tables simultaneously. This segmentation reduces the processing time from hours to minutes while maintaining accuracy, as demonstrated in the OLAP cube publishing workflow where the dataset is split before being processed by the data wrangling server.
2Reliability
If all columns and rows are processed regardless of changes, then complete data processing is ensured, but memory usage increases significantly
Solution Approach 1:
The patent extracts and processes only the changed columns and rows from the multi-dimensional dataset. The system identifies which specific portions of the data have changed since the last publish operation and creates sub-tables containing only those changed elements. This extraction approach ensures that all necessary changes are processed completely while minimizing memory utilization by excluding unchanged data from the processing workflow.
3Ease of operation
If data wrangling scripts are sent for each table in sequence, then the process is simple to manage, but the time to publish OLAP cube increases significantly
Solution Approach 1:
The patent segments the data processing workflow by dividing the multi-dimensional dataset into multiple sub-tables that can be processed in parallel. Instead of sending one large table through sequential processing, the system creates multiple smaller sub-tables with different operation types and processes them simultaneously on the data wrangling server. This parallel processing approach dramatically reduces the time required to publish OLAP cubes while maintaining manageable process control through organized sub-table handling.
Data Source
AI summary
Disclosed are methods, systems, and computer-readable medium for data wrangling. For instance, the method may include: determining one or more data operations to be performed on a multi-dimensional dataset; determining at least one operation type associated with the one or more data operations to be performed; dividing the multi-dimensional dataset into a plurality of sub-tables based on the operation type; providing the plurality of sub-tables to a data wrangling process, the one or more data operations being performed on each of the plurality of sub-tables in parallel to determine a processed plurality of sub-tables; and determining, based on the processed plurality of sub-tables, a finished multi-dimensional dataset. The data wrangling process may process sub-tables (e.g., change-only columns/rows) of tables of the multi-dimensional dataset and/or process sub-tables in parallel for data operations that are not related.