Big Data Read-Write Reduction via Tall Variables
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Processing big data is time-consuming and resource-intensive due to the need for numerous read-write operations, especially when data exceeds local memory limits and is distributed across multiple devices, requiring different algorithms for various data formats and locations.
Innovation Solution
The use of tall variables, partitioned arrays, and big data primitives that facilitate reduced read-write operations by generating and storing data structures to combine operations, identify parallel operations, and optimize data access, allowing for efficient processing across different data formats and locations.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If traditional processing methods are used for big data, then data can be processed, but the processing is time-consuming and resource-intensive due to numerous read-write operations
Solution Approach 1:
The patent combines multiple read and write operations into fewer, more efficient operations by using tall variables and partitioned arrays. The system generates and stores data structures that consolidate operations, reducing the total number of read-write cycles needed to process big data across multiple devices.
Solution Approach 2:
The system performs preliminary actions by generating and storing optimized data structures before actual data processing occurs. These pre-computed structures enable faster access and reduce the need for repeated read-write operations during data processing tasks.
2Adaptability or versatility
If data is distributed across multiple devices, then big data can be processed, but different algorithms are needed for various data formats and locations
Solution Approach 1:
The patent creates a universal approach using tall variables and partitioned arrays that can handle multiple data formats and locations through a single unified interface. The system generates standardized data structures that work across different devices and formats, eliminating the need for separate algorithms for each data type.
Solution Approach 2:
The system introduces tall variables and partitioned arrays as intermediary structures between the data storage layer and processing layer. These intermediaries abstract away the complexities of different data formats and locations, providing a consistent interface for data processing across heterogeneous environments.
3Ease of operation
If numerous read-write operations are performed, then data can be accessed and processed, but disk access operations become time-consuming and resource-intensive
Solution Approach 1:
The patent merges multiple sequential read-write operations into fewer batch operations. By using partitioned arrays and pre-generated data structures, the system consolidates access patterns, reducing the total number of disk access operations and associated resource consumption while maintaining full data access capability.
Data Source
AI summary
A device may include one or more processors. The device may receive program code that identifies operations to be performed with regard to a data set to determine an output value. The device may identify a plurality of read operations corresponding to the operations. The plurality of read operations may be performed to obtain respective inputs of the operations. The device may generate a first data structure that identifies the operations and the plurality of read operations. The device may identify two or more operations, of the operations, that receive a particular input. The device may generate a second data structure that includes a combined operation. The combined operation may be generated based on the two or more operations, and may be associated with fewer read operations than the two or more operations. The device may determine the output value based on the second data structure.