Columnar Index Data Format for Big Data Processing
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional big data processing techniques are inefficient due to high disk and network input/output operations, leading to long execution times and unsuitable for fast response queries, as they often require full scans of unnecessary data and lack effective filtering mechanisms, resulting in increased resource utilization and latency.
Innovation Solution
A columnar index data format is introduced, which generates a dictionary by sorting data by column units, classifying it into data blocks, creating an index with first data values, and assigning column IDs based on row order, allowing for efficient data retrieval and processing by loading the index into memory, comparing query data, and reading only necessary data blocks.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If conventional full scan methods are used for big data processing, then all data can be processed, but disk and network input/output operations increase significantly leading to long execution times
Solution Approach 1:
The patent segments data into row groups and further divides them into column partitions. This segmentation allows the system to process only relevant column partitions rather than scanning entire row groups, significantly reducing I/O operations and execution time while maintaining complete data processing capability.
Solution Approach 2:
The patent extracts and stores statistical information (min/max values, null counts) separately from the main data. This extraction enables the query optimizer to quickly determine whether column partitions need to be read, avoiding unnecessary I/O operations and reducing execution time for queries that don't require all data.
2Ease of operation
If conventional data formats are used, then data can be stored, but reading only required data becomes inefficient due to lack of effective filtering mechanisms
Solution Approach 1:
The patent performs preliminary actions by pre-computing and storing statistical information (min/max values, null counts, distinct value counts) for each column partition during data loading. This preliminary action enables efficient filtering and data retrieval operations without requiring complex runtime analysis, improving ease of operation while managing complexity through structured pre-processing.
3Productivity
If map reduce operations are used on big data, then distributed processing can be achieved, but relatively long execution preparation time is required
Solution Approach 1:
The patent performs preliminary actions by pre-computing statistical information and organizing data into columnar format with row groupings during data loading. This preliminary structuring eliminates the need for complex map reduce setup and execution preparation, enabling direct and efficient distributed query processing while maintaining scalability.
Solution Approach 2:
The patent changes the fundamental data organization parameters from row-based to column-based storage, with additional row group abstraction. This parameter change enables more efficient distributed processing by allowing independent column partition processing and reducing data movement requirements, thereby reducing execution preparation time while maintaining distributed processing capability.
Data Source
AI summary
Provided are systems, apparatuses, methods and non-transitory computer readable media for efficiently processing and analyzing big data using a columnar index data format. A method of processing big data at a processing system configured as a computer may include generating a dictionary by sorting data based on a column unit of the big data; classifying the sorted data into one or more data blocks for each dictionary based on a data size; generating an index that includes first data values of the respective data blocks in order of the data blocks, for each dictionary; and generating a column ID for each column based on row order of the big data.