Object Storage Querying with Block Merging for Unstructured Big Data
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing big data systems struggle to efficiently process unstructured data in cloud environments due to the requirement of a fixed schema, making it difficult to handle large-scale log data at high speeds.
Innovation Solution
A method involving recording data in block storage, merging it based on column names, and storing it in object storage, along with metadata management, allows for fast querying and analysis of unstructured big data without fixing the schema.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Speed
If a fixed schema is used for column acceleration in object storage, then query speed is improved, but the ability to process unstructured big data is worsened
Solution Approach 1:
The patent segments data into blocks with flexible schemas, allowing each block to have its own column definitions. This enables the system to handle unstructured data without requiring a global fixed schema, while still maintaining column acceleration capabilities within each block for fast querying.
Solution Approach 2:
The patent introduces dynamic schema evolution where column schemas can change over time as new data types are encountered. The system automatically adapts the schema of data blocks to accommodate evolving data structures, enabling both fast querying and flexibility for unstructured data processing.
2Adaptability or versatility
If data is stored in small blocks in object storage, then data flexibility is improved, but query processing time is worsened
Solution Approach 1:
The patent merges multiple small data blocks into larger merged blocks while preserving columnar organization. This merging process consolidates data that can be queried together, reducing the number of individual block access operations and thereby decreasing query processing time while maintaining the flexibility to handle various data types.
Solution Approach 2:
The patent performs preliminary merging of data blocks during the data ingestion phase, organizing data into optimized merged blocks before queries are executed. This preliminary organization reduces the workload during query processing, as the system can work with pre-merged blocks rather than individually processing many small blocks.
3Adaptability or versatility
If unstructured data is processed without schema fixation, then data versatility is improved, but processing speed is worsened
Solution Approach 1:
The patent applies local quality by allowing each data block to have its own optimized schema and column definitions tailored to the specific data it contains. This enables the system to maintain fast processing speeds within each block through columnar storage and indexing, while the overall system remains versatile enough to handle diverse unstructured data types across different blocks.
Data Source
AI summary
A computer implemented method for recording data includes a first step of recording data of a predetermined first time-range in a block storage; and a second step of performing a batch operation of merging data in the block storage to record the generated block data in an object storage, if there is no data inflow for a predetermined second time-range or if the sum of the capacities of new blocks recorded in the block storage exceeds a predetermined capacity or the sum of the number of records exceeds a predetermined number of data. The block data recorded in the second step is data where the time range of a merged column is placed at the front and the data is merged based on a column name.