Semi-Structured Data Ingestion in Columnar Storage
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing columnar storage systems are inefficient for semi-structured data, as they require a priori knowledge of the schema to store and query data effectively, which is not feasible when dealing with diverse and unknown data schemas.
Innovation Solution
The system breaks down data into row groups with individual schemas, allowing for the storage of semi-structured data in a columnar format without prior knowledge of the schema, using techniques like encoding repetition and definition levels to efficiently manage nested data structures and handle heterogeneous records.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If columnar storage systems use a fixed schema approach to store data, then query efficiency is improved, but the system cannot handle semi-structured data with unknown or varying schemas
Solution Approach 1:
The patent implements dynamic schema evolution by allowing row groups to be created with different schemas based on the actual data being ingested. Each row group maintains its own schema definition, enabling the system to adapt to varying data structures while preserving query efficiency through schema-aware processing. This resolves the contradiction by making the schema flexible rather than fixed, allowing the system to handle both structured and semi-structured data effectively.
2Adaptability or versatility
If the system creates a new row group for each unique schema, then schema diversity is supported, but the number of row groups increases reducing storage efficiency
Solution Approach 1:
The patent segments the data storage into row groups with individual schemas, allowing each segment to handle its specific data structure independently. This segmentation enables efficient storage of semi-structured data by creating row groups only when necessary, rather than forcing all data into a single schema. The segmentation principle resolves the contradiction by organizing data into manageable units that balance schema diversity with storage efficiency.
3Quantity of substance
If the system buffers records to reduce row group creation, then storage efficiency is improved, but query performance may degrade due to increased buffering complexity
Solution Approach 1:
The patent applies preliminary action by pre-defining schemas for row groups before data ingestion. This allows the system to buffer records efficiently by knowing in advance which row group schema will accommodate incoming data, reducing the need for dynamic schema creation during buffering. The preliminary schema definition simplifies buffering management while maintaining storage efficiency, resolving the contradiction between storage efficiency and buffering complexity.
Data Source
AI summary
An apparatus, computer-readable medium, and computer-implemented method for ingesting semi-structured data in a columnar format, including parsing a data record, the data record comprising semi-structured data including one or more data fields and one or more data values corresponding to the one or more data fields, generating a schema corresponding to the data record, the schema comprising the one or more data fields and one or more data types corresponding to the one or more data fields, determining whether the schema corresponding to the data record is homogenous with a schema corresponding to a current row group, wherein the current row group comprises one or more current column names and one or more current column types, and storing the data record as a new record in the current row group based at least in part on a determination that the schema is homogenous.