Flex Tables for Sparse Semi-Structured Data Storage
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional relational databases face inefficiencies and performance issues when handling large quantities of sparse, heterogeneous data due to rigid schemas and the presence of numerous empty columns, which are exacerbated by the disparate data sources in modern computer and communication networks.
Innovation Solution
The implementation of a flexible record definition through flex tables, which include an abstraction layer that dynamically adjusts column sizes and definitions based on data variants, allowing for efficient storage and querying of semi-structured data while maintaining relational model capabilities.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If conventional relational databases use rigid schemas with allocated columns for each row, then data structure consistency is maintained, but storage space is wasted and performance deteriorates due to numerous empty columns in sparse tables
Solution Approach 1:
The patent segments the column structure into fixed schema columns and flexible variant columns. Each row can have its own set of columns defined by a variant identifier, allowing the table to be divided into a consistent base structure and variable data portions. This resolves the contradiction by maintaining structure consistency in the base schema while enabling storage efficiency through row-specific column definitions.
Solution Approach 2:
The patent introduces dynamic column definitions through variant identifiers. Instead of static schema definitions, each row can dynamically define its own columns based on the data type being stored. The system dynamically allocates and deallocates columns as needed, allowing the schema to adapt to each row's specific data requirements while maintaining overall structural integrity through the variant mechanism.
2Ease of operation
If relational databases allocate columns for all possible data fields, then data normalization and consistent querying are enabled, but storage requirements increase and performance decreases when handling heterogeneous data from multiple sources
Solution Approach 1:
The patent applies local quality by allowing different rows to have different column sets tailored to their specific data types. Instead of uniformly allocating columns for all possible fields across all rows, each row locally defines its own columns based on its data variant. This enables efficient storage by allocating space only where needed while maintaining normalization capabilities through the standardized variant interface.
3Adaptability or versatility
If semi-structured data formats like JSON, XML, or Avro are used to handle heterogeneous data, then flexibility and adaptability improve, but query performance and processing speed deteriorate
Solution Approach 1:
The patent introduces a variant identifier as an intermediary between the rigid relational schema and flexible semi-structured data. The variant ID acts as a mediator that enables the database to interpret and process heterogeneous data in a standardized way. This intermediary layer allows the system to maintain the speed advantages of relational databases while gaining the flexibility to handle diverse data formats through the variant mechanism.
4Adaptability or versatility
If non-relational database approaches like Resource Description Framework are used for semi-structured data, then flexibility in storing heterogeneous data improves, but the ability to use relational models and SQL querying is lost
Solution Approach 1:
The patent creates a universal data storage mechanism that combines the benefits of both relational and non-relational approaches. The flex table structure with variant identifiers provides multi-functionality, enabling the system to handle semi-structured data with the flexibility of non-relational databases while maintaining SQL compatibility and relational model capabilities. This universal approach allows a single system to perform both relational and flexible data operations.
Data Source
AI summary
Techniques related to flexible record definitions for semi-structured data in a relational database system are disclosed. The techniques include a machine readable medium, on which are stored instructions for accessing flexible record definitions for efficient database storage, comprising instructions that when executed cause a device to: receive a first request to create a first row in a database table, the first request indicating a first set of columns associated with the first row, determine that columns of the first set of columns are included in a second set of columns associated with the database table, select a first variant, from a set of variants, the first variant associated with a third set of columns, the third set of columns including at least the first set of columns, and write the first row to the database table based on the first variant and the third set of columns.