Delimited Column Qualifier NoSQL Database Ingestion
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Relational databases face inefficiencies in scalability and storage due to hierarchical data structures, leading to increased storage needs for null characters and slower query performance, especially when scaling across multiple servers.
Innovation Solution
A method and system for transforming data from relational databases into a delimited column qualifier NoSQL database, using a reference table to define the data format and store only non-null data, reducing storage requirements and improving data access efficiency.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If data is stored in a relational database with hierarchical tables, then data structure integrity is maintained, but storage efficiency deteriorates due to null character storage and query complexity increases when scaling across multiple servers
Solution Approach 1:
The patent extracts only the non-null data values from the relational database structure and stores them in a flattened columnar format. By taking out only the meaningful data and eliminating the hierarchical table structure with its null character requirements, the system achieves superior storage efficiency while maintaining data integrity through the column qualifier encoding scheme
Solution Approach 2:
The patent changes the fundamental parameters of data representation by transitioning from a hierarchical table structure with fixed columns and rows to a flattened structure with column qualifiers that encode both data type and hierarchical position. This parameter change eliminates the need for null character storage while preserving data structure relationships through the encoding scheme
2Quantity of substance
If relational databases are horizontally scaled across multiple servers, then storage capacity increases, but query performance deteriorates due to the need to access and join data from many different servers
Solution Approach 1:
The patent segments data into independent columnar units with self-descriptive column qualifiers that encode hierarchical relationships. This segmentation allows each server to store and process specific data segments independently without requiring joins to other servers, as the column qualifiers contain all necessary information about data origin and relationships
Solution Approach 2:
The column qualifier acts as an intermediary that encodes hierarchical relationships and data type information. Instead of requiring actual joins between tables on different servers, the column qualifier provides all necessary contextual information locally, eliminating the need for complex inter-server data joining operations
3Speed
If a positional column qualifier is used to reference data in a non-relational database, then data access efficiency improves, but storage requirements increase due to the need for a byte for each data table
Solution Approach 1:
The column qualifier is designed as a universal encoding scheme that simultaneously provides multiple functions: it identifies the data type, encodes hierarchical position information, and references the original table structure. This multi-functionality eliminates the need for separate storage mechanisms for each data table, reducing overall storage requirements while maintaining efficient data access
Solution Approach 2:
The patent changes the column qualifier from a simple positional byte reference to an encoded structure that contains multiple pieces of information (data type, hierarchical position, table reference). This parameter change increases the information density of each qualifier byte, reducing the number of bytes needed per data element while improving data access efficiency
Data Source
AI summary
Methods and systems for ingesting data stored in a relational database into a non-relational delimited column qualifier NoSQL database are provided. According to certain aspects, an ingestion server may receive a request to transform the data stored in the relational database into a delimited column qualifier format. The ingestion server may then identify and query a reference table that defines the delimited column qualifier format. In accordance with the reference table, the ingestion server may transform the data into the delimited column qualifier format. Subsequently, the ingestion server may store the transformed data in the delimited column qualifier NoSQL database. By storing data in the column qualifier database, fewer computer resources are used to store and access the same data that is stored in the relational database. Accordingly, all the transformed data is stored in a single delimited CQ row.