Key-MiniTable Data Storage for Agile Big Data Querying
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional data management systems struggle to support both horizontal scaling and agile development, especially in the context of Big Data and rapid product lifecycles, as existing techniques like Key-Value and Key-Row Stores lack the complexity to handle complex logic and SQL compatibility hinders horizontal scaling.
Innovation Solution
A data storage and query method that transforms data into 2-tuples (Key, MiniTable) with pre-defined structures, allowing for analogous SQL grammar support while maintaining horizontal scaling through a Key-MiniTable mode with specific primitives and distributed storage, utilizing a Key server and MiniTable servers for request handling and execution.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If conventional Key-Value or Key-Row Store techniques are used for horizontal scaling, then data storage capacity can be expanded, but the system cannot support complex queries and agile development
Solution Approach 1:
The patent segments the data structure into Keys and MiniTables, where each MiniTable contains multiple data columns with pre-defined modes. This segmentation allows the system to maintain simple horizontal scaling through Key distribution while enabling complex queries within each MiniTable through SQL-like syntax support, thus resolving the contradiction between storage capacity expansion and query complexity support
Solution Approach 2:
The patent implements a nested structure where MiniTables are nested within Keys, and each MiniTable contains multiple data columns. This nesting allows the system to combine the simplicity of Key-Value storage at the outer level with the complexity of relational data structures at the inner level, enabling both horizontal scaling and complex query support simultaneously
2Adaptability or versatility
If SQL semantics are used to support agile development, then complex queries can be executed, but horizontal scaling becomes impossible due to mutual dependency among data
Solution Approach 1:
The patent segments data into independent Key-MiniTable units where each unit can be stored and retrieved independently. This segmentation eliminates mutual dependency among data while preserving SQL-like query capabilities within each unit, allowing the system to support complex queries without compromising horizontal scaling ability
Solution Approach 2:
The patent applies local quality by allowing SQL-like complex queries within each MiniTable context while maintaining simple Key-based distribution at the global level. This local complexity with global simplicity enables the system to provide rich query capabilities locally without affecting the overall horizontal scaling architecture
3Quantity of substance
If simple primitives are provided for horizontal scaling, then data storage can be scaled, but the system cannot rapidly construct applications with complex logic
Solution Approach 1:
The patent implements preliminary action by pre-defining data column modes and structures within MiniTables before data insertion. This pre-configuration enables rapid application development with complex logic while maintaining horizontal scalability, as the structural framework is established in advance and does not require complex operations during runtime
Data Source
AI summary
A data storage and query method supporting an agile development and horizontal scaling, in a field of computer applications, includes steps of: transforming data into a 2-tuple (Key, MiniTable) and identifying any MiniTable (having a data set removed) with a unique Key, wherein the MiniTable includes multiple data columns; a structure of each data column is limited by a pre-defined mode; and the data columns are sorted according to a primary key defined by the mode. The data storage and query method obtains a capability of the horizontal scaling via removing dependency among the data sets, and a capability of the agile development via supporting an analogous SQL grammar by saving dependency among the data within the data sets. The data storage and query method can be applied to a data management system which supports the horizontal scaling of big data and the agile development of short development lifecycles.