Distributed Relational Dictionaries for Database Encoding
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing database systems face inefficiencies in generating and managing global dictionaries, leading to high computational and memory overhead, latency, and network resource taxation, particularly in distributed environments, due to the lack of viable mechanisms for decentralized dictionary generation and the 'one-size-fits-all' approach of hard-coded dictionary generation processes.
Innovation Solution
Modifying the query optimizer in a distributed database system to generate a query execution plan for efficient global dictionary generation and encoding, utilizing a relational dictionary implemented as a database table that can serve as both an encoding and decoding dictionary, allowing for optimal sequence selection based on column characteristics, and distributing dictionary generation across nodes to reduce latency and network communication.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If a centralized database system generates global dictionaries, then dictionary consistency is maintained, but computational overhead and memory usage increase significantly
Solution Approach 1:
The patent divides the centralized dictionary generation process into distributed segments across multiple nodes. Each node generates local dictionaries independently, and these are later merged into global dictionaries. This segmentation reduces the computational burden on any single node while maintaining overall system consistency through coordinated merging operations.
Solution Approach 2:
The patent transitions from a single-dimensional centralized generation approach to a multi-dimensional distributed approach. Multiple nodes simultaneously generate dictionaries in parallel, adding a spatial dimension (across nodes) to the generation process. This dimensional change enables load distribution and reduced computational overhead per node.
2Reliability
If dictionary generation is performed centrally, then global consistency is achieved, but network resources are heavily taxed
Solution Approach 1:
The patent segments the dictionary generation workload across multiple nodes, allowing local dictionary creation without centralized coordination for every operation. This reduces network traffic by eliminating the need for constant central-node communication, while periodic merging operations maintain global consistency.
Solution Approach 2:
Each node performs self-service by independently generating its own local dictionaries based on its local data. This eliminates the need for network communication during local dictionary generation, reducing network resource consumption. Nodes only communicate during merging operations to achieve global consistency.
3Ease of manufacture
If hard-coded dictionary generation processes are used, then implementation is straightforward, but adaptability to different column characteristics is poor
Solution Approach 1:
The patent replaces static hard-coded generation processes with dynamic, adaptive processes that respond to column characteristics. The system automatically adjusts generation parameters based on the specific properties of each column, enabling adaptability while maintaining ease of implementation through automated decision-making.
Solution Approach 2:
The patent modifies generation parameters dynamically based on column characteristics such as data type, cardinality, and distribution patterns. By changing parameters like dictionary size, compression level, and generation algorithm selection based on measured column properties, the system achieves adaptability without complex hard-coded logic for each scenario.
4Device complexity
If local dictionaries are used instead of global dictionaries, then memory overhead is reduced, but query execution efficiency decreases
Solution Approach 1:
The patent merges local dictionaries from multiple nodes into global dictionaries that are shared across the distributed system. This combining approach reduces total memory overhead compared to maintaining separate full dictionaries at each node, while still enabling efficient query execution through the use of these consolidated global dictionaries for operations requiring cross-node data access.
Data Source
AI summary
Techniques related to distributed relational dictionaries are disclosed. In some embodiments, one or more non-transitory storage media store a sequence of instructions which, when executed by one or more computing devices, cause performance of a method. The method involves generating, by a query optimizer at a distributed database system (DDS), a query execution plan (QEP) for generating a code dictionary and a column of encoded database data. The QEP specifies a sequence of operations for generating the code dictionary. The code dictionary is a database table. The method further involves receiving, at the DDS, a column of unencoded database data from a data source that is external to the DDS. The DDS generates the code dictionary according to the QEP. Furthermore, based on joining the column of unencoded database data with the code dictionary, the DDS generates the column of encoded database data according to the QEP.