Database Engine Shard Local ID Parallel Processing
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing database query methodologies face challenges in reducing response time and improving resource efficiency, particularly in large enterprise systems with complex data, leading to sub-optimal usage of memory and processing power.
Innovation Solution
The implementation of a database engine with an acceleration layer that includes a query processing system using shards, thread pools, and Local IDs to optimize query execution, allowing for parallelization of transactions while maintaining ACID properties and minimizing locks, and utilizing Reactive Operators and index structures for efficient data retrieval.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of time
If existing database query methodologies are used, then data can be stored and retrieved, but query response time becomes unacceptably long in large enterprise systems
Solution Approach 1:
The database is divided into multiple shards distributed across different slave machines. Each shard contains a partition of the data, allowing queries to be processed in parallel across multiple nodes. This segmentation enables the system to handle large datasets by distributing the query load, thereby reducing overall query response time while maintaining high productivity.
Solution Approach 2:
The patent introduces Local IDs as an additional dimension for data identification within each shard. Instead of relying solely on Global IDs, the system uses Local IDs that are specific to each shard's context, enabling more efficient local lookups and reducing the complexity of cross-shard queries. This dimensional addition optimizes data retrieval efficiency without sacrificing response time.
2Speed
If query processing is accelerated using parallelization, then response time is reduced, but system complexity increases
Solution Approach 1:
The Master machine acts as an intermediary that coordinates queries between client applications and slave machines. It receives queries, determines the appropriate shards and slave machines, and manages the distribution of query tasks. This intermediary layer simplifies the overall system architecture by centralizing coordination logic, allowing parallel processing on slave machines without requiring complex peer-to-peer communication protocols.
Solution Approach 2:
Each slave machine independently processes assigned query tasks using its local shard data and thread pool resources. The slaves autonomously execute queries without requiring continuous coordination with other slaves, reducing inter-node communication overhead and simplifying the distributed system architecture while maintaining high processing speeds.
3Adaptability or versatility
If more processing resources are allocated to handle complex queries, then query capability improves, but resource usage efficiency decreases
Solution Approach 1:
The system dynamically allocates thread pool resources on each slave machine based on query workload and availability. Thread pools are created and managed adaptively, allowing the system to scale processing capacity up or down according to actual demand. This dynamic resource management maintains high query handling capability while optimizing processing resource efficiency by avoiding static over-provisioning.
Solution Approach 2:
The patent changes the parameter of resource allocation from fixed to variable by implementing thread pools that can be dynamically created, scaled, and terminated. This parameter change allows the system to adapt resource usage to query complexity and volume, maintaining versatile query handling capability while improving processing efficiency through demand-based resource allocation.
Data Source
AI summary
In response to a query to a database, the identity of a slave machine which stores the requested data is resolved, and the query is parsed to identify one or more shards at the slave machine. Each shard corresponds to a partition of the requested data stored at the slave, and rows inside of the shard are respectively assigned a fixed-size local ID. Tasks corresponding to each shard are placed into a thread pool, and the threads are processed using the respective local IDs to obtain the requested data.