Kodiak Distributed Database Parallel Table Loading
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current systems face challenges in efficiently managing and querying large volumes of user data for online advertising, particularly in handling big data applications where tens of terabytes are generated daily, requiring high performance and availability while supporting latency SLAs for front-end applications.
Innovation Solution
A table-level distributed database system, referred to as Kodiak, is introduced, which is highly available, self-healing, and horizontally scalable, supporting aggregation and join operations, and capable of asynchronously loading thousands of cache tables in parallel from a file system, improving loading performance by 60% compared to traditional relational database management systems.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If traditional relational database management systems are used to manage large volumes of user data, then data consistency and reliability are maintained, but data loading performance and handling capacity for big data applications deteriorate
Solution Approach 1:
The system segments data into distributed tables stored across multiple nodes in a distributed file system, allowing parallel loading and processing of large datasets. Each node independently manages its portion of data, enabling the system to handle big data volumes while maintaining loading performance through concurrent operations.
Solution Approach 2:
The patent transitions from a single-node relational database to a multi-node distributed database system, adding the dimension of distribution across multiple servers. This dimensional change enables horizontal scaling to handle larger data volumes while maintaining loading performance through parallel processing capabilities.
2Reliability
If data is stored in a distributed file system and loaded asynchronously, then data loading speed and system availability are improved, but data consistency and synchronization complexity worsen
Solution Approach 1:
The system implements feedback mechanisms where the loading interface monitors the status of data tables in the distributed file system and automatically manages loading operations. This feedback loop enables asynchronous loading while maintaining consistency by detecting when data is ready and coordinating loading across multiple nodes without manual intervention.
Solution Approach 2:
The distributed database system performs self-service by automatically managing its own data loading, distribution, and consistency operations. The system autonomously tracks data availability, determines loading priorities, and coordinates table loading across nodes without requiring external management, thereby reducing operational complexity while maintaining reliability.
3Adaptability or versatility
If the system supports complex queries and aggregation operations on large datasets, then query capability and analytical functionality are improved, but processing time and system resource consumption worsen
Solution Approach 1:
The system segments query processing across distributed nodes, where each node processes local data and returns results to coordinating nodes. This segmentation enables complex queries to be executed in parallel across the distributed file system, reducing overall processing time while maintaining the ability to handle sophisticated analytical operations on large datasets.
Solution Approach 2:
The system performs preliminary actions by pre-processing and indexing data in the distributed file system, allowing complex queries to be executed more efficiently. By preparing data structures and metadata in advance, the system reduces query processing time while maintaining comprehensive query capabilities for aggregation and analysis operations.
Data Source
AI summary
A method of managing on-line user data in a table-level distributed database system is disclosed. At a loading interface, a load request from a client device specifying a new or updated data table, which has been populated in a distributed file system, is received. The load request also specifies a location in the distributed file system and a schema of the data table. The loading interface creates new records in a metadata store, and each new record specifies a mapping between the data table and a particular instance that is to load such data table. Each record also specifies the location and schema for such data table. Based on the new records, each instance also loads the data table. Each instance publishes, in the metadata store, load status specifying that loading of the data table is complete so that such data table can be queried by a query client.