Resource-Efficient Schema Object Creation via DDL-DML Workload Segmentation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Computational bottlenecks and inefficiencies arise when data definition language (DDL) operations, which may include data manipulation language (DML) operations, are performed by resource-light computing devices not equipped to handle DML operations efficiently, leading to starvation and system crashes.
Innovation Solution
A resource-efficient schema object creation system (SCS) offloads computationally intensive DML operations from a coordinator node to resource-rich write nodes, which perform DML operations efficiently, while the coordinator handles lighter DDL operations, ensuring data consistency through a coordinated update of the schema and data using forward and reverse transaction logs.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of energy
If a resource-light computing device is used to perform DDL operations, then resource consumption is optimized, but computational bottlenecks and starvation occur when DDL operations include DML operations
Solution Approach 1:
The system segments DDL operations into two types: lightweight DDL operations that are executed by the coordinator node, and heavy DDL operations that include DML operations and are executed by write nodes. This segmentation allows each node type to perform operations appropriate to its computational capacity, preventing the coordinator from being overwhelmed while maintaining efficient resource utilization.
Solution Approach 2:
The system introduces write nodes as intermediary components between the coordinator node and the data storage layer. Write nodes receive heavy DDL operations from the coordinator, execute the computationally intensive DML operations, and then report back to the coordinator. This intermediary structure offloads computational burden from the coordinator while maintaining centralized coordination.
2Adaptability or versatility
If a resource-light computing device performs DDL operations with DML operations, then device utilization is maximized, but computational bottlenecks and starvation arise
Solution Approach 1:
The system segments DDL operations by weight and assigns them to appropriate node types. Lightweight DDL operations go to the coordinator node, while heavy DDL operations with DML components are routed to write nodes. This segmentation ensures that no single node is overloaded, maintaining system stability while keeping all nodes utilized.
Solution Approach 2:
Different nodes in the system are given different functional qualities matched to their computational capabilities. The coordinator node is optimized for coordinating lightweight operations, while write nodes are optimized for executing heavy operations. This local quality differentiation ensures each node operates within its capabilities, preventing starvation and maintaining reliability.
3Adaptability or versatility
If DML operations are included in DDL operations, then schema object creation is enhanced, but processing time increases
Solution Approach 1:
The system segments DDL operations into lightweight and heavy categories based on whether they include DML operations. Lightweight DDL operations are executed quickly by the coordinator node, while heavy DDL operations with DML components are executed by write nodes with appropriate computational resources. This segmentation minimizes overall processing time by ensuring each operation type receives appropriate computational power.
Solution Approach 2:
Write nodes serve as intermediaries that handle heavy DDL operations with DML components. These write nodes receive the heavy DDL operations from the coordinator, execute the time-consuming DML operations, and return results to the coordinator. This intermediary approach ensures that schema object creation with DML operations is handled efficiently without delaying lightweight operations.
Data Source
AI summary
System, method, and various embodiments for a resource efficient schema object creation system are described herein. An embodiment operates by receiving a schema-change request at a first node of a cluster of computing nodes. The first node creates and populates a schema object in a shared storage, based on the schema-change request. An entry is generated in a reverse transaction log, the entry corresponding to the schema object stored in the shared storage device. A coordinator node is configured to read the entry from the reverse transaction log, update a schema, and generate an entry in a forward transaction log. The first node executes the entry in the forward transaction log, wherein the executing comprises updating a catalog of the first node to include the schema object stored in the shared storage, wherein upon a completion of the updating, the schema object is accessible by the first node.