Schema Object Creation via Write-Node Offloading for Heavy DDL
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Computational bottlenecks and inefficiencies arise when data definition language (DDL) operations, which may include data manipulation language (DML) operations, are performed by resource-light computing devices not equipped to handle DML operations efficiently, leading to starvation and system crashes.
Innovation Solution
A resource-efficient schema object creation system (SCS) offloads computationally intensive DML operations from a coordinator node to resource-rich write nodes, performing enhanced DDL processing by distinguishing between normal and heavy DDL operations, allowing efficient resource utilization and reducing computational burden on the coordinator.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of energy
If a resource-light computing device is used to perform DDL operations, then resource consumption is optimized, but computational bottlenecks and starvation occur when DDL operations include DML operations
Solution Approach 1:
The system segments DDL operations into two categories: normal DDL operations that are computationally lightweight and heavy DDL operations that include embedded DML operations. Normal DDL operations are executed by the coordinator node, while heavy DDL operations are offloaded to write nodes, enabling efficient resource utilization without computational bottlenecks
Solution Approach 2:
The system introduces write nodes as intermediary components that assist the coordinator node in executing heavy DDL operations. Write nodes receive heavy DDL operations from the coordinator, execute the embedded DML operations, and return results, thereby mediating the computational burden and preventing starvation
2Productivity
If a resource-rich computing device is used to perform both DDL and DML operations, then computational capability is sufficient, but resource consumption increases
Solution Approach 1:
The system segments computational tasks by assigning different types of operations to different nodes: the coordinator node handles normal DDL operations, while write nodes handle heavy DDL operations with embedded DML operations. This segmentation allows each node to operate within its resource constraints while maintaining overall system productivity
Solution Approach 2:
Different nodes in the system are assigned different computational capabilities and roles based on their local characteristics. The coordinator node is optimized for coordinating and executing normal DDL operations, while write nodes are optimized for executing heavy DDL operations and embedded DML operations, creating a heterogeneous system that optimizes resource consumption
3Device complexity
If DDL operations with embedded DML operations are executed by the coordinator node, then operation simplicity is maintained, but computational burden increases leading to system crashes
Solution Approach 1:
Write nodes serve as intermediaries that receive heavy DDL operations from the coordinator node, execute the embedded DML operations, and return results. This intermediary approach maintains the simplicity of the coordinator node's operation processing while preventing system crashes by distributing computational burden
Solution Approach 2:
The system extracts the computationally intensive DML operations from the coordinator node and assigns them to write nodes. By taking out the heavy computational tasks, the coordinator node maintains operational simplicity while the write nodes handle the computational burden that prevents system crashes
Data Source
AI summary
System, method, and various embodiments for a resource efficient schema object creation system are described herein. An embodiment operates by receiving a schema-change request at a first node of a cluster of computing nodes. The first node creates and populates a schema object in a shared storage, based on the schema-change request. An entry is generated in a reverse transaction log, the entry corresponding to the schema object stored in the shared storage device. A coordinator node is configured to read the entry from the reverse transaction log, update a schema, and generate an entry in a forward transaction log. The first node executes the entry in the forward transaction log, wherein the executing comprises updating a catalog of the first node to include the schema object stored in the shared storage, wherein upon a completion of the updating, the schema object is accessible by the first node.