HTAP System Decoupling Row-Column Storage Formats
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing Hybrid Transaction/Analytical Processing (HTAP) systems face challenges such as interference between analytic and transactional workloads, compromised performance due to using a single data format for both, and high costs associated with extensive memory usage, which affects data consistency and scalability.
Innovation Solution
A modularized HTAP system that decouples major components, allowing for separate query processing engines and data formats (row-based for OLTP and columnar for OLAP), with a distributed delta store for real-time data changes and global snapshot isolation, enabling flexible scaling and strong data consistency.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Device complexity
If a single data format is used for both OLTP and OLAP workloads, then device complexity is reduced, but manufacturing precision deteriorates because OLTP requires row-based format while OLAP requires columnar format
Solution Approach 1:
The patent divides the storage system into separate OLTP storage and OLAP storage components, each optimized for its specific workload type. OLTP storage maintains row-based data formats while OLAP storage uses columnar formats, allowing each segment to operate at peak performance without compromising the other.
Solution Approach 2:
Different data formats are applied locally to different storage components based on their specific requirements. The OLTP portion uses row-based storage for efficient transactional operations, while the OLAP portion uses columnar storage for optimized analytical queries, ensuring each component has the quality characteristics it needs.
2Productivity
If extensive memory is used to handle both workloads, then productivity improves, but loss of substance increases due to high costs
Solution Approach 1:
The memory resources are segmented and allocated separately to OLTP and OLAP workloads. This allows each workload type to receive appropriate memory allocation optimized for its specific needs, improving overall productivity while preventing wasteful over-provisioning that would increase costs.
Solution Approach 2:
The storage system is designed to handle both OLTP and OLAP workloads within a unified architecture that shares common infrastructure components. This multi-functionality allows the system to serve dual purposes without requiring completely separate hardware installations, reducing overall resource requirements and costs.
3Adaptability or versatility
If OLAP and OLTP workloads are processed in the same system, then adaptability improves, but object-generated harmful factors increase due to workload interference
Solution Approach 1:
The system separates OLTP and OLAP workloads into distinct processing paths and storage components. This segmentation prevents workload interference by isolating the resource-intensive analytical queries from transactional operations, eliminating the harmful effects of mutual interference while maintaining the ability to handle both workload types.
Solution Approach 2:
The patent introduces an intermediary layer that manages data flow and resource allocation between OLTP and OLAP components. This mediator coordinates access to shared resources, prevents conflicts, and ensures that neither workload type negatively impacts the other, thereby eliminating interference while preserving system versatility.
Data Source
AI summary
The present disclosure describes hybrid transactional and analytical processing (HTAP) techniques. A HTAP system comprises a first processing engine configured to perform online transactional processing, a second processing engine configured to perform online analytical processing, and a storage in communication with the first processing engine and the second processing engine. The first processing engine, the second processing engine, and the storage may be modularized and configured to be decoupled from each other. The system may be configured to capture data by the first processing engine in real time, organize the data in a first format in a first part of the storage for use by the first processing engine, propagate the data to a second part of the storage subsystem, and organize the data in a second format in the second part of the storage for use by the second processing engine.