Distributed Shared Log Storage Adapter for Heterogeneous Big Data
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Cloud computing providers face inefficiencies and management challenges due to the need for different clusters to support various big data paradigms like Hadoop, SQL, and Kafka, leading to inefficient resource utilization and difficulty in handling diverse workloads.
Innovation Solution
A distributed shared log storage system, such as Corfu, partitions the namespace across log unit servers by time rather than namespace, allowing for extreme scaling and richer ordering guarantees while using a central sequencer to prevent concurrent writes and employing adapters to translate APIs across different paradigms, enabling support for heterogeneous big data applications.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If different clusters are dedicated to different big data paradigms (Hadoop, SQL, Kafka), then each paradigm can operate with optimized performance, but resource utilization becomes inefficient and cluster management becomes complex
Solution Approach 1:
The patent merges multiple big data paradigms (Hadoop, SQL, Kafka) into a single unified cluster by implementing a common log-based storage layer that all paradigms can access. This eliminates the need for separate dedicated clusters while maintaining paradigm-specific performance characteristics through adaptive query execution plans.
Solution Approach 2:
The unified cluster is designed with multi-functionality to serve multiple big data paradigms simultaneously. The system provides universal access to the log storage layer through different API interfaces, allowing Hadoop, SQL, and Kafka workloads to coexist and share resources within the same cluster infrastructure.
2Reliability
If different clusters are dedicated to different big data paradigms, then each paradigm has dedicated resources, but resource utilization becomes inefficient
Solution Approach 1:
By combining multiple paradigms into one cluster with a shared log storage layer, the system enables efficient resource utilization through workload consolidation while maintaining performance guarantees for each paradigm through isolated query execution contexts and adaptive planning.
Solution Approach 2:
The system dynamically adjusts execution parameters and resource allocation based on the specific paradigm and workload characteristics. Query execution plans are adapted according to the paradigm type (Hadoop, SQL, or Kafka), allowing optimal resource utilization while maintaining paradigm-specific performance requirements.
3Device complexity
If a unified cluster supports multiple big data paradigms, then resource utilization improves and management simplifies, but the system must handle heterogeneous workloads with different access patterns and requirements
Solution Approach 1:
The patent introduces an intermediary layer consisting of a common log storage interface and adaptive query execution engine that mediates between different big data paradigms and the underlying storage system. This intermediary handles the heterogeneity of workloads while presenting a unified interface to the cluster management layer, simplifying operations.
Solution Approach 2:
The system dynamically changes execution parameters and access patterns based on the incoming workload type. The adaptive query execution engine modifies execution plans according to whether the workload is Hadoop-style batch processing, SQL-style analytical queries, or Kafka-style stream processing, thereby handling heterogeneous workloads effectively within a unified cluster.
Data Source
AI summary
A distributed shared log storage system employs an adapter that translates APIs for a big data application to APIs of the distributed shared log storage system. The adapter is configured for different big data applications in accordance with a profile thereof, so that storage performance using the distributed shared log storage system can be comparable to the storage performance of the profiled big data application. An over-utilized adapter instance is detected and the workload assigned to the over-utilized adapter instance is either moved to a different adapter instance that can handle the workload or split among two or more adapter instances.