Distributed Shared Log Storage Adapter for Heterogeneous Big Data
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Cloud computing providers face inefficiencies and management challenges due to the need for different clusters to support various big data paradigms like Hadoop, SQL, and Kafka, which leads to inefficient resource utilization and complexity in handling diverse workloads.
Innovation Solution
A distributed shared log storage system, such as Corfu, partitions the namespace across log unit servers by time rather than namespace, allowing for extreme scaling and richer ordering guarantees while using a central sequencer to prevent concurrent writes and employing an adapter layer to translate different APIs into Corfu APIs, enabling support for heterogeneous big data applications.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If different clusters are dedicated to different big data paradigms (Hadoop, SQL, Kafka), then each paradigm can operate with optimized performance, but resource utilization becomes inefficient and management complexity increases
Solution Approach 1:
The patent implements a universal storage cluster that can serve multiple big data paradigms (Hadoop, SQL, Kafka) simultaneously through a unified namespace and adapter layer. The shared log storage system provides a common infrastructure that adapts to different paradigms without requiring separate dedicated clusters, thereby improving resource utilization while maintaining paradigm-specific performance requirements.
2Reliability
If different clusters are dedicated to different big data paradigms, then each paradigm operates independently, but the number of clusters to manage increases significantly
Solution Approach 1:
The patent merges multiple paradigm-specific clusters into a single shared storage cluster with a unified namespace. By combining Hadoop, SQL, and Kafka workloads into one infrastructure with centralized management, the system reduces the total number of clusters while maintaining the independence and reliability of each paradigm through virtualization and adapter layers.
3Ease of operation
If namespace is partitioned by namespace across storage units, then data access is organized by application type, but scaling is limited and ordering guarantees are weaker
Solution Approach 1:
The patent inverts the traditional namespace partitioning approach by partitioning the namespace across storage units in time rather than by namespace. This temporal partitioning enables extreme scaling capabilities and richer ordering guarantees while still allowing organized data access through the unified namespace, resolving the contradiction between access organization and scaling capability.
4Productivity
If multiple clients can concurrently write to different partitions, then throughput increases, but maintaining data reliability and ordering becomes more difficult
Solution Approach 1:
The patent introduces a central sequencer as an intermediary component that coordinates concurrent writes from multiple clients across different partitions. The sequencer maintains global ordering guarantees and ensures data reliability by sequencing operations atomically, allowing high concurrent throughput while preventing ordering conflicts that would arise from uncoordinated parallel writes.
Data Source
AI summary
A distributed shared log storage system employs an adapter that translates Application Programming Interfaces (APIs) for a big data application to APIs of the distributed shared log storage system. An instance of an adapter is configured for different big data applications in accordance with a profile thereof, so that the big data applications can take on a variety of added characteristics to enhance the application and/or to improve the performance of the application. Included in the added characteristics are global or local ordering of operations, replication of operations according to different replication models, making the operations atomic and caching.