Hyperscale Data Analytics via Binary Record Storage
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current data analytics systems face inefficiencies in storing and processing large-scale, high-velocity datasets due to limitations in legacy file formats, leading to data loss, increased complexity, and latency, which hampers real-time analysis and correlation of hyperscale data streams.
Innovation Solution
A High-Performance Data Analytics (HPDA) system utilizing a Record Storage Allocation System (RSAS) format that stores data as non-serialized binary sequences across block storage devices without file constructs, enabling efficient distribution, aggregation, and querying of hyperscale datasets through a microservices architecture, using bitmask filters and remote direct memory access (RDMA) for optimized data processing.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If legacy standardized file formats are used for data storage, then data transportability and sharing are improved, but processing efficiency and latency are worsened
Solution Approach 1:
The patent segments the data processing architecture by introducing a dedicated ingestion layer that operates independently from storage and query layers. This segmentation allows each layer to be optimized separately - the ingestion layer for speed using columnar formats, storage for efficiency using binary formats, and queries for flexibility - thereby resolving the contradiction between standardized transportability and processing efficiency.
Solution Approach 2:
The patent changes the fundamental parameter of data representation by transitioning from row-based to columnar storage formats, and further to binary serialized formats. This parameter change enables dramatically improved processing efficiency and reduced latency while maintaining data accessibility through the layered architecture that preserves standard interfaces.
2Ease of operation
If data is stored in generic file formats, then shared resource access is improved, but processing performance and data fidelity are worsened
Solution Approach 1:
The patent introduces an intermediary ingestion layer that acts as a mediator between data sources and storage systems. This intermediary layer handles the conversion to optimized binary formats and manages the trade-off between shared access capability and processing performance, allowing both requirements to be satisfied simultaneously through the layered architecture.
3Quantity of substance
If distributed computing resources are scaled to handle large datasets, then data capacity is improved, but system complexity and synchronization overhead are worsened
Solution Approach 1:
The patent segments the distributed system into independent, loosely-coupled layers (ingestion, storage, query) that can be scaled independently. This segmentation reduces system complexity by eliminating tight synchronization requirements between components, allowing each layer to process and manage data at its own pace while maintaining overall system coherence through standardized interfaces.
Data Source
AI summary
Systems and methods for storing and querying voluminous “big data” are described, with application to append-only hyperscale databases. The methods dispense with blob abstractions such as objects and file systems, instead storing data as addressable binary sequences at all stages of storage and querying. In particular, a microservices architecture system arrangement combines with a powerful data format to create an architecture for executing an external query using a distributed process.