Distributed Data Sketching for Large Dataset Storage and Querying
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing technologies struggle to efficiently manage and analyze large datasets generated by networked computing systems, particularly in distributed environments, due to the volume and dispersed nature of the data, which makes querying and analysis infeasible and prone to erroneous analytics.
Innovation Solution
A system and process for compressing operation data using multidimensional data sketches that preserve statistical properties and trends, allowing for compact storage and efficient analysis across multiple devices, including operation servers, index servers, and aggregation servers, utilizing coloring techniques to reduce data size while maintaining mergeability and statistical integrity.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If data is stored in distributed servers without compression, then data completeness is maintained, but storage space requirements and data transmission volume increase significantly
Solution Approach 1:
The patent extracts only the essential statistical properties and trends from the complete dataset using sketching algorithms. Instead of storing all raw operation data, the system extracts key statistical characteristics (such as quantiles, averages, and distribution patterns) that preserve analytical value while occupying minimal storage space. This extraction process resolves the contradiction by maintaining information quality without requiring complete data retention.
Solution Approach 2:
The patent transforms the data representation from storing complete raw values to storing compressed statistical parameters and sketches. By changing the parameter representation method - using probabilistic data structures and aggregated statistics instead of individual data points - the system achieves significant space reduction while preserving the ability to perform meaningful analysis on the distributed data.
2Quantity of substance
If data is compressed using traditional methods, then storage space is reduced, but analytical accuracy and trend preservation deteriorate
Solution Approach 1:
The patent introduces data sketches and statistical intermediaries as mediating structures between the raw distributed data and the analysis queries. These sketches act as intermediaries that capture essential data characteristics while occupying minimal space. When analysis is needed, the system queries these intermediate sketches rather than processing raw data, thereby maintaining analytical accuracy despite significant compression. The intermediary structures preserve trend information and statistical properties that traditional compression methods would lose.
3Ease of operation
If all operation data is collected and stored centrally, then complete analysis is possible, but network bandwidth consumption and system complexity increase
Solution Approach 1:
The patent segments the data processing and storage function across multiple distributed servers rather than consolidating everything centrally. Each server maintains local sketches of its operation data, and these sketches can be independently queried and analyzed. This segmentation allows analysis capabilities to be distributed throughout the system, reducing network bandwidth requirements while maintaining ease of operation through localized processing. The system architecture becomes simpler because each node operates independently with its own compressed data representation.
Data Source
AI summary
A plurality of operation-servers are configured to perform operations that produce operation-data having at least two dimensions of information. A second plurality of index-servers are configured to: maintain a datastore; receive a stream of the incoming operation-data; determine if the ingestion buffer contains sufficient incoming operation-data to be moved to a storage layer as a databank; move the ingestion buffer to a storage layer as a databank; determine if a given storage layer contains sufficient databanks to be compacted and moved to a higher layer; combine each databank in the given layer into a databank-union; identify pairs of operation-data in the databank-union based on a multidimensional distance measure between operation-data of the databank-union; color a first operation-data of the pair a first color; color a second operation-data of the pair a second color; create in a higher storage layer, a new databank.