Probabilistic Data Structures for Browser Big Data Analytics
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current big data analytics systems in cloud computing environments face substantial storage costs and query serving times due to the need to store and process large volumes of data, leading to operational inefficiencies and increased maintenance burdens.
Innovation Solution
Implementing a probabilistic data structure, such as a sketch, on a data analysis server to represent big data streams, which allows for efficient storage and real-time updates, combined with WebAssembly modules and WebSocket protocols for secure and approximate data analytics, reducing the need for extensive infrastructure and improving query performance.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If traditional columnar storage or distributed frameworks like MapReduce are used to perform big data analytics, then data can be stored and processed, but storage costs increase substantially and query serving time becomes substantial
Solution Approach 1:
The patent extracts only the essential data characteristics needed for analytics by using probabilistic data structures (sketches) that capture statistical properties of big data streams without storing complete records. This extraction approach retrieves only necessary information patterns, enabling fast query serving while maintaining analytics capability.
Solution Approach 2:
The patent changes the data representation parameter from storing complete data records to storing compressed probabilistic sketches that approximate data distributions. This parameter change enables O(1) query time while maintaining sufficient analytics accuracy through controlled approximation.
2Reliability
If complete data records are stored for long-term analytics, then comprehensive analysis is possible, but storage costs increase substantially
Solution Approach 1:
The patent extracts essential statistical properties from complete data records using probabilistic sketches, storing only these extracted characteristics rather than full records. This reduces storage requirements from storing 100 million complete records to storing compact sketch representations that preserve analytics accuracy.
Solution Approach 2:
The patent uses approximate probabilistic data structures that trade exactness for efficiency, accepting controlled approximation errors in exchange for dramatically reduced storage costs and faster query performance.
3Productivity
If distributed frameworks like MapReduce are deployed on cloud infrastructure, then big data processing capability is achieved, but system complexity and maintenance overhead increase
Solution Approach 1:
The patent extracts the core analytics function from complex distributed frameworks by using simplified probabilistic data structures that can be updated and queried with basic operations, eliminating the need for complex MapReduce jobs and distributed coordination.
Solution Approach 2:
The probabilistic data structures are self-contained and can be updated incrementally with simple operations, eliminating the need for complex distributed task management and coordination that characterizes frameworks like MapReduce.
4Measurement precision
If traditional database engines process queries across complete storage, then accurate results are obtained, but query execution becomes time-consuming
Solution Approach 1:
The patent changes the fundamental parameter of data representation from complete records to probabilistic sketches, enabling O(1) query execution time while maintaining controlled approximation accuracy through mathematical guarantees of the sketch structures.
Data Source
AI summary
Some embodiments may be associated with a cloud computing environment. A computer processor of a data analysis server may receive a stream of big data associated with the cloud computing environment. The data analysis server may then create a probabilistic data structure (e.g., a sketch) representing the received stream of big data and store the created probabilistic data structure into the probabilistic data structure repository. A request for data analytics is then received from a remote client browser. Responsive to the received request, the data analysis server may retrieve the probabilistic data structure from the probabilistic data structure repository and push a WebAssembly (“Wasm”) module and the retrieved probabilistic data structure to the client browser. According to some embodiments, the data analysis server may then receive new updates to the stream of big data and send delta changes to the client browser via a WebSocket protocol.