Entity Snapshot Partitioning for Memory-Efficient Historical Analytics
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing data analytics systems face challenges in efficiently managing large sets of entity data with varying update frequencies, leading to computationally intensive queries due to the need to reconstruct entire entity states from log files, which increases computational load while saving space.
Innovation Solution
The system partitions entity properties into partial historical snapshots and combines them on demand to execute queries, minimizing data replication and optimizing memory usage.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If the entire entity state is logged every time one state value changes, then complete entity state information is available, but storage space is wasted due to redundant logging of unchanged values
Solution Approach 1:
The patent segments the entity state into multiple partial snapshots, where each snapshot contains only the properties that have changed since the previous snapshot. This segmentation allows the system to store only the necessary changed values rather than logging the entire entity state, thus reducing storage space while maintaining information completeness.
2Quantity of substance
If only updated values and timestamps are logged, then storage space is optimized, but computational intensity increases when reconstructing entity state for queries
Solution Approach 1:
The patent applies preliminary action by pre-computing and storing partial snapshots that capture only the changed properties at each update point. When a query is executed, the system can efficiently reconstruct the entity state by combining only the relevant partial snapshots rather than processing the entire log file, thus reducing computational intensity while maintaining storage efficiency.
3Measurement precision
If the log file is processed to find the latest updated value for every entity property, then accurate entity state is obtained, but query execution time increases
Solution Approach 1:
The patent extracts only the necessary information from the log file by organizing data into partial snapshots that contain only the changed properties. When executing a query, the system extracts and combines only the relevant partial snapshots needed to reconstruct the entity state at the requested time point, rather than processing the entire log file, thus reducing query execution time while maintaining accuracy.
Data Source
AI summary
Embodiments relate to improving efficiency of data analytics performed on sets of entity data in which different entity properties having very different update frequencies. Time-based analytical queries track the entity states at each moment within a given time window. Analytical queries are executed over a massive number of entity states while using a reasonable memory footprint. The technique partitions the entity properties into partial historical snapshots of data and combines the partial snapshots on demand only as needed to execute analytical queries over business entities. A complete entity state having values for all entity properties is not required to execute most queries. Only partial snapshots including values referenced by the query need to be combined to satisfy the query. Using partial snapshots minimizes data replication, and the snapshots can be efficiently combined into entity states sufficient for query execution.