Data Snapshot Encoding for Large Scale Trend Analysis
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Data analysis systems face challenges in reducing execution time when processing large datasets spanning across long periods, leading to inefficiencies in tracking data changes and generating daily trends.
Innovation Solution
The system generates a second set of data containing encoded values for the status of items over subperiods, allowing for efficient tracking and presentation of statistical information, using techniques such as snapshot views, physical tables, or bitmap tables to reduce computational burden and memory overhead.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of time
If traditional data processing methods are used on large datasets spanning long periods, then complete data analysis is achieved, but execution time becomes excessively long
Solution Approach 1:
The patent divides a long time period into multiple subperiods and processes data from each subperiod separately. Instead of loading and processing the entire large dataset at once, the system segments the data by time intervals, generates snapshots for each subperiod, and combines them. This segmentation dramatically reduces the execution time for tracking data changes while maintaining complete analysis coverage.
2Measurement precision
If data is processed in detail to track changes over long periods, then accurate trend analysis is achieved, but computational burden increases
Solution Approach 1:
The patent performs preliminary processing by generating snapshot views for each subperiod before the final analysis. These snapshots pre-compute and store intermediate results, including encoded status values and change tracking information. When the final trend analysis is needed, the system simply combines these pre-prepared snapshots rather than reprocessing the entire dataset, significantly reducing computational burden while maintaining accuracy.
3Productivity
If complete datasets are loaded into memory for analysis, then comprehensive processing is achieved, but memory overhead becomes excessive
Solution Approach 1:
The patent extracts only the necessary data elements needed for trend analysis from the complete dataset. Instead of loading entire records into memory, the system extracts specific status changes and encodes them into compact representations. The snapshot views store only the essential information (status transitions, time periods, encoded values) required for analysis, dramatically reducing memory overhead while maintaining processing capability.
4Loss of information
If detailed status tracking is performed for each item over time, then comprehensive trend information is obtained, but device complexity increases
Solution Approach 1:
The patent transforms complex status tracking data into simplified encoded representations. Each status change is encoded into compact values that capture the essential information without requiring complex data structures. The snapshot views use parameter encoding to represent status transitions efficiently, reducing device complexity while preserving complete trend information through the encoded parameters.
Data Source
AI summary
A system may include a processor configured to receive a first set of data indicating a first period of time during which a item has a first status, and a second period of time during which the item has a second status. The processor may generate, based on the first set of data, a second set of data containing a plurality of logical rows, each indicating an encoded value of the first status of the item and an encoded value of the second status of the item during a respective period of time that is smaller than or equal to each of the first period of time and the second period of time. The processor may cause a display to present, based on the first set of data and the second set of data, statistical information of the first status and the second status during a third period of time.