Selective Data Exposure via Summarization Scope Locks
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing data consumption methods face inefficiencies, particularly in handling varying data volumes, as they often require manual optimizations and struggle with concurrent operations between bulk and incremental data processing, leading to suboptimal performance and resource management.
Innovation Solution
A method that determines the scope of summarization, obtains locks on data sets, and exposes data through a view object, allowing for bulk, incremental, and refresh summarization methods, ensuring efficient metadata-based consumption by selectively exposing data based on the summarization program's parameters and avoiding unnecessary join operations.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Device complexity
If a data consumer uses a single table for both bulk and incremental data, then device complexity is reduced, but productivity decreases due to forced determination of refresh scope and potential reset requirements
Solution Approach 1:
The patent segments the data storage structure into separate tables: a bulk data table for large volumes of historical data and an incremental data table for recent changes. This segmentation allows the data consumer to efficiently determine whether to perform a full bulk refresh or a selective incremental refresh based on data age and volume, eliminating the need to reset the entire data consumer and improving processing productivity without excessive structural complexity.
2Productivity
If a data consumer is designed to perform well on particular data volumes, then productivity is improved for specific scenarios, but adaptability worsens when data volumes vary significantly
Solution Approach 1:
The patent implements dynamic adaptability by enabling the data consumer to automatically select between bulk processing and incremental processing modes based on the actual data volume and characteristics. The system dynamically determines the appropriate refresh strategy (full reload vs. selective update) without requiring manual configuration or reset, allowing optimal performance across varying data volumes while maintaining a single unified data consumer structure.
3Adaptability or versatility
If manual optimizations are required for data consumption, then adaptability to specific scenarios is improved, but ease of operation worsens due to manual maintenance requirements
Solution Approach 1:
The patent implements self-service automation where the data consumer automatically determines whether to perform bulk or incremental data consumption based on metadata about the data source (such as data volume, age, and change frequency). The system autonomously selects the optimal processing mode, performs appropriate locks on data sets, and executes the correct refresh strategy without requiring manual intervention, configuration, or maintenance from operators, thereby maintaining high adaptability while significantly improving ease of operation.
Data Source
AI summary
A method for consuming data includes determining a scope of summarization for a data set in response to receiving a request from a summarization program, obtaining a lock on the data within the scope of summarization and consuming the data from the summarization program within the scope of data summarization. The summarization program includes methods configured to obtain concurrent locks on different portions of the data set. The consumption of the data may be governed by specific rules.