Cache Management for Data Processing Workflows
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current data processing workflows require significant time to process large amounts of data, and existing caching methods necessitate a large cache area, making efficient data caching in limited spaces challenging.
Innovation Solution
A computer system with a storage device and processor that manages cache data by storing run-time logs and management information to determine if cache data for input subsets is present, allowing for efficient cache data reuse and reduced processing time.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Speed
If a large cache area is used to cache all possible data, then data processing speed is improved, but the cache area requirement increases
Solution Approach 1:
The patent segments the cache management into multiple components: run-time logs that record input-output relationships, management information that tracks data unit mappings, and a determination mechanism that checks for cache hits. This segmentation allows the system to efficiently manage cache data without requiring a large cache area, as it only caches and retrieves specific data subsets based on run-time logging.
Solution Approach 2:
The system performs preliminary actions by logging run-time information during data processing workflows. It records the relationships between input subsets and output data units in advance, creating management information that enables future cache determinations. This preliminary logging allows the system to quickly determine whether cache data exists without requiring exhaustive cache storage.
2Measurement precision
If data is processed using a workflow with multiple steps, then analysis accuracy is improved, but processing time increases
Solution Approach 1:
The patent implements feedback mechanisms through run-time logs that record the execution results of each step in the data processing workflow. These logs provide feedback information about input-output relationships, which is stored as management information. This feedback enables the system to determine whether subsequent processing can reuse cached results, thereby reducing processing time while maintaining analysis accuracy through the complete multi-step workflow.
Solution Approach 2:
The system creates copies of run-time log information and management information that document the relationships between input data subsets and output data units. These copied records allow the system to determine cache hits without reprocessing data, effectively copying the results of expensive multi-step workflows and avoiding redundant processing while maintaining analytical accuracy.
Data Source
AI summary
A computer system acquires information on a first present input subset selected from first present input data for a first step from a run-time log of the first step, determines whether or not first cache data corresponding to the first present input subset for the first step is present in a cache area with reference to management information, and determines the first cache data as present output data for the first present input data in a case where the first cache data is present.