Step Editor Data Preparation Caching
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Large-scale web applications with millions or billions of records face computational intensity issues during data evaluation, leading to slow application response times due to the need for re-computation of data operations, causing operator-users to wait for extended periods.
Innovation Solution
A distributed computing platform like Apache Spark is utilized to perform sequenced data preparation operations efficiently, generating transformation results on-the-fly without prior indexing and partitioning, using a step editor to manage and cache results for reuse.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If re-computation is performed on large-scale data, then accurate evaluation results are obtained, but application response time increases significantly
Solution Approach 1:
The system performs data partitioning and indexing operations in advance before the actual evaluation query is executed. By pre-processing the data into manageable partitions with appropriate indexes, the system avoids performing these computationally intensive operations during the evaluation phase, thus reducing response time while maintaining accuracy.
Solution Approach 2:
The patent divides large-scale data into smaller partitions that can be processed independently and in parallel. This segmentation allows the evaluation computation to be distributed across multiple processing units, significantly reducing the time required to process billions of records while maintaining the same evaluation accuracy as processing the entire dataset as a single unit.
2Speed
If data is partitioned and indexed for fast access, then query speed improves, but device complexity increases
Solution Approach 1:
The system dynamically adjusts the partitioning strategy based on the specific evaluation query being executed. Rather than using a fixed partitioning scheme, the system can reorganize data partitions on-the-fly to match the query requirements, optimizing access speed for each specific operation without requiring complex pre-defined structures for all possible query types.
Solution Approach 2:
The patent introduces an intermediary layer (the partitioning and indexing system) that sits between the raw data and the evaluation queries. This intermediary handles the complexity of data organization transparently, providing fast access to users while hiding the underlying structural complexity from both users and the core evaluation logic.
3Loss of time
If cached results are used for reuse, then computation time is reduced, but memory resources are consumed
Solution Approach 1:
The system caches evaluation results at the partition level rather than requiring centralized caching of entire datasets. Each partition can maintain its own cache of recently accessed or frequently evaluated results, allowing memory resources to be distributed locally where they are most needed. This reduces overall memory consumption while still providing fast access to commonly used data through localized caching.
Data Source
AI summary
Using a step editor for data preparation includes: receiving an indication of a user input with respect to at least some of a set of sequenced data preparation operations on a set of data; generating, using one or more processors, a signature based at least in part on the set of sequenced data preparation operations, references to the set of data, and the user input; using the generated signature to determine whether there exists a cached result associated with the set of sequenced data preparation operations, the references to the set of data, and the user input; based at least in part on the determination, obtaining a data traversal program representing a result associated with the set of sequenced operations, the references to the set of data, and the user input; and providing output based at least in part on the result represented by the obtained data traversal program.