Query Time Data Cleansing for Sequence-Based RFID Anomalies
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional data cleansing methods for sequence-based data, such as RFID data, face challenges in removing anomalies upfront due to lack of available rules and business context, leading to inefficient data processing and maintenance of multiple cleaned versions, especially when application-specific rules and dynamic changes are involved.
Innovation Solution
A computer-implemented method that cleanses anomalies from sequence-based data at query time by loading data into a database, receiving cleansing rules, converting them into templates, and rewriting user queries to compensate for anomalies, allowing for efficient data retrieval and preservation of original data.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If conventional eager data cleansing approaches are used to remove all anomalies upfront during ETL process, then data quality is improved, but data processing time and storage requirements increase significantly
Solution Approach 1:
The patent applies preliminary action by pre-computing and storing cleansing rules as templates during data loading, but defersthe actual cleansing execution until query time. This allows the system to prepare cleansing logic in advance without immediately processing all data, thus improving data quality while avoiding excessive processing time upfront.
Solution Approach 2:
The system dynamically applies cleansing rules based on query requirements. Different queries can trigger different cleansing rules from the stored templates, allowing the data cleansing process to adapt to specific query needs rather than applying a static upfront cleansing approach. This dynamic behavior resolves the contradiction by making cleansing time-variable and query-dependent.
2Adaptability or versatility
If multiple cleaned versions of data are maintained for different application-specific rules, then adaptability to different applications is improved, but system complexity and maintenance burden increase
Solution Approach 1:
The patent implements a universal cleansing rule template system that can serve multiple applications. Instead of maintaining separate cleaned data versions for each application, the system stores cleansing rules as reusable templates that can be applied dynamically to support different application-specific requirements. This single universal system resolves the contradiction by eliminating the need for multiple data versions while maintaining adaptability.
Solution Approach 2:
The system creates virtual copies of cleansing rules as templates rather than creating physical copies of cleaned data. Each cleansing rule template can be instantiated and applied to different queries as needed, providing application-specific cleansing without duplicating the actual data. This copying approach reduces system complexity by working with rule templates rather than multiple data versions.
3Measurement precision
If upfront data cleansing is performed to ensure data quality, then measurement precision is improved, but the ability to preserve original tracking information is reduced
Solution Approach 1:
The system performs preliminary preparation of cleansing rules as templates during data loading, but preserves the original sequence-based data unchanged. The actual cleansing action is deferred to query time when it can be applied selectively without losing original tracking information. This timing strategy resolves the contradiction by separating rule preparation from data transformation.
Solution Approach 2:
The patent introduces cleansing rule templates as an intermediary between the original data and the query processing. These templates act as a mediator that can be applied selectively to preserve tracking information while still enabling precise analytical results when needed. The intermediary templates allow the system to maintain original data integrity while providing cleansing capability on demand.
Data Source
AI summary
A method and system for cleansing anomalies from sequence-based data at query time. Sequence-based data such as RFID data is loaded into a database. One or more cleansing rules are received at a cleansing rules engine. The cleansing rule engine converts the cleansing rule(s) to a template that includes logic to compensate for anomalies in the sequence-based data. A query to retrieve the sequence-based data is received by a query rewrite engine. The query rewrite engine rewrites the query by applying the template logic. The rewritten query is executed at query time. The result of the rewritten query execution is identical to the result of executing the original query on a data set generated by applying the cleansing rule to all of the sequence-based data.