Derived Dimensions for Schema Correction Without Re-ingestion
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional data tracking and analysis systems are inflexible and inaccurate due to rigid data organization schemas, leading to computational inefficiencies and flawed analytical insights, as schema errors persist after data ingestion and require costly and time-consuming post-ingestion patches.
Innovation Solution
The system generates derived dimensions that modify the schema organization after data ingestion without re-ingesting raw data, allowing for schema corrections and improvements at query time through user-defined definitions and fallback expressions, enabling the merging, removal, or addition of dimensions and elements.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If conventional systems use a rigid data organization schema at data ingestion time, then data can be efficiently ingested and stored, but schema errors (data mis-labeling, incorrect data categorization) inflexibly persist and cannot be corrected for previously ingested data
Solution Approach 1:
The system segments the data organization into two independent layers: the original rigid schema structure for efficient storage, and an overlay of derived dimensions that provide flexibility. The derived dimensions are generated from fallback expressions that map to the original schema, allowing corrections without modifying the underlying data structure.
Solution Approach 2:
The system creates derived dimensions as virtual copies of the original schema dimensions. These derived dimensions are generated from user-provided definitions and fallback expressions, allowing the system to present corrected or modified dimension views without altering the original ingested data or schema structure.
2Measurement precision
If conventional systems allow schema correction, then accuracy can be improved for new data, but previously ingested data remains incorrectly configured creating a split in the data
Solution Approach 1:
The derived dimensions serve multiple functions simultaneously: they provide corrected data views, maintain consistency across all time periods, and enable retroactive schema corrections. A single derived dimension can replace multiple original dimensions, unifying the data view across different time periods while maintaining accuracy.
Solution Approach 2:
The derived dimensions act as an intermediary layer between the original ingested data and the user queries. This intermediary allows the system to present corrected, consistent data views without modifying the original data or creating splits, by translating queries against derived dimensions into appropriate data retrievals.
3Measurement precision
If conventional systems implement post-ingestion schema patches, then schema accuracy can be improved, but vast amounts of processing power and digital memory are spent running queries, generating displays, and storing mid-correction analysis results
Solution Approach 1:
Instead of physically modifying or re-processing the original data, the system creates virtual copies in the form of derived dimensions. These derived dimensions are generated from fallback expressions that reference the original data, allowing accurate schema corrections without duplicating or re-processing the actual data, thus minimizing computational resource usage.
Solution Approach 2:
The system changes the parameter representation rather than the underlying data. By generating derived dimensions with modified parameters (dimension names, categorizations, groupings) that map to the original data through fallback expressions, the system achieves schema accuracy improvements without the computational cost of actual data transformation or re-processing.
Data Source
AI summary
The present disclosure relates to systems, non-transitory computer-readable media, and methods for generating derived dimensions within ingested data collections. In particular, in one or more embodiments, the disclosed systems generate derived dimensions that modify or correct one or more organizational schemas of an ingested data collection when querying the ingested data collection. For example, the disclosed systems receive a user-defined definition, and generates a fallback expression based on the user-defined definition and aligned to the schema of a particular ingested data collection. In at least one embodiment, the disclosed systems generate the derived dimension by identifying and modifying one or more existing dimensions in the schema based on the fallback expression when reading data from the ingested data collection.