Automatic Data Organization Correction Without Re-ingestion
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional data tracking and analysis systems are inflexible, inaccurate, and inefficient due to rigid data organization schemas, leading to persistent errors and computational inefficiencies, requiring expert knowledge and resource-intensive corrections.
Innovation Solution
An automatic modification system that generates and executes suggestions to modify data organization elements at query time without re-ingesting data, using machine learning models to identify and correct errors such as merging, renaming, or removing dimension items, allowing for accurate data reading and analysis without altering the underlying data.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If conventional systems use rigid data organization schemas at ingestion time, then data is organized systematically, but errors in the schema persist inflexibly and cannot be corrected without re-ingestion
Solution Approach 1:
The system dynamically modifies data organization schemas after ingestion by generating and executing suggestions to merge, remove, or rename dimension items. This allows the schema to adapt and correct errors without re-ingesting data, transforming the rigid static schema into a dynamic one that can self-correct while maintaining data integrity.
Solution Approach 2:
The system introduces an intermediary layer of suggestion generation and execution that sits between the ingested data and the final data organization. This intermediary processes the data through generated suggestions without requiring direct modification of the original ingested data, enabling schema corrections while preserving the original data structure.
2Reliability
If conventional systems allow schema correction, then errors can be fixed, but corrections are applied rigidly only to newly ingested data, leaving previously ingested data incorrectly configured
Solution Approach 1:
The system performs preliminary actions by generating suggestions that can be applied retroactively to previously ingested data. Instead of waiting for new data ingestion to apply corrections, the system proactively generates and executes suggestions that correct errors in existing data organizations, ensuring consistent accuracy across all data regardless of when it was ingested.
3Reliability
If conventional systems re-ingest data to correct schema errors, then data organization can be updated, but time and computing resources are wasted and originally ingested data is destroyed
Solution Approach 1:
The system creates a copy or representation of the correction process through generated suggestions rather than physically re-ingesting the original data. These suggestions capture the necessary organizational changes and apply them to the existing ingested data, preserving the original data while implementing corrections without the time-consuming re-ingestion process.
Solution Approach 2:
The system discards the inefficient approach of re-ingestion and recovers the valuable originally ingested data by applying corrections through generated suggestions. This methodology eliminates the need to destroy and re-process original data, recovering both time and computational resources while achieving the same correction goals.
4Reliability
If conventional systems implement post-ingestion patches to overcome schema errors, then data accuracy can be improved, but vast amounts of processing power and digital memory are wasted
Solution Approach 1:
The system enables self-service correction by automatically generating suggestions based on the analyzed data organization errors. Rather than requiring external expert intervention or complex post-ingestion patches, the system autonomously identifies issues and generates appropriate correction suggestions, reducing the need for resource-intensive manual correction processes.
Solution Approach 2:
The system extracts only the essential correction information needed to fix schema errors through generated suggestions, rather than processing and storing entire datasets multiple times. By extracting and applying only the necessary organizational corrections, the system significantly reduces computational resource consumption while maintaining data accuracy.
Data Source
AI summary
The present disclosure relates to systems, non-transitory computer-readable media, and methods for generating automatic suggestions to effectively modify the organization of an ingested data collection without destruction of the underlying raw data. In particular, in one or more embodiments, the disclosed systems utilize multiple machine learning models in sequence to determine likelihoods that the organizational structure of an ingested data collection should be modified in various ways. In response to generating these likelihoods, the disclosed systems generate corresponding automatic suggestions to modify the organization of the ingested data collection. In response to a detected selection of one or more of the automatic suggestions, the disclosed systems read data out of the ingested data collection in accordance with the selected automatic suggestions to effectively modify the organization of the ingested data collection.