AI Data Enrichment for Interactive Raw Dataset Exploration
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing data mining systems lack the ability to collect predictions in a centralized place, improve training and testing datasets with user corrections, and require data scientists for intuitive data exploration and visualization, leading to inefficiencies and overfitting.
Innovation Solution
A platform that utilizes machine learning and AI to transform datasets into enriched datasets, provides visual representations with explorable and navigable widgets, allows user interaction for insights, and includes an ongoing learning function to improve models with user corrections.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If multiple enriched datasets are created for each enrichment job, then data processing can be performed on distinct datasets, but predictions cannot be collected in a centralized place for convenient review
Solution Approach 1:
The patent merges predictions from multiple enrichment jobs into a single centralized collection. Instead of maintaining separate enriched datasets for each job, the system consolidates all predictions into one unified structure that can be reviewed conveniently, while still allowing processing of distinct datasets through the enrichment pipeline.
Solution Approach 2:
The centralized predictions collection serves multiple functions: it stores predictions from different enrichment jobs, enables unified review and analysis, and supports various downstream tasks such as model training and evaluation. This multi-functional approach eliminates the need for separate data collection mechanisms for each enrichment job.
2Productivity
If foundation models and transfer learning are used, then AI tools can parse data faster, but effective AI requires careful fine-tuning to perform well across different datasets
Solution Approach 1:
The system performs preliminary actions by pre-processing and enriching data before it reaches the AI models. Enrichers prepare the data in advance with relevant features and transformations, reducing the fine-tuning burden on models and enabling them to perform well across different datasets with minimal adaptation.
Solution Approach 2:
The patent introduces enrichers as intermediary components between raw data and AI models. These enrichers transform raw data into enriched formats that are more suitable for model processing, acting as a mediator that bridges the gap between diverse data sources and the requirements of foundation models, thereby reducing the need for extensive fine-tuning.
3Measurement precision
If users must provide testing datasets each time evaluation is run, then model performance can be evaluated, but it requires careful tracking of training and testing datasets using external tools
Solution Approach 1:
The system implements self-service by automatically managing testing datasets. Instead of requiring users to manually provide and track testing datasets, the system automatically selects, prepares, and manages test data from the centralized predictions collection, eliminating the need for external tracking tools and reducing time spent on dataset management.
Solution Approach 2:
The patent establishes a feedback loop where model evaluation results are automatically fed back into the system. The centralized predictions collection serves as both training and testing data, allowing the system to automatically evaluate model performance and use the results to improve future predictions, creating a continuous self-improving cycle without manual intervention.
Data Source
AI summary
A system and method for data exploring enables a user to utilize existing raw and unstructured data, upload it to the system explore it to gain relevant insights. The system is comprised of enrichers that utilize ML and AI to transform the data into enriched data. Once the data has been enriched, the system displays the results through widgets, alongside structured and unstructured data. The widgets are explorable and navigable by a user in the sense that selecting one datapoint on one widget filters the other widgets accordingly, such that the user can gain insights on their original dataset. The system also has an interactive Q&A functionality that leverages LLM for users to query their data. In the Q&A, the system uses RAG methodology to retrieve semantically similar results and run them through a LIM to provide insightful, helpful and cited answers.