Data Attribute Consolidation Using Metadata Insights for Faster Processing
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional data analytics solutions are becoming complex and inefficient due to increasing data sizes and varieties, leading to high operational and maintenance costs, inadequate information derivation, and increased risk of incorrect business decisions.
Innovation Solution
A system and method for consolidating data attributes that focus on experience-driven data mining, utilizing metadata insights to filter and generate experience-based metadata, reducing data volume through predefined rules, and generating consolidated sample data for optimized processing.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If huge data sets are processed to handle data growth and variety, then data coverage and completeness are improved, but processing time and operational costs increase
Solution Approach 1:
The patent extracts and processes only the most relevant and valuable data attributes from huge datasets using automated attribute identification and selection algorithms. This extraction approach maintains data coverage while significantly reducing the volume of data that requires intensive processing, thereby resolving the contradiction between comprehensive data handling and processing efficiency
Solution Approach 2:
The patent segments data processing into multiple stages: initial data ingestion, attribute identification, attribute selection, and focused processing of selected attributes. This segmentation allows the system to handle huge datasets by breaking down the processing task into manageable segments, reducing overall processing time while maintaining data completeness
2Ease of operation
If huge data sets are stored in data warehouses, then data availability is improved, but operational and maintenance costs increase
Solution Approach 1:
The patent performs preliminary attribute identification and selection before data processing and storage. By pre-identifying relevant attributes and filtering out unnecessary data, the system reduces the volume of data that needs to be stored and maintained in warehouses, thereby lowering operational and maintenance costs while maintaining data availability for critical attributes
Solution Approach 2:
The patent applies different quality standards and processing levels to different data attributes based on their relevance and value. High-priority attributes receive full processing and storage resources, while lower-priority attributes receive minimal processing. This local quality approach optimizes resource allocation and reduces operational costs while maintaining data availability where it matters most
3Productivity
If data processing focuses on available data within business units, then processing speed is improved, but data quality and decision accuracy deteriorate
Solution Approach 1:
The patent creates a universal attribute identification framework that can process data from multiple business units and data sources simultaneously. This multi-functional approach enables the system to quickly identify and consolidate relevant attributes across the entire organization, maintaining processing speed while ensuring comprehensive data quality and decision accuracy by leveraging data from all relevant sources
Data Source
AI summary
The present disclosure relates to system and method for consolidating data attributes for optimizing data processing. The method comprises defining objectives associated with data along with data outcome expected after processing of data from a sample dataset through processor. The data outcome is predicted based on metadata associated with the data from the sample dataset. The method further comprises identifying metadata insights for classifying metadata into a plurality of metadata types through metadata crawler, generating an experienced based metadata by filtering metadata with incomplete information from the metadata according to the objectives and data output expected through the processor. The experience-based metadata comprises one or more metadata types from the plurality of metadata types and generating consolidated sample data by using the experienced based metadata through the processor. The consolidated sample data is generated based on validation of the consolidated sample data through predefined rules.