Dynamic Data Profiling for Unstructured Big Data
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing systems for profiling large data sets require converting semi-structured and unstructured data into a relational database format, which is time-consuming and inefficient, especially for big data sets that are growing rapidly and stored in disparate locations.
Innovation Solution
The method involves dynamically determining the format of data objects and applying data transformation policies to generate a relational-type representation of the data on-the-fly, allowing for interactive profiling without preloading data into a relational database, and enabling profiling across disparate locations without physical data collection.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If semi-structured and unstructured data are converted into relational database format before profiling, then the data can be analyzed using traditional relational tools, but the process is time-consuming and inefficient for big data sets
Solution Approach 1:
The system performs preliminary format detection and transformation policy selection on sample data objects before processing the complete data set. By analyzing a sample to determine the data format and selecting an appropriate transformation policy in advance, the system can efficiently process the entire data set without time-consuming conversion operations, thus resolving the contradiction between reliable data analysis and time efficiency.
Solution Approach 2:
The invention extracts only the necessary profile information from the original non-structured data directly, rather than converting all data to relational format first. The profiling operation extracts statistics, patterns, and characteristics directly from the data in its native format, eliminating the time-consuming conversion step while maintaining analysis reliability.
2Productivity
If all data is preloaded into a relational database before profiling, then the profiling process can be executed, but the system cannot handle rapidly growing big data sets stored in disparate locations
Solution Approach 1:
The system segments the data processing task by working with data objects in their original locations rather than loading all data into a central database. The profiling operation processes data objects individually or in batches from disparate locations, enabling the system to handle rapidly growing data sets without requiring complete data preload, thus resolving the contradiction between profiling execution capability and adaptability to growing data.
Solution Approach 2:
The invention introduces a data transformation policy as an intermediary that bridges the gap between non-structured data in disparate locations and the profiling operation. This intermediary layer enables the system to access and process data from various sources without requiring physical data collection or relational database conversion, maintaining both productivity and adaptability.
3Ease of manufacture
If a fixed mapping method or user-supplied mapping schema is used to convert data, then data can be stored in relational databases, but the process requires predefined schemas and metadata which reduces adaptability to data changes
Solution Approach 1:
The system dynamically determines the data format by analyzing sample data objects and automatically selects appropriate transformation policies without requiring predefined schemas. This dynamic approach allows the system to adapt to data format changes and variations, resolving the contradiction between ease of data storage structure and adaptability to data changes.
Solution Approach 2:
The invention enables the system to self-determine data formats and select transformation policies automatically based on sample data analysis. Rather than requiring user-supplied mapping schemas or fixed mapping methods, the system performs self-service format detection and policy selection, maintaining structured storage capability while gaining adaptability to data changes.
Data Source
AI summary
A system, a method, and a non-transitory computer readable for generating a profile of one or more data objects comprising determining a format of the at least one data object and selecting a data transformation policy based on the format of the at least one data object and generating a model descriptive of the non-structured data contained in the at least one data object based on the data transformation policy and selecting at least a portion of the model indicative of a portion of the non-structured data and generating a profile of the portion of the non-structured data contained in all of the one or more data objects.