Dynamic Schema Generation for Heterogeneous Data
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current systems for managing structured, semi-structured, and unstructured data from heterogeneous sources are inflexible and costly due to the use of rigid data models like relational databases, which complicate the processing of diverse data types.
Innovation Solution
A dynamic schema generation system that analyzes attributes of data items to determine mandatory, expected, and optional attributes based on occurrence frequencies, allowing for the creation of flexible data templates that adapt to irregularly structured data.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Stability of the object's composition
If a rigid data model like relational database is used, then data structure stability is improved, but adaptability to heterogeneous data sources deteriorates
Solution Approach 1:
The patent implements dynamic schema generation that automatically adapts to different data sources and formats. The system learns schemas from incoming data items and updates the data model dynamically, allowing the structure to evolve with new data types while maintaining stability through learned patterns and frequency-based attribute classification.
Solution Approach 2:
The system changes the parameter of schema rigidity by introducing dynamic schema generation. Instead of using a fixed rigid schema, the system generates schemas on-demand based on data analysis, transforming the static parameter into a dynamic one that can adapt to heterogeneous data sources while maintaining structural integrity through learned attributes.
2Manufacturing precision
If a static predefined data schema is enforced, then data processing consistency is improved, but flexibility in handling irregularly structured data deteriorates
Solution Approach 1:
The system transitions from static schema enforcement to dynamic schema generation. The schema is generated dynamically based on the analysis of incoming data items, allowing the system to maintain processing consistency through learned patterns while adapting to irregularly structured data through automatic schema evolution.
Solution Approach 2:
The system performs preliminary analysis of incoming data items to generate schemas before actual data processing occurs. This preliminary schema generation ensures that subsequent processing maintains consistency while the schema itself is tailored to the specific characteristics of the data being processed.
3Ease of manufacture
If distinct systems for database content and files are maintained, then data organization clarity is improved, but system complexity and cost deteriorates
Solution Approach 1:
The patent merges the handling of structured database content and semi-structured/unstructured files into a single unified system. The dynamic schema generation capability allows the same system to process both types of data by learning appropriate schemas from the data itself, eliminating the need for separate systems while maintaining organizational clarity through learned data structures.
Solution Approach 2:
The system achieves multi-functionality by enabling a single data processing system to handle both structured database content and semi-structured/unstructured files. The dynamic schema generation provides universal adaptability, allowing the system to process diverse data types through a unified architecture rather than requiring distinct specialized systems.
Data Source
AI summary
One or more data items of one or more types are received, wherein each data item is associated with one or more attributes, and wherein each attribute is associated with an attribute name and an attribute value. A key is calculated for each attribute, and a determination as to whether the key is located in a dictionary is made. The key is added to the dictionary based on the determination along with an attribute description, and a determination of a number of occurrences of one of the one or more items to the number of all the one or more items in a set is made. Based on this number of occurrences attributes are classified as either mandatory, expected or optional in the generated template for the corresponding info item type.