Graph Dataset Schema Determination via Partitioning and ILP
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing data management systems face challenges in identifying the actual schema of datasets, particularly in semi-structured data like RDF graphs, where there is a mismatch between the declared schema and the actual data structure, affecting data analysis, querying, and storage efficiency.
Innovation Solution
A framework is developed to specify structuredness functions using a formal language, allowing users to define rules for gauging dataset conformance to a schema, with bitmaps encoding entity information to reduce memory and CPU usage, and employing Integer Linear Programming (ILP) to optimize type system refinement, partitioning datasets into subsets that meet specified thresholds.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If schema flexibility is provided to allow users to add new predicates without breaking the schema, then adaptability is improved, but the ability to identify the actual schema for data analysis and querying deteriorates
Solution Approach 1:
The patent changes the parameter of schema representation from rigid predefined structures to flexible statistical models that capture the actual distribution and relationships of predicates in the data. This allows the schema to adapt to data variations while maintaining identifiability through statistical parameters.
Solution Approach 2:
The patent introduces statistical models as an intermediary layer between the flexible data and the schema identification process. These models serve as mediators that can handle the variability in predicates while still providing a structured representation for analysis and querying.
2Adaptability or versatility
If semi-structured datasets are used with inherent incompleteness in data, then adaptability is improved, but manufacturing precision of uniform data conformance deteriorates
Solution Approach 1:
The patent transforms the approach from requiring uniform data conformance to using statistical parameters that naturally accommodate incompleteness. By modeling data in terms of distributions and relationships rather than fixed schemas, the system handles incomplete data while maintaining analytical precision.
3Productivity
If bitmaps are used to encode entity information, then productivity is improved through reduced memory and CPU usage, but device complexity increases
Solution Approach 1:
The patent creates simplified bitmap representations (copies) of the entity information that capture the essential structural properties without storing the full complexity of the original data. These bitmap copies enable efficient processing while the encoding complexity is managed through systematic generation rules.
4Manufacturing precision
If Integer Linear Programming is used to optimize type system refinement, then manufacturing precision of schema conformance is improved, but productivity deteriorates due to computational complexity
Solution Approach 1:
The patent formulates the schema refinement problem as an Integer Linear Programming task that can be solved efficiently for each dataset instance. While ILP is computationally intensive, it provides optimal solutions that can be computed on-demand rather than maintaining complex persistent schema refinement structures.
Data Source
AI summary
A schema for a dataset is identified by identifying a dataset comprising data and relationships between data pairs. An original schema is identified for the dataset. This original schema comprises an organizational structure. An initial fit between the dataset and the original schema is determined. The initial fit quantifying a conformity of the data in the dataset to the organizational structure of the original schema. A plurality of additional schemas are identified. Each additional schema is a distinct organizational schema. The dataset is partitioned into a plurality of subsets. Each subset comprises a modified fit quantifying a modified conformity of subset data in each subset to one of the original schema and the additional schemas. The modified fit is greater than the original fit.