Per-object Indexes for Automated Data Denormalization
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Data denormalization in existing systems is typically performed manually, leading to difficulties in maintaining data consistency and updating data, as it often requires manual intervention by individuals other than those responsible for data updates, which can cause inconsistencies and inefficiencies in search operations due to the need for joining multiple tables during querying.
Innovation Solution
A system that embeds denormalization information in data store schemas as annotations, allowing a denormalization engine to generate per-object indexes (POIs) for data instances, which include target and source sub-POIs, and automatically updates these indexes in storage, enabling quick search operations without the need for manual intervention.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If data is normalized and stored in separate tables, then data redundancy is reduced and updating becomes easier, but querying becomes time-consuming due to the need to join multiple tables
Solution Approach 1:
The patent segments the indexing task into multiple independent per-object indexes (POIs), each handling specific data instances. This allows parallel processing of index generation and maintenance, reducing query time while maintaining normalized data storage structure.
Solution Approach 2:
The system performs preliminary denormalization by generating POIs that contain pre-computed join results. When data instances are inserted or updated, the corresponding POIs are automatically generated or updated in advance, so that query operations can directly access pre-processed data without performing expensive join operations at query time.
2Speed
If data is manually denormalized to reduce search time, then query speed improves, but maintaining data consistency becomes difficult
Solution Approach 1:
The patent implements a feedback mechanism where the system automatically detects when source data instances are inserted or updated, and triggers corresponding updates to the POIs. This closed-loop approach ensures that denormalized data in POIs always reflects the current state of normalized source data, maintaining consistency without manual intervention.
Solution Approach 2:
The system enables self-service denormalization by automatically generating and maintaining POIs based on changes to source data instances. The denormalization process is automated and self-managing, eliminating the need for manual denormalization operations and the associated consistency maintenance problems.
3Productivity
If manual denormalization is performed by individuals other than data update personnel, then search operations can be optimized, but updating and maintaining data consistency becomes more complex
Solution Approach 1:
The patent creates a universal POI generation mechanism that works with any data instance type and relationship structure. The same automated process handles insertion, update, and deletion operations across different data domains, eliminating the need for specialized manual denormalization procedures and reducing operational complexity.
Solution Approach 2:
The system introduces POIs as intermediary structures between normalized source data and query operations. These POIs act as mediators that automatically reflect source data changes through the feedback mechanism, separating the concerns of data maintenance and search optimization into distinct, automated layers.
Data Source
AI summary
Technologies are described herein for denormalizing data instances. Schemas for data instances are embedded with annotations indicating how the denormalization is to be performed. Based on the annotations, one or more sub per object indexes (“sub POIs”) can be generated for each data instance and stored. The sub POIs can include a target sub POI containing data from the data instance, and at least one source sub POI containing data from another data instance, if the data instance depends on the other data instance. Data instance updates can be performed by identifying sub POIs that are related to the updated data instance in storage, and updating the related sub POIs according to the update to the data instance. The sub POIs can be sent to an indexing engine to generate an index for a search engine to facilitate searches on the data instances.