Non-materialized Tables with Standing Queries for Data Regeneration
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
In network environments, accessing and storing data across multiple computing devices is inefficient due to the lack of effective methods for managing data consistency and resource utilization, particularly in scenarios where data needs to be regenerated after consumption in non-materialized tables.
Innovation Solution
A network device system that includes a database with materialized and non-materialized tables, where a standing query service manager generates outputs stored in non-materialized tables, regenerating data upon request if it has been consumed, and utilizing observers and modifiers to propagate changes and manage storage efficiently.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If data is stored in non-materialized tables to reduce resource usage, then resource utilization is improved, but data availability and access efficiency deteriorate when data needs to be regenerated
Solution Approach 1:
The system performs preliminary actions by maintaining standing queries that can regenerate data on-demand. When data is consumed from a non-materialized table, the standing query is triggered to regenerate the data automatically, ensuring data availability without permanently storing it. This resolves the contradiction by preparing the regeneration mechanism in advance while keeping resource usage low.
Solution Approach 2:
The system dynamically switches between materialized and non-materialized storage based on data access patterns and resource availability. Data can be temporarily materialized when frequently accessed, and transitioned to non-materialized storage when resources are constrained. This dynamic approach allows the system to optimize both resource usage and data access efficiency according to actual needs.
2Productivity
If data is materialized to improve data access speed, then data access efficiency is improved, but resource consumption increases
Solution Approach 1:
The system applies different storage qualities to different data based on local requirements. Frequently accessed data is materialized (higher quality storage) while less frequently accessed data remains in non-materialized tables (lower quality storage). This local differentiation allows the system to improve data access efficiency for critical data without unnecessarily increasing overall resource consumption.
Solution Approach 2:
The system changes the materialization parameter of data based on access patterns, resource availability, and priority levels. Data can be dynamically materialized or dematerialized by changing the storage parameter, allowing flexible optimization of both access efficiency and resource usage without permanent commitment to a single storage state.
3Quantity of substance
If standing queries regenerate data on-demand from non-materialized tables, then resource usage is reduced, but data consistency and reliability may deteriorate
Solution Approach 1:
The system implements feedback mechanisms where standing queries monitor data source changes and automatically trigger regeneration when underlying data is modified. This feedback loop ensures that regenerated data remains consistent with the source, maintaining data reliability while still using non-materialized tables to reduce resource usage.
Solution Approach 2:
The standing queries are designed to be self-service, automatically detecting when data needs regeneration and executing the regeneration process without external intervention. This self-service capability ensures data consistency is maintained proactively, reducing the risk of inconsistent data states while keeping resource usage low through on-demand regeneration.
Data Source
AI summary
A network device includes a database that includes a materialized table and a non-materialized table. The network device also includes a standing query instance that generates an output that is stored in the non-materialized table. The database obtains a request that specifies the output, and provides the output in response to the request.