Virtualized Database Catalog for Multi-Format Data Access
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Traditional relational databases struggle to handle large volumes of big data, leading to difficulties in managing and accessing data across various storage formats, which results in data inconsistencies, incorrect access controls, and unnecessary data copies.
Innovation Solution
A virtualized database system that provides a unified interface for multiple data storage types, using a catalog to track metadata and locations of variables across different storage formats, allowing for uniform query language access and reducing the need for multiple data copies.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If traditional relational databases are used to store big data, then data can be managed in a structured format, but the system cannot handle the large volume of data efficiently
Solution Approach 1:
The patent segments the monolithic relational database into multiple specialized storage formats (columnar storage, key-value stores, document stores) that can handle different types of big data workloads efficiently. Each storage format is optimized for specific query patterns and data types, allowing the system to manage large volumes of data across distributed storage nodes.
Solution Approach 2:
The patent creates a universal interface layer that provides consistent access to multiple different storage formats. This multi-functional interface allows users to interact with various storage types (relational, non-relational, structured, unstructured) through a single standardized API, enabling the system to handle diverse big data workloads while maintaining ease of use.
2Adaptability or versatility
If multiple data storage formats are used to handle different big data requirements, then data can be stored in optimized formats, but users face difficulty learning various interface protocols and query syntaxes
Solution Approach 1:
The patent introduces an intermediary layer (catalog and unified interface) that mediates between users and multiple storage formats. This intermediary translates standardized user queries into format-specific operations, hiding the complexity of different storage protocols from users while maintaining access to optimized storage backends.
Solution Approach 2:
The unified interface provides universal access to multiple storage formats through a single consistent API. This multi-functional interface can adapt to different storage types (SQL, NoSQL, columnar, document stores) while presenting a uniform interaction model to users, eliminating the need to learn multiple query syntaxes.
3Adaptability or versatility
If users make copies of tables and data storage structures to support separate applications, then data can be accessed by multiple applications, but additional disk space is consumed and data consistency cannot be maintained
Solution Approach 1:
The patent uses virtual copying through the catalog system rather than physical data duplication. The catalog maintains references to the same underlying data stored once in optimized formats, allowing multiple applications to access identical data without creating physical copies. This virtual copying mechanism supports multiple applications while consuming minimal additional disk space.
Solution Approach 2:
The unified interface and catalog system provide universal data access to multiple applications through a single data storage structure. Instead of requiring separate copies for each application, the system enables one storage structure to serve multiple applications simultaneously through standardized access protocols and view management.
4Adaptability or versatility
If users make copies of data storage structures, then data can be accessed by multiple applications, but incorrect access controls and metadata may result from unmaintained copies
Solution Approach 1:
The catalog acts as an intermediary that maintains centralized control over data access and metadata. Instead of allowing applications to maintain independent copies with potentially inconsistent metadata, the catalog serves as the single source of truth for access controls, metadata, and data location information, ensuring consistency across all applications.
Solution Approach 2:
The system implements feedback mechanisms where the catalog continuously tracks and updates metadata and access controls for all data accesses. This feedback loop ensures that any changes to data structures, permissions, or metadata are immediately reflected across all applications accessing the data, maintaining reliability and consistency.
Data Source
AI summary
A system may store a plurality of first records comprising a first variable in a first data storage format. The system may also store a plurality of second records comprising a second variable in a second data storage format. The first data storage format may have a different data structure than the second data storage format. The system may then generate a catalog including a first location of the first variable and a second location of the second variable. The first location identifies the first data storage format, which is where the first variable is stored. The second location identifies the second data storage format, which is where the second variable is stored. The system may then receive a request to access at least one of the first variable or the second variable and access the first variable and/or the second variable in response to the request.