Object Store Metadata Generation for Data Query Optimization
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing object data stores face challenges in efficiently transferring and processing large data objects for big data analytics and machine learning applications, as they lack a schema or structure to selectively retrieve only the required data subsets.
Innovation Solution
Generating metadata from a scan of a data object in an object data store allows for the identification of record boundaries and statistical information, enabling efficient querying and data retrieval by supporting range queries and parallel processing.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If large data objects are retrieved from object storage service for processing, then the data can be accessed and processed by analytics applications, but the amount of data transferred is excessive and time-consuming even though only a subset is needed
Solution Approach 1:
The patent segments the large data object into smaller manageable units by introducing metadata that describes the internal structure and organization of data within the object. This metadata enables the system to identify and retrieve only the specific subsets of data needed for processing, rather than transferring the entire object. The segmentation is achieved through scanning the data object to generate metadata that captures record boundaries, data formats, and other structural information.
Solution Approach 2:
The patent performs preliminary scanning of the data object to generate metadata before actual data retrieval and processing occurs. This preliminary action creates a map or index of the data object's structure, allowing subsequent queries to efficiently locate and retrieve only the required data subsets. The metadata generation is performed in advance, enabling faster and more targeted data access in future operations.
2Ease of operation
If object storage service maintains no schema or structure for data objects, then web-services-based access is simplified, but selective retrieval of data subsets becomes difficult
Solution Approach 1:
The patent introduces metadata as an intermediary layer between the object storage service and the data processing applications. This metadata acts as a mediator that provides structural information about the data objects without requiring the object storage service itself to maintain schemas. The metadata bridges the gap by enabling structured access and selective retrieval while preserving the simplicity of the underlying object storage model.
Solution Approach 2:
The patent enables the data object itself to provide its own structural information through self-scanning. When a data object is stored or accessed, the system automatically scans the object to generate metadata that describes its internal structure. This self-service approach eliminates the need for external schema management while still providing the structural information needed for efficient data retrieval and processing.
Data Source
AI summary
An object data store may generate metadata responsive to a request that causes a scan of a data object for subsequent use in performing queries to the data object. A request may be received that causes a scan operation of the data object. As part of performing the scan one or multiple types of metadata describing the data object may be generated. The generated metadata may be applied to access the data object and perform a subsequently received query to the data object at the object data store.