Manifest-Based Data Persistence in Object Stores
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Big Data analytics systems face challenges in persisting and accessing distributed data sets due to the eventual consistency of list operations in object stores, which can lead to inaccurate results and increased system complexity and cost.
Innovation Solution
A method that generates a manifest with unique identifiers for successfully persisted data parts, allowing for accurate tracking and retrieval of data without relying on eventually consistent list operations, and persists this manifest alongside the data, reducing the need for external strongly consistent metadata storage.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If list operations are used to identify persisted data parts in object stores, then data tracking is enabled, but accuracy deteriorates due to eventual consistency
Solution Approach 1:
The patent introduces a manifest file as an intermediary data structure that records the mapping between task identifiers and persisted data part names. This manifest acts as a mediator between the distributed tasks and the object store, providing a reliable source of truth for data tracking that is not affected by eventual consistency issues of object store list operations.
Solution Approach 2:
The patent performs preliminary actions by recording data part information in the manifest file at the time of persistence, before any reading operations occur. This advance recording ensures that the metadata is captured in a strongly consistent manner, eliminating the need for later list operations that would be subject to eventual consistency delays.
2Reliability
If external strongly consistent data stores are used for metadata, then consistency is improved, but system complexity and cost increase
Solution Approach 1:
The patent enables the object store to serve its own metadata needs through the manifest file mechanism. Instead of requiring external strongly consistent storage systems, the object store uses its own capabilities to store and retrieve manifest files, which contain all necessary metadata about persisted data parts. This self-service approach eliminates the need for additional external storage infrastructure.
Solution Approach 2:
The manifest file serves multiple functions: it acts as a metadata repository, a transaction log, and a data catalog all in one structure. This multi-functional approach consolidates what would otherwise require separate systems, reducing overall system complexity while maintaining strong consistency for metadata operations.
3Stability of the object's composition
If temporary files are used for data persistence, then atomicity is improved, but storage overhead and complexity increase
Solution Approach 1:
The patent extracts the atomicity guarantee mechanism from the temporary file approach and implements it directly in the object store using the manifest file. Instead of creating temporary files and then renaming them, the system directly persists data parts to their final locations in the object store and records the successful persistence in the manifest. This extraction eliminates the temporary file overhead while maintaining atomicity through the manifest's authoritative record.
Data Source
AI summary
Embodiments of the present systems and methods may provide the capability ensure that data is persisted and accessed correctly without depending on eventually consistent list operations on the object store. For example, in an embodiment, a computer-implemented method for data distribution may comprise attempting to persist a plurality of data parts from a plurality of processing tasks, generating a manifest including information indicating those attempts to persist data parts that have succeeded, and persisting the manifest with the data parts that have been successfully persisted.