Data Storage API for Asynchronous Archival Transfer Integrity
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional data storage technologies, such as magnetic tapes, fail to leverage advances in data compression and error correction, leading to high costs and resource inefficiencies in archival data storage, particularly for large datasets.
Innovation Solution
A data storage service accessible through an application programming interface (API) allows customers to upload, retrieve, and manage data asynchronously, providing features like checksums for partial data transmission, job identifiers for asynchronous processing, and customizable notifications for job status, along with capabilities for data verification and flexible metadata logging.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If conventional magnetic tape storage is used, then cost is reduced, but data storage efficiency and reliability are worsened due to inability to leverage data compression and error correction advances
Solution Approach 1:
The patent introduces a storage service system with API as an intermediary layer between users and the physical storage infrastructure. This mediator handles complex operations like data compression, error correction, and distributed storage management, allowing users to access reliable storage without dealing with the underlying complexity. The service abstracts the sophisticated storage techniques into simple upload/download operations.
Solution Approach 2:
The patent replaces traditional mechanical tape-based storage systems with a software-defined storage system that uses computational methods for data compression, error correction, and management. Instead of relying on physical tape media and linear sequential access, the system uses digital processing, distributed storage nodes, and intelligent data placement to achieve superior reliability and efficiency.
2Productivity
If synchronous data processing is used, then simplicity is maintained, but resource efficiency is worsened due to retransmissions and inability to handle large datasets asynchronously
Solution Approach 1:
The system performs preliminary actions by breaking data into chunks and pre-processing them before full transmission. It calculates checksums for data chunks in advance, allowing for efficient verification and partial retransmission if needed. The system also pre-establishes storage locations and prepares receive buffers, reducing overall transmission time and improving productivity.
Solution Approach 2:
The patent implements periodic action through asynchronous job processing where data operations are submitted as jobs that execute independently over time. The system periodically checks job status and provides notifications at intervals, allowing resource-efficient processing of large datasets without blocking the user interface or requiring continuous synchronous communication.
3Reliability
If data is stored without verification mechanisms, then storage speed is increased, but data integrity is worsened due to lack of checksums and verification
Solution Approach 1:
The patent implements feedback mechanisms through checksum calculations and verification. After data is stored or transmitted, the system calculates checksums and compares them against stored values to verify integrity. Error notifications are sent back to users when verification fails, allowing for corrective action. This feedback loop ensures data integrity without requiring overly complex verification hardware.
Solution Approach 2:
The system creates redundant copies of verification data (checksums, error correction codes) alongside the primary data. These copies are stored with the data and used for verification without requiring complex real-time computation during storage operations. The copying approach simplifies the verification process while maintaining high reliability.
Data Source
AI summary
An application programming interface for a data storage service provides a convenient mechanism for clients of the data storage service to access its various capabilities. An API call may be made to initiate a job and in response a job identifier may be provided. A separate API call specifying the job identifier may be made and a response providing information related to the job may result. Various API calls may be used to store data, retrieve data, obtain an inventory of stored data, and to obtain other information relating to stored data.