Data Transfer System Using Messaging Platform Offsets
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing data transfer methods between different storage locations are inefficient, requiring significant resources and causing system delays due to the need to reallocate resources to correct errors, which leads to unnecessary bandwidth and resource consumption.
Innovation Solution
A data transfer and resource management system (DMS) that extracts schema information for data validation, encrypts sensitive data, performs data quality checks, and uses an intermediary messaging platform to manage data transfers, allowing for targeted and efficient data retrieval and re-retrieval, minimizing resource usage and maintaining data continuity.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If data transfer methods are used to move or copy data between storage locations, then data can be transferred, but large amounts of memory, processing power, and bandwidth are required
Solution Approach 1:
The patent segments data into batches and tracks transfer progress using offset records. When errors occur, only the affected batch needs to be retransferred rather than the entire dataset. This segmentation approach maintains data transfer capability while significantly reducing resource consumption by avoiding redundant transfers of unaffected data.
Solution Approach 2:
The system performs preliminary actions by extracting schema information before data transfer and storing offset records to track progress. This preliminary preparation enables efficient error detection and targeted retransfer, reducing the need to reallocates resources for complete data transfers when errors occur.
2Reliability
If all data is reallocated to move all data again to resolve errors, then data quality can be maintained, but system delays and unnecessary bandwidth consumption occur
Solution Approach 1:
The patent divides data into manageable batches with individual error tracking. When errors are detected in one batch, only that specific batch is marked for retransfer while other batches continue uninterrupted. This segmentation maintains data quality through error isolation while minimizing system delays by avoiding complete data retransfer.
Solution Approach 2:
The system implements feedback mechanisms through offset records that track transfer progress and error status for each batch. This feedback enables selective retransfer of only affected data portions, maintaining data quality while reducing system delays compared to complete data reallocation approaches.
3Manufacturing precision
If schema information is extracted and used for data validation, then data quality can be ensured, but additional processing steps are required
Solution Approach 1:
The patent extracts schema information as a preliminary action before data transfer begins. By storing schema details and offset records in advance, the system enables efficient validation and error detection during transfer without adding significant processing complexity. The preliminary extraction allows rapid verification during actual data transfer operations.
Data Source
AI summary
Disclosed herein are various embodiments for a data transfer and resource management system. An embodiment operates by retrieving both data and a schema from a table. A first offset corresponding to the retrieved data indicating a retrieval time of the retrieved data and a number of records is identified. It is determined that the retrieved data corresponds to the schema. The data is stored on a messaging platform and made available to be pulled by one or more end users prior to a transfer to one or more cloud servers. A second offset corresponding to a number of records of the data retrieved by the one or more end users from the messaging platform is determined. It is validate that the second offset corresponds to the first offset. The records and the schema retrieved from the intermediary messaging platform are transferred to the one or more cloud servers.