Distributed Storage Data Block Modification via Local Copying
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional distributed storage systems face bandwidth and latency issues due to the unprecedented scale and complexity of data generated by IoT devices and increasing complexity of machine learning models, leading to inefficient data processing and high network traffic as they copy all data, including unmodified data, back to central locations without tracking updates.
Innovation Solution
A method that involves receiving data blocks from a central storage location, modifying only the updated data blocks by creating copies, and sending these modified blocks back to the central location, where they replace the original data, thereby reducing compute throughput and network traffic by avoiding the transfer of unmodified data.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If all data blocks are copied back to central storage location, then data consistency is maintained, but network traffic and computational overhead increase significantly
Solution Approach 1:
The patent extracts only the modified data blocks from the full dataset and sends them back to the central storage location, rather than transferring the entire dataset. This selective extraction approach maintains data consistency for changed portions while significantly reducing network traffic and computational overhead for unmodified data.
Solution Approach 2:
The patent creates a local copy of data blocks at the remote location, modifies only the necessary portions of this copy, and then transfers only the modified blocks back to the central location. This copying mechanism enables local processing while maintaining synchronization with the central storage without requiring full data transfers.
2Reliability
If all data blocks are transferred to central location, then data is updated centrally, but bandwidth consumption increases
Solution Approach 1:
The system extracts and transfers only the modified data blocks to the central storage location, filtering out unmodified data from the transfer process. This maintains accurate data updates at the central location while dramatically reducing the volume of data transmitted over the network.
Solution Approach 2:
The patent performs preliminary modification of data blocks at the remote location before transferring them to the central location. By preparing and validating the modifications locally in advance, the system ensures data update accuracy while minimizing the amount of data that needs to be transmitted centrally.
3Productivity
If data is processed at remote locations, then processing speed improves, but network infrastructure strain increases
Solution Approach 1:
The patent extracts only the essential modified data blocks for network transmission, leaving the bulk of data processing to occur locally at remote positions. This approach enhances processing speed by performing computations where data resides while minimizing network infrastructure strain through selective data transfer.
Solution Approach 2:
The patent uses local data blocks as intermediaries that facilitate processing at remote locations without requiring constant network communication with the central storage location. These intermediary copies enable autonomous local processing while maintaining synchronization with the central system through selective updates.
Data Source
AI summary
A computer-implemented method, according to one embodiment, includes: receiving multiple data blocks from a central data storage location, and modifying data in one or more of the received data blocks. Modifying data in one or more of the data blocks further includes creating copies of the one or more data blocks and the data included therein, and modifying the data in the copies of the one or more data blocks. A copy of the modified data blocks are sent to the central data storage location. Moreover, the modified data blocks are used to replace corresponding ones of the data blocks received from the central data storage location in response to receiving an indication that the modified data blocks have been implemented at the central data storage location.