Distributed Data Placement Using Data Intimacy for Transaction Efficiency
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Inefficient data storage in distributed systems leads to increased migration costs and reduced transaction execution efficiency, particularly in distributed databases with large data volumes.
Innovation Solution
A distributed data storage method that determines intimacy between data based on historical usage patterns, optimizing storage locations to minimize migration costs and improve efficiency by storing data with high intimacy in the same server.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If data is stored in different devices in a distributed system, then data distribution and scalability are improved, but data migration costs increase and transaction execution efficiency deteriorates
Solution Approach 1:
The system performs preliminary analysis of historical task logs to determine intimacy relationships between data pieces before transactions occur. Based on these pre-determined relationships, data is proactively relocated to optimal storage locations in advance, so that when transactions execute, data with high intimacy is already co-located, eliminating the need for costly real-time migrations during transaction processing.
Solution Approach 2:
The system dynamically changes the storage location parameter of data based on the intimacy value calculated from historical usage patterns. By adjusting this parameter according to measured intimacy relationships, the system optimizes data placement to minimize migration costs while maintaining distributed storage benefits.
2Ease of operation
If data is frequently migrated between devices, then data can be kept in optimal locations for current tasks, but migration costs increase and system performance deteriorates
Solution Approach 1:
The system continuously monitors task execution logs to measure intimacy relationships between data pieces. This feedback information is used to dynamically adjust data storage locations, creating a closed-loop control system that optimizes data placement based on actual usage patterns while avoiding unnecessary migrations.
Solution Approach 2:
Instead of migrating all data to a single location or frequently repositioning data, the system selectively migrates only data pairs with high intimacy values that are frequently accessed together. This partial action approach optimizes the most critical data relationships while minimizing overall migration overhead.
3Productivity
If data is aggregated into one device for transaction execution, then transaction efficiency improves, but data distribution benefits are lost and system scalability deteriorates
Solution Approach 1:
The system creates local data clusters on distributed devices where data with high intimacy relationships is co-located. This allows each device to have optimized local data arrangements for efficient transaction execution while the overall system maintains distributed architecture and scalability through selective data placement rather than centralization.
Data Source
AI summary
A distributed data storage method includes obtaining a log, where the log indicates N pieces of data used in M historical tasks executed by a server cluster, M is a positive integer, and the N pieces of data are stored in the server cluster in a distributed manner; determining intimacy between the N pieces of data based on the log, where intimacy corresponding to any two of the N pieces of data indicates a possibility that the two pieces of data are used in a same task, and N is an integer greater than 1; and updating storage locations of the N pieces of data based on the intimacy between the N pieces of data.