Distributed Data Placement Using Data Intimacy to Cut Migration
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
In distributed storage scenarios, improper data storage methods lead to increased migration costs and reduced transaction execution efficiency, particularly in distributed databases with large data volumes.
Innovation Solution
A distributed data storage method that determines intimacy between data based on historical usage patterns, optimizing storage locations to minimize data migration and ensure balanced distribution across servers.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If data is stored in a distributed manner across different devices, then data storage capacity and system reliability are improved, but data migration costs increase and transaction execution efficiency deteriorates
Solution Approach 1:
The system performs preliminary analysis of historical task logs to determine intimacy relationships between data pieces before actual data storage. By pre-calculating which data should be co-located based on their joint access patterns, the system prepares an optimized storage layout in advance, avoiding the need for frequent data migrations during transaction execution.
Solution Approach 2:
The patent applies local quality by storing data with high intimacy relationships in the same server or storage location, while data with low intimacy is distributed across different servers. This creates non-uniform local storage characteristics that match the access patterns, reducing cross-server data migrations while maintaining distributed storage benefits.
2Productivity
If data is migrated to aggregate in one device for transaction execution, then transaction execution efficiency is improved, but migration costs increase
Solution Approach 1:
The system performs preliminary analysis of historical task logs to determine intimacy relationships between data pieces before actual data storage. By pre-calculating which data should be co-located based on their joint access patterns, the system prepares an optimized storage layout in advance, avoiding the need for frequent data migrations during transaction execution.
Solution Approach 2:
The patent merges data pieces with high intimacy relationships into the same storage location or server. By combining data that is frequently accessed together, the system reduces the number of separate data retrieval operations and minimizes migration costs while maintaining efficient transaction execution.
3Loss of energy
If data storage layout is optimized based on intimacy relationships, then data migration costs are reduced, but system complexity increases
Solution Approach 1:
The system implements feedback by continuously monitoring actual data access patterns and comparing them with the intimacy-based storage layout. Based on this feedback, the system can dynamically adjust the storage configuration, adding or removing intimacy constraints as needed. This feedback mechanism allows the system to adapt to changing workloads while maintaining manageable complexity through incremental adjustments rather than complete reconfigurations.
Data Source
Figure 1
Figure 2~3
Figure 4~5
AI summary
Embodiments of this application provide a distributed data storage method and a controller. The method includes: obtaining a log, where the log indicates N pieces of data used in M historical tasks executed by a server cluster, M is a positive integer, and the N pieces of data are stored in the server cluster in a distributed manner; determining intimacy between the N pieces of data based on the log, where intimacy corresponding to any two of the N pieces of data indicates a possibility that the two pieces of data are used in a same task, and N is an integer greater than 1; and updating storage locations of the N pieces of data based on the intimacy between the N pieces of data. Solutions in embodiments of this application help implement proper data storage, thereby improving running performance.