Object Replica Placement via Bitmask Categories
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing techniques are inefficient in placing objects in a planet-wide distributed storage system that stores trillions of objects and petabytes of data, as they often require scanning all objects to ensure timely satisfaction of placement constraints, leading to resource overload and poor utilization.
Innovation Solution
A scalable scheme that categorizes objects based on their replica placement constraints, allowing periodic scanning of categories rather than individual objects, prioritizing actions, and managing resources to prevent overload, ensuring that all objects in a category have the same set of possible actions to satisfy their constraints.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If all objects are scanned individually to ensure timely satisfaction of placement constraints, then constraint satisfaction is improved, but system performance deteriorates due to resource overload
Solution Approach 1:
The patent divides the system state into discrete segments represented by bitmask categories. Each category represents a unique combination of placement constraint satisfaction states across multiple dimensions (e.g., minimum replicas, maximum replicas, geographic distribution). By segmenting the problem this way, the system can track and manage placement constraints efficiently without scanning all individual objects, thus maintaining reliability while improving productivity.
2Reliability
If periodic scanning of all objects is performed, then placement constraint satisfaction is improved, but resource utilization deteriorates
Solution Approach 1:
Instead of performing complete scanning and evaluation of all objects periodically, the patent applies partial action by only processing objects that have changed state or are relevant to current placement constraints. The bitmask category system allows the daemon to identify and process only those objects requiring attention, avoiding wasteful full-system scans and improving resource utilization while maintaining constraint satisfaction.
Solution Approach 2:
The patent transforms the placement constraint satisfaction problem from object-by-object evaluation to a state-based parameter system. By changing parameters such as replica locations, constraint configurations, and category bitmasks, the system can efficiently track satisfaction states without repeatedly scanning all objects. This parameter-based approach significantly reduces resource utilization while maintaining reliable constraint enforcement.
3Speed
If actions are executed immediately for each object, then responsiveness is improved, but system overload increases
Solution Approach 1:
The patent implements preliminary action by pre-computing and caching placement decisions based on current system state and constraints. When objects are added, removed, or modified, the daemon updates the bitmask categories and determines required actions in advance, rather than reacting to each individual object change in real-time. This preliminary processing improves responsiveness while preventing system overload by batching and optimizing action execution.
Solution Approach 2:
The patent introduces an intermediary mechanism in the form of the bitmask category system that mediates between object changes and action execution. Instead of directly processing each object change, the daemon updates category states as an intermediary step, then derives actions from these aggregated states. This intermediary layer smooths out peak loads and distributes processing more evenly, improving responsiveness without causing system overload.
Data Source
AI summary
A location assignment daemon (LAD) manages placement of object replicas in a distributed storage system. The distributed storage system may include a plurality of instances, which may be at distinct geographic locations. The LAD determines placement categories for objects stored in the distributed storage system. A placement category for an object corresponds to the object's placement policy and current replica locations. There are substantially fewer placement categories than objects. The LAD determines an action plan for each placement category whose associated objects require either creation or removal of object replicas. Each action plan includes either creating or removing an object replica. The LAD prioritizes the action plans and implements at least a subset of the action plans in priority order in accordance with available resources in the distributed storage system. Each action plan is applied to objects in the placement category corresponding to the action plan.