Universal Storage Handler for Dynamic Overflow Data Storage
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional data storage systems face inefficiencies due to unpredictable high traffic periods leading to wasted storage resources and the need for additional capacity, while data processing elements often exceed their input limits, resulting in data loss or inefficient resource allocation.
Innovation Solution
Implementing a universal storage handler that manages distributed storage systems by utilizing excess storage capacity across different formats and providers, temporarily storing data in alternative systems when intended storage is unavailable, and managing data spikes to optimize resource utilization.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If storage capacity is increased to handle high traffic periods, then data availability is improved, but storage resource waste occurs during low traffic periods
Solution Approach 1:
The system dynamically allocates storage resources by switching between primary and secondary storage systems based on real-time traffic conditions. During high traffic periods, data is routed to the primary storage system; during low traffic periods, data is routed to the secondary storage system, optimizing resource utilization while maintaining data availability.
Solution Approach 2:
A load manager acts as an intermediary between data sources and storage systems, intelligently routing data to appropriate storage destinations based on current system conditions. This mediator coordinates between primary and secondary storage systems, ensuring seamless data availability while optimizing resource usage patterns.
2Reliability
If additional storage capacity is provisioned for unpredictable high traffic periods, then data loss is prevented, but storage resource allocation efficiency decreases
Solution Approach 1:
The system performs preliminary actions by pre-configuring secondary storage systems with sufficient capacity to handle high traffic periods. The load manager monitors traffic patterns and proactively switches data routing to secondary storage before capacity is exhausted, preventing data loss without requiring permanent over-provisioning of primary storage.
Solution Approach 2:
The system changes operational parameters by dynamically adjusting which storage system is active based on traffic conditions. The load manager modifies data routing parameters in real-time, switching between primary and secondary storage systems to optimize both data loss prevention and resource allocation efficiency under varying load conditions.
3Productivity
If data is temporarily stored in alternative storage systems, then resource utilization is optimized, but system complexity increases
Solution Approach 1:
The load manager serves multiple functions: it monitors traffic conditions, makes routing decisions, manages data flow between storage systems, and coordinates capacity allocation. This universal component handles diverse tasks through a unified architecture, optimizing resource utilization while managing system complexity through functional consolidation.
Data Source
AI summary
Aspects of the present disclosure enable a storage manager to provide access to storage systems of varying types, such as database storage systems or indexed data storage systems, to a data source and allow for use of reserved storage capacity of the storage systems. The storage manager may provide access to storage systems under the control of different cloud storage providers. The storage manager may provide for temporary data storage in a reserved portion of a storage system when processing capacity of a data processing system is exceeded. The storage manager may provide for temporary storage of query results retrieved from a database system in a reserved portion of a data store. The storage manager may provide for temporary storage of data used for data enrichment in a reserved portion of a data store.