Dynamic Cache Storage for Big Data Retrieval Bottlenecks
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The retrieval of big data is limited by bottlenecks caused by the sheer volume of data and the time it takes to execute calls, leading to inefficiencies in storage and retrieval processes, which can degrade system performance.
Innovation Solution
A dynamically configured responsive storage system that caches data from a data source based on criteria such as update frequency, using a data management component with a processor, input/output device, and memory to store and retrieve data efficiently, including the use of JSON responses and an enhanced control, orchestration, management, and policy (ECOMP) platform with AI and machine learning for optimized data management.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Speed
If data is retrieved directly from the data source, then data integrity is maintained, but retrieval speed deteriorates due to bottlenecks caused by data volume and call execution time
Solution Approach 1:
The system pre-loads and caches data from the data source before it is actually needed by consumers. The data management component proactively retrieves data and stores it in the data store, so that when consumers request data, it is already available in cache form, eliminating the need for time-consuming calls to the data source at retrieval time.
Solution Approach 2:
The patent introduces a data management component and data store as an intermediary layer between the data source and data consumers. This intermediary caches data and manages retrieval requests, so consumers interact with the cache rather than directly with the data source, reducing the burden on the data source and improving retrieval speed.
2Productivity
If all data from the data source is cached, then retrieval efficiency is improved, but storage resource consumption increases
Solution Approach 1:
The data management component selectively caches data based on local conditions such as consumer request patterns, data access frequency, and specific consumer criteria. Instead of uniformly caching all data, the system identifies which portions of data should be cached and stores only those in the data store, optimizing the balance between retrieval efficiency and storage resource usage.
Solution Approach 2:
The system dynamically adjusts caching parameters such as cache size, retention period, and selection criteria based on changing conditions. The data management component monitors data access patterns and modifies caching behavior accordingly, changing which data is cached and for how long, thereby optimizing storage resource consumption while maintaining retrieval efficiency.
3Reliability
If the data store is continuously updated with latest data, then data freshness is maintained, but system complexity and update overhead increase
Solution Approach 1:
The data management component updates the data store periodically or at scheduled intervals rather than continuously. Data is refreshed at predetermined times or when specific conditions are met, reducing the complexity of update management while still maintaining acceptable data freshness for consumer applications.
Solution Approach 2:
The system implements feedback mechanisms where the data management component monitors data access patterns, consumer requirements, and data source changes to intelligently determine when updates are necessary. This feedback-driven approach optimizes update timing and frequency, maintaining data freshness without requiring continuous updates, thereby reducing system complexity.
Data Source
AI summary
Storage systems and method to store data, from a data source, which is responsive to one or more requests from at least one data consumer. A data store is configured to cache at least some data from the data source. A data management component is configured to store the at least some data in the data store based on at least one criteria of the data consumer. At least one criteria is identified based on the one or more requests. Data is stored in the at least some data from the data source in response to the identifying the at least one criteria.