Cognitive Data Filters for Storage Query Latency
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Large file systems face resource-intensive delays and latency due to scanning massive data sets for queries, especially in big data analytics, where unnecessary data is often copied and processed, wasting resources like bandwidth and memory.
Innovation Solution
Implementing cognitive data filters to pre-group data based on definable characteristics before queries are received, allowing for efficient data retrieval and adaptation over time based on usage patterns.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If all data is copied to execute queries on another device, then query processing can be distributed, but network bandwidth and memory resources are wasted due to copying unnecessary data
Solution Approach 1:
The system performs preliminary filtering of data based on query criteria before copying data to executing devices. Cognitive data filters are generated and applied to pre-fetch only relevant data portions, eliminating the need to copy entire data sets and thereby reducing network bandwidth and memory resource consumption.
Solution Approach 2:
The system extracts and separates relevant data from the global data repository based on query-specific criteria. By using cognitive filters to identify and extract only the necessary data portions, the system avoids copying unnecessary data, thus reducing resource waste while maintaining query processing capability.
2Speed
If data is cached locally to speed up query processing, then query response time improves, but enormous local storage capacity is required
Solution Approach 1:
The system performs preliminary filtering and pre-fetches only the data portions that are likely to be needed for query execution. By applying cognitive filters in advance, the system prepares relevant data subsets without requiring full data caching, thus improving query response time while minimizing local storage requirements.
Solution Approach 2:
The system implements location-specific data caching where each executing device stores only the filtered data relevant to its specific query needs. This localized approach ensures that each device has the necessary data for fast processing without requiring all devices to maintain enormous storage capacity.
3Reliability
If the global data repository processes all queries directly, then data access is centralized, but resource-intensive delays and latency occur
Solution Approach 1:
The system segments the monolithic query processing task into distributed filtering and execution components. Cognitive data filters are generated and applied in a preprocessing stage, separating the filtering function from the query execution function. This allows queries to be processed in parallel across multiple devices with reduced latency while maintaining centralized data access control.
Solution Approach 2:
The system performs preliminary data filtering and preparation before query execution. By generating cognitive filters and pre-fetching relevant data subsets in advance, the system reduces the processing burden during actual query execution, thereby reducing delays and latency while maintaining reliable centralized control over data access.
Data Source
AI summary
In one embodiment, a method for cognitively filtering data in a storage environment includes generating a plurality of cognitive data filters based on one or more factors to create a plurality of groups. The method also includes cognitively filtering data received by a global data repository to place independent portions of the data into the plurality of groups using the plurality of cognitive data filters prior to receiving a query on the global data repository. Data within each of the plurality of groups share a common definable characteristic. Other systems, methods, and computer program products for cognitively filtering data within storage environments are presented in accordance with more embodiments.