Job Classification System for Multi-Tier Storage Scheduling
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing job scheduling systems in multi-tier storage environments face challenges in balancing storage capacity and performance requirements, particularly in allocating computational tasks to appropriate storage resources based on their characteristics, leading to suboptimal execution efficiency and reliability.
Innovation Solution
A job classification system that assigns different classes of jobs to corresponding storage resources, including performance, capacity, and key-value storage classes, with prefetching and future-aware data eviction techniques to optimize resource utilization.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of operation
If multi-tier storage environments are provided transparently to users and applications, then ease of operation is improved, but performance improvements that applications could obtain when multiple tiers are visible are lost
Solution Approach 1:
The patent introduces a job scheduler as an intermediary component that sits between applications and the multi-tier storage system. The scheduler classifies computational jobs and intelligently assigns them to appropriate storage tiers (performance tier with memory or capacity tier with storage) based on job characteristics, thereby maintaining transparent operation for applications while optimizing performance through informed resource allocation
2Device complexity
If computational tasks are allocated without job classification, then device complexity is reduced, but execution efficiency and reliability deteriorate
Solution Approach 1:
The patent segments computational jobs into different classes based on their characteristics and requirements. The job scheduler divides the task allocation process into classification and assignment phases, matching specific job types to appropriate storage tiers. This segmentation enables optimized execution efficiency and reliability without requiring complex application-level modifications
3Device complexity
If storage resources are not classified into predefined classes, then device complexity is reduced, but the ability to match jobs with suitable storage resources deteriorates
Solution Approach 1:
The patent applies local quality by classifying storage resources into predefined classes (performance class with memory, capacity class with storage) and matching specific job types to appropriate storage classes. This creates localized optimization where each job class receives storage resources with properties specifically suited to its requirements, thereby improving execution reliability without requiring uniform complexity across the entire system
Data Source
AI summary
Techniques are provided for scheduling computational tasks among multiple classes of storage resources based on a job classification. A job to be executed is classified into one of a plurality of predefined job classes. Each predefined job class is associated with a corresponding one of a plurality of predefined storage classes. The job is then assigned based on the classification to one of the storage resources of the predefined storage class associated with the classified predefined job class. Exemplary predefined storage classes include a performance class, a capacity class, a key-value storage class, and a shingled disk drive class. Exemplary predefined job classes include a CPU Intensive job class, an IO Intensive job class and a Small IO job class. Data required for a job is optionally prefetched before the job is assigned to a storage device. Data objects to be evicted from a storage device are optionally selected based on an anticipated future access.