Data Object Classification via Rule Evaluation Job Scheduling
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Data management systems face bottlenecks during peak workloads due to the inefficiencies in classifying large numbers of data objects, which is exacerbated by the need to apply different retention and security policies to various categories of data.
Innovation Solution
The implementation of systems and methods that create rule evaluation jobs for data objects based on classification rules, prioritize processing using estimated times, and leverage multi-core processors to evaluate rules in parallel, optimizing the order of operations to minimize processing time.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If traditional sequential classification methods are used to ensure accurate categorization of data objects, then classification accuracy is maintained, but processing time increases significantly during peak workloads
Solution Approach 1:
The system pre-calculates and stores the estimated processing time for each classification rule before actual classification occurs. This preliminary action enables the queue management system to optimize job scheduling in advance, allowing the system to process rules in an order that minimizes total processing time while maintaining accurate classification results.
Solution Approach 2:
The system dynamically adjusts the processing order of classification rules based on real-time queue conditions and pre-calculated estimated processing times. By implementing a dynamic scheduling mechanism that considers both accuracy requirements and time constraints, the system can adaptively optimize classification performance during varying workload conditions.
2Adaptability or versatility
If multiple classification rules are applied to ensure comprehensive data categorization, then classification completeness is improved, but processing complexity increases
Solution Approach 1:
The system segments the classification process into independent rule evaluation jobs, each handling a specific classification rule. By dividing the comprehensive classification task into smaller, manageable segments that can be processed independently and in parallel, the system maintains classification completeness while reducing the complexity of individual processing units.
Solution Approach 2:
The system introduces a queue management mechanism as an intermediary between data objects and classification rules. This intermediary layer orchestrates the application of multiple classification rules by managing job queues, scheduling processing order, and coordinating rule evaluation, thereby handling complexity centrally while keeping individual rule evaluations simple.
3Ease of operation
If classification rules are processed in sequential order to simplify job management, then system simplicity is maintained, but processing efficiency decreases
Solution Approach 1:
The system creates copy instances of the classification rule evaluation process for each rule, allowing multiple evaluations to proceed in parallel rather than sequentially. Each rule evaluation job is an independent copy that can be processed simultaneously by different processing units, dramatically improving efficiency while the queue management system maintains operational simplicity through standardized job handling.
Solution Approach 2:
The system changes the processing parameter from sequential to parallel execution by implementing a queue-based job management system. By transforming the processing mode parameter and utilizing pre-calculated estimated processing times as scheduling parameters, the system achieves both improved efficiency and maintained operational simplicity through automated queue management.
Data Source
AI summary
The disclosed computer-implemented method for efficiently classifying data objects may include (1) receiving a data object to be classified according to a group of rules, where each rule includes one or more clauses, (2) creating, for each rule, a rule evaluation job that directs a rule evaluation processor to evaluate the data object according to the clauses within the rule, where the rule evaluation processor evaluates the clauses in increasing order of estimated processing time, (3) submitting the rule evaluation jobs created for the rules to rule evaluation queues for processing by the rule evaluation processor, where the rule evaluation jobs are submitted in decreasing order of estimated processing time, (4) receiving an evaluation result for each rule evaluation job, and (5) in response to receiving the evaluation results, classifying the data object according to the evaluation results. Various other methods, systems, and computer-readable media are also disclosed.