Continuous Workload Tracking Using Paged Command Batches
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing data processing systems face challenges in efficiently tracking large volumes of data processing workloads due to storage limitations and the need for accurate monitoring, especially in compliance with regulations, as current solutions require excessive storage space and fail to scale effectively.
Innovation Solution
The system divides received commands into discrete units, assigning unique identifiers to each unit, which are sorted into pages and monitored in an ordered fashion, reducing storage requirements by expressing completion using fewer data points.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If each command is tracked individually in current solutions, then accurate monitoring is achieved, but storage requirements become excessive (90 terabytes)
Solution Approach 1:
Multiple individual command tracking records are merged into aggregate batch records. The system groups commands into batches and tracks them collectively using a single completion status per batch, rather than maintaining separate tracking entries for each individual command. This merging reduces the total number of tracking records from potentially millions of individual command records to a manageable number of batch records.
Solution Approach 2:
The continuous stream of commands is segmented into discrete batches or groups. Each batch is assigned a unique identifier and tracked as a separate unit. This segmentation allows the system to manage large volumes of commands by dividing them into smaller, more manageable units that can be tracked efficiently without requiring excessive storage space for individual command tracking.
2Adaptability or versatility
If large volumes of workload data are monitored, then comprehensive compliance tracking is achieved, but system scalability is limited due to storage constraints
Solution Approach 1:
The system combines multiple command tracking operations into aggregate batch operations. By merging individual command monitoring into batch-level monitoring, the system can handle larger volumes of workloads without proportionally increasing storage requirements, thereby improving scalability while maintaining comprehensive compliance tracking capability.
Solution Approach 2:
The system changes the granularity parameter of tracking from individual command level to batch level. This parameter change allows the system to scale more effectively by adjusting batch sizes and frequencies, enabling comprehensive monitoring of large workload volumes while managing storage consumption through optimized data aggregation parameters.
3Reliability
If detailed command-level tracking is maintained, then complete execution monitoring is achieved, but data processing efficiency decreases due to excessive storage and retrieval operations
Solution Approach 1:
Individual command tracking operations are merged into batch-level tracking operations. This merging reduces the total number of storage and retrieval operations required, as the system performs fewer, larger aggregate operations instead of numerous small individual operations. This improves data processing efficiency while maintaining complete execution monitoring through batch completion status tracking.
Solution Approach 2:
The system performs preliminary grouping and batching of commands before tracking begins. By organizing commands into batches in advance and assigning batch identifiers beforehand, the system eliminates the need for individual command tracking during execution, thereby improving processing efficiency while maintaining monitoring completeness through batch-level status tracking.
Data Source
AI summary
A stream of continuous data processing commands or workloads may be tracked for completeness. A stream of signals is received that includes a plurality of commands to be executed. At least one timeslice is generated in the received stream which corresponds to a period of time and includes the plurality of commands received in the period of time. The plurality of commands is categorized into at least one page and a unique identifier is assigned to the at least one page. The at least one page, including the unique identifier, is transmitted to a command executing agent and in response to receiving an indication that the at least one page is completed, each of the plurality of commands included in the at least one page is marked as completed.