Asynchronous Command Service for Active Data Warehousing

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Active data warehouses require improved mechanisms for real-time monitoring and control of job performance and resource management in parallel processing environments to handle high demand for tactical decision queries and ensure continuous data availability and consistency.

Innovation Solution

An asynchronous command processing service that allows users to issue commands to dynamically manage and control running jobs and resources within a parallel processing data warehousing environment, enabling real-time processing, job monitoring, and resource optimization through demand-based, time-based, event-based, and data-driven commands.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Ease of operation

If data warehousing components are made more intelligent and controllable in real-time, then job monitoring and control capability is improved, but system complexity increases

Engineering Contradiction:

Improvejob monitoring and control capabilityVSAvoidsystem complexity

Core Design Contradiction:

Ease of operationVSDevice complexity

Solution Approach 1:

The patent introduces an asynchronous command processing service as an intermediary layer between users and the parallel processing data warehousing environment. This service receives, validates, queues, and processes commands asynchronously, mediating between user control needs and system complexity by handling command routing and coordination internally without requiring users to manage underlying complexity

Inventive Principle:

Principle #24Intermediary (Mediator)

Solution Approach 2:

The system implements feedback mechanisms where the asynchronous command processing service monitors job execution status and provides updates to users. The service tracks command processing states, job performance metrics, and resource utilization, then feeds this information back through the command interface enabling continuous monitoring and dynamic control adjustments

Inventive Principle:

Principle #23Feedback

2Speed

If real-time command processing is implemented, then responsiveness to tactical queries is improved, but system resource consumption increases

Engineering Contradiction:

Improvecommand processing speedVSAvoidsystem resource consumption

Core Design Contradiction:

SpeedVSUse of energy by moving object

Solution Approach 1:

The asynchronous command processing service processes commands in periodic batches rather than immediately for each incoming command. Commands are queued and processed at scheduled intervals using available system resources, transforming continuous real-time processing into periodic action that reduces peak resource consumption while maintaining acceptable responsiveness

Inventive Principle:

Principle #19Periodic action

Solution Approach 2:

The system dynamically adjusts command processing behavior based on current system resource availability and workload conditions. The asynchronous service monitors resource states and adapts its processing rate, queue management, and execution scheduling dynamically, allowing fast processing when resources are abundant and throttling when resources are constrained

Inventive Principle:

Principle #15Dynamics

3Reliability

If continuous data loading and querying is supported, then data availability is improved, but data consistency challenges increase

Engineering Contradiction:

Improvedata availabilityVSAvoiddata consistency

Core Design Contradiction:

ReliabilityVSStability of the object's composition

Solution Approach 1:

The system performs preliminary actions by pre-validating commands, pre-checking data prerequisites, and pre-coordinating load and query operations before execution. The asynchronous command processing service prepares transaction scripts, validates data states, and schedules operations in advance to prevent consistency issues before they occur during continuous data loading and querying

Inventive Principle:

Principle #10Preliminary action

Solution Approach 2:

The system implements beforehand cushioning through transaction logging, checkpoint mechanisms, and error recovery preparations that are set up in advance of potential consistency problems. These protective measures cushion against data inconsistency risks during continuous operations by enabling rollback and recovery if consistency issues arise

Inventive Principle:

Principle #11Beforehand cushioning (Prior cushioning)

Data Source

PatentUS8027962B2Techniques for asynchronous command interface for scalable and active data warehousing

Publication Date: 2011.09.27 TERADATA US INC

AI summary

Techniques for asynchronous command processing within a parallel processing environment are provided. A command is raised or received within a parallel processing data warehousing environment. A job or a component of the job is dynamically monitored, controlled, or modified in response to the real-time processing of the command. The job is actively processing within the parallel processing data warehousing environment when the command is received and processed against the job or the component of the job.