Asynchronous Batch Processing via Callback Functions

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Cloud data systems face inefficiencies in processing large batches of data due to synchronous processing methods, which require significant resources and custom configurations for each target data system, leading to wasted processing resources and increased complexity.

Innovation Solution

Implementing asynchronous batch processing using unique resource identifiers and callback functions that allow for dynamic return endpoints, enabling the cloud data system to transition to a sleeping state during processing and simplifying authentication across different target systems.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Productivity

If synchronous processing methods are used to process large batches of data, then data processing can be completed, but processing resources are significantly consumed and waiting times increase

Engineering Contradiction:

Improvedata processing efficiencyVSAvoidprocessing resource consumption

Core Design Contradiction:

ProductivityVSUse of energy by moving object

Solution Approach 1:

The system dynamically transitions between active and sleeping states based on processing phase. During asynchronous batch processing, the source system sends requests and enters a sleeping state, then wakes up to receive results, avoiding continuous resource consumption while maintaining processing capability.

Inventive Principle:

Principle #15Dynamics

Solution Approach 2:

The system implements a feedback mechanism where the target system sends processing results back to the source system through callback functions. This allows the source system to efficiently retrieve results without continuous polling or waiting, reducing resource consumption while ensuring data completeness.

Inventive Principle:

Principle #23Feedback

2Productivity

If synchronous processing methods are used, then data processing can be completed, but waiting times are increased

Engineering Contradiction:

Improvedata processing throughputVSAvoidwaiting time

Core Design Contradiction:

ProductivityVSLoss of time

Solution Approach 1:

The source system performs preliminary actions by sending all batch processing requests to the target system in advance, then immediately transitions to a sleeping state. This eliminates the need for continuous waiting, as the system has already initiated all necessary processing operations before entering low-resource mode.

Inventive Principle:

Principle #10Preliminary action

Solution Approach 2:

The system introduces a callback function mechanism as an intermediary for result transmission. The target system stores results and automatically sends them back when ready, eliminating the need for the source system to continuously wait or poll, thus reducing waiting time while ensuring complete data retrieval.

Inventive Principle:

Principle #24Intermediary (Mediator)

3Adaptability or versatility

If custom configurations are implemented for each target data system, then specific system requirements can be met, but system complexity increases

Engineering Contradiction:

Improvecompatibility with different target systemsVSAvoidconfiguration complexity

Core Design Contradiction:

Adaptability or versatilityVSDevice complexity

Solution Approach 1:

The system implements a universal callback function mechanism that can be applied to any target data system regardless of its specific characteristics. The callback interface standardizes result retrieval across different systems, eliminating the need for custom configuration code for each target system while maintaining adaptability to various data formats and processing requirements.

Inventive Principle:

Principle #6Universality (Multi-functionality)

Solution Approach 2:

The system segments the integration complexity by separating the processing request mechanism from the result retrieval mechanism. The callback function acts as an independent, reusable component that handles result retrieval universally, while the source system maintains simple, system-agnostic configuration. This segmentation reduces overall complexity while preserving adaptability.

Inventive Principle:

Principle #1Segmentation

Data Source

PatentUS11226979B2Data system with asynchronous batch processing

Publication Date: 2022.01.18 SALESFORCE INC

AI summary

A data processing manager receives a batch of data for real-time data processing. The batch of data is associated with a correlation identifier and includes a set of data objects. The data processing manager assigns, to each data object, a unique resource identifier. The data processing manager transmits a data processing request to a target data system, and the data processing request includes a set of callback functions corresponding to the set of data objects. The callback functions identify the unique resource identifier. As the target data system processes the data objects, the target data system executes the callback functions, which operate as dynamic return endpoints for the data processing results.