Serverless Function Control Over Poller Data Ingestion
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing systems face inefficiencies when processing data streams due to the need for intermediaries between data streams and serverless function executions, leading to overhead and resource duplication, especially when handling malformed data items or varying data volumes.
Innovation Solution
Implementing a control mechanism within serverless function executions to manage the intermediary system, allowing serverless functions to notify and instruct the intermediary on data resubmission, batch processing, and other operational aspects, thereby reducing inefficiencies and eliminating the need for pre-configured complex logic on the intermediary.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If an intermediary system is used between data streams and serverless function executions, then data processing can be enabled, but overhead and resource duplication occur
Solution Approach 1:
The patent extracts the control logic from the intermediary system and places it directly within the serverless function executions. The serverless functions now directly control the ingestion of data items from the stream, eliminating the need for complex pre-configured intermediary logic and reducing overhead while maintaining processing capability.
Solution Approach 2:
The patent creates an asymmetric relationship where serverless functions have direct control over data ingestion while the intermediary system provides only basic infrastructure. This asymmetric architecture allows the processing logic to be where it is most needed (in the serverless functions) rather than distributed equally across multiple components.
2Reliability
If complex pre-configured logic is implemented on the intermediary system, then error handling and resubmission control can be achieved, but device complexity increases
Solution Approach 1:
The patent inverts the traditional architecture by placing control logic in the serverless functions rather than in the intermediary system. The serverless functions now directly manage error handling, resubmission control, and data ingestion decisions, while the intermediary system provides only basic infrastructure support.
Solution Approach 2:
The serverless functions become self-sufficient by directly controlling their own data ingestion and error handling. Each function independently manages its execution flow, determines when to resubmit data items, and handles errors without requiring complex coordinated logic in the intermediary system.
3Productivity
If serverless functions directly control data ingestion, then computational efficiency is improved, but control mechanism complexity increases
Solution Approach 1:
The serverless functions directly control their own data ingestion by evaluating data items and determining when to invoke themselves. This self-service approach eliminates the need for external control mechanisms while maintaining computational efficiency, as each function independently manages its execution based on stream conditions.
Solution Approach 2:
The system implements feedback loops where serverless functions evaluate processed data items and use the results to control subsequent ingestion decisions. The functions receive feedback about processing outcomes and adjust their behavior accordingly, creating a closed-loop control system that is simple yet effective.
Data Source
AI summary
Systems and methods are described controlling ingestion of data items within a data stream by executions of a serverless function on a serverless compute system. A poller device can act as an intermediary between the data stream and the serverless function, iteratively retrieving data items from the data stream and passing them in invocations of the serverless function. To allow for fine-grained control of ingestion without requiring implementation of complex logic at the poller device, the poller device can enable the serverless function to pass instructions controlling subsequent operation of the poller device. Each execution of the serverless function may determine whether subsequent operation of the poller device should be altered, and if so, instruct the poller device accordingly. The poller device can then modify its operation pursuant to the instructions, enabling highly granular control of streaming data ingestion without inhibiting existing benefits of serverless computing.