Data Lake Partitioning for Complete Stream Payload Recovery
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional stream-processing software in networked computing systems fails to consume a full payload of data, leading to incomplete data sets and dropped messages, necessitating improved fault tolerance in streaming architectures.
Innovation Solution
A system that generates data partitions within data lakes, identifies offset numbers for each payload, and performs quality checks to automatically reconcile missing data, reducing manual intervention and computational resources.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If stream-processing software consumes data from stream-processing logs, then real-time data processing is achieved, but data completeness deteriorates due to application server failures
Solution Approach 1:
The system performs preliminary actions by writing data to both the stream-processing log and the data lake before application servers consume it. This redundant preliminary storage ensures that even if application servers fail to consume data completely, the original data remains preserved in the data lake for later reconciliation, thus maintaining both processing speed and data completeness.
Solution Approach 2:
The system implements a feedback mechanism where the application server periodically reconciles data between the data lake and stream-processing log by comparing offset numbers. This feedback loop detects missing payloads and triggers automatic re-consumption from the log, ensuring data completeness is restored after any consumption failures, thereby resolving the contradiction between fast processing and complete reliability.
2Reliability
If application servers consume full data payloads, then data completeness is improved, but system complexity increases due to fault tolerance mechanisms
Solution Approach 1:
The data lake serves as an intermediary between the stream-processing log and application servers. It provides a stable, immutable copy of the original data that simplifies the reconciliation process. The application server only needs to compare offsets between the log and data lake, and selectively re-consume missing data, rather than implementing complex distributed consensus protocols, thus reducing overall system complexity while maintaining reliability.
Solution Approach 2:
The system segments the data consumption responsibility: the stream-processing log maintains the authoritative sequence, the data lake preserves immutable copies, and application servers selectively re-consume only missing payloads based on offset comparison. This segmentation allows each component to focus on its specific function, reducing the complexity any single component must handle while achieving overall fault tolerance.
3Reliability
If manual intervention is used to recover missing data, then data completeness is restored, but loss of time increases due to manual recovery processes
Solution Approach 1:
The application server performs self-service data recovery by automatically detecting missing payloads through offset number comparison and autonomously re-consuming data from the stream-processing log. This eliminates the need for manual intervention entirely, as the system automatically identifies and recovers from its own consumption failures, restoring data completeness without human involvement and minimizing recovery time.
Solution Approach 2:
The system maintains preliminary records of all consumed data in the data lake with associated offset numbers. When reconciliation is needed, these preliminary records enable rapid identification of missing payloads through simple offset comparison, avoiding the need for time-consuming manual data reconstruction or full data re-processing, thus enabling fast automatic recovery.
Data Source
AI summary
In conventional networked computing systems, stream-processing software is utilized to handle real-time data feeds. For example, a stream-processing software may sort incoming data feeds into data logs, which are then consumed by various application servers within the computing system. However, this consumption process is susceptible to failure as application servers often fail to consume a full payload of data comprising a particular message. As such, a need exists for a streaming architecture with improved fault tolerance. The system provided herein solves the problem via a reliable streaming process, which regularly generates partitions within a series of data lakes. As streaming data is received, the payloads are synced into the different data lakes based on metadata associated with the streaming data. The system can then perform individual quality checks on each partition of data, enabling the system to continuously monitor data quality over time.