Redundant Ingestion Endpoints with Deduplication for Data Intake Systems
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing data intake and query systems face issues with data loss due to network congestion and device failures during machine data ingestion from remote sources, leading to undesirable latency and resource inefficiencies in failover processes.
Innovation Solution
Implementing a redundant set of ingestion endpoints with deduplication techniques to ensure reliable data ingestion across potentially unreliable networks, allowing messages to be sent to multiple endpoints and processed only once, while dynamically scaling and sharding endpoints to adapt to changing workload conditions.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If a single ingestion endpoint is used, then device complexity is reduced, but data loss occurs due to network congestion and device failures
Solution Approach 1:
The system segments the ingestion functionality into multiple independent ingestion endpoints distributed across different devices. Each endpoint can independently receive and process messages, so if one endpoint fails or experiences network congestion, other endpoints continue to operate, preventing data loss while maintaining manageable individual device complexity.
Solution Approach 2:
The system changes the parameter of endpoint redundancy from single to multiple, transforming the ingestion architecture from a single point of failure to a distributed resilient system. This parameter change enables the system to tolerate network congestion and device failures without compromising data ingestion reliability.
2Reliability
If redundant ingestion endpoints are implemented, then data loss is prevented, but resource usage increases due to multiple endpoints processing messages
Solution Approach 1:
The system creates copies of messages and distributes them to multiple ingestion endpoints for redundant processing. Each endpoint receives a copy of the same message, ensuring that at least one copy is successfully processed even if some endpoints fail or experience network issues, thereby preventing data loss while managing resource consumption through controlled replication.
Solution Approach 2:
The system merges the processing results from multiple ingestion endpoints into a single unified output. By combining the efforts of multiple endpoints and consolidating their results, the system achieves reliable message delivery without duplicating the entire processing pipeline, thus reducing overall resource usage while maintaining redundancy.
3Reliability
If failover processes are implemented, then endpoint outages are mitigated, but latency increases due to failover detection and switching
Solution Approach 1:
The system performs preliminary actions by pre-configuring multiple ingestion endpoints and establishing their availability before failures occur. This proactive approach allows the system to immediately route messages to available endpoints without requiring detection and switching processes, thereby maintaining high endpoint availability while minimizing failover latency.
Solution Approach 2:
The system introduces an intermediary component that manages message routing between data sources and multiple ingestion endpoints. This intermediary handles the complexity of endpoint selection and failover logic centrally, allowing individual endpoints to operate independently with minimal latency, while the intermediary coordinates overall system reliability and availability.
4Reliability
If multiple ingestion endpoints are used, then data loss is reduced, but device complexity increases due to endpoint management
Solution Approach 1:
The system implements self-service mechanisms where each ingestion endpoint autonomously manages its own operation, including receiving messages, processing data, and reporting status. This decentralized self-management approach reduces the overall complexity of endpoint management by eliminating the need for complex centralized coordination, while still maintaining high data ingestion reliability through redundant endpoint operation.
Data Source
AI summary
Techniques are described for providing a highly available data ingestion system for ingesting machine data sent from remote data sources across potentially unreliable networks. To provide for highly available delivery of such data, a data intake and query system provides users with redundant sets of ingestion endpoints to which messages sent from users' computing environments can be delivered to the data intake and query system. Users' data sources, or data forwarding components configured to obtain and send data from one or more data sources, are then configured to encapsulate obtained machine data into discrete messages and to send copies of each message to two or more of the ingestion endpoints provisioned for a user. The ingestion endpoints receiving the messages implement a deduplication technique and provide only one copy of each message to a subsequent processing component (e.g., to an indexing subsystem for event generation, event indexing, etc.).