Distributed Data Ingestion via Management Node Buffering
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Modern distributed information systems face data loss during log data ingestion due to unavailability of the underlying data platform, which undermines operational visibility and troubleshooting efficiency.
Innovation Solution
A distributed-computing system processes data at a management node, replicates it across multiple management nodes, and selects a data core for ingestion, enabling reinitiation of the ingestion process in case of failures and providing immediate confirmation to software agents, thus reducing data loss and improving reliability.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If log data is aggregated and ingested from multiple sources using software agents, then operational visibility and troubleshooting capability are improved, but data loss occurs when the data platform becomes unavailable
Solution Approach 1:
The patent implements preliminary action by having software agents store log data locally in a buffer or queue before ingestion requests are sent to the data platform. This local storage ensures that data is preserved even when the platform is unavailable, preventing data loss during platform outages or maintenance windows.
Solution Approach 2:
The patent introduces an intermediary component (buffer/queue system) between the software agents and the data platform. This intermediary decouples the data collection process from the data ingestion process, allowing agents to continue collecting and buffering data independently of platform availability, thereby eliminating data loss when the platform is unavailable.
2Productivity
If the data platform processes large volumes of log data from multiple sources, then comprehensive operational monitoring is achieved, but the platform becomes vulnerable to unavailability due to schema failures, hardware upgrades, or ingestion traffic
Solution Approach 1:
The patent segments the data ingestion architecture into independent components: software agents that collect data, a buffer/queue system that temporarily stores data, and the data platform that processes data. This segmentation allows each component to operate independently, so platform unavailability due to schema failures, hardware upgrades, or high ingestion traffic does not affect data collection or cause data loss.
Solution Approach 2:
The buffer/queue system performs preliminary storage of log data before the data platform processes it. This preliminary action decouples data collection from data processing, allowing the platform to handle large volumes of data at its own pace without risking data loss during platform maintenance or high-traffic periods.
3Productivity
If software agents periodically request ingestion of log data, then data is transmitted to the data platform, but data loss occurs during ingestion requests when the platform is unavailable
Solution Approach 1:
The patent introduces a buffer/queue intermediary between software agents and the data platform. Agents send data to this intermediary, which holds data temporarily and forwards it to the platform when available. This intermediary prevents data loss during platform unavailability by storing data locally, while maintaining efficient batch processing when the platform is operational.
Solution Approach 2:
The buffer system performs preliminary storage of incoming log data before ingestion requests are attempted. This preliminary action ensures data is captured and preserved even when the platform is unavailable, eliminating data loss during ingestion requests while maintaining efficient batch processing when the platform is operational.
Data Source
AI summary
Techniques for data ingestion by a distributed-computing system are provided. In one embodiment, data received from one or more data sources is processed at a management node of the distributed-computing system. The data is generated by one or more components of an information processing system external to the distributed-computing system. The data is stored at the management plane. The management plane selects, from a plurality of data cores, a data core to ingest the data. The plurality of data cores are stored across one or more data plane containers of a data plane of the distributed-computing system. The management plane processes the data to generate one or more event records corresponding to the data. The one or more event records are sent to the selected data core, which causes the data plane to store the one or more event records in the selected data core.