High Availability Scheduler for Map-Reduce Search Clusters
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Modern data centers face challenges in processing and indexing large volumes of machine-generated data due to its unstructured nature, which complicates semantic meaning application and efficient searching operations, especially as the number of hosts and clients grows.
Innovation Solution
A high availability scheduler is implemented in a distributed system that elects a leader among server devices to schedule tasks, ensuring continuous operation even if the leader fails, by allowing other server devices to assume the scheduling role, and uses a flexible schema for data processing and searching that applies extraction rules at search time rather than ingestion time.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If a single leader server is used to schedule tasks in a distributed system, then scheduling efficiency and decision-making speed are improved, but system availability deteriorates when the leader fails
Solution Approach 1:
The system performs preliminary actions by establishing a leader election mechanism before failure occurs. When the current leader fails, the remaining servers can immediately elect a new leader without system downtime, ensuring continuous scheduling operations. This preliminary preparation of election protocols resolves the contradiction by maintaining scheduling efficiency while improving reliability.
Solution Approach 2:
The leader role is made dynamic rather than static. The system allows the leader position to change dynamically through election mechanisms when failures occur or when leaders step down. This dynamic approach ensures that scheduling efficiency is maintained by always having an active leader while reliability is improved through the ability to transition leadership seamlessly.
2Productivity
If extraction rules are applied at data ingestion time, then data processing speed is improved, but flexibility in schema adaptation deteriorates
Solution Approach 1:
The system performs preliminary data ingestion and storage without applying extraction rules immediately. Data is ingested at high speed and stored in a raw or semi-raw format. The extraction rules are applied later at search time, allowing the system to maintain fast ingestion speeds while retaining the flexibility to apply different extraction rules based on search requirements.
Solution Approach 2:
The extraction rule application is made dynamic, shifting from a static upfront application to a flexible on-demand application. The system dynamically determines when and how to apply extraction rules based on the specific search query and data type, allowing both fast initial processing and flexible subsequent adaptation.
3Manufacturing precision
If a rigid data schema is used for indexing, then indexing precision is improved, but difficulty in processing unstructured data worsens
Solution Approach 1:
The data schema is made dynamic rather than rigid. The system uses a flexible schema that can adapt to different data types and structures. Extraction rules are applied dynamically at search time to transform unstructured data into structured formats suitable for precise indexing, allowing the system to handle both structured and unstructured data effectively.
Solution Approach 2:
The system changes the parameters of data processing by applying different extraction rules and transformation parameters based on the data type and search requirements. This allows the same indexing infrastructure to handle diverse data formats with varying degrees of structure, maintaining indexing precision while reducing processing complexity.
Data Source
AI summary
A high availability scheduler of tasks in a cluster of server devices is provided. A server device of the cluster of server devices enters a leader state based upon the results of an election process in which the server device participates with others of the cluster of server devices. Upon entering the leader state, the server device schedules one or more tasks by assigning each of the one or more tasks to a device, wherein the one or more tasks involve initiating a search of time stamped events.