Dynamic Shard Management and TMO Scheduling for Big Data Concurrency
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current distributed computing systems for big data analysis face performance degradation, inefficient management, and limitations in real-time stream processing, particularly with large datasets, leading to response delays and complexity in environment settings, as well as bottlenecks and increased error costs due to the inability to support various data types simultaneously.
Innovation Solution
A dynamic big data processing method that adjusts the number of shards in a distributed database and employs Time-triggered Message-triggered Object (TMO) scheduling to ensure real-timeliness, while selecting between micro-batching and event stream processing methods based on data type to optimize concurrency and processing speed.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of time
If event stream processing method is used to guarantee short delay time and process all logics, then delay time is reduced and state management is simplified, but bottleneck phenomenon occurs when data is concentrated on certain key and error treatment costs increase
Solution Approach 1:
The patent merges event stream processing and micro-batching processing into a unified hybrid processing framework. The system dynamically selects between event stream processing (for low-latency requirements) and micro-batching processing (for high-throughput requirements), combining the advantages of both methods to resolve the contradiction between delay time and processing throughput.
2Productivity
If micro-batching method is used to bundle data in array units for reduced error treatment costs and high processing rates, then processing throughput increases, but delay time increases and limitations in processing logics occur
Solution Approach 1:
The patent implements dynamic processing mode selection where the system can switch between event stream processing and micro-batching processing based on real-time data characteristics and service requirements. This dynamic adaptation allows the system to optimize between throughput and delay time depending on the specific processing scenario.
3Ease of manufacture
If conventional framework is used for micro-batching processing, then implementation is straightforward, but delay time significantly increases due to data concentration on certain time zone
Solution Approach 1:
The patent introduces an intermediary layer (the hybrid processing framework) that sits between data ingestion and processing. This intermediary dynamically routes data to appropriate processing modes (event stream or micro-batching) based on timing characteristics, preventing the delay time increase issue of conventional micro-batching while maintaining implementation feasibility.
4Reliability
If real-time processing framework is altered or integrated with another platform according to service characteristics or structural classification, then performance issues are solved, but device complexity increases
Solution Approach 1:
The patent creates a universal hybrid processing framework that can handle multiple service characteristics and data structures through a single unified architecture. The framework automatically adapts to different service requirements (real-time, near-real-time, batch) without requiring separate platform integrations, thereby reducing complexity while maintaining performance predictability.
Data Source
AI summary
Provided herein is an In-Memory DB connection support type scheduling method and system for real-time big data analysis in distributed computing environment. The data processing method according to an embodiment of the present disclosure analyzes data based on a distributed computing environment using a distributed system and dynamically alters a structure of a distributed DB constituting the distributed system based on the distributed computing environment. By this method, it is possible to secure concurrency adaptively to the distributed computing environment by dynamically managing the number of shards, and secure real-timeliness through TMO-based scheduling, thereby ultimately improving the speed/efficiency of big data analysis.