Intelligent Messaging Grid for Big Data Ingestion
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current messaging technologies are inefficient in handling large volumes of Big Data, particularly in real-time processing, and fail to utilize available processing resources effectively, leading to underutilization and increased processing burdens due to rigid and proprietary network compositions.
Innovation Solution
An intelligent messaging grid that utilizes complex event processing (CEP) engines with dynamic configuration options for brokered or brokerless communication models, enabling data classification and routing based on metadata, and supports flexible, real-time data ingestion across geographically distributed nodes with multiple protocols and formats.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If current messaging technologies are used for Big Data ingestion, then data can be distributed to end-points, but processing resources along the way are underutilized and real-time processing capability is diminished
Solution Approach 1:
The patent segments the centralized analytics processing into distributed analytics capabilities at multiple nodes throughout the messaging grid. Each node can perform local analytics operations, filtering, and aggregation, rather than all data being funneled to a single endpoint. This segmentation enables real-time processing at the source and intermediate points, utilizing processing resources distributed across the network infrastructure.
Solution Approach 2:
The patent adds a spatial dimension to analytics processing by enabling analytics capabilities at multiple geographic locations and network levels simultaneously. Instead of a single-dimensional flow from source to endpoint, the system creates multi-dimensional processing paths where analytics can occur at edge nodes, intermediate brokers, and central endpoints, enabling real-time processing closer to data sources.
2Adaptability or versatility
If uniformly capable nodes are used across the network, then messaging functionality is consistent, but computing power available in different devices is not taken into account
Solution Approach 1:
The patent implements local quality by allowing each node in the messaging grid to have customized analytics capabilities matched to its specific computing resources. Nodes with greater processing power can handle more complex analytics operations, while nodes with limited resources perform simpler filtering and routing. This heterogeneous configuration optimizes computing power utilization across the network while maintaining consistent messaging protocols and functionality through standardized interfaces.
Solution Approach 2:
The patent enables dynamic parameter changes in node capabilities, allowing the system to adapt analytics processing intensity, data filtering complexity, and aggregation operations based on available computing resources. Nodes can adjust their processing parameters dynamically, scaling analytics operations up or down based on current resource availability, workload conditions, and data characteristics, thereby optimizing resource utilization while maintaining operational consistency.
3Adaptability or versatility
If proprietary formats and rigid network composition are used, then data ingestion is limited to specific formats, but flexibility and dynamic routing capabilities are reduced
Solution Approach 1:
The patent implements universality by designing the messaging grid to handle multiple data formats, protocols, and transmission types through a unified architecture. The system provides format-agnostic data ingestion capabilities, supporting structured, unstructured, and semi-structured data through standardized schemas. Multiple messaging protocols and transport mechanisms can coexist on the same network infrastructure, enabling flexible data exchange without requiring proprietary network compositions for each data type.
Solution Approach 2:
The patent introduces dynamics into the network composition through adaptive routing and flexible data format handling. The messaging grid dynamically adjusts data routing paths, processing operations, and format conversions based on real-time conditions, data characteristics, and node capabilities. This dynamic behavior enables the system to handle diverse data formats and protocols flexibly without requiring rigid, pre-configured network compositions for each scenario.
4Measurement precision
If all analytics are performed at the endpoint after data collection, then complete analytics can be performed, but processing burden increases and analytics speed slows down
Solution Approach 1:
The patent applies preliminary action by performing analytics operations, filtering, aggregation, and data preparation at intermediate nodes before data reaches the final endpoint. Analytics capabilities are distributed throughout the messaging grid, enabling preprocessing operations such as schema validation, data enrichment, anomaly detection, and aggregation to occur en route. This preliminary analytics processing reduces the volume and complexity of data requiring full analytics processing at the endpoint, maintaining analytics completeness while significantly reducing overall processing time and endpoint burden.
Data Source
AI summary
Certain example embodiments relate to an intelligent messaging grid for Big Data ingestion and/or associated methods. Each node in a network of nodes is dynamically configurable to send and/or receive messages using one of brokered and brokerless communication models. At least some of the nodes have a complex event processing (CEP) engine deployed thereto, the CEP engines being configured to operate on messages received by the respective nodes and being classified as one of at least two different types of CEP engines. For each message received by a given node that is to be forwarded to a further node along one of multiple possible paths, the given node is configured to route the message to be forwarded to an intermediate node in one of the possible paths. The intermediate node is selected by the CEP engine of the given node based on metadata associated with the message to be forwarded.