Containerized Data Layers Using Pub-Sub Filtering for Low-Latency Access
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Distributed computing systems face issues with high volumes of irrelevant data leading to resource strain, latency, and network congestion due to continuous data collection and filtering, which hinders system performance and application efficiency.
Innovation Solution
A system that builds containerized layers using publish-subscribe topics to aggregate and filter data based on predefined parameters, storing them in a container registry for efficient access by software applications, reducing the need for continuous data collection and improving resource utilization.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of information
If continuous data collection and filtering is performed at edge devices, then data availability for software applications is improved, but resource consumption (CPU, memory, battery) increases and causes latency
Solution Approach 1:
The patent extracts the data collection and filtering functions from edge devices and relocates them to a centralized data lake. Edge devices only need to publish raw data to the data lake, while complex filtering and data preparation are performed centrally, reducing edge device resource consumption while maintaining data availability.
Solution Approach 2:
The patent introduces a data lake as an intermediary layer between data sources and software applications. This data lake acts as a buffer that pre-processes and organizes data, allowing edge devices to avoid continuous filtering operations while ensuring data is readily available when needed by applications.
2Quantity of substance
If high volume data is continuously received and processed at edge devices, then data completeness is improved, but network congestion increases and hinders data transmission
Solution Approach 1:
The patent extracts the data aggregation function from edge devices and consolidates it at the centralized data lake. Edge devices publish data asynchronously to the data lake without waiting for processing or acknowledgment, eliminating the need for continuous data push operations that cause network congestion while ensuring complete data capture.
Solution Approach 2:
The patent implements asynchronous data publishing where edge devices send data in discrete events rather than continuous streams. This periodic/event-driven approach reduces network traffic intensity and allows the system to handle high volumes of data without congestion, as the data lake can process and store data at its own pace.
3Speed
If software applications continuously listen to data sources, then real-time data access is improved, but system latency increases due to constant data processing
Solution Approach 1:
The patent implements preliminary data preparation by the data lake, which pre-filters, validates, and organizes data before it becomes available to software applications. This preliminary action ensures that when applications need data, it is already processed and ready for consumption, reducing access latency without requiring continuous processing.
Solution Approach 2:
The patent creates a copy of the data pipeline where the data lake maintains a replicated version of the data stream. Software applications subscribe to this replicated stream from the data lake rather than directly to edge devices, allowing them to access real-time data without the latency of direct continuous processing at the edge.
4Loss of information
If edge devices filter and process high volume incoming data, then relevant data identification is improved, but application performance deteriorates due to processing overhead
Solution Approach 1:
The patent extracts the complex filtering and data relevance determination logic from edge devices and implements it centrally at the data lake. Edge devices simply publish all raw data events, while the data lake applies sophisticated filtering algorithms and data validation rules, reducing edge device processing overhead while improving the accuracy of relevant data identification.
Solution Approach 2:
The data lake serves as an intermediary that handles the computationally intensive task of identifying relevant data. By placing this filtering function in the intermediary layer rather than at the edge devices, the system achieves both accurate relevant data identification and improved application performance, as edge devices are freed from heavy processing loads.
Data Source
AI summary
A computing system can be provided for building containerized layers using publish-subscribe topics to facilitate software application data consumption. For example, the computing system may receive data that satisfies a set of data parameters using a publish-subscribe topic. The publish-subscribe topic can be a logical channel for receiving data streams with the data from computing nodes in a distributed computing environment. The computing system can then assemble one or more containerized layers, which can include the data. Additionally, the computing system can deploy the one or more containerized layers in a container at one of the computing nodes.