Dynamic Execution Environment Selection for Sensor Data Analytics
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing data analytics solutions for sensor data face challenges in scalability, efficiency, and resource management across different computing environments, leading to inefficiencies and suboptimal performance due to limitations in data conversion, preprocessing, and query execution.
Innovation Solution
A processor-implemented method and system that dynamically selects and switches between distributed and non-distributed computing environments based on input parameters and metadata, optimizing resource utilization and query execution by populating and processing sensor data in the most suitable environment.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If existing solutions utilize a single computing environment, then the implementation is simple, but the scalability and adaptability across different computing environments is limited
Solution Approach 1:
The patent implements a universal framework that can execute sensor data analytics across multiple computing environments (distributed and non-distributed) through a common interface. The system defines environment-agnostic data structures and execution models that work uniformly across different infrastructure types, eliminating the need for environment-specific implementations while maintaining full functionality.
Solution Approach 2:
The patent introduces an intermediary layer consisting of abstracted execution environments and translation mechanisms that mediate between the user's analytics code and the underlying computing infrastructure. This intermediary handles environment-specific details, allowing users to write environment-agnostic code while the system automatically adapts to the target computing environment.
2Productivity
If manual configuration of environment functionalities is required, then the system can be optimized for specific environments, but the development time and code base changes increase significantly
Solution Approach 1:
The patent pre-defines standardized execution environments with common sensor data analytics workloads and configurations. These pre-configured environments include typical data processing pipelines, query patterns, and resource allocations that can be directly applied to common use cases, eliminating the need for manual configuration while maintaining optimized performance.
Solution Approach 2:
The patent implements dynamic parameter adjustment mechanisms that automatically configure execution environments based on the specific query requirements and available resources. Instead of manual configuration, the system adjusts parameters such as compute resources, memory allocation, and processing parallelism automatically, achieving environment optimization without development time investment.
3Reliability
If computing resources are over-provisioned, then the system can handle peak loads, but the resource utilization efficiency decreases
Solution Approach 1:
The patent implements dynamic resource allocation that adjusts computing resources in real-time based on actual workload demands. The system monitors query execution patterns, data volume, and resource usage metrics to automatically scale resources up or down, ensuring that sufficient capacity is available for peak loads while minimizing resource consumption during lower-demand periods.
Solution Approach 2:
The patent incorporates feedback mechanisms that continuously monitor system performance, resource utilization, and query execution outcomes. This feedback is used to automatically adjust resource allocation decisions, learning from past performance to optimize future resource distribution, thereby maintaining reliability while improving overall efficiency.
Data Source
Figure 1
Figure 2
Figure 3
AI summary
Sensor data (or IoT) analytics plays a critical role in taking business decisions for various entities (e.g., organizations, project owners, and the like). However, scaling of such analytical solutions beyond certain point requires adopting to various computing environments which seems to be challenging with the constrained resources available. Embodiments of the present disclosure provide system and method for analysing and executing sensor observational data in computing environments, wherein extract, transform, load (ETL) workflow pipeline created by users in the cloud, can be seamlessly deployed to job execution service available in cloud/edge without any changes in the code/config by end user. The configuration changes are internally handled by the system based on the selected computing environment and queries are executed either in distributed or non-distributed environments to output data frames. The data frames are further pre-processed in a desired computing environment and thereafter visualized accordingly.