Serverless Data Ingestion and Transformation for Scalable Endpoints
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional computing systems and architectures for ingesting, transforming, and delivering data to endpoints, such as security event logs and system audit logs, are costly, difficult to scale, and prone to downtimes, leading to inefficiencies and resource wastage.
Innovation Solution
Implementing a serverless computing environment with computing code and microservices to dynamically allocate resources for data ingestion, transformation, and loading, using pre-coded data packages and configurations to create processing threads for efficient data delivery.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If specifically configured and designated servers are used for data processing and delivery, then data processing can be performed with dedicated infrastructure, but the system becomes costly, difficult to change, and hard to scale
Solution Approach 1:
The patent applies universality by using a single cloud-based data processing platform that can serve multiple endpoints and handle various types of data processing tasks. Instead of having specifically configured servers for each endpoint, the invention uses a universal cloud infrastructure that dynamically allocates resources to different data processing flows, making the system adaptable and easier to modify without requiring changes to dedicated hardware configurations.
2Productivity
If specifically configured servers are deployed for data delivery, then data can be processed with dedicated resources, but the system becomes difficult to update and maintain
Solution Approach 1:
The patent applies copying by virtualizing data processing resources in the cloud, where multiple virtual instances can be created and managed without affecting physical hardware. This allows the system to maintain high productivity through dedicated virtual resources while enabling easy updates and maintenance by simply modifying or replacing virtual instances without disrupting the underlying physical infrastructure.
3Reliability
If conventional server systems are used for data processing, then data can be ingested and delivered, but the system slows down or becomes unusable during downtimes, servicing, and updates
Solution Approach 1:
The patent applies the intermediary principle by introducing a cloud-based data processing platform as a mediator between data sources and endpoints. This cloud intermediary handles data processing, transformation, and delivery, isolating the endpoints from server downtimes and maintenance activities. The cloud platform ensures continuous availability by managing failover and load balancing, maintaining both reliability and productivity even during infrastructure servicing.
4Adaptability or versatility
If dedicated servers are allocated for specific data processing tasks, then resources can be assigned to specific functions, but scalability becomes limited and costly
Solution Approach 1:
The patent applies dynamics by implementing dynamic resource allocation in the cloud-based platform, where computing resources are automatically adjusted based on demand. Instead of static dedicated server allocations, the system dynamically provisions and de-provisions resources as needed, enabling both adaptability for different data processing requirements and scalability to handle varying workloads without being constrained by fixed infrastructure allocations.
Data Source
AI summary
There are provided systems and methods for serverless data ingestion for transforming and delivering data in system endpoints. An entity, such as company or business, may utilize computing services provided by a service provider. When providing these services, one or more computing services, processors, or the like of the service provider's computing architecture may be used. This may include use of serverless computing environment to delivery data to internal and/or external endpoints. To automate data delivery in a serverless environment, computing code for data processing and delivery pipelines may be configured and deployed in the serverless environment. This allows applications and microservices to be executed in the serverless environment to provide a computing architecture for without requiring servers and designation of server clusters. The data pipeline may provide operations for ingesting, transforming, and loading data when received for delivery to endpoints.