REST ETL Step with Dynamic Schema and Reusable Connections
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Traditional applications like RESTClient are not designed to handle multiple REST service calls, which are often required for complex use cases, and lack flexibility to support customizations defined by REST service providers, failing to effectively parse and compose documents in various formats.
Innovation Solution
A computer-implemented method for executing a computing job with multiple steps that includes receiving events, transmitting REST requests, and generating new events based on responses, with configurable steps via a graphical user interface (GUI) that dynamically generates input and output schemas, allowing for reusable connections and parameterized URLs to support multiple REST service calls.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of operation
If traditional RESTClient applications are used for making individual REST service calls, then simplicity of operation is maintained, but inability to handle multiple REST service calls and lack of flexibility for customizations occurs
Solution Approach 1:
The patent segments the REST service calling process into distinct steps: connection management, request formulation, response parsing, and event generation. Each step is handled by a dedicated component (ConnectionManager, RequestBuilder, Parser), allowing the system to manage multiple service calls systematically while maintaining operational simplicity through modular architecture.
Solution Approach 2:
The patent creates a universal REST service framework that can handle multiple different REST service calls through a single integrated system. The framework supports various service types (cloud services, enterprise services, NoSQL databases) through common interfaces and event-driven mechanisms, providing versatility without requiring separate applications for each service type.
2Device complexity
If traditional RESTClient applications are used, then device complexity is kept low, but inability to parse and compose documents in various formats occurs
Solution Approach 1:
The patent introduces a Parser as an intermediary component between the REST response and the rest of the system. This Parser handles the complex task of parsing and composing documents in various formats (JSON, XML, URL-encoded) and generating events, shielding the rest of the system from format complexity while maintaining high adaptability to different data structures.
3Ease of operation
If REST services are called individually using traditional tools, then operation simplicity is maintained, but inability to support customizations defined by REST service providers occurs
Solution Approach 1:
The patent implements dynamic configuration capabilities where REST service calls can be adapted to support customizations defined by service providers. The system allows dynamic adjustment of request parameters, headers, and response parsing logic through configuration mechanisms, enabling support for various customization schemes without changing the core operational simplicity of the application.
Data Source
AI summary
Techniques are disclosed for consuming and integrating with representational state transfer (REST) based services. In one embodiment, a multiple-step extract, transform, load (ETL) job includes one or more REST steps. Each step in the ETL job performs a particular well-defined task, and the REST step in particular constructs REST requests, invokes REST services, and retrieves REST responses. The REST step may be mapped to a vector of inputs such that multiple REST service calls are made for each of the inputs. Further, a reusable connection may be employed so that one connection can be used to make multiple REST service calls. URLs used to make REST service calls may be dynamically constructed based on input data parameters and ETL job parameters. Additionally, a graphical user interface (GUI) allows users to configure aspects of a REST step.