Data Extraction Job Specification Segregating Business Logic
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Deploying data extraction systems is time-consuming and error-prone due to the need for custom scripting, which intermingles business logic with data extraction logic, compromising system integrity and security.
Innovation Solution
A data extraction system that uses an extraction job specification to programmatically extract data without custom scripting, separating business logic from data extraction, allowing for rapid deployment and expansion, and ensuring data integrity and security by applying business logic solely at the data recipient.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If custom scripts are used for data extraction, then data extraction can be performed, but deployment becomes time-consuming and error-prone
Solution Approach 1:
The patent segments the data extraction system into distinct components: a standardized extraction engine that handles the extraction logic separately from business logic. This segmentation allows the extraction engine to be deployed once and reused across multiple data sources, eliminating the need for custom scripting for each deployment while reducing errors through standardized processes.
Solution Approach 2:
The patent introduces an intermediary layer (the standardized extraction engine) that sits between the data sources and the business logic. This intermediary handles all extraction operations using consistent, pre-tested logic, preventing errors that would otherwise propagate from custom scripts while maintaining the ability to extract from various sources.
2Ease of operation
If business logic is intermingled with data extraction logic, then data can be extracted and processed, but system integrity and security are compromised
Solution Approach 1:
The patent applies segmentation by clearly separating business logic from data extraction logic into distinct system components. The extraction engine handles only extraction operations with standardized logic, while business logic resides in separate processing systems. This separation maintains system integrity by preventing contamination of extraction logic with untested business rules while preserving full data processing capability through defined interfaces.
Solution Approach 2:
The patent extracts business logic from the data extraction process entirely. By taking out business logic as a separate component that operates on extracted data rather than during extraction, the system maintains integrity of the extraction engine while still enabling comprehensive business processing through the standardized data interface.
3Adaptability or versatility
If custom solutions are used for data extraction, then specific client requirements can be met, but the solutions become more error-prone and bug-prone
Solution Approach 1:
The patent creates a universal extraction engine that can handle multiple data sources and client requirements through configuration rather than custom code. The standardized engine provides multi-functionality by supporting various extraction scenarios through a common platform, reducing bugs by eliminating the need for multiple custom solutions while maintaining adaptability through flexible configuration options.
Data Source
AI summary
Computer-implemented techniques for data extraction are described. The techniques include a method and system for retrieving an extraction job specification, wherein the extraction job specification comprises a source repository identifier that identifies a source repository comprising a plurality of data records; a data recipient identifier that identifies a data recipient; and a schedule that indicates a timing of when to retrieve the plurality of data records. The method and system further include retrieving the plurality of data records from the source repository based on the schedule, creating an extraction transaction from the plurality of data records, wherein the extraction transaction comprises a subset of the plurality of data records and metadata, and sending the extraction transaction to the data recipient.