Data Synchronization Without Middleware Using Delta Loads
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional data synchronization approaches face challenges in accessing and processing data from different source systems due to varying APIs, data storage formats, reliance on middleware systems, inefficiencies in initial and delta synchronization, and the inability to handle multiple source systems simultaneously.
Innovation Solution
A system and method for synchronizing data from a source system to an application without middleware, utilizing a synchronization manager that performs a three-step process: data retrieval, mapping, and updating, with custom and REST APIs, and employing custom and template mappers to handle diverse data formats and systems, while using metadata files for configuration and managing delta loads and deletions.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If conventional data synchronization approaches are used, then data can be replicated from source systems, but middleware systems are required which increase system complexity and operational overhead
Solution Approach 1:
The patent extracts and removes the middleware component from the data synchronization architecture. The application directly connects to source systems using standard protocols (SFTP, JDBC, ODBC, RESTful APIs) to retrieve data, eliminating the need for middleware while maintaining data replication capability. This directly resolves the contradiction by removing the complexity-introducing component while preserving the core function.
Solution Approach 2:
The application performs data synchronization operations autonomously without requiring intermediate middleware services. The system self-manages data retrieval, transformation, and loading processes using built-in capabilities and standard communication protocols, thereby eliminating middleware dependency and reducing system complexity while maintaining reliable data replication.
2Adaptability or versatility
If conventional data synchronization approaches are used, then data can be accessed from source systems, but different APIs require additional adaptation layers increasing complexity
Solution Approach 1:
The patent implements a universal data access approach where the application can connect to multiple different source systems (ERP, CRM, data warehouses, cloud systems) using standard, widely-supported protocols such as SFTP, JDBC, ODBC, and RESTful APIs. This multi-functional capability allows the system to access diverse data sources without requiring custom adaptation layers for each API, thereby maintaining versatility while reducing complexity.
3Productivity
If conventional data synchronization approaches are used, then data can be replicated, but handling both initial and delta synchronization efficiently is challenging
Solution Approach 1:
The patent implements periodic action by distinguishing between initial full data loads and subsequent delta (incremental) synchronization operations. The system performs an initial comprehensive data retrieval and replication, then schedules periodic delta synchronization operations that only transfer changed data since the last synchronization. This periodic approach significantly reduces data transfer time and processing overhead for routine updates while maintaining data currency.
Solution Approach 2:
The system performs preliminary actions by establishing connection parameters, authentication credentials, and synchronization schedules in advance. Metadata files are pre-configured with source system connection details, data field mappings, and synchronization preferences. This preliminary setup enables rapid execution of both initial and delta synchronization operations without requiring complex runtime decision-making or adaptive configuration.
4Adaptability or versatility
If conventional data synchronization approaches are used, then data can be synchronized, but handling multiple source systems simultaneously increases complexity
Solution Approach 1:
The patent segments the multi-source data synchronization task into independent, manageable units. Each source system connection and data retrieval operation is handled as a separate, modular process. The application can independently connect to multiple source systems (ERP, CRM, data warehouses, cloud systems) using the same standard protocols, processing each source separately without requiring complex coordination mechanisms. This segmentation maintains adaptability for handling multiple sources while minimizing coordination complexity.
Data Source
AI summary
Disclosed herein are system, method, and computer program product embodiments for synchronizing data in a source system to an application without relying on middleware. An embodiment operates by performing an initial load of records for an object for an application from a source system, thereby retrieving a first set of records. The embodiment first maps the first set of records to the object using a transform template. The embodiment creates the object in the application using the first set of records based on the first mapping. The embodiment performs a delta load of records for the object from the source system, thereby retrieving a second set of records. The embodiment second maps the second set of records to the object using the transform template. The embodiment then updates the object using the second set of records based on the second mapping.