Intelligent data replication

Abstract

Embodiments include systems and methods for data replication. In some embodiments, the method includes analyzing, by at least one scanning agent, metadata and application code associated with at least one data source, the at least one scanning agent and the at least one data source deployed on a mainframe computer system; transmitting information about the analyzed metadata and application code; receiving, by a replication agent, a replication request specifying at least one source data record in the at least one data source; transforming the at least one source data record based on the transmitted information; and transmitting the at least one transformed source data record to a target destination.

Description

UTILITY PATENT APPLICATION DOCKET NO. P0068W01INTELLIGENT DATA REPLICATION CROSS REFERENCE TO RELATED APPLICATIONS

[0001] The present application claims the benefit of and priority to co-pending United States Provisional Application No. 63 / 756,738, filed on February 10, 2025, the content of which is hereby incorporated by reference as if set forth in its entirety herein.TECHNICAL FIELD

[0002] Embodiments described herein generally relate to systems and methods for providing intelligent data replication and, more particularly but not exclusively, to systems and methods for replicating data sets from mainframes to modern computing systems.BACKGROUND

[0003] Large-scale enterprises and organizations frequently rely on mainframe computer systems to perform critical data processing tasks. These tasks often include large data processing operations such as census data aggregation, industry and consumer statistics compilation, enterprise resource planning (ERP), and / or high-volume transactional processing. Many of the mainframe systems currently in use were developed in the 1960s, prior to the advent of widespread internet usage and adoption of standardized formats. As such, these systems are often characterized by proprietary formats, rigid data structures, and operational paradigms that require specialized technical knowledge and expertise to operate effectively.

[0004] In modern times, data infrastructure paradigms, such as data fabrics, have emerged to support data integration, accessibility, and interoperability across diverse platforms and data sources. A data fabric typically includes a unified architecture that employs data services, application programming interfaces (APIs), and standardized data models to facilitate seamless access to data from a wide range of sources. These sources may include legacy systems, data lakes, data warehouses, relational databases (e.g., SQL), and / or contemporary software applications. One of the key advantages of a data fabric is its ability to present data in open, standardized formats that are readily consumable by data analysts and other non-technical users.DOCKET NO. P0068W01

[0005] Integrating mainframe computer systems with data fabrics and other modern systems, however, may be challenging. Replicating data from mainframes often requires extensive manual intervention. For example, experienced mainframe operators may have to conduct detailed interviews with various subject matter experts, perform a thorough review of various documentation and legacy system configurations, and manually identify data sets appropriate for replication. This may be overly cumbersome and inefficient, particularly for operators with little experience with mainframe systems. The steep learning curve, combined with the need for precision in data mapping and metadata generation, can result in substantial delays even for relatively basic replication tasks. Moreover, once the required information is collected, the information must be correctly structured and formatted in accordance with the target computing system to ensure that the replication process operates as intended.

[0006] Accordingly, there exists a need for improved methods and systems for replicating data sets from mainframes to modern computing systems.SUMMARY

[0007] This summary is provided to introduce a selection of concepts in a simplified form that are further described below in the Detailed Description section. This summary is not intended to identify or exclude key features or essential features of the claimed subject matter, nor is it intended to be used as an aid in determining the scope of the claimed subject matter.

[0008] In some aspects, the techniques described herein relate to a method for data replication, the method including: analyzing, by at least one scanning agent, metadata and application code associated with at least one data source, the at least one scanning agent and the at least one data source deployed on a mainframe computer system; transmitting information about the analyzed metadata and application code; receiving, by a replication agent, a replication request specifying at least one source data record in the at least one data source; transforming the at least one source data record based on the transmitted information; and transmitting the at least one transformed source data record to a target destination.

[0009] In some aspects, the analyzing of the metadata and application code includes identifying available data records at the at least one data source; and mapping the available data records to corresponding data formats.DOCKET NO. P0068W01

[0010] In some aspects, the mapping of the available data records includes identifying a logical structure regarding at least one available data record; and mapping the at least one available record to a corresponding data format based on the logical structure.

[0011] In some aspects, the mapping of the available data records includes translating at least one field name of the available data records to be more readable.

[0012] In some aspects, the method further includes providing a user interface for receiving selections of data sources and target destinations, wherein the replication request is received at the user interface.

[0013] In some aspects, the method further includes identifying available data sources; and presenting, at the user interface, a list of the available data sources.

[0014] In some aspects, the target destination is located at a cloud computing environment.

[0015] In some aspects, the method further includes analyzing, by at least one target scanning agent, metadata and application code associated with the target destination.

[0016] In some aspects, the at least one scanning agent and the at least one target scanning agent are deployed at the same time.

[0017] In some aspects, the metadata and application code are associated with a proprietary format usable by the mainframe computer system, and wherein the at least one transformed source data record is in a different format usable by the target destination.

[0018] In some aspects, the proprietary format includes at least one of VSAM, DB2, AD AB AS, IMS, IDMS, or SEQUENTIAL.

[0019] In some aspects, the method further includes storing an inventory of available data sources at a persistent data repository, the inventory including the transmitted information.

[0020] In some aspects, the techniques described herein relate to a system for data replication, the system including: a mainframe computer system including: at least one data source; a scanning agent configured to: analyze metadata and application code associated with the at least one data source; and transmit information about the analyzed metadata and application code; and a replication agent configured to: receive a replication request specifying at least one source data record in the at least one data source; and transform at least one source data record based on theDOCKET NO. P0068W01transmitted information; and a replicator server configured to transmit the at least one transformed source data record to a target destination.

[0021] In some aspects, the replicator server is further configured to provide a user interface for receiving selections of data sources and target destinations for data replication; and manage replication between selected data sources and target destinations.

[0022] In some aspects, the analyzing of the metadata and application code includes identifying available data records at the at least one data source; and mapping the available data records to corresponding data formats.

[0023] In some aspects, the mapping of the available data records includes identifying a logical structure regarding at least one available data record; and mapping the at least one available record to a corresponding data format based on the logical structure.

[0024] In some aspects, the system further includes a persistent data repository for storing an inventory of available data sources.

[0025] In some aspects, the target destination is located at a cloud computing environment.

[0026] In some aspects, wherein the metadata and application code are associated with a proprietary format usable by the mainframe computer system, and wherein the at least one transformed source data record is in a different format usable by the target destination.

[0027] In some aspects, the techniques described herein relate to a computer program product embodied in a non-transitory computer readable storage medium and including computer instructions for analyzing metadata and application code associated with at least one data source, the at least one data source deployed on a mainframe computer system; transmitting information about the analyzed metadata and application code; receiving a replication request specifying at least one source data record in the at least one data source; transforming the at least one source data record based on the transmitted information; and transmitting the at least one transformed source data record to a target destination.DOCKET NO. P0068W01BRIEF DESCRIPTION OF DRAWINGS

[0028] Non-limiting and non-exhaustive embodiments of the invention are described with reference to the following figures, wherein like reference numerals refer to like parts throughout the various views unless otherwise specified.

[0029] FIG. 1 illustrates a system for replicating data in accordance with one embodiment.

[0030] FIG. 2 illustrates another system for replicating data in accordance with one embodiment.

[0031] FIG. 3 illustrates a flowchart of a method for replicating data in accordance with one embodiment.

[0032] FIG. 4 illustrates a flowchart of another method for replicating data in accordance with one embodiment.

[0033] FIG. 5 illustrates a graph for replicating data in accordance with one embodiment.

[0034] FIG. 6 illustrates an output file for replicating data in accordance with one embodiment.

[0035] FIG. 7 illustrates another system for replicating data in accordance with one embodiment.DETAILED DESCRIPTION

[0036] Various embodiments are described more fully below with reference to the accompanying drawings, which form a part hereof, and which show specific exemplary embodiments. However, the concepts of the present disclosure may be implemented in many different forms and should not be construed as limited to the embodiments set forth herein; rather, these embodiments are provided as part of a thorough and complete disclosure, to fully convey the scope of the concepts, techniques and implementations of the present disclosure to those skilled in the art. Embodiments may be practiced as methods, systems or devices. Accordingly, embodiments may take the form of a hardware implementation, an entirely software implementation or an implementation combining software and hardware aspects. The following detailed description is, therefore, not to be taken in a limiting sense.

[0037] Reference in the specification to “one embodiment” or to “an embodiment” means that a particular feature, structure, or characteristic described in connection with the embodiments isDOCKET NO. P0068W01included in at least one example implementation or technique in accordance with the present disclosure. The appearances of the phrase “in one embodiment” in various places in the specification are not necessarily all referring to the same embodiment. The appearances of the phrase “in some embodiments” in various places in the specification are not necessarily all referring to the same embodiments.

[0038] Some portions of the description that follow are presented in terms of symbolic representations of operations on non-transient signals stored within a computer memory. These descriptions and representations are used by those skilled in the data processing arts to most effectively convey the substance of their work to others skilled in the art. Such operations typically require physical manipulations of physical quantities. Usually, though not necessarily, these quantities take the form of electrical, magnetic or optical signals capable of being stored, transferred, combined, compared and otherwise manipulated. It is convenient at times, principally for reasons of common usage, to refer to these signals as bits, values, elements, symbols, characters, terms, numbers, or the like. Furthermore, it is also convenient at times, to refer to certain arrangements of steps requiring physical manipulations of physical quantities as modules or code devices, without loss of generality.

[0039] However, all of these and similar terms are to be associated with the appropriate physical quantities and are merely convenient labels applied to these quantities. Unless specifically stated otherwise as apparent from the following discussion, it is appreciated that throughout the description, discussions utilizing terms such as “processing” or “computing” or “calculating” or “determining” or “displaying” or the like, refer to the action and processes of a computer system, or similar electronic computing device, that manipulates and transforms data represented as physical (electronic) quantities within the computer system memories or registers or other such information storage, transmission or display devices. Portions of the present disclosure include processes and instructions that may be embodied in software, firmware or hardware, and when embodied in software, may be downloaded to reside on and be operated from different platforms used by a variety of operating systems.

[0040] The present disclosure also relates to an apparatus for performing the operations herein. This apparatus may be specially constructed for the required purposes, or it may comprise a general-purpose computer selectively activated or reconfigured by a computer program stored inDOCKET NO. P0068W01the computer. Such a computer program may be stored in a computer readable storage medium, such as, but is not limited to, any type of disk including floppy disks, optical disks, CD-ROMs, magnetic-optical disks, read-only memories (ROMs), random access memories (RAMs), EPROMs, EEPROMs, magnetic or optical cards, application specific integrated circuits (ASICs), or any type of media suitable for storing electronic instructions, and each may be coupled to a computer system bus. Furthermore, the computers referred to in the specification may include a single processor or may be architectures employing multiple processor designs for increased computing capability.

[0041] The processes and displays presented herein are not inherently related to any particular computer or other apparatus. Various general -purpose systems may also be used with programs in accordance with the teachings herein, or it may prove convenient to construct more specialized apparatus to perform one or more method steps. The structure for a variety of these systems is discussed in the description below. In addition, any particular programming language that is sufficient for achieving the techniques and implementations of the present disclosure may be used. A variety of programming languages may be used to implement the present disclosure as discussed herein.

[0042] In addition, the language used in the specification has been principally selected for readability and instructional purposes and may not have been selected to delineate or circumscribe the disclosed subject matter. Accordingly, the present disclosure is intended to be illustrative, and not limiting, of the scope of the concepts discussed herein.

[0043] As discussed above, there remains a need for improved methods and systems for replicating data sets from mainframes to modern computing systems. The embodiments here provide techniques for replicating data from a data source deployed at a mainframe computer system. For example, certain embodiments may include analyzing, by at least one scanning agent, metadata and application code associated with at least one data source, the at least one scanning agent and the at least one data source deployed on a mainframe computer system; transmitting information about the analyzed metadata and application code; receiving, by a replication agent, a replication request specifying at least one source data record in the at least one data source; transforming the at least one source data record based on the transmitted information; and transmitting the at least one transformed source data record to a target destination.DOCKET NO. P0068W01

[0044] FIG. 1 illustrates a system 100 for replicating data in accordance with one embodiment. The system 100 may include any number of components for performing operations related to replicating data. As shown, for example, the system 100 may include or otherwise deploy a mainframe system 102, a replicator 104, and a cloud system 106. The system 100 may be implemented within any number of suitable computing environments, such as a cloud-based computing environment, a distributed computing environment, and / or a virtualized computing environment. Although one mainframe system 102 and one cloud system 106 are shown, it is to be appreciated that the system 100 may include any suitable number of computing systems.

[0045] The mainframe system 102 may include any suitable computing system used to manage enterprise data. The mainframe system 102 may act as a centralized computing platform designed to support large-scale data processing tasks, transaction management, and / or centralized storage functionalities. The mainframe system 102 may support tasks for any suitable enterprise application such as financial processing, insurance policy management, inventory control, and / or telecommunications billing.

[0046] In some embodiments, the mainframe system 102 may include distributed computing systems. In this manner, any number of components of the mainframe system 102 may be distributed among any number of computing devices, such as a cluster of computing nodes. The distributed systems may support extensive parallel processing for large-scale data processing.

[0047] In some embodiments, the mainframe system 102 may act as a centralized storage platform to support various data processing and transactional tasks. As shown, for example, the mainframe system 102 may include at least one data source 108. The data source 108 may represent the available data sources on the mainframe system 102. The data source 108 may include at least one repository for storing data records. The data source 108 may store various types of semi-structured and / or structured data records, such as sequential datasets, hierarchical or relational databases, message queues, log files, system files, application code, metadata files, and / or transactional data files. While one data source 108 is shown, it is to be appreciated that mainframe system 102 may include any number of data sources.

[0048] In some embodiments, the data records may be stored in various mainframe formats. Such formats may be proprietary to the mainframe system 102 and / or other legacy systems. As a specific example, the data source 108 may include VS AM (Virtual Storage Access Method) files.DOCKET NO. P0068W01VSAM is a high-performance file storage access format designed for mainframe systems. VSAM may support efficient access to data records through both direct and sequential access modes, with indexing typically based on key fields. VSAM is particularly well-suited for high-throughput batch processing and transactional systems frequently utilized in legacy enterprise environments. Other examples of mainframe formats may include AD AB AS, DB2, IMS, SYSLOG, MQ, SEQUENTIAL, SMF, IDMS, and / or Datacom.

[0049] Accessing and integrating data in mainframe formats with modem data systems (e.g., cloud-based systems) may be challenging, however, as such formats may be inherently optimized for use within mainframe systems. For example, VSAM often relies on metadata to define the structure, layout, and / or semantics of VSAM files. The metadata may include, for example, descriptions of record layouts, record formats, field formats, data lineage, and / or access controls. Such metadata is often not centralized and may be distributed across various locations within the mainframe system. For example, the metadata may reside within datasets, in catalog entries, within packaged applications, and / or even within application code that interfaces with the VSAM files, such as COBOL copybooks, JCL procedures, and / or programmatic definitions. The metadata may be stored at the data source 108 and / or at a separate repository associated with the mainframe system 102.

[0050] In some embodiments, the mainframe system 102 may use and / or otherwise be compatible with various legacy operating systems. For example, the mainframe system 102 may be compatible with IBM z / OS, IBM I, Z / VM, and / or z / VSE. Such operating systems may be proprietary to the mainframe system and highly optimized for high-throughput and reliable data processing.

[0051] As shown, the mainframe system 102 may deploy at least one scanning agent 110. The scanning agent 110 may be configured to capture and analyze various data files associated with the data source 108. For example, the scanning agent 110 may analyze metadata (e.g., VSAM metadata files), application code (e.g., for the mainframe system 102), system configuration files, and / or control scripts that define and / or interact with the data source 108. The captured data may be in various formats, such as Job Control Language (JCL), COBOL, utility configuration formats (e.g., SORT, IDCAMS), and / or database schemas (e.g., DB2, IMS DBD).DOCKET NO. P0068W01

[0052] In some embodiments, the scanning agent 110 may build a comprehensive inventory of data sources that are available on the mainframe system 102 (e.g., based on the captured data). For example, the scanning agent 110 may reverse engineer the data source 108 and access methods and field mappings within the captured data to identify data records, fields, types, data structures, logical structures, and / or hierarchical patterns within the data source 108. In this manner, the scanning agent 110 may map data records at the data source 108 to corresponding data formats.

[0053] In some embodiments, multiple mappings may be possible for a particular data record. For example, a data record (e.g., a VSAM file) may contain logical redefinitions and / or other logical structures that allow for multiple possible mappings. The scanning agent 110 may detect such cases using the captured data and determine the proper mapping. In some embodiments, the scanning agent 110 may use heuristics to determine the proper mapping.

[0054] As shown, the mainframe system 102 may deploy at least one replication agent 112. The replication agent 112 may be configured to perform various data replication tasks for the mainframe system 102. The replication agent 112 may be implemented as any suitable agent and / or software component.

[0055] In some embodiments, the replication agent 112 may receive data commands relating to the mainframe system 102, such as commands for data replication, data synchronization across multiple systems, and / or data virtualization across multiple systems. The commands may specify various parameters, such as a data source, a target destination, data identifiers (e.g., filenames), conditional expressions (e.g., for selecting data records), replication modes (e.g., full replication, incremental replication, etc ), replication timing, and / or replication frequency. For example, the replication agent 112 may receive a command that certain data records at the data source 108 should be replicated daily at a specified time. The replication agent 112 may parse the command to extract the parameter values, identify data records associated with the command, and initiate a corresponding replication operation (e.g., a daily copy operation at the specified time). In some embodiments, the replication agent 112 may provide a user interface for receiving the data commands. In some embodiments, the replication agent 112 may support incremental replication by monitoring data records for data changes, such as insertions and / or deletions. The replication agent 112 may replication only the changed data in incremental replication tasks.DOCKET NO. P0068W01

[0056] In some embodiments, the replication agent 112 may interact with the scanning agent 110 to process data replication tasks. For example, the replication agent 112 may use the outputs of the scanning agent 110 (e.g., identified fields, types, data structure, etc.) to identify the data records corresponding to a replication request and accurately replicate the data records.

[0057] In some embodiments, the replication agent 112 may monitor data replication tasks, log data operations, and / or handle error conditions to ensure reliable data operations. For example, the replication agent 112 may monitor the scanning agent 110.

[0058] As shown, the mainframe system 102 may communicate with the replicator 104. The replicator 104 may be configured to orchestrate and manage data operations across disparate computing systems, such as the mainframe system 102 and the cloud system 106. The data operations may include operations such as data replication, data synchronization, and / or data virtualization. The replicator 104 may be implemented by any suitable system, such as a distributed server.

[0059] In some embodiments, the replicator 104 may build inventories of available data sources from multiple computing systems, such as the mainframe system 102 and the cloud system 106. The replicator 104 may interact with various tools associated with each computing system, such as the scanning agent 110 and the replication agent 112. The tools may provide the replicator 104 with information needed to build the inventories and perform other tasks. The replicator 104 may interact with the tools via standard network protocols and / or application programming interfaces (APIs).

[0060] In some embodiments, the replicator 104 may provide a user interface for enabling discovery, inspection, and / or selection of data sources and target destinations for data replication. The user interface may present a variety of information related to the data sources, such as names, data records, origins, lineage relationships, and / or usage contexts. The user interface may allow users to search for and select data sources and target destinations based on the presented information. The user interface may allow users to input a replication request with selected parameters. The replication request may specify any number of parameters, such as data sources, data records, target destinations, address data, replication frequencies, start and / or end times, replication types (e.g., incremental replication), and / or types of access methods. In some examples, replication requests may be generated automatically by a policy engine.DOCKET NO. P0068W01

[0061] In some embodiments, the user interface may be a unified user interface for initiating and managing the entire workflow for a replication task. In this manner, a single user interface may be used to manage an entire replication task without the need to switch between multiple interfaces.

[0062] The replicator 104, in response to a replication request, may aggregate the required information into a standardized configuration format, such as a JSON schema, and queue the request for execution. For example, the replicator 104 may request the replication agent 112 to copy data corresponding to the request (e.g., by transmitting the aggregated information to the replication agent 112).

[0063] The replicator 104 may configure data replication tasks with any suitable frequency or timing. For example, the data replications may be performed on a one-time basis or on a recurring schedule (e.g., daily, weekly, etc.). In some embodiments, the replicator 104 may support incremental data replication, where only data that has changed since the most recent replication task is replicated.

[0064] In some embodiments, the replicator 104 may include a repository 114. The repository 114 may store instructions to be executed by the replicator 104, such as instructions for providing a user interface and communicating with the scanning agent 110 and the replication agent 112. In some embodiments, the repository 114 may store information from the scanning agent 110 and / or replication agent 112, such as an inventory of available data sources and / or metadata from the scanning agent 110. In this manner, the replicator 104 may communicate with the scanning agent 110 and the replication agent 112 to perform data replication tasks.

[0065] In some embodiments, the replicator 104 may transform copied data to be compatible with the target destination of a replication request, such as the cloud system 106. The transformation may account for differences between differing computing systems, such as different data formats, encoding schemes, record layouts, and / or access protocols. In some embodiments, the replication agent 112 may perform the transformation before sending the transformed data to the replicator 104. The replicator 104 may subsequently send the data to the target destination.

[0066] The cloud system 106 may be implemented by any suitable computing system deployed in a cloud computing environment, such as Striim, Kafka, Elastic, MongoDB, Snowflake,DOCKET NO. P0068W01Databricks, WatsonX, Amazon Web Services, Azure, and / or Google Cloud Platform. As shown, the cloud system 106 may include at least one data target 116. The data target 116 may include a cloud storage repository and / or server hosted on the cloud system 106. The data target 116 may serve as the target destination for a data replication task. For example, the replication agent 112 may receive a replication request to replicate data from the data source 108 to the data target 116.

[0067] While the cloud system 106 is shown as including the data target 116, it is to be appreciated that the cloud system 106 may include any number of data sources and / or data targets. That is, data may be replicated from the cloud system 106 to the mainframe system 102 (and / or any other suitable target destination). In some embodiments, the cloud system 106 may include replication agents and / or scanners to carry out data replication tasks. In other embodiments, the cloud system 106 may carry out data replication tasks without the need for replication agents and / or scanners, as certain embodiments of the cloud system 106 may inherently support network connectivity and replication operations.

[0068] FIG. 2 illustrates a system 200 for replicating data in accordance with one embodiment. As shown, the system 200 may include multiple mainframe systems 202 (e.g., mainframe systems 202-1 and 202-2) and multiple cloud systems 204 (e.g., cloud systems 204-1 and 204-2). Each mainframe system 202 and cloud system 204 may include any number of data sources 206 and / or data targets 208. For example, as shown, the system 200 may include data sources 206-1, 206-2 and 206-3, and data targets 208-1 and 208-2.

[0069] In some embodiments, data may be replicated between any suitable pair of data sources 206 and data targets 208. For example, data may be replicated from the data source 206-1 to the data target 208-2. The mainframe systems 202 and cloud systems 204 may be configured to communicate with the replicator 104 for data replication. Although not illustrated, any of the mainframe systems 202 and / or cloud systems 204 may deploy various components for replicating data, such as the scanning agent 110 and the replication agent 112. The components may be deployed at the same time or may be deployed as needed at particular systems identified by scanning agents.

[0070] FIG. 3 illustrates a flowchart of a method 300 for replicating data in accordance with one embodiment. While FIG. 3 shows illustrative operations according to one embodiment, other embodiments may omit, add to, reorder, and / or modify any of the operations shown in FIG. 3.DOCKET NO. P0068W01Moreover, each of the operations depicted in FIG. 3 may be performed in any of the ways described herein. The operations shown in FIG. 3 may be performed by any of the illustrative systems described herein, such as the system 100.

[0071] At operation 302, the method 300 may include analyzing, by at least one scanning agent, metadata and application code associated with at least one data source. The scanning agent and / or data source may be deployed at a mainframe computer system, such as the mainframe system 102 deploying the scanning agent 110 and the data source 108. The metadata and application code may be associated legacy or proprietary formats compatible with the mainframe computer system. The scanning agent may analyze the metadata and application code to identify available data records at the at least one data source and map the available data records to corresponding data formats, such as by identifying types, data structures, and / or logical structures. The scanning agent may handle large amounts of metadata and application code across hundreds of applications. For example, the analyzed code may include millions of lines of code (e.g., at least 100 million lines). In some embodiments, the scanning agent may capture the metadata and application code and then transmit the capture data to a parsing station (e.g., a distributed parsing server, a cloud-based server, etc.) for further analysis and processing.

[0072] At operation 304, the method 300 may include transmitting information about the analyzed metadata and application code. The information may be transmitted to various other components, such as a storage repository and / or a replicator (e.g., the replicator 104).

[0073] At operation 306, the method 300 may include receiving, by a replication agent, a replication request. The replication agent may receive the request from a user, such as via a user interface, and / or automatically, such as from a policy engine configured to schedule automatic replication tasks. The replication request may specify at least one parameter, such as a data source and / or a target destination for the replication request.

[0074] At operation 308, the method 300 may include transforming the at least one source data record based on the analyzed metadata and application code. The at least one source data record may be transformed from a mainframe format usable by the mainframe computer system to a different format usable by the target destination for the replication request.

[0075] At operation 310, the method 300 may include transmitting the at least one transformed source data record to the target destination. The target destination may be any suitable computingDOCKET NO. P0068W01system and / or environment external to the mainframe computer system, such as the cloud system 106. The target destination may be compatible with modem data formats that are not supported by the mainframe computer system.

[0076] Although not illustrated, it is to be appreciated that the method 300 may include any number of additional and / or alternative operations. For example, the method 300 may include generating data samples based on the analyzed metadata and application code. The data samples may include portions of the transformed source data record (and / or any other requested data). The data samples may be generated before and / or in lieu of any of the operations described herein, such as operations 308 and 310.

[0077] In some embodiments, the data samples may be used to test and / or validate a replication operation and / or process, such as the analysis of the metadata and application code in operation 302. For example, the data samples may be analyzed to identify samples that deviate significantly from learned patterns (e.g., using a machine learning model). In some examples, the identified samples may be flagged and presented at a user interface. In other examples, replication operations may be automatically adjusted in response to the identified samples.

[0078] In some embodiments, the method 300 may include generating and providing lineage data regarding the transformed source data record. The lineage data may include lineage information such as where the data record originated, how the data record was transformed, and / or where the data record is currently located at the target destination. The lineage data may be structured as at least one lineage path representing the flow of data between an original source and the transformed data record. The lineage data may be visualized at a user interface (e.g., by the replicator 104 and / or any other suitable component). For example, the lineage data may be visualized as part of a graph showing various lineage paths.

[0079] In some embodiments, the method 300 may include generating a replication template that encodes predefined replication workflows. The template may include predefined mappings and / or other information that are commonly used for specific types of workflows, such as workflows in particular business domains or use cases. For example, the template may be tailored for financial data replication, customer relationship management (CRM) systems, and / or healthcare records migration.DOCKET NO. P0068W01

[0080] FIG. 4 illustrates a flowchart of a method 400 for replicating data in accordance with one embodiment. As shown, the method 400 may include the operations 302 and 304 as illustrated in FIG. 3. While FIG. 4 shows illustrative operations according to one embodiment, other embodiments may omit, add to, reorder, and / or modify any of the operations shown in FIG. 4. The operations show in FIG. 4 may be performed by any of the illustrative systems described herein, such as the system 100.

[0081] As described above, at operation 302, the method 400 may include a system (e.g., the system 100) analyzing metadata and application code associated with at least one data source. The system may perform various tasks to analyze the metadata and application data. For example, as shown, operation 302 may include sub-operations 402 to 410. At sub-operation 402, the method 400 may include the system accessing the metadata and application code associated with the data source. The metadata and / or application code may contain definitions and mappings for data stored at the data source, such as VSAM files. The application code may be legacy and / or mainframe-oriented code written in languages such as JCL, COBOL, PL / I, Assembler, Easytrieve and / or CLIST.

[0082] At sub-operation 404, the method 400 may include the system parsing the metadata and application code. The system may use a parsing engine to parse the data.

[0083] At sub-operation 406, the method 400 may include the system identifying entities associated with the at least one data source based on the metadata and application code. The entities may include available data records, types, and / or structures. For example, the system may extract data elements, analyze control flow and conditional statements, determine dynamic field mappings, and / or identify record groupings that share access or transformation logic. The system may perform additional queries to the metadata to retrieve relevant supplemental information concerning the data source. These queries may seek attributes such as data types, usage context, data lineage, access controls, dependencies, and / or previously defined transformation logic. By identifying the entities, the system may identify mappings for the data records at the data source to corresponding data formats.

[0084] In some embodiments, the system may identify records that are accessed using multiple access points. For example, a particular database may be accessed via either a JDBC alias or a COBOL copybook using a COBOL short name. The system may recognize that the access pointsDOCKET NO. P0068W01all reference the same record. The system may generate a composite identifier (e.g., a composite fingerprint) for the record based on the access points. For example, the system may retrieve and aggregate metadata associated with the access points, standardize conflicting metadata formats, and generate the composite identifier based on metadata attributes (e.g., by concatenating and / or hashing attributes). The composite identifier may be used to target the correct physical record for replication regardless of which access point is used. This may help match data across multiple environments and avoid unnecessary duplication of data. In some embodiments, users may access a record using any suitable access point via a unified user interface.

[0085] In some embodiments, the system may be configured to identify logical structures from the metadata and application code. The metadata and application code may define logical structures where the format or structure of a particular data record is determined at runtime based on the value of particular fields. VS AM files, for example, may contain record redefinitions that are defined conditionally. The mapping of a VSAM file to record types and formats may depend on the value of markers such as discriminator fields. For example, a VSAM file may be linked to possible two record types, one for “cars” and one for “vessels.” The mapping of the VSAM file may depend on the value of a discriminator field. One particular value for the discriminator field (e g., ‘C’) may map the VSAM file to the “cars” record type and another value (e.g., ‘V’) may map the VSAM file to the “vessel” record type.

[0086] To resolve ambiguities resulting from such logical structures, the system may identify the logical structures and map data records to particular record types or formats based on the identified logical structures. The system may identify markers used by the application code to decide which record type applies at runtime. The system may tag and annotate records with markers and connect the records to the corresponding logical structures. This may enable the system to intelligently route a single record (e.g., a VSAM file) to multiple possible target formats (e.g., multiple tables for different record types).

[0087] In some embodiments, the system may identify entities associated with sensitive or confidential data. For example, certain field values may be identified as containing personally identifiable information (PII), protected health information (PHI), payment card information (PCI), and / or confidential business identifiers. The system may apply enforcement actions based on the identified entities. The actions may include at least one of redacting the entities, encryptingDOCKET NO. P0068W01the entities, and / or preventing unauthorized access through access control (e.g., using attributebased access control, role-based access control or policy -based access control).

[0088] At sub-operation 408, the method 400 may include the system generating a graphical representation of the identified entities (e.g., types, fields, structures, etc.) and their relationships. The graphical representation may serve as a unified model for the layout of data records and the program logic for processing the data records.

[0089] To illustrate, FIG. 5 illustrates a graph 500 for replicating data in accordance with one embodiment. As shown, the graph 500 may include nodes and edges. The nodes may represent various entities and structures identified from the metadata and application code, such as identified files, structures, record redefinitions, data fields, annotations, and / or descriptions. The nodes may include elements directly parsed from the metadata and application code and / or elements inferred by the system. The inferred elements may enrich or augment the graph 500 with additional information. The inferred elements may include elements such as annotations, semantic tags, markers, statistical summaries, inferred field mappings, and / or other suitable inferences. The inferred elements may be tagged by corresponding enrichment nodes in the graph 500. For example, the graph 500 may include marker nodes that tag (i.e., identify) other nodes as representing markers. The enrichment nodes (and / or other selected types of nodes) may be presented at a user interface as color-coded nodes (e.g., as green nodes).

[0090] In some embodiments, the edges of the graph 500 may represent connections between nodes. For example, the edges may represent connections between nodes, such as conditional paths, control flow paths, and / or data flow paths. As shown, for example, the “marker” node and the “RECTYPE EVAL4” node may be connected by an edge to represent a connection between the two nodes.

[0091] In some embodiments, the system may generate shortcut links associated with the data source. As used herein, shortcut links may refer to generated connections between entities that are not explicitly connected according to the metadata and application code. For example, as shown, the graph 500 includes the “PP TRUE CONDITION DEPENDENCY" shortcut link connecting the “WHEN EVAL4 [TEST]” node and the “MAKE EVAL4” node. In some embodiments, shortcut links can connect top-level entities (e.g., top-level record structures) to lower-level entities (e g., deeply nested sub-elements) in hierarchical record layouts. Such shortcut links may allowDOCKET NO. P0068W01faster processing of lineage and mapping queries by avoiding the need for multiple hops through hierarchical levels.

[0092] Returning to FIG. 4, at sub-operation 410, the method 400 may include the system generating an output file containing the information needed for replication. The output file may contain the information identified at operation 304, such as the shortcut links and markers. The output file may be generated based on the generated graphical representation. In some embodiments, the output file may include a JSON file. The output file may include fields for structuring the information.

[0093] At operation 304, as described above, the method 400 may include the system transmitting the information. The information may be transmitted to a storage repository for storage. The information may be later used (e.g., by accessing and interpreting a JSON file) to process replication requests associated with the data source.

[0094] FIG. 6 illustrates an output file 600 for replicating data. As described above, the output file 600 may be a JSON file that includes information generated to handle replication requests, such as the information represented in the graph 500. The output file 600 may include any number of fields for organizing the information. For example, as shown, the output file 600 may contain a “body” field indicating record redefinitions, a “maxlevel” field indicating a record level where record redefinitions occur, a “recordtype” field indicating names of record types (e.g., “CARS”), and a “value” field indicating values of record types (e.g., “C”).

[0095] FIG. 7 illustrates a system 700 for replicating data in accordance with one embodiment. The system 700 may include any number of components for performing operations related to replicating data. As shown, for example, the system 700 may include or otherwise deploy a source system 702 including a data source 704, a target system 706 including a data target 708, an analysis repository 710, a replication repository 712, and a user interface 714. The source system 702 and target system 706 may be similar to any of the systems described herein, such as the mainframe system 102 and cloud system 106.

[0096] At operation 720, the system 700 may analyze the data source 704 and data target 708. For example, scanning agents deployed at the data source 704 and / or data target 708 may capture and analyze information associated with the data source 704 and / or data target 708, such as metadata and / or application code associated with the data source 704 and / or data target 708. TheDOCKET NO. P0068W01system 700 may store the information at the analysis repository 710. The analysis repository 710 may contain an inventory of identified data records and / or other relevant information.

[0097] At operation 722, the system 700 may present information stored at the analysis repository 710 at the user interface 714. For example, the user interface 714 may present information regarding the data source 704 and data target 708, such as data records available for replication at the data source 704. The user interface 714 may allow a user to search for and select data sources and target destinations for replication requests.

[0098] At operation 724, the system 700 may receive, at the user interface 714, a request to replicate data from the data source 108 to the data target 116. The user interface 714 may send the request to the source system 702.

[0099] At operation 726, the system 700 (e.g., the source system 702) may perform various tasks to prepare for the requested replication, such as additional metadata searches, extraction of types and structures (e.g., logical conditions), mappings of source and target formats, calculations of any required transformations or mappings, and / or preparation of aggregated metadata in support of the requested replication. The resulting information may be stored (e.g., as a compiled document) in the replication repository 712, which may serve as a replication queue management system.

[0100] In some embodiments, the system 700 may identify multiple possible mappings for a source data record to a target format suitable for the target system 706. In such scenarios, the system may apply heuristics to select an optimal mapping from the possible mappings. For example, the system may evaluate the possible mappings using a heuristic algorithm and calculate a score for each mapping. The heuristic algorithm may include any suitable algorithm, such as field-name similarity, tag consistency, and / or byte-length concordance (e.g., comparing total and individual field lengths).

[0101] In some embodiments, the system 700 may select the mapping with the highest score as the optimal mapping. In other embodiments, the system 700 may compare the highest-scoring mapping against a threshold. If the highest score meets or exceeds the threshold, the system 700 may automatically select the highest-scoring mapping. Otherwise, the corresponding data record may be flagged for administrative review.DOCKET NO. P0068W01

[0102] In some embodiments, the names associated with the source data record may be constrained by historical naming conventions, such as length restrictions or alphanumeric codification. This can result in opaque legacy names that are difficult for modem users to interpret, such as “PN,” “ACTCD,” and “ACCTNUM.” Such names may also vary across different representations of the same data source.

[0103] Accordingly, in some embodiments, the system 700 may translate the names associated with the data source into corresponding names that are more readable and / or user-friendly for modern systems such as the target system 706. For example, the system 700 may translate various types of legacy names for source data records into descriptive equivalents, such as file identifiers, table names, column names, and / or row names. The system may handle various types of ambiguous or opaque names, such as abbreviated names (e.g., “PN”), technical identifiers (e.g., “DB9D”), numerical fields (e.g., “123456789C”), date fields (e.g., “2405”), and / or encoded names (e.g., “X7ACRF”).

[0104] In some embodiments, the system may translate the names using a machine learning model. The model may be trained on past mappings between legacy names and modern equivalents. The model may receive as input at least one legacy name and corresponding metadata or descriptors. The model may output at least one suggested translation based on the inputs. For example, the model may receive the legacy field name “PO” and metadata describing “PO” as a field indicating a purchase order number. The model may then output suggested column names that are more user-friendly, such as “po_purchase_order_number,” “purchase_order_number_po,” and “po number.” The suggested names may be presented at a user interface.

[0105] In some embodiments, the system 700 may receive feedback from users regarding suggested translations. For example, the user may either accept a suggested name, reject the name and ask for other suggestions, or edit the name manually. The system may fine tune the machine learning model based on the feedback.

[0106] At operation 728, the system 700 may execute the requested replication. The stored data at the replication repository 712 may be transmitted to various replication tools for executing the requested replication, such as replicators and / or replication agents. In some embodiments, the replication components may receive data via API calls (e.g., without the need to store the data in a repository and / or queue management system).DOCKET NO. P0068W01

[0107] The methods, systems, and devices discussed above are examples. Various configurations may omit, substitute, or add various procedures or components as appropriate. For instance, in alternative configurations, the methods may be performed in an order different from that described, and that various steps may be added, omitted, or combined. Also, features described with respect to certain configurations may be combined in various other configurations. Different aspects and elements of the configurations may be combined in a similar manner. Also, technology evolves and, thus, many of the elements are examples and do not limit the scope of the disclosure or claims.

[0108] Embodiments of the present disclosure, for example, are described above with reference to block diagrams and / or operational illustrations of methods, systems, and computer program products according to embodiments of the present disclosure. The functions / acts noted in the blocks may occur out of the order as shown in any flowchart. For example, two blocks shown in succession may in fact be executed substantially concurrent or the blocks may sometimes be executed in the reverse order, depending upon the functionality / acts involved. Additionally, or alternatively, not all of the blocks shown in any flowchart need to be performed and / or executed. For example, if a given flowchart has five blocks containing functions / acts, it may be the case that only three of the five blocks are performed and / or executed. In this example, any of the three of the five blocks may be performed and / or executed.

[0109] A statement that a value exceeds (or is more than) a first threshold value is equivalent to a statement that the value meets or exceeds a second threshold value that is slightly greater than the first threshold value, e.g., the second threshold value being one value higher than the first threshold value in the resolution of a relevant system. A statement that a value is less than (or is within) a first threshold value is equivalent to a statement that the value is less than or equal to a second threshold value that is slightly lower than the first threshold value, e.g., the second threshold value being one value lower than the first threshold value in the resolution of the relevant system.

[0110] Specific details are given in the description to provide a thorough understanding of example configurations (including implementations). However, configurations may be practiced without these specific details. For example, well-known circuits, processes, algorithms, structures, and techniques have been shown without unnecessary detail in order to avoid obscuring theDOCKET NO. P0068W01configurations. This description provides example configurations only, and does not limit the scope, applicability, or configurations of the claims. Rather, the preceding description of the configurations will provide those skilled in the art with an enabling description for implementing described techniques. Various changes may be made in the function and arrangement of elements without departing from the spirit or scope of the disclosure.

[0111] Having described several example configurations, various modifications, alternative constructions, and equivalents may be used without departing from the spirit of the disclosure. For example, the above elements may be components of a larger system, wherein other rules may take precedence over or otherwise modify the application of various implementations or techniques of the present disclosure. Also, a number of steps may be undertaken before, during, or after the above elements are considered.

[0112] Having been provided with the description and illustration of the present application, one skilled in the art may envision variations, modifications, and alternate embodiments falling within the general inventive concept discussed in this application that do not depart from the scope of the following claims.

Claims

DOCKET NO. P0068W01CLAIMSWhat is claimed is:

1. A method for data replication, comprising:analyzing, by at least one scanning agent, metadata and application code associated with at least one data source, the at least one scanning agent and the at least one data source deployed on a mainframe computer system;transmitting information about the analyzed metadata and application code; receiving, by a replication agent, a replication request specifying at least one source data record in the at least one data source;transforming the at least one source data record based on the transmitted information; and transmitting the at least one transformed source data record to a target destination.

2. The method of claim 1, wherein the analyzing of the metadata and application code comprises:identifying available data records at the at least one data source; andmapping the available data records to corresponding data formats.

3. The method of claim 2, wherein the mapping of the available data records comprises: identifying a logical structure regarding at least one available data record; and mapping the at least one available record to a corresponding data format based on the logical structure.

4. The method of claim 2, wherein the mapping of the available data records comprises translating at least one field name of the available data records to be more readable.

5. The method of claim 1, further comprising providing a user interface for receiving selections of data sources and target destinations, wherein the replication request is received at the user interface.

6. The method of claim 5, further comprising:identifying available data sources; andDOCKET NO. P0068W01presenting, at the user interface, a list of the available data sources.

7. The method of claim 1, wherein the target destination is located at a cloud computing environment.

8. The method of claim 1, further comprising analyzing, by at least one target scanning agent, metadata and application code associated with the target destination.

9. The method of claim 8, wherein the at least one scanning agent and the at least one target scanning agent are deployed at the same time.

10. The method of claim 1, wherein the metadata and application code are associated with a proprietary format usable by the mainframe computer system, and wherein the at least one transformed source data record is in a different format usable by the target destination.

11. The method of claim 10, wherein the proprietary format comprises at least one of VSAM, DB2, CICS, or SEQUENTIAL.

12. The method of claim 1, further comprising storing an inventory of available data sources at a persistent data repository, the inventory including the transmitted information.

13. A system for data replication, comprising:a mainframe computer system comprising:at least one data source;a scanning agent configured to:analyze metadata and application code associated with the at least one data source; andtransmit information about the analyzed metadata and application code; anda replication agent configured to:DOCKET NO. P0068W01receive a replication request specifying at least one source data record in the at least one data source; andtransform at least one source data record based on the transmitted information; anda replicator server configured to transmit the at least one transformed source data record to a target destination.

14. The system of claim 13, wherein the replicator server is further configured to:provide a user interface for receiving selections of data sources and target destinations for data replication; andmanage replication between selected data sources and target destinations.

15. The system of claim 13, wherein the analyzing of the metadata and application code comprises:identifying available data records at the at least one data source; andmapping the available data records to corresponding data formats.

16. The system of claim 15, wherein the mapping of the available data records comprises: identifying a logical structure regarding at least one available data record; and mapping the at least one available record to a corresponding data format based on the logical structure.

17. The system of claim 13, further comprising a persistent data repository for storing an inventory of available data sources.

18. The system of claim 13, wherein the target destination is located at a cloud computing environment.

19. The system of claim 13, wherein the metadata and application code are associated with a proprietary format usable by the mainframe computer system, and wherein the at least one transformed source data record is in a different format usable by the target destination.DOCKET NO. P0068W0120. A computer program product embodied in a non-transitory computer readable storage medium and comprising computer instructions for:analyzing metadata and application code associated with at least one data source, the at least one data source deployed on a mainframe computer system;transmitting information about the analyzed metadata and application code; receiving a replication request specifying at least one source data record in the at least one data source;transforming the at least one source data record based on the transmitted information; and transmitting the at least one transformed source data record to a target destination.

Images