Data sharing method

Abstract

This disclosure relates to a system for sharing confidential data between a data provider and a data consumer. The system comprises a matching service prevented from accessing plain data stored by the data provider to: receiving from the data provider obfuscated data that comprises at least part of the plain data; storing the obfuscated data on a data storage; a matching engine to match the obfuscated data; an extraction engine to extract response data related to the obfuscated data matched from the data storage; and a broker application to: receive a request from the data consumer, the request comprising obfuscated query data; provide the obfuscated query data to the matching engine to query the data storage for the obfuscated query data; receive, from the extraction engine the response data; and provide the response data to the data consumer.

Description

"Data sharing method"Cross-Reference to Related Applications

[0001] The present application claims priority from Australian Provisional Patent Application No 2025900483 filed on 20 February 2025, the contents of which are incorporated herein by reference in their entirety.Technical Field

[0002] This disclosure relates to methods and systems for sharing data, such as, but not limited to personally identifiable data.Background

[0003] As more and more data is collected by various agents, it becomes more important to perform operations on joint datasets to combine the data from the various agents. However, often the collected data is confidential and may contain personally identifiable information and therefore, cannot be shared in plain text.

[0004] Encryption can be used to encrypt data so that it can be sent to other services. However, encryption of large datasets takes a large amount of processing power. Further, the sending of encrypted data requires large network capacity.

[0005] Therefore, there is a need for methods and systems for sharing data in a computationally more efficient way.

[0006] Any discussion of documents, acts, materials, devices, articles or the like which has been included in the present specification is not to be taken as an admission that any or all of these matters form part of the prior art base or were common general knowledge in the field relevant to the present disclosure as it existed before the priority date of each of the appended claims.

[0007] Throughout this specification the word "comprise", or variations such as "comprises" or "comprising", will be understood to imply the inclusion of a stated element, integer or step, orgroup of elements, integers or steps, but not the exclusion of any other element, integer or step, or group of elements, integers or steps.Summary

[0008] This disclosure provides a system for sharing confidential data that is stored by a first data provider. The system comprises a matching service that runs on the infrastructure of the first data provider and that selects data for matching and encrypts the selected data. This way, the size of the dataset is reduced, the encryption is performed locally and the dataset, encrypted or in plain text, remains on the data providers’ infrastructure. The matching service receives encrypted query data and matches the query data against the encrypted stored data. The matching result is then returned in encrypted form. A broker application, that may be outside the first data providers infrastructure and may serve multiple data providers, receives the encrypted results and may aggregate them before sending them to a data consumer. The disclosed system is decentralised, which means the datasets remain with the data providers. This is especially advantageous in the case of multiple data providers because they each retain their datasets, which is in contrast to centralised systems where the entire dataset is copied to a central server that then performs the matching.

[0009] A system for sharing confidential data between a first data provider and a data consumer comprises:a first matching service that is prevented from accessing first plain data stored by the first data provider, the first matching service configured to perform the steps of:receiving from the first data provider first obfuscated data that comprises at least part of the first plain data obfuscated to prevent access to the first plain data by the data consumer;storing the first obfuscated data on a first data storage;a first matching engine configured to match the first obfuscated data stored on the first data storage;a first extraction engine configured to extract response data related to the first obfuscated data matched from the first data storage; anda broker application, configured to:receive a request from the data consumer, the request comprising obfuscated query data;provide the obfuscated query data to the first matching engine to query the first data storage for the obfuscated query data;receive, from the first extraction engine the response data; andprovide the response data to the data consumer.

[0010] It is an advantage that the matching service receives data that is obfuscated because this means that no confidential data leaves the first data provider. Further, the first data provider can send only a subset of the data that contains the fields that are used for the matching and in the returned data. In other words, the first data provider can send only some data fields, which can significantly reduce the size of the dataset and reduce the query time. It is a further advantage that the system comprises a matching engine that matches the obfuscated data. This way, the matching engine does not need to be deployed at the first data provider but can be deployed and administered centrally at the first matching service. As a result, the first data storage and the first matching engine can be administered by the same service provider, which increases robustness and therefore security.

[0011] In some embodiments, the first data storage prevents access to the first obfuscated data by the data consumer.

[0012] In some embodiments, the first obfuscated data is generated by encrypting the first plain data using a first public key corresponding to a first private key for decrypting the first obfuscated data, and the first private key is stored private to the first data provider.

[0013] In some embodiments, the system further comprises:a second matching service that is prevented from accessing second plain data stored by a second data provider, the second matching service configured to perform the steps of:receiving from the second data provider second obfuscated data that comprises at least part of the second plain data obfuscated to prevent access to the second plain data by the data consumer and by the first data provider;storing the second obfuscated data on a second data storage;a second matching engine configured to match the second obfuscated data stored on the second data storage;a second extraction engine configured to extract response data related to the second obfuscated data matched from the second data storage; andthe broker application is further configured to:provide the obfuscated query data to the second matching engine to query the second data storage for the obfuscated query data;receive, from the second extraction engine the response data; and provide the response data to the data consumer.

[0014] In some embodiments, the broker application is further configured to consolidate the response data from the response data from the first extraction engine and the response data from the second extraction engine.

[0015] In some embodiments, the first obfuscated data and the second obfuscated data are generated using a one-way obfuscation method applied to the first plain data and the second plain data, respectively.

[0016] In some embodiments, the first obfuscated data, the second obfuscated data and the obfuscated query data are generated by encrypting the first plain data, the second plain data and plain query data, respectively.

[0017] In some embodiments, the first obfuscated data is generated by using an encryption key that is identical to the encryption key used to generate the second obfuscated data and the plain query data.

[0018] In some embodiments, the system further comprises a key generator that is configured to generate the encryption key, and the first data provider, the second data provider and the data consumer are prevented from accessing a decryption key corresponding to the encryption key.

[0019] In some embodiments, the key generator is configured to discard the decryption key.

[0020] In some embodiments, encrypting the first plain data, the second plain data and the plain query data comprises using homomorphic encryption.

[0021] In some embodiments, each of the first obfuscated data and the second obfuscated data comprise multiple data records and each of the multiple data records comprises multiple data fields comprising obfuscated query fields for matching against the obfuscated query data and obfuscated value fields contained in the response data.

[0022] In some embodiments, the broker application is configured to perform arithmetic operations on the obfuscated value fields encrypted using homomorphic encryption to aggregate the first obfuscated data with the second obfuscated data.

[0023] In some embodiments, the homomorphic encryption is configured to generate different cypher text for identical plain text.

[0024] In some embodiments, preforming the homomorphic encryption comprises adding a random salt value to the plain data to generate obfuscated data.

[0025] In some embodiments, the first matching engine and the second matching engine are configured to apply matching rules that perform approximate string matching.

[0026] In some embodiments, the homomorphic encryption comprises mapping text characters to number values and encrypting the number values, and matching comprises calculating a difference between the number values.

[0027] In some embodiments, the request from the data consumer comprises a data transfer key and the first extraction engine is configured to send the response data to the first data provider for encryption of the first plain data using the data transfer key and send the response data, encrypted using the data transfer key, to the broker application.

[0028] In some embodiments, the data transfer key is a data transfer public key associated with a data transfer private key stored on the data consumer.

[0029] In some embodiments, the system further comprises a data selection module configured to select first plain data from an unencrypted data storage private to the first data provider and provide the selected first plain data, in encrypted form, to the first data storage.

[0030] In some embodiments, the first matching service is hosted on computing infrastructure of the first data provider.

[0031] In some embodiments, the broker service is hosted on computing infrastructure outside the first data provider.

[0032] A method for sharing confidential data between a first data provider and a data consumer as performed by a matching service comprises:receiving from the first data provider first obfuscated data that comprises at least part of first plain data obfuscated to prevent access to the first plain data by the data consumer, wherein the matching service is prevented from accessing the first plain data stored by the first data provider;storing the first obfuscated data on a first data storage;receiving obfuscated query data from a broker application;matching the first obfuscated data stored on the first data storage to the obfuscated query data;extracting response data related to the first obfuscated data matched from the first data storage; andsending the response data to the broker application.Brief Description of Drawings

[0033] An example will now be described with reference to the following drawings:

[0034] Figure 1 illustrates a system for sharing confidential data between one data provider and a data consumer.

[0035] Figure 2 illustrates a system for sharing confidential data between two data providers and a data consumer.

[0036] Figure 3 illustrates a method for sharing confidential data.Description of Embodiments

[0037] As set out above, there is a need for an improved data sharing system and method to share data from a data provider with a data consumer. One option may be to encrypt the entire dataset and send it outside the data provider’s infrastructure to an independent custodian or governance module. The custodian can then receive another encrypted dataset from another data provider and then perform matching between the two encrypted datasets. However, this solution requires a large amount of processing power for encryption, large bandwidth for sending theencrypted dataset and updates to the dataset are difficult to propagate to the encrypted dataset held by the custodian.

[0038] Therefore, this disclosure provides an improved system that does not require the sending of the entire encrypted data set to a custodian.System with one data provider

[0039] Figure 1 illustrates a system 100 for sharing confidential data between a data provider 101 and a data consumer 102. Only one data provider 101 is shown noting that multiple data providers can participate as explained further below. Data provider 101 hosts a database 103 of sensitive data. The sensitive data is stored in unencrypted form, which is also referred to as plain data in order to enable fast and computationally efficient access to the sensitive data by the data provider 101. Database 103 is protected from external access by any entity outside the data provider 101. Data provider 101 also operates a data selection module 104, which may be an algorithm that selects data from the sensitive data on database 103. In particular, the data selection module 104 stores rules on which data is selected. For example, the database 103 may store multiple data records and each data record contains multiple fields containing sensitive data of one person. The data selection algorithm may then have rules defined that select a number of fields from each record.

[0040] The selection rules may depend on the what matches are intended to be performed on the data. For example, if it is intended to match only an email address, then only the email address is selected from the data fields. However, if multiple fields are intended to be matched, such as first name, last name, and email address, then those fields are selected. The selection rules may also define to select an return field that is of interest to the data consumer 102. For example, the return field may be a numerical value field, such as a currency field, which may indicate outstanding dept, expenditure, or other details. The selection rules may also select multiple return fields. The one or more selected return fields are selected together with the matching fields.

[0041] The data provider 101 obfuscates the selected data, which may be by way of a one-way function, such as a hash function or keyed hash function, or by way of a two-way function, such as an encryption function. The encryption function may be a symmetric encryption function or an asymmetric encryption function. In the case of an asymmetric encryption function, the obfuscated data is generated by encrypting the plain data on database 103 using a public keycorresponding to a private key for decrypting the first obfuscated data. The private key may be stored private to data provider 101, that is, only the data provider 101 can access the private key. As also explained herein, the private key may also be discarded, deleted or otherwise made inaccessible to any entity or not generated at all, so that no entity can decrypt the encrypted data.

[0042] The public key may be available to both the data provider 101 and the data consumer 102 (and potentially other entities in the system 100). It is not necessary that the private key is available to any entity in the system 100. This way, the sensitive data can be encrypted but no entity can decrypt the sensitive data. This changes the two-way function into a one-way function.

[0043] The system 100 further comprises a matching service 105, which may be hosted on the data provider infrastructure 106. This means that the database 103 storing the sensitive data and the matching service 105 are hosted on the same infrastructure. Therefore, any data selected by selection algorithm and encrypted does not need to be sent outside the data provider infrastructure 106. This means that external bandwidth requirements are reduced and even the encrypted data is not accessible from anywhere outside the data provider infrastructure 106 to launch attacks that use the entire set of encrypted data. In particular, by using this configuration the escrow data storage 107 prevents access to the obfuscated data by the data consumer 102.

[0044] As an extra security measure, matching service 105 is prevented from accessing the plain data stored by the first data provider on database 103. This way, the risk of accidental leakage of plain text data is reduced. The matching service may be implemented as a web service or any other service architecture.Network technology

[0045] The matching service can be implemented as a web service by creating an API that allows the data provider to interact with it over the internet. This web service can be designed to receive encrypted queries from the data consumer, process them, and return the encrypted results. The web service implementation ensures that the data remains secure and is only accessible by authorized entities. It leverages HTTPS for secure communication and can be hosted on the data provider's infrastructure to maintain proximity to the sensitive data stored in the database. This approach reduces latency and enhances security by ensuring that encrypted data does not leave the data provider's environment.

[0046] Alternatively, the matching service can be implemented using a microservices architecture. In this setup, the matching functions can be broken down into smaller, independent services that communicate with each other through lightweight protocols such as REST or gRPC. Each microservice can handle a specific part of the matching process, such as data selection, encryption, and querying. This modular approach allows for better scalability and maintainability, as individual services can be updated or replaced without affecting the entire system.

[0047] Another alternative is to implement the matching service as a serverless function on cloud platforms such as AWS Lambda, Azure Functions, or Google Cloud Functions. In this scenario, the matching logic is encapsulated in small, stateless functions that are triggered by events such as receiving a new query. Serverless functions can automatically scale to handle varying loads and ensure cost-efficiency by only consuming resources when actively processing requests.

[0048] Lastly, the matching service can also be implemented within a containerized environment using container orchestration platforms like Kubernetes or Docker Swarm.Containers provide a lightweight and portable way to deploy and manage the matching service, ensuring consistency across different environments. Container orchestration platforms offer capabilities such as automated scaling, load balancing, and self-healing, which enhance the reliability and efficiency of the matching service.

[0049] Each of these implementation strategies offers distinct advantages and can be chosen based on the specific requirements and constraints of the data provider's infrastructure and operational needs.Matching service

[0050] Matching service 105 is configured to perform the steps explained below. “Configured to” in this context may mean that there is a processor that is programmed to perform the steps described below in the sense that each step is implemented by program code stored on a nonvolatile computer readable medium. Other hardware can equally be used, including integrated circuits, such as cryptography chips, field programmable gate arrays, cloud computing architectures and others.

[0051] Matching service 105 receives from the data provider 101 the obfuscated (e.g. encrypted) data. As described above, the obfuscated data comprises at least part of the plain data stored on database 103 that is selected by data selection module 104 and then obfuscated to prevent access to the first plain data by the data consumer 102 (and other entities other than data provider 101).

[0052] Matching service 105 stores the obfuscated data on an escrow data storage 107. In one example, the escrow data storage 107 is protected from access from outside the data provider infrastructure 106.

[0053] Preventing access as used throughout this disclosure may refer to network segmentation, which involves dividing a computer network into smaller, isolated segments to enhance security. By isolating the escrow database within a specific network segment, access can be restricted to only those systems and users within the data provider's infrastructure.Implementing firewalls between network segments can prevent unauthorized access by controlling the traffic that can pass between these segments. Using virtual local area networks (VLANs) to separate the escrow database from other parts of the network ensures that only authorized devices within the same VLAN can communicate with the database. Deploying access control lists (ACLs) on network devices can further restrict access by specifying which users or systems can access the escrow database based on IP addresses and other criteria.

[0054] Physical security measures can also be used for protecting the hardware that stores the escrow database. Hosting the escrow database in a data centre that has robust physical security measures, such as surveillance cameras, access control systems, and security personnel, can prevent unauthorized physical access. Using hardware security modules (HSMs) to store encryption keys and perform cryptographic operations ensures that the keys are protected by physical tamper-resistant devices.

[0055] While the escrow data storage 107 itself is not accessible from outside the data provider infrastructure, there is a matching engine 108 that can be contacted from the outside. More specifically, matching engine 108 can receive requests from a broker application 109. The request includes obfuscated request data that the matching engine 108 can match against the obfuscated data is the escrow data storage 107. This way, obfuscated data can enter the data provider infrastructure 106 but the escrow data storagel07 itself is not exposed. In some examples, the key used by data provider 101 to encrypt the plain data to be stored on escrow datastorage 107 is the same as the key used by data consumer 102 to encrypt the query data. As a result, the cypher is identical and the query data matches the obfuscated data in escrow data storage 107. In other examples, the encryption is altered to change the cypher for identical plain text to prevent phone book attacks where an attacker simply encrypts every record of a dataset of people and attempts to match the obfuscated data against that encrypted dataset. To prevent this attack, a salt value can be added to the plain text before encryption. The salt value may be a random value and results in the cypher being different each time the plain data or the query data is encrypted. The match can still be performed using homomorphic encryption and fuzzy matching as described below.

[0056] Matching engine 108 may perform the matching based on a number of rules that govern which matches, and in particular partial matches, are acceptable. For example, each data field may be associated with a significance value, such as a high significance value of 1 for social security number data fields and a low significance value of 0.1 for first name data fields. The query data may contain values for multiple fields and the matching engine may calculate a matching score indicative of the degree of a match for each record in escrow data storage 107. Matching engine 108 may then select the record with the highest matching score. For example, matching engine 108 may add the significance value for each field that matches a corresponding value in the query data. The match for each field may also be partial, which may result in a value that indicates the degree of match. That value can be combined, such as multiplied, with the significance score to provide the contribution of that field to the total matching score.

[0057] For example, the escrow data storagel07 may hold two recordsJohn | Smith | jsmith@gmail.com |Johnathan | Smith | | +61401234567and the query data may beJohnathan | Smith | +61401234567 | jsmith@hotmail.comThe first record has a full match of second name and email address, while the second record has full match of first name, second name and phone number but a mismatch in the email address. It is now possible to assign a higher significance value to the phone number than to the email address and together with the full match in the first name, the second record would result in a higher matching score and would be selected. Various different combinations of significance scores and matching scores are possible.

[0058] The matching engine 108 may further implement security rules to check the requests before they are executed in a search. For example, the received queries can be assessed for suspicious behaviour, such as based on the number of requests. In one example, if the number of requests within a give time frame is above a threshold, this indicates that an end user is potentially trying to combine the results into the entire dataset. In that case, matching engine 108 may block queries that are determined as being suspicious.Extraction engine

[0059] System 100 further comprises an extraction engine 110, which is configured to extract response data related to the first obfuscated data matched from the escrow data storage 107. Extraction engine 110 performs one or more extraction rules that define which fields are extracted from the matching records and provided as output to broker 109. For example, escrow data store 107 may comprise, for each record, a first set of data fields that are used for matching and a second set of data fields that are returned to broker 109. It is also possible that broker 109 requests only one or only a subset of the second set of data fields. In that case, only the requested data fields are returned. In a further example, extraction engine 110 performs checks on the matched data records before passing them to broker 109. For example, extraction engine 110 may check that there are a minimum number of data records to avoid re-identification or other checks.

[0060] It is noted again that the data in escrow data storage 107 may be encrypted, which means the extraction engine 110 returns encrypted data to broker 109. In one example, the data is encrypted homomorphically, so that broker 109 can still perform arithmetic operations, such as additions, on the returned data fields. In other examples, the data to be returned is not encrypted in escrow database 107. That is, there are fields (‘columns’) that are not sensitive on their own without a link to the sensitive fields. This may include values such as outstanding debt, which is not sensitive by itself without being linked to a name. These data fields that are not sensitive, can be stored in plain text and returned to the broker 109 in plain text. Broker 109 can then aggregate the plain text return values before sending the aggregated value back to data consumer 102. It is noted that there may still be fields (or ‘columns’) that store sensitive information and are encrypted and those fields are used for matching (such as names, personal identifiers, etc.). This way, the query data is obfuscated (e.g., encrypted) to prevent leakage of sensitive information.

[0061] Further examples may include encrypted fields that are encrypted with different keys. That is, each field may have its own key so that is decryptable by different parties depending onthe sensitivity of the data. The keys that can decrypt the data may be held by the broker 109, the data consumer 102 or elsewhere.Broker application

[0062] As mentioned above, system 100 comprises broker application 109, which receives a request from the data consumer 102. The request comprises obfuscated query data, which may be created by the data consumer 102 using hashing or encryption as described herein. Data consumer 102 may use the same hashing or encryption method / algorithm as data provider 101 uses to obfuscate the data stored on escrow data storage 107. This way, the obfuscated data can be matched efficiently. Broker application 109 provides the obfuscated query data to the matching engine 108 to query the escrow data storage 107 for the obfuscated query data. In response, broker application 109 receives, from extraction engine 110 the response data and provides the response data to the data consumer 102.

[0063] In one example, broker application 109 is implemented outside the data provider’s infrastructure 106, which means that access rights, firewalls, etc. are separate between the data provider’s infrastructure 106 and broker application 109. This way, the broker application 109 has no direct access to the obfuscated data on escrow data store 107. Further, an attacker with full access to the broker application 109 is not able to access the obfuscated data stored on escrow data storage 107. Finally, as described further below, a single broker application 109 can communicate with multiple matching engines of multiple data providers. At the same time, data consumer 102 only communicates with a single broker application 109, which improves efficiency in operation and communication.

[0064] Further, the matching service 105 receives data that from broker 109 that is obfuscated, which means that no confidential data leaves the first data provider 101. Further, the first data provider 101 can send only a subset of the data that contains the fields that are used for the matching and in the returned data. In other words, the first data provider can send only some data fields, which can significantly reduce the size of the dataset and reduce the query time. It is a further advantage that the system comprises a matching engine 108 that matches the obfuscated data. This way, the matching engine 108 does not need to be deployed at the first data provider 101 but can be deployed and administered centrally at the first matching service 105. As a result, the escrow data storage 107 and the matching engine 108 can be administered by the same service provider, which increases robustness and therefore security.System with two or more data providers

[0065] While system 100 in Figure 1 comprises one data provider 101, it is equally possible to add further data providers that store their own data in obfuscated form on their own, respective escrow data storage. For example, Figure 2 illustrates a system 200 for sharing confidential data comprising first data provider 101 from Figure 1 and now also including second data provider 201. It is noted that more than two data providers can be integrated and the above description related to the elements of first data provider 101 equally apply to any further data providers. That is, similar to first data provider 101, there is a second database 203 to store sensitive data, which is confidential to second data provider 201 and a data selection algorithm 204 that selects data to be obfuscated and stored in escrow. System 200 further comprises a second matching service 205 that may be hosted on the second data provider’s infrastructure 206. Second matching service 205 is prevented from accessing plain data stored by second data provider 201 on database 203. Similar to the first matching service 105, he second matching service 205 receives from the second data provider 201 obfuscated data that comprises at least part of the second plain data stored on database 203. The data is obfuscated to prevent access to the plain data by the data consumer 102 and by the first data provider 101 and by any other entity within our outside system 200. Second matching service 205 also stores the obfuscated data from second data provider 201 on escrow data storage 207.

[0066] A second matching engine 208 may also be hosted on second data provider’s infrastructure 206 and matches the obfuscated data stored on the escrow data storage 207 and second extraction engine 210 extracts response data related to the second obfuscated data matched from the second escrow data storage 207 as described above with reference to Figure 1.

[0067] In this case, with more than one data provider, broker application 109 provides the obfuscated query data to the second matching engine 208 to query the second escrow data storage 207 for the obfuscated query data and receives, from the second extraction engine 210 the response data. Finally, broker application 109 provides the response data from second extraction engine 210 to the data consumer 102.

[0068] As mentioned above, the broker application may consolidate the response data from the response data from the first extraction engine 110 and the response data from the second extraction engine 210 into a consolidated or single response to data consumer 102. This way, data consumer 102 receives an insight relating to the original query where that insight is basedon both sensitive data sets stored securely on databases 103 and 203 without accessing any of the plain data.

[0069] In relation to the data obfuscation, it is noted that the first obfuscated data stored on first escrow data storage 107 is generated by using an encryption key that is identical to the encryption key used to generate the second obfuscated data stored on second escrow data storage 207 and identical to the encryption key used to encrypt the plain query data. That is, all entities, the first data provider 101, the second data provider 201 and the data consumer 102 may use the same encryption key, such as the same public key to encrypt their database or query data.

[0070] To that end, system 200 may further comprise a key generator that generates this encryption key. Further, all other entities, including the first data provider 101, the second data provider 201 and the data consumer 102 are prevented from accessing the decryption key corresponding to the encryption key. This may be achieved by storing the private key corresponding to the published public key safely on the key generator or by destroying, discarding or never generating the private key.

[0071] Using identical encryption keys is useful if combined operations are being performed on the data. In other cases where the broker returns data from 107 and 207 without combining them, data storages 107 and 207 can be encrypted with different keys. It is also possible to encrypt the matching fields with different fields - in this scenario the query is encrypted twice (one sent to first data storage 107 and one sent to second data storage 207).Homomorphic encryption

[0072] As also mentioned above, encryption of the data for escrow databases 107 / 207 and the query data may be performed using homomorphic encryption, which allows the performing of operations on the encrypted data. More particularly, the obfuscated data stored on first and second escrow data storage 107 / 207, may comprise multiple data records. Each of the multiple data records comprises multiple data fields comprising obfuscated query fields for matching against the obfuscated query data and obfuscated value fields contained in the response data. Broker application 109 can then perform arithmetic operations on the obfuscated value fields encrypted using homomorphic encryption to aggregate the first obfuscated data with the second obfuscated data.

[0073] While homomorphic encryption effectively obfuscates the plain data from an attacker, it may be possible to extract the plain data by a dictionary attack. More particularly, since the public encryption key is available to an attacker, the attacker can calculate the cypher for all common first names and second names. The attacker can then read the query data and match the query data to the calculated cyphers. When they match, the attacker has determined the actual plain data of the query.

[0074] In order to address this vulnerability, the obfuscation method may be configured to generate different cypher text for identical plain text. This may be achieved by a non-deterministic obfuscation method. For example, the method may introduce random data into the process, such as into the plain text each time it is encrypted. This random data may be referred to as a random salt value added to the plain data to generate the obfuscated data. This way, the generated obfuscated data is different each time the plain data is obfuscated. In that case, the homomorphic encryption method can be configured to find matches even where the input plain text has been randomly altered.

[0075] In some examples, the first matching engine 108 and the second matching engine 208 are configured to apply matching rules that perform approximate string matching. This is useful because in many cases, field values are not exactly the same as there is some amount of noise in the data. For example, first names may be varied, abbreviated or misspelled (e.g. “John”, “Jonny”, “Jonathan”). In order to account for this variation in the string space, the homomorphic encryption may comprise mapping the text characters to number values and encrypting the number values. Then, the matching engine can match entries in escrow data storage 107 / 108 by calculating a numerical difference between the number values using the concept of homomorphic encryption (e.g. calculating a subtraction of homomorphically encrypted number values).

[0076] There is an advantage that the approximate string matching can be combined with other matching rules in matching engine 108 / 208. For example, the matching rules may state that a match of either email address or phone number is sufficient for a match. This can be assigned a numerical value, e.g. +1 for phone number match and +1 for email address match. The first name and second name matches may be represented by a numerical difference d. The match between some fields (e.g. phone number and email address) may be added to obtain an aggregate match score. This match score may then the be aggregated with the numerical difference d, such as by adding the result of a monotonically decreasing function (e.g. \!d) to the total match score.Matching engines 108 / 208 can then apply a threshold (e.g. 2.0) to each data record for a match or simply return the data record with the highest total score (or none if no score is above the threshold). Numerous variations regarding matching scores aggregations are possible to suit the requirements of a particular scenario.

[0077] The system is not limited to text-based inquiries. Information included in media formats such as images may be extracted using feature detection and / or machine learning algorithms and passed onto matching engine 108 / 208. The match score can be calculated from pixel / feature matches across images and a minimum threshold applied before match will be accepted. For example, if a data consumer wished to find all records that matched a given persons face amongst images, the matching algorithm can search the database of multiple data providers for all images that having similar detected features and provide a quality score to the consumer. In one example, in the case of non-textual queries, such as image queries, the image may be transformed into numerical features, such as eigenface feature vectors, embeddings calculated by a autoencoder / transf ormer network or others. In that case, the matching engine 108 / 208 can calculate the distance between the features of the face in the query to the features in escrow data storage 107. The transformation of non-textual data, such as images, to numerical feature values may be performed by data providers when storing the data in escrow data storage 107 as well as by data consumer 102, broker application 109, or matching engine 108. It is noted that the matching of features also works when using obfuscated feature values, such as homomorphically encrypted feature values as explained herein.

[0078] It is noted that in some applications, it may not be required that the matching is perfect. That is, in many applications, the “ground truth” is not known in the sense that two records in plain databases 103 / 203 may be similar and may relate to the same person or may relate to different persons. Matching engines 108 / 208 find the most likely outcome, which may not always be the correct outcome. Therefore, the additional noise and potential inaccuracy introduced by the use of homomorphic encryption and added randomness, may affect the matching scores but may not significantly change the most likely outcome. In other words, the noise introduced by the employed encryption algorithm may be less than the difference between the most likely and the next most likely outcome. As a result, the record that matches the query best without added noise, may also match the query best with added noise, even if there is no perfect match in one or both of these cases. Even if there are some changed matches due to added noise, the aggregation over potentially many records by the broker 109 reduces the influence of a single or small number of mismatched records.Plain data retrieval

[0079] While broker application 109 returns an aggregated result to data consumer 102, there may be instances where the consumer desires to obtain the actual plain data of the matched data records even though the data is encrypted in escrow data storage 107. In that case, data consumer 102 can request the plain data from data providers 101 / 201. The returned plain data may be encrypted so that only data consumer 102 can decrypt the plain data. To that end, there may be a separate key that is available for encryption and decryption. In one example, this transfer key is a symmetric key that is exchanged between the data consumer and the data providers 101 (all entities involved have the same key). This key may be specific to first data provider 101 and different to the transfer key used with second data provider 201. In yet another example, the plain data is encrypted for transfer with a public key of the data consumer 102. In that case, the request from the data consumer 102 comprises a data transfer (public) key and the plain data is encrypted using the public key. Data consumer 102 can then use the corresponding private key to decrypt the data. In that flow, the extraction engines 108 / 208 may send the response data to the first data provider 101 / 201 for encryption of the plain data using the data transfer key and send the response data, encrypted using the data transfer key, to the broker application 109.

[0080] The individual records to be transferred may be identified by a record identifier that is unencrypted in the matches or may be determined from matching the encrypted records against the datasets in escrow data storage 107. That is, data consumer 102 may send a request for plain data to the data provider 101. That request may include the encrypted data (cypher). This encrypted data matches exactly against one of the entries in the escrow data storage 107 which determines the record for the data provider 101. The data provider 101 can then retrieve that data record from database 103, encrypt it and send it to the data consumer 102. It is noted that this request for plain data may be enabled or disabled depending on the requirements of the data collaboration.

[0081] Figure 3 illustrates a method 300 for sharing confidential data stored on database 103 between first data provider 101 and data consumer 102 as performed by matching service 105. First, matching service 105 receives 301 from first data provider 101 first obfuscated data. The first obfuscated data comprises at least part of first plain data stored on database 103. The first obfuscated data is obfuscated to prevent access to the first plain data by the data consumer 102. Further, matching service 105 is prevented from accessing the first plain data stored by the first data provider 101 in improved security of the system. Matching service 105 stores 302 the firstobfuscated data on a escrow data storage 107. These steps are part of the initial set-up of the method and are typically only performed once, noting that the sensitive data on database 103 may be updated and those updates may be applied to the obfuscated data 107 by way of updating records that have been altered on database 103.

[0082] Then, during operation of the data sharing, matching service 105 receives 303 obfuscated query data from broker application 109 and matches 304 the first obfuscated data stored on the first escrow data storage 107 to the obfuscated query data received from broker application 109. Matching service 105 then extracts response data related to the first obfuscated data matched from the first escrow data storage 107 as described above and sends the response data to the broker application. Throughout this process, advanced encryption techniques are employed to protect the data at rest and in transit, ensuring that sensitive information remains secure and inaccessible to unauthorized parties. The method 300 performed by matching service 105 offers several advantages, including robust data security, enforced separation of data roles, and the use of encryption to safeguard information. By preventing direct access to plain data and maintaining control over data interactions, the system provides a secure and efficient way to share confidential information between data providers and consumers. This approach not only protects privacy but also builds trust among users who rely on the integrity and security of their data-sharing mechanisms.Obfuscation details

[0083] The following description provides further detail on obfuscation methods that may be used. In particular, the disclosed methods may involve the use of full homomorphic encryption (FHE). Full homomorphic encryption's mathematical foundation rests on lattice-based cryptography, where messages are mapped to points in ideal lattices with added noise. These lattices form regular arrangements in n-dimensional space, and the security derives from the computational hardness of solving certain lattice problems, particularly Ring Learning With Errors (Ring-LWE). The actual operations take place in polynomial rings, typically denoted as R = Z[X] / (XAN + 1), where N is chosen as a power of 2 for computational efficiency.

[0084] Messages are encoded as polynomials in this ring, with coefficients taken modulo some value q. One aspect is noise management - a ciphertext takes the form c = m + e (mod q), where m is the message and e represents small random noise. When operations are performed on ciphertexts, this noise grows. Addition of ciphertexts results in addition of their respective noises, while multiplication causes more noise growth. The noise is kept below a threshold forcorrect decryption. When noise grows too large, a process called bootstrapping can "refresh" noisy ciphertexts. For string operations specifically, characters may be encoded in a way that preserves their relationships and allows the desired operations while maintaining reasonable noise growth - this may involve representing them as binary values or polynomials. Each operation on encrypted strings balances the mathematical requirements of the FHE scheme with the practical needs of string manipulation, ensuring that the noise growth doesn't exceed the scheme's capacity for correct decryption.

[0085] String operations in homomorphic encryption use encoding since FHE natively works with numbers. The steps may comprise

[0086] 1. Character Encoding:' python# First, characters need to be converted to numbers# Example encoding :'A' -> 1' B ' -> 2' Z ' -> 26

[0087] 2. Common String Operations:String Equality:- Convert strings to vectors of numbers- Subtract corresponding elements- If all differences are 0, strings are equal' python# Pseudocode for encrypted string comparisondef compare strings (enc strl, enc str2 ) :difference = enc strl - enc str2 # Element-wise subtractionresult = all zeros (difference) # Check if all differences are 0

[0088] String Search:- Use sliding window techniques- Convert pattern matching to numerical operations- May involve polynomial evaluation

[0089] Pattern Matching:' python# Basic pattern matching in FHE ( simplified pseudocode)def pattern match ( encrypted text, pattern) :n = len ( encrypted text)m = len (pattern)pattern encoded = encode to numbers (pattern)matches = [ ]for i in range (n-m+1 ) :window = encrypted text [i : i+m]diff = homomorphic subtract (window, pattern encoded)is match = homomorphic check zero (diff )matches . append ( is match)

[0090] Optimization Techniques:a. Batching:- Process multiple characters in parallel using SIMD (Single Instruction Multiple Data)- Improves throughput significantlyb. Custom Encoding:' python# Example of efficient binary encodingdef binary encode ( char) :# Convert char to binary representation# Each bit can be processed independentlyreturn format (ord ( char) , ' 08b ' )

[0091] c. Preprocessing:- Convert common operations to lookup tables- Use optimized circuits for frequent patterns

[0092] This disclosure provides a system for sharing confidential data between a data provider and a data consumer. This system employs a first matching service, a first matching engine, and a first extraction engine to ensure that sensitive information is securely handled while facilitating necessary data exchanges.

[0093] One of the significant advantages of this system is its robust security mechanism. By obfuscating the plain data before any processing occurs, the system prevents unauthorized access and ensures that sensitive information remains confidential throughout the data sharing process. This obfuscation method is useful in protecting the data provider's information from potential breaches, thereby maintaining the integrity and privacy of the data.

[0094] Another advantage lies in the system's efficiency. The use of lookup tables for common operations and optimized circuits for frequently encountered patterns enhance the processing speed and overall performance. This efficiency is particularly beneficial in scenarios where large volumes of data are processed and matched, as it reduces the time and computational power required to achieve accurate results.

[0095] The system's versatile architecture also allows for numerous variations and modifications, making it adaptable to different use cases. For example, it can be employed in healthcare to share patient information securely between hospitals and research institutions or in finance to exchange transaction data between banks and regulatory bodies. Additionally, its application can extend to any industry where the confidentiality of data is desired, such as legal services, government agencies, and personal data management systems.

[0096] In conclusion, this system offers a secure, efficient, and adaptable solution for confidential data sharing, making it a valuable tool in various sectors requiring stringent data protection measures. The ability to obfuscate data and the incorporation of optimized processing techniques ensure that the system meets the high standards of security and performance necessary for handling sensitive information.

[0097] It will be appreciated by persons skilled in the art that numerous variations and / or modifications may be made to the above-described embodiments, without departing from the broad general scope of the present disclosure. The present embodiments are, therefore, to be considered in all respects as illustrative and not restrictive.

Claims

1. CLAIMS:

1. A system for sharing confidential data between a first data provider and a data consumer, the system comprising:a first matching service that is prevented from accessing first plain data stored by the first data provider, the first matching service configured to perform the steps of:receiving from the first data provider first obfuscated data that comprises at least part of the first plain data obfuscated to prevent access to the first plain data by the data consumer;storing the first obfuscated data on a first data storage;a first matching engine configured to match the first obfuscated data stored on the first data storage;a first extraction engine configured to extract response data related to the first obfuscated data matched from the first data storage; anda broker application, configured to:receive a request from the data consumer, the request comprising obfuscated query data;provide the obfuscated query data to the first matching engine to query the first data storage for the obfuscated query data;receive, from the first extraction engine the response data; andprovide the response data to the data consumer.

2. The system of claim 1, wherein the first data storage prevents access to the first obfuscated data by the data consumer.

3. The system of claim 1 or 2, wherein the first obfuscated data is generated by encrypting the first plain data using a first public key corresponding to a first private key for decrypting the first obfuscated data, and the first private key is stored private to the first data provider.

4. The system of any one of claims 1 to 3, wherein the system further comprises:a second matching service that is prevented from accessing second plain data stored by a second data provider, the second matching service configured to perform the steps of:receiving from the second data provider second obfuscated data that comprises at least part of the second plain data obfuscated to prevent access to the second plain data by the data consumer and by the first data provider;storing the second obfuscated data on a second data storage;a second matching engine configured to match the second obfuscated data stored on the second data storage;a second extraction engine configured to extract response data related to the second obfuscated data matched from the second data storage; andthe broker application is further configured to:provide the obfuscated query data to the second matching engine to query the second data storage for the obfuscated query data;receive, from the second extraction engine the response data; and provide the response data to the data consumer.

5. The system of claim 4, wherein the broker application is further configured to consolidate the response data from the response data from the first extraction engine and the response data from the second extraction engine.

6. The system of claim 4 or 5, wherein the first obfuscated data and the second obfuscated data are generated using a one-way obfuscation method applied to the first plain data and the second plain data, respectively.

7. The system of claim 4 or 5, wherein the first obfuscated data, the second obfuscated data and the obfuscated query data are generated by encrypting the first plain data, the second plain data and plain query data, respectively.

8. The system of claim 7, wherein the first obfuscated data is generated by using an encryption key that is identical to the encryption key used to generate the second obfuscated data and the plain query data.

9. The system of claim 8, wherein the system further comprises a key generator that is configured to generate the encryption key, and the first data provider, the second data provider and the data consumer are prevented from accessing a decryption key corresponding to the encryption key.

10. The system of claim 9, wherein the key generator is configured to discard the decryption key.

11. The system of any one of claims 7 to 10, wherein encrypting the first plain data, the second plain data and the plain query data comprises using homomorphic encryption.

12. The system of any one of claims 7 to 11, wherein each of the first obfuscated data and the second obfuscated data comprise multiple data records and each of the multiple data records comprises multiple data fields comprising obfuscated query fields for matching against the obfuscated query data and obfuscated value fields contained in the response data.

13. The system of claim 12, wherein the broker application is configured to perform arithmetic operations on the obfuscated value fields encrypted using homomorphic encryption to aggregate the first obfuscated data with the second obfuscated data.

14. The system of claims 11 to 13, wherein the homomorphic encryption is configured to generate different cypher text for identical plain text.

15. The system of claim 14, wherein preforming the homomorphic encryption comprises adding a random salt value to the plain data to generate obfuscated data.

16. The system of claim 11 to 15, wherein the first matching engine and the second matching engine are configured to apply matching rules that perform approximate string matching.

17. The system of claim 16, wherein:the homomorphic encryption comprises mapping text characters to number values and encrypting the number values, andmatching comprises calculating a difference between the number values.

18. The system of any one of the preceding claims, wherein the request from the data consumer comprises a data transfer key and the first extraction engine is configured to send the response data to the first data provider for encryption of the first plain data using the data transfer key and send the response data, encrypted using the data transfer key, to the broker application.

19. The system of claim 18, wherein the data transfer key is a data transfer public key associated with a data transfer private key stored on the data consumer.

20. The system of any one of the preceding claims, wherein the system further comprises a data selection module configured to select first plain data from an unencrypted data storage private to the first data provider and provide the selected first plain data, in encrypted form, to the first data storage.

21. The system of any one of the preceding claims, wherein the first matching service is hosted on computing infrastructure of the first data provider.

22. The system of any one of the preceding claims, wherein the broker service is hosted on computing infrastructure outside the first data provider.

23. A method for sharing confidential data between a first data provider and a data consumer as performed by a matching service, the method comprising:receiving from the first data provider first obfuscated data that comprises at least part of first plain data obfuscated to prevent access to the first plain data by the data consumer, wherein the matching service is prevented from accessing the first plain data stored by the first data provider;storing the first obfuscated data on a first data storage;receiving obfuscated query data from a broker application;matching the first obfuscated data stored on the first data storage to the obfuscated query data;extracting response data related to the first obfuscated data matched from the first data storage; andsending the response data to the broker application.

Images