Methods, systems, interfaces and computer readable media for training machine learning models

Abstract

A method for extracting data from a data object comprising receiving, at an extraction application programming interface (API), an extraction request comprising at least one data object or at least one data object reference and an extraction schema comprising one or more target data elements. The method includes determining for each target data element, whether a target model exists for extracting the target data element. Responsive to determining that a target model exists, the target model is applied to the data object to extract one or more values associated with the target data element from the data object. Responsive to determining that a target model does not exist, a general extractor model is applied to the data object to extract one or more values associated with the target data element from the data object. The method further includes generating and outputting an extraction payload.

Description

"Methods, systems, interfaces and computer readable media for training machine learning models"Cross-reference to related applications

[0001] This application claims the benefit of Australian Provisional Patent Application No. 2025900911, filed 20 March 2025, and Australian Provisional Patent Application No. 2025903601, filed 12 August 2025, which are incorporated by reference herein in their entirety, and are hereby expressly made a part of this specification.Technical Field

[0002] Embodiments generally relate to systems, methods and computer readable media for training machine learning models, and in some embodiments, for configuring and / or training machine learning models using reference machine learning models and / or extracted data.Background

[0003] The deployment and training of machine learning models faces several challenges and difficulties that impede their efficiency, accuracy and scalability. Large machine learning models, such as Large Language Models (LLMs) often require substantial computational resources and storage, making them challenging to deploy in resource-constrained environments. These models, due to their size and complexity, often need significant server space for both the model weights and the data used for training and inference.

[0004] Training time is also an issue for developing high-performing large machine learning models. Training large machine learning models from scratch requires extensive computational power and time, as well as adequate training data. This increases the cost of development and delays the development cycle, making it challenging to iterate and improve models rapidly.

[0005] Further, adequate training data can be hard to obtain. The performance of machine learning models is heavily dependent on the quality, diversity and relevance of the training data. However, many existing systems are trained on limited datasets that do not adequately represent the full range of possible data. The process of obtaining and annotating large, diverse datasets for training purposes is resource-intensive and timeconsuming, as it typically requires manual labelling of individual data. As a result, machine learning models often have to be trained on limited datasets and as a result exhibit poor generalisation and low accuracy. Models trained on limited data may also suffer from overfitting and fail to perform well on new or dynamically changing data.

[0006] Any discussion of documents, acts, materials, devices, articles or the like which has been included in the present specification is not to be taken as an admission that any or all of these matters form part of the prior art base or were common general knowledge in the field relevant to the present disclosure as it existed before the priority date of each claim of this application.Summary

[0007] Some embodiments relate to a computer-implemented method for extracting data from a data object, the method comprising:receiving, at an extraction application programming interface (API), an extraction request comprising:at least one data object or at least one data object reference; and an extraction schema, the extraction schema comprising one or more target data elements;determining, by the extraction API, for each target data element in the extraction schema:whether a target model exists for extracting the target data element; and responsive to determining that a target model exists for extracting the target data element, applying the target model to the data object to extract one or more values associated with the target data element from the data object;responsive to determining that a target model does not exist for extracting the target data element, applying a general extractor model to the data object to extract one or more values associated with the target data element from the data object;generating an extraction payload comprising the one or more extracted values; andoutputting the extraction payload.

[0008] The method may further comprise generating the extraction request. In some embodiments, the extraction request may be generated by: determining the at least one data object; and generating the extraction schema based on one or more inputs received through a user interface that defines the one or more target data elements.

[0009] Generating the extraction schema may comprise: receiving at least one input through a user interface that represents a target data element, creating one or more structured objects representing the target data element, and serialising the one or more structured objects.

[0010] In some embodiments, the one or more structured objects may be serialised into JSON. The user interface may be configured to display a command-line interface (CLI) command or a code snippet for execution of the extraction request. The extraction request may be a POST request.

[0011] In some embodiments, determining whether a target model exists may comprise querying a repository of target models to determine whether a target model associated with the target data element is located in the repository.

[0012] Each target data element may include an element type, a name, a data type and / or a description. The element type may be a single value type or a multi-instance value type.

[0013] In some embodiments, the target model may be a domain-specific model. The general extractor model may be a large language model (LLM).

[0014] Outputting the extraction payload may comprise providing the extraction payload for display on a user interface. In some embodiments, the extraction payload may comprise an extraction identifier associated with each of the one or more extracted values.

[0015] In some embodiments, the method may further comprise: receiving at least one validation input associated with an extracted value of the one or more extracted values, and generating an annotation data record corresponding to the extracted value. The annotation data record may comprise the extracted value, the data object and the target data element associated with the extracted value.

[0016] The method may further comprise outputting the annotation data record to a data store. The data store may be an Online Analytical Processing (OLAP) database.

[0017] The method may further comprise: generating a labelled set of training data using the annotation data records associated with the target data element, and training a new target model using the labelled set of training data, the new target model configured to extract values corresponding to the target data element associated with the annotation data records.

[0018] The method may further comprise querying the data store to determine a quantity of annotation data records for a target data element. The method may further comprise outputting the new target model to be accessible by the extraction API.

[0019] The at least one data object reference may be a locator or an identifier. The at least one data object reference may be used to access the at least one data object.

[0020] Some embodiments of the present disclosure provide a non-transitory computer-readable storage medium storing instructions which, when executed by a processing device, cause the processing device to perform any of the methods as described herein.

[0021] Some embodiments of the present disclosure provide a computing device comprising: the non-transitory computer-readable storage medium as described herein, and a processor configured to execute the instructions stored in the non-transitory computer-readable storage medium.

[0022] Throughout this specification the word "comprise", or variations such as "comprises" or "comprising", will be understood to imply the inclusion of a stated element,integer or step, or group of elements, integers or steps, but not the exclusion of any other element, integer or step, or group of elements, integers or steps.Brief Description of Drawings

[0023] Various ones of the appended drawings merely illustrate example embodiments of the present disclosure and cannot be considered as limiting its scope.

[0024] Figure 1 is a process flow diagram of a method 100 for training a machine learning model, according to some embodiments;

[0025] Figure l is a process flow diagram of a method for training a target model until the target model reaches convergence, according to some embodiments;

[0026] Figure 3A is a schematic diagram of a first method for creating a labelled set of training data using a plurality of reference models, according to some embodiments;

[0027] Figure 3B is a schematic diagram of a second method for creating a labelled set of training data using a plurality of reference models, according to some embodiments;

[0028] Figure 4 is a schematic diagram of a data engine configured to create a labelled set of training data, according to some embodiments;

[0029] Figure 5 is process flow diagram of an example method of training a target model to extract addresses from documents, according to some embodiments;

[0030] Figure 6 is a block diagram of a system configured to perform the methods of Figures 1, 2 and 5, according to some embodiments;

[0031] Figure 7 is a process flow diagram of a method for extracting data from a data object, according to some embodiments;

[0032] Figure 8 is a flow diagram of an extraction API configured to extract data from a data object, according to some embodiments;

[0033] Figure 9A is a first configuration of an example user interface, according to some embodiments;

[0034] Figure 9B is a second configuration of the example user interface of Figure 9A displaying extracted values, according to some embodiments;

[0035] Figure 10 is a first configuration of a second example user interface, according to some embodiments;

[0036] Figure 11 is a second configuration of the second example user interface of Figure 10 displaying extracted values, according to some embodiments;

[0037] Figure 12 is a configuration of a third example user interface showing an extraction schema and extraction request, according to some embodiments;

[0038] Figure 13 is a process flow diagram of a method for creating annotation data records based on a validation input, according to some embodiments;

[0039] Figure 14 is a flow diagram of the extraction API of Figure 8 configured to create annotation data records, according to some embodiments;

[0040] Figure 15 is a process flow diagram of a method for training a new target model to be used by an extraction API, according to some embodiments; and

[0041] Figure 16 is a flow diagram showing how new models are trained for use by the extraction API of Figure 8, according to some embodiments.Description of Embodiments

[0042] Embodiments generally relate to methods, systems and computer-readable media for training machine learning models. In some embodiments, embodiments relate to methods, systems and computer-readable media for training machine learning models using reference machine learning models. Improved methods disclosed herein for training machine learning models may enable smaller, or target, machine learning models to be trained using larger, or reference, machine learning models, for example, by using theoutput of one or more reference machine learning models. Some embodiments relate to the creation of a training dataset used to train a machine learning model. Some embodiments relate to the iterative training process of training a machine learning model by iteratively refining a training set, which may be created based on the outputs of the reference machine learning models.

[0043] Some embodiments provide a method for training a machine learning model. The method may include determining a plurality of input data objects from a first unlabelled set of data, and providing the plurality of input data objects to one or more reference models configured to generate a first output for each of the plurality of input data objects. The generated outputs are then used to create a labelled set of training data, and a target machine learning model is trained using the created labelled set of training data. As such, the target machine learning model may be trained by using a set of training data formed from the outputs of the reference machine learning models. That is, the target machine learning model may be trained using a dataset which has been labelled by one or more reference machine learning models.

[0044] In some embodiments, the trained target model may then be applied to a second unlabelled set of data, and may be configured to generate a target response for each unlabelled data object in the unlabelled set of data. The method may then determine a plurality of training data objects from the target responses generated by the trained target model. These training data objects may indicate a low level of confidence from the trained target model in the generated response. For example, the low level of confidence may be represented by a confidence metric. In some embodiments, the confidence metric may be based on a probability related to a token produced by the target model.

[0045] In some embodiments, the trained data objects may then be input back into the reference models to generate a second output for each of the training data objects, and the labelled set of training data may be updated by combining the second outputs with the labelled set of training data. That is, the labelled set of training data may be iteratively updated to improve the training of the target machine learning model.

[0046] In some embodiments, the methods, systems and computer-readable media described herein may be used to train machine learning models for domain specific tasks,or specific functionalities. For example, machine learning models may be trained to perform specific functionalities relating to analysing and / or extracting information from documents. In some embodiments, machine learning models may be trained to perform functionalities such as, but not limited to, extracting addresses from documents, extracting line items from documents and / or extracting dates from documents. The methods disclosed herein allow the training of smaller machine learning models for domain specific tasks, thereby improving the speed of training models, and the accuracy with which the trained models can perform specific tasks.

[0047] As used herein, a set of data (also referred to as a “dataset” herein) may refer to a set of related or unrelated data and / or the way the related or unrelated data is organised, for example, a collection of data. In some embodiments, a set of data may refer to a collection of data that is stored and accessed electronically. A set of data may include data from a database, or a subset of data from a database. For example, as used herein, a set of data may refer to a set of tables or relations of a database. In some embodiments, a set of data may refer to a plurality of data objects. In some embodiments, the data objects within the set of data may or may not be related and may, or may not, have one or more common features. For example, the plurality of data objects may be grouped together based on one or more common features between the data objects to define the set of data. In some embodiments, the set of data may be structured or unstructured. For example, the set of data may have a structured form, such as being organised in a tabular format having rows and columns. The set of data may be a labelled set of data or an unlabelled set of data.

[0048] A labelled set of data may refer to a collection of data object where each data object is associated with, or comprises, one or more labels, annotations, categorisations or characterisations. The labels, annotations, categorisations or characterisations may be associated with the data object or a part of the data object, and may provide additional information about the data object, or the content of the data object. For example, in a labelled set of data which comprises a plurality of images of animals, each image might be labelled as "cat", "dog", or "bird", indicating the content of the image. In another example, in a labelled set of data which comprises a plurality of invoice documents, portions of the text content in the invoice documents may be labelled as “address”, “price”, “line item”, or“date”, indicating the type of content within the document. A labelled set of data may be used for training machine learning models.

[0049] In some embodiments, “labelled” may refer to a set of data that has been pre-processed or categorised, or that has been partially pre-processed or categorised. In some embodiments, labelled may refer to a set of data that contains annotations, information or labels related to a particular type or characteristic of the data objects within the labelled set of data, such as metadata or file type. In some embodiments, labelled may refer to a set of data which is associated with a set of labels corresponding to each object in the set of data.

[0050] An unlabelled set of data may refer to a set of data that does not include any associated labels or annotations that would provide context or meaning. In some embodiments, an unlabelled set of data may refer to a set of data that does not include any associated labels or annotations relating to a specific characteristic of the data object. For example, an unlabelled dataset may comprise a plurality of image objects, without any accompanying information about the objects depicted in those images. Similarly, unlabelled dataset may comprise a list of numerical measurements without any indication of what those measurements represent, or a collection of documents, without any indication of the content of the documents.

[0051] In some embodiments, “unlabelled” may refer to a set of data that has not been pre-processed or categorised, or that has been partially pre-processed or categorised. In some embodiments, unlabelled may refer to a set of data that does not contain any annotations, information or labels related to the data objects within the set of data. In some embodiments, unlabelled may refer to a set of data that does not contain any annotations, information or labels related to a particular type of information of the data objects within the set of data, but may contain, for example, other types of information, such as metadata. In one example, the unlabelled dataset may refer to a set of data that is made up of a plurality of data objects in the form of a plurality of images or photographs of receipts. The unlabelled set of data may include no information regarding the content of the receipts in the photographs, but may include metadata related to each photograph. In another example, the unlabelled set of data may include information that the image relates to a photograph of a receipt, that is, the image contains the structure and format consistent with a receipt, but may not include any information relating to the content of the receipt (for example, the textwhich appears on the receipt in the image). An unlabelled set of data may be used to test or evaluate machine learning models.

[0052] A data object as referred to herein may comprise a unit of data that is identifiable and manipulable within a digital environment. A data object may be any data which is processed, stored, retrieved, or transmitted as a single entity. A data object may include an instance of a data structure that is defined by a particular data type. This could be a simple data type, such as an integer or a string, or a more complex data type, such as an array, a list, or a custom-defined object in an object-oriented programming language. A data object may include code to be interpreted and executed by a computer. A data object may include a file or a packet of data. For example, a file may be a document, an image file, a video file, a spreadsheet, a web page, or any other type of file that can be stored on a computer or transferred over a network. A data object may also include a table, a record, or a field within a record and may be used to organise and manipulate data within a database. In some embodiments, a data object may refer to an object made up of two or more data types combined together. In some embodiments, the data object may comprise a plurality of primitive data types, or may comprise a number of objects that may be broken down into primitive data types.

[0053] Data objects that are suitable for input to a machine learning model may be referred to as input data objects. Examples of data objects include, but are not limited to, single characters or a number, strings, dates, object data types, files (such as text files, image files, audio files, video files, zip files and archive files), documents (including word documents, PDFs, spreadsheets, presentations), code and code files (which may include text written using a programming language including Python, Java, Ruby, C, C++, C#, Perl, HyperText Markup Language (HTML), Cascading Style Sheets (CSS)), text, databases (including SQL databases, NoSQL databases, relational databases, graph databases), images (including PNG, JPEG, GIFF, TIFF, SVG, BMP, RAW, Psd, EPS), audio and video, web pages, emails, posts, and the like.

[0054] Determining the plurality of input data objects from a first unlabelled set of data may include at least one of retrieving, extracting, receiving, querying, reading, viewing, fetching, accessing, loading, selecting, scanning, and / or exporting the plurality of input data objects from a data store. In some embodiments, the data store may be directly orindirectly in communication with the reference machine learning model to which the plurality of input data object will be input. In some embodiments, determining the plurality of input data objects from a first unlabelled set of data may include receiving the plurality of input data objects from an external system or another machine learning model. In some embodiments, the plurality of input data objects may be randomly determined or sampled from the first unlabelled set of data.

[0055] The term “machine learning model” encompasses a wide range of techniques and architectures including, but not limited to, transformer models, convolutional neural networks (CNNs), recurrent neural networks (RNNs), long short-term memory (LSTM) networks, gated recurrent units (GRUs), conditional random fields (CRFs), hidden Markov models (HMMs), logistic regression, random forests, gradient boosting machines, support vector machines (SVMs), decision trees, Naive Bayes, K-nearest neighbour (KNN), and autoencoders. The term machine learning model may also be used to refer to the use of a plurality of models in combination, for example, ensemble methods, including bagging, boosting, blending, extreme gradient boosting (XGBoost), and stacking.

[0056] The machine learning model may be configured for natural language processing tasks. In some embodiments, the machine learning model may be a language model. For example, the machine learning model may comprise a statistical language model or a neural language model, or a combination of both.

[0057] The machine learning model may be configured for language processing tasks and operate in combination with other machine learning models or features. In some embodiments, the language model may encompass a variety of architectures and techniques, including but not limited to, autoregressive language models, encoder-decoder models, transformer based models, pre-trained and / or fine-tuned models, multilingual models, multimodal large language models, and / or vision language models. For example, machine learning models may include transformer models such as GPT (Generative Pretrained Transformer), BERT (Bidirectional Encoder Representations from Transformers), and / or T5 (Text-To-Text Transfer Transformer). In some embodiments, the machine learning model may include any large scale pretrained generative transformer models. In some embodiments, the machine learning model may include, but is not limited to, Sonnet,Haiku, Llama3, Flan-T5, GPT-2, LiLt, Gemini and / or LayoutLM. In some embodiments, one or more of these models may be used as a reference model.

[0058] In some embodiments, the target model may be a small machine learning model. For example, the target model may be a Flan-T5 model, GPT-2 model, LiLt model, or LayoutLM model. It will be appreciated that certain machine learning models, such as Flan-T5, GPT-2, LiLt, and LayoutLM, may be configured as either a reference model or a target model depending on the implementation. For example, a smaller or fine-tuned version of such models may be used as a target model, while a larger or more general version may be used as a reference model. In some embodiments, the reference model may comprise a large language model (LLM), such as Sonnet, Haiku, Llama3, or Gemini, which may be configured to generate outputs for training the target model.

[0059] In the context of the present disclosure, the terms "small" and "large" when used to describe machine learning models are relative terms that may refer to the comparative size, complexity, or computational requirements of the models. A "small" model may be considered small relative to a "large" model in terms of the number of parameters, the amount of training data required, the computational resources needed for training or inference, or the memory footprint of the model. For example, a model with tens of millions of parameters may be considered small when compared to a model with hundreds of billions of parameters. The designation of a model as small or large may also depend on the specific application or deployment environment, such that a model considered large in one context may be considered small in another.

[0060] The machine learning model may comprise a large language model (LLM) or a small language model (SLM). LLMs may refer to language models that are trained on large amounts of data. SLMs may be used to refer to language models that are trained on less data, configured with fewer parameters and / or have less computational complexity compared to LLMs. In some embodiments, SLMs may refer to a language model that is smaller in scale and / or scope than a LLM. For example, in some embodiments, a SLM may have fewer parameters than a LLM. That is, the parameters for a SLM may range from a few million to a few billion, which is smaller when compared with the number of parameters for a LLM which may range from hundreds of billions to trillions. In some embodiments, a SLM may be more compact and / or more efficient than a LLM in someaspect such as memory or power consumption. For example, SLMs may require less memory and less computational power than a large language model to perform the same task.

[0061] Figure 1 is a process flow diagram of a method 100 for training a machine learning model, according to some embodiments. The method 100 comprises, at 102, determining a plurality of input data objects from a first unlabelled set of data. The plurality of data objects are unlabelled data objects. In some embodiments, the plurality of input data objects may form all or part of the first unlabelled set of data. In some embodiments, the plurality of input objects may be manually or automatically selected or extracted from the first unlabelled set of data. At 104, the plurality of input data objects are provided to a reference machine learning model. The reference machine learning model, or reference model, is a pre-trained model that is configured such that its output may be used to provide a reference for training other machine learning models. The reference model may also be referred to as a teacher model. The reference model may, for example, be configured to output responses to guide the training of a target machine learning model (also referred to as a student model).

[0062] In some embodiments, the method 100 may provide the plurality of input data objects to a single reference model for processing. The reference model is configured to generate an output in response to each of the plurality of input data objects. In some embodiments, a plurality of generated outputs may define a generated output set or a generated response set. The generated output set comprises outputs from the reference model corresponding to each of the plurality of data objects provided to the reference model. For example, if a first data object is provided to the reference model, the reference model will generate an output in response to the first data object, and this output will be aggregated into the generated output set. The generated output and / or the generated outputs in the generated output set may comprise information such as one or more labels, annotations or categorisations corresponding to one or more features of the input data objects. For example, the generated output set may comprise information that the reference model has extracted from the input data objects. For example, the generated output may comprise an address that the reference model has extracted from a document. In one example, the generated output set may comprise a plurality of strings, where each string isan output response from the reference model that corresponds to an input data object. In another example, the generated output set may comprise a plurality of objects in JSON format, where each object in JSON format is an output response from the reference model that corresponds to an input data object.

[0063] At 106, a labelled set of training data is created using the generated outputs for each of the plurality of input data objects from the reference model. In some embodiments, the labelled set of training data is created using the generated output set. In some embodiments, creating the labelled set of training data may include filtering the generated output set to provide a reduced output set. For example, the generated output set may be filtered to remove lower probability responses. In another example, the generated output set may be filtered based on a ground truth dataset, such as to remove outputs which do not substantially align with a ground truth dataset.

[0064] At 108, the labelled set of training data created at 106 is used to train a target model. The target model may also be referred to as a student model. In some embodiments, the target model is a small machine learning model, for example, a small language model. In some embodiments, the target model may be suitable to be configured to perform a specific task. That is, the target model may be a suitable model to be trained for a specific function or environment, such as a domain specific model. By training the target model using the labelled set of training data created from the outputs of the reference model, the target model is able to quickly identify the knowledge and patterns learned by the reference model. Since the target model is being trained for a specific task and is computationally smaller than the reference model, the target model may have improved performance over the reference model for a particular task or function, with less data required for training and less computational resources required than the reference model. This technique has the advantage of increasing the speed of the training process for small machine learning models by utilising a synthetically created labelled data set as training data.

[0065] Once the target model has been trained, at 110 the trained target model is applied to a second unlabelled set of data. The second unlabelled set of data comprises a plurality of unlabelled data objects. In some embodiments, the second unlabelled set of data may be different to the first unlabelled set of data from which the plurality of input data objects originated. In some embodiments, the second unlabelled set of data may be determined, forexample, it may be extracted, obtained, accessed or received. In some embodiments, determining the second unlabelled set of data may comprise sampling the first unlabelled set of data without replacement. The second unlabelled set of data is used to test the function of the trained target model. The trained target model is configured to generate a target response for each unlabelled data object in the second unlabelled set of data. That is, the target model takes in a plurality of unlabelled data objects from the second unlabelled set of data and generates an output in response to each unlabelled data object. For example, if the target model is configured for the specific task of address extraction from documents, then the target model will, in response to receiving a plurality of unlabelled documents, generate an output corresponding to identifying and extracting an address within the unlabelled data object. That is, the target model will perform its trained functionality on the unlabelled data object. In some embodiments, the target responses output by the target model may define a target response set.

[0066] After the target model has generated a target response for each unlabelled data object in the second unlabelled set, the target responses are then evaluated to identify unlabelled data objects where the target model provided a low probability of confidence in the target response associated with the unlabelled data object. At 112, the method 100 determines a plurality of training data objects from the plurality of unlabelled data objects. Training data objects are unlabelled data objects from the second unlabelled set of data which may be suitable for training the target model. The training data object may be determined based on a confidence metric associated with the target response of the target model corresponding to the unlabelled data object. For example, where the target model has output a target response corresponding to an unlabelled data object which has a low confidence metric, this may indicate that the target model is not confident in the target response associated with this unlabelled data object. The unlabelled data object may then be categorised as a training data object, and / or aggregated into a set of training data objects. This advantageously provides a mechanism in which weaknesses or areas of low accuracy in the target model performance can be identified and supplemented with further training. For example, the target model may consistently output target responses with a low confidence metric for unlabelled data objects in the form of invoices that have a particular format. By identifying such low-confidence unlabelled data objects as training data objects, these training data objects can be used to identify weaknesses in the target modeloperations, and may be used to provide further related training to the target model on that particular format of invoice.

[0067] The one or more training data objects identified at 112 are then input to the reference model at 114. The reference model generates an additional output, or a second output, for each of the plurality of training data objects provided to the reference model. In some embodiments, the plurality of additional outputs corresponding to each of the training data objects may define an additional output set. At 116, the labelled set of training data originally produced by the reference model at 106 is updated by adding or combining the additional outputs with the labelled set of training data to define an updated labelled set of training data. In some embodiments, combining the additional output set with the labelled set of training data may include pre-processing or filtering the additional outputs before combining the additional outputs with the labelled set of training data.

[0068] In some embodiments, the method may be iteratively repeated such that the target model is trained with the updated training data set each time it is updated until the target model reaches convergence. In some embodiments, the method may be iteratively repeated by training the target model with the updated labelled set of training data, testing the trained target model with a second unlabelled dataset, and then updating the labelled set of training data based on the results of testing the trained target model.

[0069] Figure l is a process flow diagram of a method 200 for training a target model until the target model reaches convergence, according to some embodiments. In some embodiments, method 200 may include repeating one or more steps of method 100. In some embodiments, method 200 may form part of method 100, for example, method 200 may form part of 108 to 116 of method 100. At 202, the target model is trained using a labelled set of training data. This labelled set of training data may be the labelled set of training data created in step 106 of method 100. At 204, the trained target model is applied to an unlabelled set of data. Applying the trained target model to an unlabelled set of data may include inputting or feeding the unlabelled set of data into the trained target model, such that the trained target model processes the data to generate target responses or outputs in accordance with the target model’s trained functionality. The trained target model generates output predictions, or target responses, for each unlabelled data object in the unlabelled set of data. These target responses or outputs may define a target response set.At 206, the target responses may be assessed to determine whether the trained target model requires further training.

[0070] To determine whether the trained target model requires further training, the method may comprise determining if the model has reached convergence. Convergence may be determined by monitoring one or more of the trained target model's performance metrics, such as loss function values, accuracy, or other relevant indicators, over successive training iterations. When these metrics stabilise and show minimal improvement across multiple iterations, it may indicate that the trained target model has reached its optimal performance given the current training data and parameters. This stabilisation suggests that further training may not yield significant improvements, thereby indicating convergence. For example, the model may be determined to reach convergence when a loss function of the target model reaches a steady state, a predetermined number of iterations has been performed, and / or changes in performance of the target model are determined to be below a predetermined threshold (for example, changes in the performance become negligible).

[0071] In some embodiments, the method may further comprise implementing validation techniques to confirm convergence. For example, in some embodiments the method 200 may comprise using a separate validation dataset to evaluate the trained target model's performance. The separate validation dataset may be the same as the unlabelled set of data, or it may be a different dataset. The separate validation dataset may be entirely independent of the training data. If the performance metrics that result from applying the trained target model to the validation dataset also stabilise and exhibit minimal variance, it may indicate that the trained target model has reached convergence.

[0072] In some embodiments, convergence may be achieved when the target responses indicate that further adjustments to the parameters of the trained target model result in negligible improvements in performance, or until a predetermined number of iterations have been performed. In some embodiments, convergence may be achieved when one or more target responses are above a predefined and / or configurable threshold, such as a threshold related to the confidence metric. In some embodiments, determining whether the parameters of the trained target model have reached convergence may comprise comparing one or more target responses from the current iteration with one or more historic targetresponses, for example, to determine whether the trained target model has improved confidence metrics in respect of target responses. In some embodiments, one or more historic target responses may comprise one or more target responses from a previous training iteration.

[0073] If the trained target model is determined to require no further training, then the method 200 finishes training the model at 214. If it is determined that the trained target model requires further training, then the method proceeds to 208.

[0074] At 208, a plurality of training data objects from the unlabelled set of data are determined. The training data objects may be determined by evaluating a confidence metric associated with the target response corresponding to an unlabelled data object. If the confidence metric is below a predetermined threshold, it may indicate that the trained target model has identified that there is a low probability of the target response being accurate. As such, training data objects may be determined by identifying unlabelled data objects which correspond to target responses having a low confidence metric. In some embodiments, the plurality of training data objects may be determined by taking a predefined number of data objects which have the lowest log -probabilities from the second unlabelled set of data.

[0075] At 210, the plurality of training data objects are provided or input to the reference model. The reference model processes the training data objects, and outputs a response for each of the training data objects to define an additional output relating to each of the training data objects. In some embodiments, the additional outputs may define an additional output set. At 212, the labelled set of training data is updated, for example, by combining the additional output set with the original labelled set of training data. Once the labelled set of training data has been updated, the method 200 reverts to 202, where the target model is trained using the updated labelled set of training data. The method 200 then repeats until the trained target model is determined to require no further training, and the method 200 terminates at 214.

[0076] In some embodiments, the methods 100 and / or 200 may provide the plurality of input data objects to a plurality of reference models. The plurality of reference models may include at least one large language model. The plurality of reference may be the samemachine learning models and / or different machine learning models. The plurality of reference models may comprise an ensemble of reference models. In some embodiments, the plurality of reference models may be in communication with each other, or may be isolated from one another. In some embodiments, the plurality of reference models may be configured to operate in combination with one another. In some embodiments, the plurality of reference models may be configured to function simultaneously, sequentially and / or in parallel.

[0077] In some embodiments, the plurality of reference models may be structured such that they have a combined and / or aggregated output. In some embodiments, the plurality of reference models may be structured such that they each operate individually, with each reference model outputting an individual output in response to a received input data object. The individual outputs of each reference model corresponding to the same received input data object may then be processed or filtered to provide a final output.

[0078] For example, where a plurality of input data objects are provided to a plurality of reference models, the reference models are each configured to receive the plurality of input data objects. The plurality of reference models are each configured to generate an individual output for each data object of the plurality of input data objects.

[0079] In some embodiments, the individual outputs or response generated by the plurality of reference models in response to an input data object may be filtered to obtain a single output or a final output. For example, where the individual outputs generated by the plurality of reference models are similar, but not exactly the same, the results may be processed by a filtering module to determine the form and / or content of the final output. In some embodiments, the final output may be a combination, aggregation, or selection of all or part of the individual generated responses from the reference models. In some embodiments, the final output may be determined by majority agreement. For example, the generated outputs of the reference models may be filtered for majority agreement. Majority agreement may be used to eliminate generated responses which do not appear to align with other common generated responses. For example, a first response generated by a first reference model and second response generated by a second reference model may be the same, whereas a third response generated by a third reference model may include major or minor differences to the first response and the second response. In this case, the finaloutput may be determined to be that of the first response and the second response, and the third response may be disregarded.

[0080] Figure 3A is a schematic diagram of a first method 300 for creating a labelled set of training data using a plurality of reference models, according to some embodiments. In the method 300, an input data object in the form of an invoice document 302 from a plurality of input data objects (not shown) may be input into reference models 304 and 306. Reference models 304 and 306 may be different large language models, each configured to take in the invoice document 302 and identify and extract any addresses from the invoice document 302. Reference model 304 processes the input data object 302 and outputs an extracted address 308 in the form of a string “PO BOX 8321 SPRING HILL QLD 4000”. Reference model 306 processes the invoice document and outputs an extracted address 310 in the form of a string “JOHN CITIZEN PO BOX 8321 SPRING HILL QLD 4000”. As reference models 304 and 306 are different language models, they have identified and extracted the address in different ways. Reference model 306 extracts the recipient name above the address from the invoice document 302 as part of the extracted address 310, whereas reference model 304 does not include the recipient name in the extracted address 308. Since the individual outputs 308 and 310 are different, the outputs are fed through a filtering module 312 to determine a final output.

[0081] In some embodiments, the final output may be determined by block Longest Common Subsequence (LCS) ensembling of the individual outputs. Block LCS ensembling may be used to enhance the performance of the reference models by combining each of their predictions in respect of a specific data object through a LCS algorithm. The LCS algorithm operates by comparing the sequences of predictions and identifying the longest subsequence present in all of them to ensure that the final output is based on the most agreed-upon predictions. As shown in method 300, filtering the extracted addresses 308 and 310 may comprise inputting the extracted address 308 and 310 into the filtering module 312 and dividing the extracted responses into smaller blocks. Extracted address 308 is divided into blocks 309a and 309b. Extracted address 310 is divided into blocks 311a and 31 lb. In some embodiments, the size of each block is determined based on the nature of the data and the computational resources available. For example, in the case of a large dataset, smaller blocks may be created to ensure efficientprocessing. In some embodiments, the size of each block may be determined based on the content, for example, a string may be divided into blocks where there is a return or a white space. Each block 309a, 309b, 31 la, 31 lb, contains a subset or a portion of the individual response 308, 310 generated by the reference models 304 and 306 to allow parallel processing and reduce computational load in the filtering process.

[0082] Following the division of data into blocks 309a, 309b, 311a, 311b, the LCS algorithm is applied to the corresponding blocks from the different reference models. The LCS algorithm identifies the longest subsequence common within the block from each reference model. In some embodiments, the LCS algorithm may be applied to a string by initialising a 2D matrix, or array, L where L[i] [j ] stores the length of the LCS of the substrings X[0...i-1] and Y[0... j - 1 ] . The dimensions of the matrix are (m+1) by (n+1), where m and n are the lengths of the two strings X and Y. The 2D matrix will enable the lengths of common subsequences to be stored as the algorithm progresses. Then, the 2D matrix is populated by iterating through each cell. If either string is empty (i == 0 or j == 0), then L[i] [j ] is set to zero (since the LCS of an empty string with any string is zero). If the characters of both strings match (X[i- 1 ] == Y[j -1 ]), then L[i] [j ] is set to L[i- 1 ] [j - 1 ] + 1, indicating that the LCS length increases by one. If the characters do not match (X[i-1 ] != Y[j - 1 ]), then L[i] [j ] is set to max(L[i-l][j], L[i] [j - 1 ]), where the LCS length is the maximum of the LCS lengths without the current character of either string.

[0083] The algorithm then traces back through the matrix to identify the LCS. Starting from L[m][n], the algorithm traces back to L[0][0] to construct the LCS string. If X[i-1] == Y[j-1], then the algorithm determines the character is part of the LCS, so it is appended to the LCS string. The algorithm then moves diagonally up-left (i— , j— ). If X[i-1] != Y[j -1 ], that is, if the characters do not match, then the algorithm moves in the direction of the larger value (L[i- 1 ] [j ] or L[i] [j - 1 ]). Accordingly, the longest common subsequence can be reconstructed by following the path of maximum values in the 2D matrix.

[0084] For example, in method 300, the LCS algorithm analyses the corresponding blocks 309a and 311a, and identifies that the longest common subsequence between 309a and 31 la is “PO BOX 8321” as this subsequence appears in both strings 309a and 311a. The LCS 314 is output from these blocks. The LCS algorithm analyses the corresponding blocks 309b and 31 lb, and identifies that the longest common subsequence between 309band 31 lb is “SPRING HILL QLD 4000” as this subsequence appears in both strings 309b and 31 lb, that is, the full strings are determined to be the same. The LCS 316 is output from these blocks. The LCSs 314 and 316 are then combined to provide the final output 318, or the final response, “PO BOX 8321 SPRING HILL QLD 4000”. The final output is therefore a consensus of the most consistent and reliable generated responses, or predictions, from the plurality of reference models. This final output 318 is then associated with the input data object 302, and aggregated to a labelled dataset 320.

[0085] In some embodiments, the outputs or responses generated by the plurality of reference models may be filtered using grounded extraction. Grounded extraction may comprise validating the extracted information, such as the addresses 308, 310, to ensure they are accurate and relevant. Validation may be performed by comparing the extracted information with known correct values or by using one or more validation models.

[0086] In some embodiments, each reference model may generate an individual output for each data object that may be collated or aggregated to define a generated output set. For example, where three input data objects are fed into a reference model, the reference model will generate an individual response for each data object. The three generated individual responses may then be collated into a generated output set. In another example, if three reference models are used, then each of the three models will be fed the plurality of data objects and generate individual responses to each of the data objects. As such, there may be three sets of generated responses once the three reference models have finished processing all input data objects. Each of the generated output sets will have outputs corresponding to each of the plurality of data objects. For example, if a first data object is provided to each of the reference models, the reference models will each generate an output in response to the first data object, and this output will be aggregated into the generated output set corresponding to each reference model. Accordingly, each generated output set will have an output corresponding to the first data object.

[0087] In some embodiments, creating the labelled set of training data may include aggregating, combining and / or filtering the generated outputs such that a set of final outputs is generated, with the final output set including an output corresponding to each of the plurality of data objects. The outputs in the final output set are each associated with the input data object form which the output was derived, and may include information such asa label, annotation or categorisation corresponding to one or more features of each input data object.

[0088] Figure 3B is a schematic diagram of a second method 350 for creating a labelled set of training data using a plurality of reference models, according to some embodiments. Method 350 provides an example method in which each reference model outputs responses to a generated output set, and the generated output sets are then filtered to create a final output set that may define a labelled training data set. In method 350, the plurality of input data objects 352 are input into each of the reference models 354, 356, 358 for processing. Each of the reference models 354, 356 and 358, process each of the plurality of input data objects 352 and, in response to receiving an input data object, each reference model generates an output associated with the input data object. For example, for input data object 360, each of reference models 354, 356, and 358 generate a respective output 362, 364 and 366 corresponding to the input data object 360. The respective output 362, 364, 366 may be a label, annotation, extraction or categorisation corresponding to all or part of the input data object 360. Each output is aggregated to a generated output set 368, 370, 372, wherein the generated output sets provide labels for the input data objects. For example, if input data object 360 is a PDF document of an invoice, the respective output 362 from reference model 354 may include an extracted address from the invoice. This output 362 which corresponds to the data object 360, may be considered a label or annotation relating to the data object. By processing the plurality of data objects, the reference models are effectively labelling or annotating the plurality of data objects from an unlabelled data set. Therefore, the generated output set for each reference model generates a labelled data set corresponding to that reference model.

[0089] The generated output sets 368, 370, and 372 are then filtered and / or processed to create a single set of labelled dataset 374. The generated output sets are filtered by filtering module 376, which may be configured to perform a specific filtering process or algorithm, such as majority agreement, grounded extraction or block LCS ensembling. The filtering module 376 may aggregate, combine and / or filter the generated output sets 368, 370 and 372 to a single labelled dataset 374, in which the labels in the dataset 374 represent the most agreed upon outputs from the reference models. By using the combination of reference models, each of which may have their own strengths and weaknesses foridentifying and / or extracting information from the data objects, the labelled data set is improved. This effectively creates a synthetic data set with a high level of accuracy, thereby reducing the cost and time of associated with manually annotating datasets. The labelled set of data 374 may therefore define a labelled set of training data.

[0090] Creating training data synthetically using ensembled reference models offers significant advantages, especially in addressing data scarcity issues. This approach allows for the easy generation of labelled datasets that are suitable for training smaller, domainspecific models with sufficient labelling accuracy. By focusing on specific tasks, such as address extraction, these domain-specific models can achieve high levels of accuracy by training on the synthetic training data. This is particularly beneficial when real-world training data is limited or difficult to obtain, ensuring that the models can be trained using training data sets that have an appropriate scope and depth.

[0091] Figure 4 is a schematic diagram of a data engine 400 configured to create a labelled set of training data, according to some embodiments. The data engine 400 may be configured to perform the methods 100, 200, 300 and 350 described herein. The data engine 400 includes a plurality of reference models 402 configured to receive a plurality of data objects 404 from a first unlabelled data set 406. The reference models 402 each output a generated output set to a filtering module 408. The filtering module 408 is configured to filter the generated output sets into a single final output that defines a labelled dataset 410. The labelled dataset 410 is used to train a target model 412. Once trained, the target model 412 is applied to a second unlabelled dataset 414. The second unlabelled dataset 414 may be provided within the data engine 400 or separate from the data engine 400. The second unlabelled dataset 414 is used to test the function of trained target model 412. As a result of applying the target model 412 to the second unlabelled dataset 414, the target model 412 outputs a set of target responses 416. From these target responses 416, the data engine 400 samples a predefined number of examples of the lowest log-probability responses from the target model 412 to identify data objects 418 from the unlabelled dataset corresponding to the lowest log-probability responses. The data engine 400 then extracts the data objects 418 from the unlabelled dataset corresponding to the determined lowest log-probability responses, and inputs the data objects 418 into each of the reference models 402.

[0092] The data engine 400 therefore provides a mechanism to continuously update the labelled dataset 410 suitable for training a target machine learning model. As the data engine 400 is configured to take in data from a first unlabelled dataset and create labelled dataset 410 using the reference models 402, the data engine 400 addresses issues of data scarcity in the field of machine learning, as well as reducing the cost and timely nature of preparing datasets suitable to train machine learning models. The synthetically created labelled dataset 410 can then be continuously and iteratively updated to address particular training deficiencies in the target machine learning model 412. Since the labelled dataset 410 is used to train the model 412, the output of target responses 416 from the model can then be evaluated to determine the lowest log-probability responses.

[0093] The log-probability is a logarithmic transformation of the probability assigned to a particular response output by target model 412. When the value of the log-probability is low, it may indicate that the target model 412 has assigned a low probability to that response, meaning it is not a typical or expected output based on the labelled dataset 410 (or training data). Such responses can occur due to various reasons, including insufficient training data, ambiguous input, or the presence of noise in the training data. Identifying low log-probability outputs from the target responses 416 of the target model 412 can assist with identifying areas where the model 412 may require further training or adjustment, such as parameter adjustments, to improve its operation and / or performance metrics such as reliability and accuracy.

[0094] Since the low log-probability responses are indicative of outputs that the target model 412 deems unlikely, or is less confident about, they can be used to identify areas where the target model 412 needs more training. By extracting the unlabelled data objects 418 where the model 412 did not perform as expected, and incorporating these into the labelled dataset 410 by feeding them through the reference model 402, the labelled dataset 410 includes additional training data relating to the areas where the originally trained model 412 did not perform as expected. As such, the labelled dataset 410 can be specifically and precisely updated to target specific training deficiencies, thereby making training of machine leaning models more efficient. When the model 412 is then trained using the updated labelled dataset 410, the trained target model 412 can be improved.

[0095] Figure 5 is process flow diagram of an example method 500 of training a target model to extract addresses from documents, according to some embodiments. Method 500 comprises, at 502, determining a sample of documents from an unlabelled dataset. The unlabelled dataset may include a plurality of documents of the same format or documents of different formats, such as photographs, spreadsheets or PDF files. In some embodiments, the unlabelled dataset may include documents from a particular field or industry, such as finance, medicine or manufacturing. In some embodiments, the unlabelled dataset may include documents having particular content, such as invoices, or it may include documents having a range of content, including receipts, invoices, shipping manifests, order confirmations, and the like. The sample of documents may be determined by random sampling, it may be automatically determined or may be determined by manual selection.

[0096] At 504, the sample documents are fed into a plurality of reference models 505. At the same time, or prior to 504, at 506 the plurality of reference models 505 receive a prompt 507. The prompt 507 may be a text prompt or a multimodal prompt. The prompt 507 defines the task for the reference model to perform in respect of the received documents. In some embodiments, the prompt 507 defines the extraction task for the reference models 505 to extract specific information, such as an address, from the data objects. In some embodiments, the prompt 507 may comprise information specifying the desired output format of the response from each of the reference models 505. For example, the prompt 507 may indicate that the extracted address should be output as a string. In some embodiments, the prompt 507 may indicate that the response should be output in JSON format, and in some embodiments, a specific JSON format may be defined. For example, a prompt provided to the reference models 505 may comprise:Find and extract the vendor address and customer address (shipping) if it exists on the document. Return the response in JSON format like {{‘vendor address str null, ‘customer address strfmll}}. Only respond in JSON nothing else.

[0097] The prompt 507 may additionally include one or more pieces of contextual information about the documents being fed to the plurality of reference models. For example, the prompt 507 may specify the type of documents, the format of the documents, the industry of the documents, and / or any other contextual information that enables thereference models 505 to recognise and interpret the relevant information within the context of the entire document.

[0098] At 508, the responses from the plurality of reference models 505 are output to a filter 510, configured to filter the responses from each of the reference models 505 into an individual response. An example 511 is shown in Figure 5. In this example 511, each of the reference models 505 has output a first address extracted from the document. The first addresses 512 extracted from a document are each different from one another. One reference model has extracted first address 514, which contains a company name, and an additional return between the lines “PO BOX 8321” and “SPRING HILL QLD 4000”. One reference model has extracted first address 516, which contains only lines “PO Box 8321” and “SPRINGHILL QLD 4000”, with no additional return between them. One reference model has extracted first address 518, which contains a single line “PO Box 8321 SPRING HILL QLD 4000”. Using a filtering method, such as block LCS ensembling, the three responses are compared and the filter determines the final output 520 of the first address 512 as being “PO Box 8321” on line 1 and “SPRING HILL QLD 4000” on line 2.

[0099] Similarly, each of the reference models 505 has output a second address 522 extracted from the document. The second addresses 522 extracted from the document by each model differ in format. One reference model has extracted second address 524, which contains a company name, followed by the address “79 Citizen Road”, “SPRINGHILL QLD”, and “4000 AUSTRALIA” across three lines. One reference model has extracted second address 526 which contains “79 Citizen Road”, “SPRING HILL QLD 4000”, and “AUSTRALIA” across three lines. One reference model has extracted second address 528, which contains a single line “79 Citizen Road SPRINGHILL QLD 4000 AUSTRALIA”. Using a filtering method, such as block LCS ensembling, the three responses are compared and the filter 510 determines the final output 530 of the second addresses 522 as being “79 Citizen Road” on line 1, “SPRINGHILL QLD 4000” on line 2, and “AUSTRALIA” on line 3.

[0100] The final outputs 520 and 530 are then, at 532, added to a labelled training dataset 534. In some embodiments, the labelled training data set may already be populated with labelled data, and the final outputs 520 and 530 may be added on to the labelled training data set. In some embodiments, the final outputs from the filter may be used to create thelabelled training data set, that is, the labelled dataset may be empty, until the final outputs 520 and 530 from the filtered responses are generated and used to populate the labelled dataset 534.

[0101] At 536, the labelled training dataset is used to train a target model 538, for example a Flan-T5 model. In some embodiments, training the target model 538 includes fine tuning the target model using the labelled training dataset. At 540, the target model is applied to a test dataset 542. The test dataset 542 is used to evaluate the models performance after training. The test dataset 542 may be an unlabelled dataset. By applying the target model 538 to the test dataset 542, the target model outputs a plurality of target responses. At 544, a sample 546 of low-log probability data from the unlabelled test dataset 542 is determined. The sample 546 of low-log probability data may be determined by identifying the data objects within the test dataset 542 to which the target model 538 has generated a low log-probability response. The low-log probability data from the unlabelled test dataset 542 may be determined by evaluating the log-probability of the responses from the trained target model. The sample of low log-probability data 546 is then, at 548, fed back to the plurality of reference models 505, where the reference models output responses in relation to the low log-probability data 546, pass it through filter 510 to generate a final output, and add the final outputs to the labelled dataset at 532 such that the target model 538 can then be trained at 536 using an updated labelled dataset 534, the updated labelled dataset containing data that was previously determined to be low log-probability data, thereby improving the process of fine tuning or training the target model 538.

[0102] Figure 6 is a block diagram of a system 600 configured to perform methods 100, 200, 300, 350 and 500 as described herein, according to some embodiments. System 600 may also be configured to perform some or all of the functionalities of the data engine 400 as described herein. The system 600 comprises one or more processor(s) 602 and / or memory 604. The processor(s) 602 may comprise one or more microprocessors, central processing units (CPUs), application specific instruction set processors (ASIPs), application specific integrated circuits (ASICs) or other processors capable of reading and executing instruction code.

[0103] The memory 604 may comprise one or more volatile or non-volatile memory types. For example, memory 604 may comprise one or more of random access memory(RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM) or flash memory. The memory 604 is configured to store program code accessible by the processor(s) 602. The program code comprises executable program code modules. In other words, memory is configured to store executable code modules, when executed by the processors to cause the system to perform certain functionalities. A module, as used herein, may be a collection of computer executable or computer intelligible code, that when executed by a processor, causes the processor to perform processes, action or functionalities, such as the processes, actions or functionalities of the methods described herein. In some embodiments, memory 604 stores computer executable instructions which when executed by the one or more processors 602, cause the system to perform the methods 100, 200, 300, 350 and 500 described herein.

[0104] The system 600 is configured to train a target machine learning model 610 using one or more reference models 608. Reference models 608 and target model 610 may be external to the system 600, or may be contained within the system 600 (as shown in Figure 6), such as in the memory 604. In some embodiments, reference models 608 and / or target model 610 may be form part of the data engine 606 (such as the configuration shown in Figure 4), or may be separate from, and in communication with, the data engine 606, as shown in Figure 6. The system 600 for training the target model 610 may be in communication with a data store (not shown) which is internal or external to the system 600. The data store may include one or more unlabelled sets of data comprising a plurality of data objects, such as documents. The system 600 may be configured to receive, extract or obtain a plurality of data objects 612 from the first unlabelled set of data for processing from the data store. The data engine 606 may be configured to determine a sample comprising the plurality of data objects 612 from the unlabelled data set. In some embodiments, the system 600 may be configured to receive and / or extract the plurality of data objects 612 from an external source, such as another system, for example, a document management system.

[0105] The processor 602, executing data engine 606, is configured to determine a plurality of data objects 612 from a first unlabelled set of data. The processor 602 causes the data engine 606 to feed the plurality of data objects 612 to the reference model(s) 608 for processing. The data engine 606 may also be configured to receive prompt 614 andfeed this to the reference models 608. In some embodiments, prompt 614 may be received directly from an external source to the reference models 608.

[0106] The processor 602, executing data engine 606, provides the plurality of data objects 612 to the reference models 608, causing the reference models 608 to generate a response for each of the plurality of data objects 612. The reference model 608 outputs these responses to the data engine 606. In some embodiments, the responses generated by the reference models 608 may be stored temporarily in a data store external to, or internal to, system 600.

[0107] The processor 602, executing filtering module 616, is configured to receive the responses generated by the reference models 608 and filter the results. The responses generated by reference models 608 may be output in the form of a generated output set, wherein the generated output set comprises a plurality of responses, each response corresponding to an individual data object from the plurality of data objects 612. The filtering module 616 is configured to apply one or more filtering mechanisms to take the responses generated by the reference models and combine, aggregate, or filter the responses to generate a final response, or a final output for each data object. For example, filtering module 616 may receive a set of responses from each of the reference models containing an address extracted from a document. For example, the filtering module 616 may be configured to apply block LCS ensembling to determine the most likely format of the address extracted, and output the address as the final response corresponding to a data object.

[0108] Processor 602, executing data engine 606, is configured to output the final responses from the filtering module 616 to the labelled set of data 618 so as to populate the labelled set of data using the final outputs. Data engine 606 is further configured to feed the labelled set of data 618 to target model 610 such that target model 610 is trained using the labelled dataset 618. In some embodiments, target model 610 may be fine tuned using the labelled dataset 618 as training data.

[0109] Target model 610 may be configured to access a second unlabelled dataset (not shown), the second unlabelled dataset comprising a plurality of unlabelled data objects. The second unlabelled dataset may be located external to system 600. In someembodiments, the second unlabelled dataset may form part of the first unlabelled dataset. In some embodiments, the second unlabelled dataset may form part of the data engine 606, and the target model 610 may be configured to access the second unlabelled dataset from the data engine 606. The target model 610, having been trained with the labelled dataset 618, is configured to generate a target response for each unlabelled data object in the second unlabelled dataset.

[0110] The processor 602, executing data engine 606, is configured to determine a plurality of training data objects from the second unlabelled set of data. The data engine 606 may be configured to receive the target responses generated by the target model 610, and identify the unlabelled data objects which correspond to target responses having a low log-probability. These identified unlabelled data objects are determined to be training data objects. The data engine 606 may be configured to extract, access and / or receive these training data objects from the second unlabelled dataset, and provide them to reference models 608. The reference models 608 receive the training data objects and each of the reference models 608 is configured to generate an additional response for the each of the training data objects.

[0111] Processor 602, executing filtering module 616, is configured to receive the additional responses generated by the reference models 608 and filter the results. The additional responses generated by the reference models 608 may be output in the form of an additional output set, wherein the additional output set comprises a plurality of responses, each response corresponding to an individual data object from the one or more training data objects. The filtering module 616 filters the additional responses received from the reference models 608 to generate an additional final response, or additional final output, for each training data object.

[0112] Processor 602, executing data engine 606 is configured to output the additional final responses from the filtering module 616 to the labelled set of data 618 to update the labelled dataset 618 with the additional final outputs.

[0113] While the preceding sections describe methods for training machine learning models using reference models to generate labelled datasets, the techniques disclosed herein are not limited to training target models using reference models to generate labelleddatasets. In some embodiments, the trained models may be deployed to perform data extraction from documents or other data objects through an extraction application programming interface. Accordingly, some embodiments relate to systems and methods for extracting data from data objects using either specialised target models or a general extractor model. Whilst the generation of data sets has been described herein by using a reference model, it will be appreciated that the data extracted from data objects using a general extractor model may be used to generate new labelled datasets of validated values, and may therefore by used to quickly collect validated and appropriate training data to train new target models for domain-specific extraction tasks.

[0114] Some embodiments relate to systems, methods and computer-implemented media for extracting data from a data object. Some embodiments relate to a computer-implemented method for extracting data from a data object. Figure 7 is a process flow diagram of a method 700 for extracting data from a data object, according to some embodiments. The method 700 comprises, at 710, receiving an extraction request, comprising at least one data object or at least one data object reference, and an extraction schema. In some embodiments, the extraction request may be received by an extraction application programming interface (API).

[0115] The extraction request may be transmitted to the extraction API. The extraction request may be generated by a request generation module which is in communication with the extraction API and a user interface. In some embodiments, the extraction request may be generated based on inputs received through a user interface. The extraction request may be generated based on a data object received through a user interface and / or an extraction schema received through a user interface. The inputs received through the user interface may be transmitted to the request generation module configured to generate the extraction request. The user interface may be in communication with the request generation module and / or the extraction API. In some embodiments, the method 700 may comprise generating an extraction request prior to 710.

[0116] The at least one data object may include, without limitation, a document or file of any type, format, or structure, including those capable of being processed, stored, transmitted, or rendered within a digital environment. References to the at least one data object that forms part of the extraction request should be understood to encompass suchdocuments and files, along with any associated metadata or content representations, and any other types of data objects as described herein.

[0117] The at least one data object reference may comprise an information element that enables the identification, retrieval, and / or resolution of a corresponding data object within a system or across systems. In some embodiments, the at least one data object reference may comprise a locator, for example, a URL, file path, or network address that represents the location of the data object. In some embodiments, the at least one data object reference may comprise an identifier, for example, a unique ID, tag, or search key. In some embodiments, the at least one data reference may be used to access the data object directly and / or to initiate a lookup or search operation that results in the retrieval of the data object.

[0118] In some embodiments, the user interface may be configured to receive a data object. The user interface may be configured to receive a data object by way of a user-initiated upload, a drag-and-drop interaction, a file selection dialogue, and / or via an automated process such as an API call or background synchronisation mechanism. In some embodiments, the user interface may be configured to access or retrieve a data object from an external data store or system. For example, the user interface may be configured to retrieve a data object from another system based on a user input providing a location or identifier of the data object. The user interface may act as a front-end layer where a user identifies a data object, such as a document, to be received by or uploaded to the user interface for the purpose of providing it to the extraction API. Upon receipt of the data object, the data object may be parsed, rendered, stored, and / or otherwise processed by the user interface or associated processing modules for further analysis, display, or interaction. In some embodiments, the user interface is configured to transmit the data object to an extraction API and / or a request generation module. In some embodiments, upon receipt of the data object, the data object may be stored in a data store and a data object reference generated corresponding to the data object and / or the data object’s location in the data store. In some embodiments, the extraction request is generated by determining the at least one data object and generating an extraction schema. Determining the data object may include at least one of retrieving, extracting, receiving, querying, reading, viewing, fetching, accessing, loading, selecting, scanning, and / or exporting the data objects from a data store and / or system. The at least one data object may be determined based on the atleast one data object reference, for example, by identifying and / or accessing the data object based on the data object reference.

[0119] The extraction schema may be generated based on one or more inputs received through a user interface that define the one or more target data elements. The extraction schema is a structured object that defines the extraction to be performed on the data object. The extraction schema may act as a prompt that guides the extraction API on how to interpret and extract values from the data object. In some embodiments, the extraction schema may be a structured format object suitable for transmission to, and interpretation by, the extraction API. The extraction schema may be any structured format that may be deserialised. For example, the extraction schema may be serialised to JavaScript Object Notation (JSON). In another example, the extraction schema may be serialised to YAML Ain’t Markup Language (YAML).

[0120] The extraction schema comprises one or more target data elements to be extracted from the data object. In some embodiments, the extraction schema may further comprise hierarchical structures, constraints and / or validations, extraction logic, post-processing instructions, confidence thresholds and / or other information to be provided to the extraction API.

[0121] The extraction schema may be generated manually, semi-automatically, or automatically. For example, the extraction schema may be generated based on user input, automated document analysis, or based on a predefined template. In some embodiments, the extraction schema may be generated automatically based on inputs received through a user interface. For example, in some embodiments, inputs may be received through a user interface that create one or more structured objects representing target data elements. The created structured objects may then be used to automatically generate the extraction schema that encompasses all target data elements. In some embodiments, the extraction schema may be generated by serialising the structured objects, for example, into JSON. In some embodiments, generating the extraction schema comprises receiving at least one input through a user interface that represents a target data element, creating one or more structured objects representing the target data element, and serialising the one or more structured objects. In some embodiments, the one or more structured objects are serialised into JSON.

[0122] In some embodiments, the one or more target data elements may define one or more elements to be extracted from a data object. A target data element may correspond to a “type” or a “field”. For example, where the data object is an invoice document, a target data element may correspond to a field that forms part of the invoice document, such as invoice number, due date, total amount, or customer address. As used herein, the term “target data element” may be used interchangeably with the term “field”.

[0123] In some embodiments, the target data element may comprise an element type, also referred to as a field type. The element type indicates the type of element to be extracted from the data object. In some embodiments, the element type may be a single value type, indicating that the extracted element will have a single value, for example, the target data element “due date” will have a single extracted value. In some embodiments, the element type may be a multi-instance value type, indicating that the extracted element will comprise a plurality of values, for example, the target data element “line items” will have multiple extracted values. In some embodiments, the target data element may comprise a name. The name represents the target data element to be extracted. For example, the name may be “due date”, “customer name” or “company address”. The name may comprise subcategories of the name of the target data element. For example, where the name is “addresses”, the name may further comprise sub-categories of “company address” and “customer address”.

[0124] In some embodiments, the target data element may comprise a data type. The data type defines the format of the value to be extracted. For example, the data type may include, but is not limited to, a string, number, character, date, currency, URL, email, array, table, or list. In one example, where the target data element is an address, the data type of the extracted value may be a string, and where the target data element is a due date, the data type of the extracted value may be a date.

[0125] In some embodiments, the target data element may comprise a description. The description may provide information about what the target data element represents. In some embodiments, the description may further provide additional information about the target data element, such as a preferred format of extraction, or reference to the data object on which the target data element appears. For example, a description for the target data element “company name” to be extracted from a data object in the form of an invoicedocument may comprise “the name of the company issuing the invoice”. In another example, the description corresponding to the target data element “total amount” may recite “the total amount due from the invoice”. In another example, the description corresponding to target data element “due date” may recite “the due date from the invoice in the format DD-MM-YYYY”.

[0126] The user interface may be further configured to receive configuration inputs that are used to create an extraction schema. In some embodiments, the configuration inputs define one or more target data elements. The target data elements may include, for example, fields to extract from a data object. For example, a user may define a set of fields to extract from a document. The configuration inputs received may include, for example, field names, field types, expected data formats, positional or structural indicators within the document, and / or any constraints applicable to the extraction process.

[0127] At 720 of method 700, the extraction API determines, for each target data element in the extraction schema, whether a target model configured to extract the target data element exists. For example, the extraction API may determine whether a target model configured to extract the target data element exists within an associated repository of target models configured to extract various target data elements. In some embodiments, determining whether a target model exists may comprise querying a repository of target models to determine whether a target model associated with the target data element is located in the repository. The extraction API may query a repository, library, database, and / or other data store comprising one or more target models. In some embodiments, the extraction API may query the repository, library, database and / or other data store by using a vector-based search mechanism. For example, the extraction API may use a vector search wherein an embedding of a target data element is employed as the query input. In some embodiments, the extraction API may use a vector search that further comprises a natural text description of the extraction model. For example, the vector search may be conducted against a set comprising natural language descriptions associated with one or more extraction models.

[0128] In some embodiments, the embedding may be generated using a machine learning model trained to capture semantic relationship between data elements and textual descriptions. In some embodiments, the set of extraction model descriptions may be pre-processed into embeddings to facilitate efficient similarity comparison. The extraction API may then compare the query embedding to embeddings of stored documents or model descriptions to identify the most relevant matches. In some embodiments, the query may identify relevant extraction models by computing similarity scores between the query embedding and the document embeddings.

[0129] In some embodiments, the extraction API may query a server hosting a plurality of target models to determine whether a target model configured to extract the target data element is deployed on the server. In such cases, the model artifacts or model weights may be stored in a centralised repository. The centralised repository may be, for example, an Artifactory repository or an Amazon S3 bucket, and may include associated metadata. When the extraction API is invoked, the server instances responsible for serving the models may be dynamically provisioned in a cloud environment, where they retrieve the relevant model weights from the centralised repository at startup. In some embodiments, selecting a model may comprise performing a metadata-based lookup within the repository. In some embodiments, selecting a model may comprise accessing a mapping between target data elements and model locations. Such mappings may be statically defined to simplify retrieval and reduce latency.

[0130] Responsive to determining that a target model exists for extracting the target data element, at 730, the determined target model is applied to the data object to extract one or more values associated with the target data element from the data object. In some embodiments, the extraction API may call the target model to apply the target model to the data object. In some embodiments, the extraction API may transmit the data object and the extraction schema to another module or system to be processed by the target model.Responsive to determining that a target model does not exist for extracting the target data element, at 740, a general extractor model is applied to the data object to extract one or more values associated with the target data element from the data object.

[0131] In some embodiments, the prompt received by the extractor model may be generated based on the extraction schema. In some embodiments, the schema may define the target data elements to be extracted and the desired output structure, and may be used to create a text prompt for input into the general extractor model. The extraction schema may be programmatically assembled into a prompt that instructs the general extractormodel to perform extraction of a particular target data element. For example, if the extraction schema specifies that target data elements “vendor address” and “customer address” are to be extracted and returned in a JSON format, this information may be used to construct and / or generate a text prompt such as:Find and extract the vendor address and customer address if it exists on the document. Return the response in JSON format like {‘vendor address str \ null, ‘customer address’: str \ null}. Only respond in JSON, nothing else.

[0132] The generated prompt maybe similar in format to, for example, prompt 507 which is used to define extraction tasks and output formats for reference models.

[0133] In some embodiments, in addition to defining the extraction task and output format, the prompt may also comprise document context, for example, to support multimodal extraction. The document context may comprise one or more elements of the document, such as, but not limited to, the underlying images of the document or the extracted text from the document. In some embodiments, the elements of the document that form the document context may be directly obtained from the source document or through another technique. For example, the extracted text from the document may be obtained directly from the source document such as a PDF file or Word file. In another example, the extracted text may be obtained through OCR techniques applied to a scanned images. In some embodiments, OCR techniques may be performed using either open-source engines or commercially available engines.

[0134] The general extractor model is a machine learning model. In some embodiments, the general extractor model may be a large language model (LLM). The general extractor model may be configured to perform data extraction across a wide range of document types and formats without requiring task-specific training. In such embodiments, the general extractor may receive a prompt that defines the extraction task to be performed. The prompt received by the general extractor model may be a text prompt or a multimodal prompt. The prompt defines the extraction task for the extractor model to perform in respect of the data object. In some embodiments, the prompt may define the extraction task for the general extractor model to extract target data elements, such as addresses, names, numbers, and the like, from the data objects. In some embodiments, the prompt maycomprise information specifying the desired output format of the response from the general extractor model. For example, the prompt may indicate that the extracted addresses from the data object should be output as a string. In some embodiments, the prompt may indicate that the response should be output in JSON format.

[0135] In some embodiments, the prompt received by the general extractor model may additionally include one or more pieces of contextual information about the at least one data object to which the general extractor model is applied. For example, the prompt may specify the type of data object, the format of the data object, and / or any other contextual information that enables the general extractor model to recognise and interpret the relevant information within the context of the data object.

[0136] In some embodiments, the general extractor model may be trained or configured to extract data. In some embodiments, the general extractor model may have no specific training or configurations to extract data, and may include, for example, a commercially available LLM. Whilst target models may be fine-tuned for extracting particular fields (e.g., invoice numbers or shipping addresses), the general extractor model may be configured to interpret an extraction schema and apply natural language understanding to identify and extract relevant values from unstructured or semi -structured data objects. The general extractor model may comprise one or more transformer-based architectures such as GPT, T5, or BERT. The general extractor model may be called by the extraction API to extract a target data element from a data object where no suitable target model is accessible or has been trained to extract that specific target data element. The use of the general extractor model in combination with one or more target models enables the extraction API to extract data from data objects relatively accurately and reliably where specific target models have not yet been trained. In some embodiments, the general extractor model supports the generation of annotated data for training new target models.

[0137] For example, if the target data element is an “employer name”, and the data object is a payslip document in PDF form, the extraction API will determine whether there exists a specialised target model configured and / or trained to specifically extract an “employer name” data element from the payslip document. The specialised target model may have been trained using all or part of any of the methods described herein, for example, methods 100, 200, 300, 350 and 500.

[0138] At 750, an extraction payload is generated. The extraction payload comprises the one or more extracted values associated with the target data element. In some embodiments, the extracted payload may comprise an extraction identifier associated with each extracted value in the extraction payload. In some embodiments, the extraction identifier is an identifier assigned to each extracted value of the extraction payload that represents the unique instance of the extraction performed by the target model and / or the general extractor model. In some embodiments, the extraction identifier may represent the model, such as the specific target model and / or the general extractor model, that performed the extraction. As a result, the extracted values extracted by the general extractor model for a particular target data element may share the same extraction identifier. The extraction identifier can therefore be used to distinguish between values extracted by a target model and values extracted by the general extractor model. In some embodiments, the extraction identifier may also link the extracted values within the extraction payload with the data object from which the extracted values were extracted, and / or the target data element.

[0139] In some embodiments, a full set of extraction-related data including the extraction identifier, data object, target data element, and extracted values may be saved in a database or data store by the extraction API. That is, the data object, target data element and extracted values may be stored alongside an extraction identifier in a database or data store by the extraction API. In some embodiments, the data object may comprise the underlying raw input document, such as images, PDFs, or other file types. In this case, when the data object is a file, it may be stored in an object datastore, with a key matching the extraction identifier. Including the extraction identifier into the extraction payload enables extraction values to be referenced in downstream processes, for example, by referencing the extraction identifier as a key to retrieve the associated extraction value. When a feedback input, or a validation input, is later received, such inputs may include the extraction identifier, which can be used to correlate the feedback input with the original extraction values.

[0140] At 760, the extraction API outputs the extraction payload. In some embodiments, the extraction API outputs the extraction payload to a user interface. The extraction API may output the extraction payload to a client application. In some embodiments, the userinterface may form part of the client application, and may be configured to display the extraction payload received by the client application.

[0141] Figure 8 is a flow diagram of an extraction API 800 configured to extract data from a data object, according to some embodiments. In some embodiments, extraction API 800 may be configured to perform all or part of method 700 described herein. The extraction API 800 is in communication with a user interface 802, which may be part of a client application 804. The extraction API is further in communication with a repository of target models 810 and a general extractor model 814. The repository of models 810 and 814 may be part of a server system 815. The extraction API 800 is configured to receive an extraction request 806 from user interface 802. In response to receiving the extraction request 806, the extraction API 800 determines whether there is a specific target model trained to extract any of the target data elements in the request 806 that exists within the repository of models 810. If there exists a trained target model from the repository of one or more trained target models 810 to extract the target data element, the extraction API 800 calls that target model 808 (which is configured to extract the target data element) to extract the values 812 of the target data element from the data object. If there does not exist a trained target model to extract the target data element, that is, if it is a new target data element, the extraction API 800 calls a general extractor model 814 to extract the values 813 of the target data element from the data object.

[0142] The extraction API 800 then generates an extraction payload 816 based on the received values 812 and / or 813, and outputs the extraction payload 816 to the user interface 802 such that the extracted values 812 and 813 can be displayed on the user interface 802.

[0143] In some embodiments, an extraction request 806 may be generated based on the inputs to the user interface 802 and the extraction API 800. For example, a user may upload a document (such as a W2 form or payslip) into the user interface 802. The user interface 802 may act as a visual configuration tool to generate the extraction request 806. Within the user interface 802, the user defines a set of fields to extract — each specified by a name, data type (e.g., string, currency), element type or kind (single or multi-instance), and a description. Each field, or target data element, of the set of fields defines the extraction schema, or extraction configuration that identifies the data elements to beextracted from the document. When the extraction is triggered, the user interface 802 packages the uploaded document and the extraction schema into an extraction request 806 and sends it to the extraction API endpoint 800. The extraction API 800 then processes the document using a general extractor model 814 or a target model 808, parses the content, and returns an extraction payload 816 that includes structured JSON formatted data that maps the defined fields to their extracted values. The user interface 802 receives the extraction payload 816 and renders it in a readable format for the user to review on the user interface 802. In some embodiments, the user interface 802 may also generate a client URL (cURL) command that replicates the request 806 programmatically. In some embodiments, the user interface 802 may support the generation of code snippets in one or more programming languages that replicate the same command logic. For example, code snippets may be generated in programming languages such as, but not limited to, Python, JavaScript, Java, C# and / or Ruby. In some embodiments, the user interface 802 may display a tabbed layout allowing users to toggle between commands in one or more programming languages. For example, a tabbed layout may allow user to toggle between a cURL command, and code snippets in one or more programming languages, such as Python requests, JavaScript fetch, or Java HttpClient examples.

[0144] Figure 9A is a first configuration of an example user interface 900, according to some embodiments. The user interface 900 includes a configuration panel 902 in an expanded position, configured to receive configuration inputs that define an extraction schema. The user interface 900 further includes an extraction panel 903 in a collapsed position. The user interface 900 further includes a display 904, configured to display a received data object. In this case, the data object 906 is a payslip document called “jane a 2014 \\ 2.pdf The payslip document 906 is shown on the display 904. The configuration panel 902 comprises a plurality of field blocks, each field block 908 is configured to define a target data element. The field block 908 includes a field type selector 910, a field name input 912, a data type selector 914, and a field description input 916. The field type selector 910 specifies whether the field is a single value field or a multi-instance field. The single value field is used to extract a single, discrete data point from a data object, such as a name, date, or number. For example, a single value field may be an employer name or a social security number, as shown in Figure 8. The field type selector 910 may comprise a dropdown control or toggle control.

[0145] The field name input 912 is configured to assign a unique identifier to the field or target data element, for example, “employ er name” or “box l wages”. The field name input may be a text input box. The data type selector 914 defines the expected data type for the target data element to be extracted. For example, the data type selector may define that the extracted values related to the field will be a string (e.g. names, descriptions) or a date (e.g. due date or paid date). The data type selector 914 may be in the form of a dropdown control or one or more radio buttons. The field description input 916 is configured to receive a description of what the associated field represents. For example, the field description input may provide the description “the name of the company issuing the payslip” that corresponds to the associated field “employer name”.

[0146] In some embodiments, the configuration panel 902 may further comprise a button 917 configured to add a new field block to the configuration panel 902.

[0147] In some embodiments, the user interface 900 may further comprise an extraction trigger 918. The extraction trigger 918 may be used to trigger the creation of an extraction schema from the configuration inputs received through the field blocks 908 of configuration panel 902. The extraction trigger 918 may further be used to trigger the creation of an extraction request for the extraction API using the configuration inputs received through the field blocks 908 of configuration panel 902 and data object 906. In some embodiments, the extraction trigger 918 may call the request generation module to generate an extraction request. In some embodiments, the extraction trigger 918 may trigger transmittal of the data object 906 and the extraction schema created from the configuration inputs of the configuration panel 902 to the extraction API.

[0148] Figure 9B is a second configuration of the example user interface 900 displaying extracted values, according to some embodiments. The user interface 900 is shown with configuration panel 902 in a collapsed position, and the extraction panel 903 In an expanded position. The extraction panel 903 is configured to display extracted values 919a, 919b, 919c, 919d resulting from an extraction process performed on the data object 906. The extracted values originate from the defined extraction schema created from the configuration inputs of field blocks 908 shown in Figure 9A. The display panel 904 shows the data object 906, with bounding boxes 920a, 920b, 920c, and 920d representing corresponding extracted values 919a, 919b, 919c, and 919d respectively.

[0149] The extraction panel 903 displays the results of the executed extraction process. Each extracted value 919a, 919b, 919c, and 919d is shown alongside its corresponding field name value, “employ er_name” 921, “social_security_number” 922, “box_l_wages” 923, and “box_2_federal_income_tax” 924. The extracted values are displayed in the extraction panel 903 in a structured format, allowing visual verification of the output of the extraction process. In some embodiments, the bounding boxes may be colour coded, and each indicator 925 shown alongside the extracted values may substantially align with the colour of each bounding box displayed in the display panel 904 on the data object 906.

[0150] Figure 10 is a first configuration of a second example user interface 1000, according to some embodiments. The user interface 1000 includes a configuration panel 1002 and a display panel 1004. The configuration panel 1002 is configured to receive configuration inputs that define an extraction schema through a set of field blocks. The display panel 1004 presents a data object 1006, in this case, a payslip document titled “Waterloo payslip demo. pdf The payslip document 1006 is shown on the display 1004.

[0151] The configuration panel 1002 comprises two field blocks 1008 and 1009, each field block 1008 and 1009 is configured to define a target data element. The field block 1008 includes a field type selector 1010, a field name input 1012, a data type selector 1014, and a field description input 1016. The field type selector 1010 of field block 1008 indicates that the field block 1008 is a single value field. The field block 1009 includes a field type selector 1011, a field name input 1013, and a field description input 1017. The field type selector 1011 of field block 1009 indicates that the field block 1009 is a multiinstance value field.

[0152] The multi-instance value field is used to extract multiple data points from a data object, such as line items, tables, lists, or other data structures. The multi -instance field may comprise a parent field that contains multiple subfields. In some embodiments, each of the multiple subfields may be defined as a single value field. In some embodiments, the multi-instance field may return an array of data points. As shown in Figure 10, field block 1009 is a multi-instance value type. As such, field block 1009 includes two values to be extracted as part of the “earnings” field: a first value defined by a sub-field name input 1013a for “description”, with a sub-data type selector 1014a as “string” and a subdescription box 1017a, and a second value defined by a sub-field name input 1013b for“amount”, with a sub-data type selector 1014b as “currency” and a sub-description box 1017b. Accordingly, the multi-instance field “earnings” will return one or more extracted values with two components: a “description” value and a corresponding “amount” value.

[0153] Figure 11 is a second configuration of the example user interface 1000 displaying extracted values, according to some embodiments. The user interface 1000 is shown with configuration panel 1002 in a collapsed position, and the extraction panel 1003 in an expanded position. The extraction panel 1003 is configured to display extracted values 1019a, 1019b, resulting from an extraction process performed on the data object 1006. The extracted values originate from the defined extraction schema created from the configuration inputs of field blocks 1008 and 1009 shown in Figure 10. The display panel 1004 shows the data object 1006, with bounding boxes 1020a and 1020b, representing corresponding extracted values 1019a and 1019b respectively.

[0154] The extraction panel 1003 displays the results of the executed extraction process. Each extracted value 1019a and 1019b is shown alongside its corresponding field name value, “employ er_name” 1021, and “deductions” 1022. As deductions is a multi -instance field, the extracted values 1019b are shown under each of the corresponding sub-field name values “description” 1023 and “amount” 1024. The extracted values are displayed in the extraction panel 1003 in a structured format, allowing visual verification of the output of the extraction process. In some embodiments, the bounding boxes may be colour coded, and each indicator 1026 shown alongside the extracted values may substantially align with the colour of each bounding box displayed in the display panel 1004 on the data object 1006.

[0155] Figure 12 is a configuration of a third example user interface 1200 showing an extraction schema and extraction request, according to some embodiments. The user interface 1200 comprises a request element 1202 displaying a code snippet of a generated extraction schema 1204 in JSON format and a code snippet of a generated extraction request 1206 comprising the extraction schema 1204, an endpoint call 1208 and a data object location 1210. The extraction schema 1204 may be generated based on inputs to a configuration panel. The request element 1202 may be displayed and / or may generate the extraction schema 1204 or extraction request 1206 in response to interaction or engagement with a trigger element on the user interface 1200, for example, a “RunExtraction” button. In some embodiments, the request element 1202 may be overlaid on other elements of the user interface 1200.

[0156] In some embodiments, the extraction request 1206 is an HTTP POST request. In some embodiments, the POST request comprises an extraction schema, an endpoint call to an extraction API and a location of at least one data object. The extraction request 1206 shown in Figure 12 provides an example client URL (CURL or cURL) command to the extraction API endpoint, that may be used to perform the extraction programmatically. In some embodiments, the extraction request is generated by a request generation module (not shown) that forms part of, or is in communication with, the user interface 1200.

[0157] The extraction schema 1204 defines the structure and intent of the data extraction process from a document, such as a payslip or invoice. Extraction schema 1204 comprises an array of fields, or target data elements, with each element representing a specific data field to be extracted. Each target data element includes an element type, name, data type and description. As shown in Figure 12, target data element 1212 is a “single_value” element type indicating that it will extract one specific piece of information. Target data element 1212 is defined with “name”: “employer_name”, “type”: “string”, and a description such as “The name of the company issuing the payslip”. This target data element 1212 of the extraction request 1206 guides the extraction API to look for and extract a single string value in the document that matches this description.

[0158] Target data element 1214 is a “multi instance” element type, indicating that it will extract multiple entries of a similar type. Target data element 1214 comprises its own nested array of sub-elements 1216 and 1218, each of which is a “single_value” type.Target data element 1214 defines an “earnings” field which includes subfields “Description” and “amount”, where the former is a string describing the type of earning (for example, “overtime”, “annual leave”, “base rate” and the like) and the latter is a currency value representing the amount earned. This target data element 1214 of the extraction request 1206 guides the extraction API to look for and extract multiple values in the document that match these descriptions.

[0159] By generating the extraction schema using a structured object in JSON that can comprise both single value and multi-instance types, the structured approach enables theextraction API to flexibly and accurately extract both simple and complex data patterns from documents. By structuring the extraction schema in this way, the extraction API is therefore highly adaptable for various types of documents and use cases. For example, the extraction API is able to extract values from a data object even where no specifically trained extraction model exists for extracting a particular target data element or field.

[0160] In some embodiments, the user interface, such as user interface 802, 900, 1000 or 1200, may include a feedback mechanism configured to receive modification inputs and / or validation inputs related to a specific extracted value that has been extracted from the data object. The feedback mechanism allows the receipt of an annotation input to modify or validate an extracted value to create labelled data related to the data object from which the extracted value was extracted. In some embodiments, the user interface may be configured to receive input to edit the extracted values. For example, user interface may be configured to enable users to correct, modify or validate each of the values extracted from a data object. In some embodiments, the feedback mechanism may be an editable element provided in the extraction panel of a user interface.

[0161] A validation input received through the feedback mechanism of the user interface or other part of a client application may include a request to modify, annotate, correct or validate the values extracted from the data object as result of the extraction request. In some embodiments, the validation input may comprise feedback on the values extracted from a data object. For example, the validation input may include inputs that correct an extracted value, redefine an extracted value, or validate an extracted value. In some embodiments, absence of a validation input relating to an extracted value may be considered as a validation of that extracted value.

[0162] Validation inputs may be provided manually, for example, by a user inputting a revised value into a user interface to override an extracted value. In some embodiments, validation inputs may be provided programmatically or automatically, for example, by the client application itself. In one example, the client application may be configured to automatically make one or more modifications to the extracted values, such as modifying an extracted “date” value to be in a particular format for display on the user interface, or modifying an extracted “currency” value to be represented as a particular data type.

[0163] In some embodiments, each validation input may further comprise additional information associated with the modification. For example, each validation input may comprise a comment from a user explaining the modification. In another example, each validation input may comprise a confidence score associated with the modification in circumstances where a modification input is automatically or programmatically applied to an original extracted value. In some embodiments, validation inputs may also include metadata relating to the modification or validation such as a timestamp, user identifier (or source application identifier) and / or an extracted value identifier which may identify the extracted value to which the validation input relates.

[0164] Figure 13 is a process flow diagram of a method 1300 for creating annotation data records based on a validation input, according to some embodiments. Method 1300 comprises at 1310, receiving at least one validation input associated with an extracted value of the one or more extracted values from an extraction payload. The validation input may comprise a modification input and / or a confirmation input. The modification input may define a specific modification action to be applied to the extracted value, for example, changing a value in part or in whole. The confirmation input may confirm the value of an extracted value, for example, it may establish a value as a ground truth value. At 1320, the method determines whether the received validation input comprises a modification input related to the extracted value. If it is determined that the validation input includes a modification input, at 1330, the extracted value is modified in accordance with the modification input. If it is determined that the validation input does not include a modification input, for example, the validation input includes only a validation input confirming the value of the extracted value is correct, then the method proceeds to 1340 without any modification to the extracted value.

[0165] At 1340, an annotation data record is generated that corresponds to the extracted value. The annotation data record comprises the extracted value, the data object from which the extracted value was extracted (and / or an identifier associated with the data object) and the target data element associated with the extracted value, that is, the target data element, or field, to which the extracted value belongs. The extracted value in the annotation data record may be a validated and / or modified extracted value. As a result, theannotation data record acts as a piece of labelled or annotation data corresponding to a particular data object.

[0166] In some embodiments, the annotation data record may be correlated with the original extraction values by using the extraction identifier. In some embodiments, the annotation data record may reference the extraction identifier, for example, as a foreign key. In some embodiments, the same database may be used to store both the extraction values and the annotation data records, with the annotation data records referencing the extraction identifier as a foreign key.

[0167] At 1350, the new annotation data record is output. In some embodiments, the annotation data record may be output to a data store. The data store may comprise one or more structured or unstructured repositories configured to store the label data records for subsequent retrieval. In some embodiments, the data store may comprise a database. In some embodiments, the data store may comprise, but is not limited to, an Online Analytical Processing (OLAP) database, a relational database, a NoSQL database, object storage, a file system, an in-memory data store and / or a graph database. The data store may comprise a plurality of data storage units configured to store data in various formats, including but not limited to structured, semi-structured, and unstructured data.

[0168] In some embodiments, the data store is an OLAP database configured to store annotation data records as training data for training machine learning models. In some embodiments, the annotation data records may be grouped within the database. For example, the annotation data records having the same target data element value may be grouped within the database, such that the number of annotation data records relating to a particular target data element may be determined.

[0169] In some embodiments, annotation data records may be generated for any extracted value. In some embodiments, annotation data records may only be generated for extracted values which have been extracted as a result of the extraction API 800 calling the general extractor model 814.

[0170] Figure 14 is a flow diagram of the extraction API 800 configured to create annotation data records, according to some embodiments. As shown in Figure 14, a user1402 may view extraction payload 816 via user interface 802. The user 1402 may, in response to viewing a representation 1403 of the extraction payload 816, provide a validation input 1404 to user interface 802 relating to an extracted value of extraction payload 816. For example, the user 1402 may identify that an extracted value recites “Corn pany Pty Ltd” when it should instead recite “Company Pty Ltd”. Accordingly, the validation input 1404 may comprise a modification request to modify the extracted value “Corn pany Pty Ltd” to “Company Pty Ltd”. In another example, user 1402 may wish to confirm that an extracted value has been correctly extracted. Accordingly, the validation input 1404 may comprise a confirmation input that validates the extracted value. Upon receiving the validation input 1404, the user interface 802 may then transmit the validation input 1404 to extraction API 800. Extraction API 800 is configured to process the validation input 1404 and generate a new annotation data record 1408 based on the received validation input 1404. The extraction API 800 then outputs the annotation data record 1408 to a data store 1410.

[0171] The extraction API 800 shown in Figures 8 and 14 may additionally be used to create annotated data records for the purposes of building up a training data set. Target models tailored to a specific extraction task may then be trained using the annotated data records generated by the extraction API 800.

[0172] The process of collecting validated annotations of confirmed extracted values relating to a particular document allows users of the API to “fire-and-forgef ’ feedback events. That is, the extraction API 800 can create these individual annotation data records based on individual instances of user feedback, or received validation inputs, such that, over time, a sufficient number of annotation data records for a particular target data element can be built up to be used for training a new target model to extract a particular type of target data element. In practice, this means that after a user reviews the extracted data and makes corrections or confirmation — such as fixing misidentified fields or adjusting values — they can submit this corrected information back to the system without needing to engage in a complex feedback loop. The process is lightweight and user-friendly, allowing the collection of annotation data records, or labelled data, as the extraction API 800 is used.

[0173] The annotation data records may be used as labelled data, which may be used to train or fine-tune a more specialised target model for extraction. The training may be performed in accordance with any of the methods described herein. The extraction API 800 initially relies on a general extractor model to interpret and extract data from documents where there is no specialised model. By building up a database of training data in the form of annotation data records, the feedback mechanism effectively helps to bridge the gap in accuracy of extractions between the general extractor model and a specialised model by capturing validated annotations relating to a data object and using them to build and / or train more accurate, domain-specific models. As these specialised models are trained, they are then able to be added to the library or database of target models that can be called by the extraction API.

[0174] Accordingly, the extraction API 800 becomes increasingly reliable and efficient as target models are trained and able to be used by the extraction API to respond to an extraction request.

[0175] Figure 15 is a process flow diagram of a method 1500 for training a new target model to be used by an extraction API, according to some embodiments. Method 1500 includes, at 1510, querying a data store to determine the number of annotation data records associated with a target data element. The data store comprises one or more annotation data records, or labelled data, relating to validated extracted values from data objects. The data store may be, for example, an OLAP database where annotation data records are stored once they have been output from an extraction API as the result of a validation input. The data store may be queried in response to a request to identify a target data element to be extracted from a data object. For example, a target element may be defined as part of an extraction request received by an extraction API. Upon determining that there does not exist a specialised target model to extract that particular target data element, the extraction API may, in addition to calling the general extractor model to extract the target element, send a request to trigger a query to the data store regarding the number of annotation data records associated with a target data element. In some embodiments, querying the data store may be manually done, for example, by a user to determine whether there exists a sufficient number of annotation data records related to a particular target data element. In some embodiments, querying the data store may be performed by anexternal system, such as system 600 described herein. In some embodiments, the extraction API may send a request to system 600 to query the data store. In some embodiments, the query may be performed by a universal training API.

[0176] In some embodiments, the query may determine whether the number of annotation data records associated with a target data element is over a predefined threshold. In some embodiments, 1510 may comprise querying the data store to determine a quantity of annotation data records for a target data element. In some embodiments, 1510 may comprise querying the data store to determine whether a threshold quantity of annotation data records for a target data element has been met. In response to determining the threshold quantity has been met, a labelled set of training data is generated using the annotation data records associated with the target data element.

[0177] If the number of annotation data records is over a predefined threshold, at 1520 of method 1500, a labelled set of training data is generated using the annotation data records associated with the target data element. The predefined threshold may be a defined number of data records associated with a target data element. For example, the predefined threshold may specify the minimum number of data records, that is, the minimum number of data points, required to adequately or sufficiently train a specialised target model for extracting the target data element. In some embodiments, the predefined threshold of data records may be specified per target data element. In some embodiments, the predefined threshold may be the same or different for different target data elements. For example, a multi-instance target data element may have a higher predefined threshold of data records for training than a single value target data element, as the multi-instance target data element extraction may be more complex. In some embodiments, the data store may be monitored in real time, for example, by a universal training API, such that once the number of data records associated with a target data element reach the predefined threshold, the universal training API is triggered to create a set of labelled training data from the data records.

[0178] The labelled set of training data generated at step 1520 may involve extracting the relevant annotation data records and organising them for model training. The method 1500 may include, between 1510 and 1520, and optional process of extracting the data records associated with the target data element and storing the records in a separate data store or apartition of the data store. In some embodiments, 1520 may comprise grouping the data records for efficient access and / or or making the data records available to downstream systems, such as system 600. Accordingly, the resulting labelled set of training data may be used to train a specialised target model for extracting the specified data element.

[0179] At 1530, a target model is trained using the labelled set of training data generated at 1520. Once trained at 1530, the new target model is configured to extract values corresponding to the target data element associated with the annotation data records. 1530 may comprise applying machine learning algorithms to the generated labelled data set in order to fine tune the parameters of the target model. In some embodiments, 1530 may comprise all or part of any of methods 100, 200, 300, 350 and 500 as described herein. As a result of using the labelled set of training data to train a target model, a domain-specific target model can be developed that may more accurately extract the specified target data element from data objects with improved precision and reliability over the general extractor model. In some embodiments, the training may be performed by a universal training API or another designated system, such as system 600. In some embodiments, the training at 1530 may comprise validation and / or tuning phases to ensure the model meets performance thresholds.

[0180] Once trained, the target model can be deployed for use by the extraction API to handle future extraction requests for that specific data element. At 1540, the trained target model is output. In some embodiments, the new target model may be output to be accessible by the extraction API. In some embodiments, the trained target model may be output for use by the extraction API, such as extraction API 800. Outputting the trained target model may comprise storing the model in a designated model repository, and / or deploying it to an environment where it can be accessed and / or called by the extraction API.

[0181] Figure 16 is a flow diagram showing how new models are trained to be used by extraction API 800, according to some embodiments. As shown in Figure 16, a training module 1600 may send a query 1602 to the data store 1410 to determine the number of annotation data records associated with a target data element. In response, the training module 1600 may receive an indication of the number of data records. In some embodiments, where the number of data records is over the predefined threshold, theannotation data records 1604 associated with the target data element are provided to the training module 1600 in response to the query 1602. The training module 1600 then generates a labelled set of training data 1606 from the annotation data records 1604 associated with the target data element. The training module 1600 then uses the labelled set of training data 1606 to train a new target model 1608. The new target model 1608, once trained with the labelled set of training data 1606, for example, by using any of the methods described herein (for example, methods 100, 200, 300, 350 and 500), the resulting trained target model 1610 is output to a repository of trained target models 810 accessible by the extraction API 800. The trained target model 1610 can therefore be called by extraction API 800 to extract values related to a specific target data model. The training module 1600 therefore completes the feedback loop, taking validated data extracted from data objects by the extraction API 800 and using it to train new models for use by the extraction API 800 to improve the accuracy and reliability of the extraction API 800.

[0182] The extraction, feedback and training methods and interfaces described herein provide a highly adaptable and scalable approach to data extraction, particularly in domains where labelled training data is scarce or unavailable. The described methods can perform schema-driven extraction across a wide variety of document types without requiring prior task-specific training by leveraging a general extractor model in combination with target models. This flexibility enables rapid deployment of extraction workflows in new domains, reducing the time and cost typically associated with manual data labelling or the development of bespoke extraction models.

[0183] An advantage of this approach is the integrated feedback mechanism, which allows users to validate or correct extracted values through a user interface. The validation inputs provided are automatically converted into annotation data records to generate labelled training data. This effectively transforms real-world usage of extraction tools into a continuous source of high-quality labelled data. The feedback loop therefore bootstraps the creation of domain-specific training datasets organically, and more efficiently, without requiring dedicated manual annotation or sourcing of appropriate annotated training data.

[0184] As the annotation data accumulates for a particular target data element over time, the methods may be used to automatically trigger the training of a new specialised target model once a predefined threshold of training data is met. Such trained target models arethen deployed to handle future extraction requests for that field, thereby improving accuracy and reducing reliance on the general extractor model. As a result, this iterative process means that the methods and interfaces become increasingly precise and efficient over time. The methods and interfaces are advantageously able to adapt to the specific constraints of each domain while maintaining a low barrier to entry for new use cases. Additionally, by collecting annotation data records to capture new target data elements, the extraction API is able to self-update to continuously reflect accurate extractions for new or evolving target data elements.

[0185] Together, the general extractor model, schema-based configuration, and feedback-driven training mechanism form a self-improving ecosystem. The methods and interfaces described herein accelerate the development of high-performing extraction models, and also provide access to intelligent document processing and extraction by enabling users to pull extraction requests to implement programmatically, as well as enable users to contribute directly to model improvement through natural interactions.

[0186] It will be appreciated by persons skilled in the art that numerous variations and / or modifications may be made to the above-described embodiments, without departing from the broad general scope of the present disclosure. The present embodiments are, therefore, to be considered in all respects as illustrative and not restrictive.

[0187] Example Clauses:

[0188] A: A method for training a machine learning model, comprising: determining a plurality of input data objects from a first unlabelled set of data; providing the plurality of input data objects to at least one reference model configured to generate a first output for each of the plurality of input data objects; creating a labelled set of training data using the generated first outputs; training a target model using the labelled set of training data; applying the trained target model to a second unlabelled set of data, the trained target model configured to generate a target response for each unlabelled data object in the second unlabelled set of data; determining a plurality of training data objects from the second unlabelled set of data, wherein the plurality of training data objects is determined based on a confidence metric associated with the target response; providing the plurality of training data objects as an input to the at least one reference model to generate a secondoutput for each of the plurality of training data objects; and updating the labelled set of training data by combining the generated second outputs with the labelled set of training data.

[0189] B: The method as clause A recites, further comprising, iteratively repeating at least part of the method until the target model reaches convergence.

[0190] C: The method as clause B recites, wherein iteratively repeating at least part of the method comprises repeating at least the steps of: training the target model using the labelled set of training data; applying the trained target model to a second unlabelled set of data; determining a plurality of training data objects from the second unlabelled set of data; and providing the plurality of training data objects to the at least one reference model; and updating the labelled set of training data; until the target model reaches convergence.

[0191] D: The method as clause B or C recites, wherein the target model is determined to converge when parameters for the target model stabilise, a loss function of the target model reaches a steady state, a predetermined number of iterations has been performed, and / or changes in performance of the target model are determined to be below a predetermined threshold.

[0192] E: The method as any one of clauses A to D recites, wherein creating a labelled set of training data using the generated first outputs includes filtering the generated first outputs.

[0193] F: The method as clause E recites, wherein filtering the generated first outputs of the at least one reference model includes filtering for majority agreement.

[0194] G: The method as clause E recites, wherein filtering the generated first outputs of the at least one reference model includes filtering using grounded extraction.

[0195] H: The method as clause E recites, wherein filtering the generated first outputs of the at least one reference model includes applying block Longest Common Subsequence (LCS) ensembling.

[0196] I: The method as any one of clauses A to H recites, further comprising filtering the generated second outputs.

[0197] J: The method as any one of clauses A to I recites, wherein determining a plurality of input data objects from a first unlabelled set of data includes determining a sample of the first unlabelled set of data.

[0198] K: The method as any one of clauses A to J recites, wherein the at least one reference model comprises a plurality of reference models.

[0199] L: The method as any one of clauses A to K recites, wherein the at least one reference model comprises at least one large language model (LLM).

[0200] M: The method as any one of clauses A to I recites, wherein the at least one reference model is provided with a prompt to process the input data objects.

[0201] N: The method as any one of clauses A to M recites, wherein training the target model includes fine-tuning the target model based on the labelled set of training data.

[0202] O: The method as any one of clauses A to N recites, further comprising determining a second unlabelled set of data.

[0203] P:The method as clause O recites, wherein determining a second unlabelled set of data comprises sampling the first unlabelled set of data without replacement.

[0204] Q: The method as any one of clauses A to P recites, wherein the confidence metric is based on a probability related to a token produced by the target model.

[0205] R: The method as any one of clauses A to Q recites, wherein the plurality of training data objects is determined by taking a predefined number of data objects from the second unlabelled set of data.

[0206] S: The method as any one of clauses A to Q recites, wherein the plurality of training data objects is determined by taking a predefined number of data objects which have the lowest log-probabilities from the second unlabelled set of data.

[0207] T: The method as any one of clauses A to S recites, wherein the target model is a small machine learning model.

[0208] U: The method as clause T recites, wherein the target model is a Flan-T5 model, GPT-2 model, LiLt model, or LayoutLM model.

[0209] V: A computer-readable storage medium storing instructions that, when executed by a computer, cause the computer to perform the method as any one of clauses A to U recites.

[0210] W: A system, including: one or more processors; and memory comprising computer executable instructions which when executed by the one or more processors, cause the system to perform the method as any one of clauses A to U recites.

[0211] X: A computer-implemented method for extracting data from a data object, the method comprising: receiving, at an extraction application programming interface (API), an extraction request comprising: at least one data object or at least one data object reference; and an extraction schema, the extraction schema comprising one or more target data elements; determining, by the extraction API, for each target data element in the extraction schema: whether a target model exists for extracting the target data element; and responsive to determining that a target model exists for extracting the target data element, applying the target model to the data object to extract one or more values associated with the target data element from the data object; responsive to determining that a target model does not exist for extracting the target data element, applying a general extractor model to the data object to extract one or more values associated with the target data element from the data object; generating an extraction payload comprising the one or more extracted values; and outputting the extraction payload.

[0212] Y: The method as clause X recites, further comprising generating the extraction request.

[0213] Z: The method as clause Y recites, wherein the extraction request is generated by: determining the at least one data object; and generating the extraction schema based on oneor more inputs received through a user interface that defines the one or more target data elements.

[0214] AA: The method as clause Z recites, wherein generating the extraction schema comprises: receiving at least one input through a user interface that represents a target data element; creating one or more structured objects representing the target data element; and serialising the one or more structured objects.

[0215] AB: The method as clause AA recites, wherein the one or more structured objects are serialised into JSON.

[0216] AC: The method as clause Z or clause AA recites, wherein the user interface is configured to display a command-line interface (CLI) command or a code snippet for execution of the extraction request.

[0217] AD: The method as any one of clauses X to AB, wherein the extraction request is a POST request.

[0218] AE: The method as any one of clauses X to AD recites, wherein determining whether a target model exists comprises querying a repository of target models to determine whether a target model associated with the target data element is located in the repository.

[0219] AF: The method as any one of clauses X to AE recites, wherein each target data element includes an element type, a name, a data type and / or a description.

[0220] AG: The method as clause AF recites, wherein the element type is a single value type or a multi-instance value type.

[0221] AH: The method as any one of clauses X to AG recites, wherein the target model is a domain-specific model.

[0222] Al: The method as any one of clauses X to AH recites, wherein the general extractor model is a large language model (LLM).

[0223] AJ: The method as any one of clauses X to Al recites, wherein outputting the extraction payload comprises providing the extraction payload for display on a user interface.

[0224] AK: The method as any one of clauses X to AJ recites, wherein the extraction payload comprises an extraction identifier associated with each of the one or more extracted values.

[0225] AL: The method as any one of claims X to AK, wherein the method further comprises: receiving at least one validation input associated with an extracted value of the one or more extracted values; and generating an annotation data record corresponding to the extracted value.

[0226] AM: The method as clause AL recites, wherein the annotation data record comprises the extracted value, the data object and the target data element associated with the extracted value.

[0227] AN: The method as clause AL or clause AM recites, further comprising outputting the annotation data record to a data store.

[0228] AO: The method as clause AN recites, wherein the data store is an Online Analytical Processing (OLAP) database.

[0229] AP: The method as any one of clauses AL to AO recites, further comprising: generating a labelled set of training data using the annotation data records associated with the target data element; and training a new target model using the labelled set of training data, the new target model configured to extract values corresponding to the target data element associated with the annotation data records.

[0230] AQ: The method as any one of claims X to AP recites , further comprising querying the data store to determine a quantity of annotation data records for a target data element.

[0231] AR: The method as any one of clauses AL to AO recites, further comprising outputting the new target model to be accessible by the extraction API.

[0232] AS: The method as any one of clauses X to AR recites, wherein the at least one data object reference is a locator or an identifier.

[0233] AT: The method as any one of clauses X to AS recites, wherein the at least one data object reference is used to access the at least one data object.

[0234] AU: A non-transitory computer-readable storage medium storing instructions which, when executed by a processing device, cause the processing device to perform the method as any one of clauses X to AT recites.

[0235] AV: A computing device comprising: the non-transitory computer-readable storage medium as clause AU recites; and a processor configured to execute the instructions stored in the non-transitory computer-readable storage medium.

Claims

CLAIMS:

1. A computer-implemented method for extracting data from a data object, the method comprising:receiving, at an extraction application programming interface (API), an extraction request comprising:at least one data object or at least one data object reference; andan extraction schema, the extraction schema comprising one or more target data elements;determining, by the extraction API, for each target data element in the extraction schema:whether a target model exists for extracting the target data element; andresponsive to determining that a target model exists for extracting the target data element, applying the target model to the data object to extract one or more values associated with the target data element from the data object;responsive to determining that a target model does not exist for extracting the target data element, applying a general extractor model to the data object to extract one or more values associated with the target data element from the data object;generating an extraction payload comprising the one or more extracted values; andoutputting the extraction payload.

2. The method of claim 1, further comprising generating the extraction request.

3. The method of claim 2, wherein the extraction request is generated by:determining the at least one data object; andgenerating the extraction schema based on one or more inputs received through a user interface that defines the one or more target data elements.

4. The method of claim 3, wherein generating the extraction schema comprises:receiving at least one input through a user interface that represents a target data element;creating one or more structured objects representing the target data element; andserialising the one or more structured objects.

5. The method of claim 4, wherein the one or more structured objects are serialised into JSON.

6. The method of claim 3 or claim 4, wherein the user interface is configured to display a command-line interface (CLI) command or a code snippet for execution of the extraction request.

7. The method of any one of claims 1 to 5, wherein the extraction request is a POST request.

8. The method of any one of claims 1 to 7, wherein determining whether a target model exists comprises querying a repository of target models to determine whether a target model associated with the target data element is located in the repository.

9. The method of any one of claims 1 to 8, wherein each target data element includes an element type, a name, a data type and / or a description.

10. The method of claim 9, wherein the element type is a single value type or a multiinstance value type.

11. The method of any one of claims 1 to 10, wherein the target model is a domain-specific model.

12. The method of any one of claims 1 to 11, wherein the general extractor model is a large language model (LLM).

13. The method of any one of claims 1 to 12, wherein outputting the extraction payload comprises providing the extraction payload for display on a user interface.

14. The method of any one of claims 1 to 13, wherein the extraction payload comprises an extraction identifier associated with each of the one or more extracted values.

15. The method of any one of claims 1 to 14, wherein the method further comprises:receiving at least one validation input associated with an extracted value of the one or more extracted values; andgenerating an annotation data record corresponding to the extracted value.

16. The method of claim 15, wherein the annotation data record comprises the extracted value, the data object and the target data element associated with the extracted value.

17. The method of claim 15 or claim 16, further comprising outputting the annotation data record to a data store.

18. The method of claim 17, wherein the data store is an Online Analytical Processing (OLAP) database.

19. The method of any one of claims 15 to 18, further comprising:generating a labelled set of training data using the annotation data records associated with the target data element; andtraining a new target model using the labelled set of training data, the new target model configured to extract values corresponding to the target data element associated with the annotation data records.

20. The method of claim 17, further comprising querying the data store to determine a quantity of annotation data records for a target data element.

21. The method of claim 19 further comprising outputting the new target model to be accessible by the extraction API.

22. The method of any one of claims 1 to 21, wherein the at least one data object reference is a locator or an identifier.

23. The method of any one of claims 1 to 22, wherein the at least one data object reference is used to access the at least one data object.

24. A non-transitory computer-readable storage medium storing instructions which, when executed by a processing device, cause the processing device to perform the method of any one of claims 1 to 23.

25. A computing device comprising:the non-transitory computer-readable storage medium of claim 24; anda processor configured to execute the instructions stored in the non-transitory computer-readable storage medium.

Images