Cost-effective tool for sequential testing

Abstract

The present invention relates to computer implemented adaptive and iterative methods to improve and accelerate a quality control decision or a performance test in a specific technical field while reducing the overall cost of the quality control or performance test process, respectively; the method comprising: constructing a sequential test; iterating the following steps using the sequential test until a decision is made, obtaining a set of samples, checking the decision of the criteria for the set of samples, determining in an adaptive way the optimal number of additional data samples required for the subsequent iterative step if no decision has been made in the current iteration.

Description

HC003P 1COST-EFFECTIVE TOOL FOR SEQUENTIAL TESTING DescriptionField of the Invention

[0001] The present invention relates to the field of quality control and product testing.Background Art

[0002] The use of methodologies whose outcome solely relies on one sample and products that are tightly linked to high stakes (e.g. human life) need to be rigorously verified. Usually, such verification is based on sequential testing and is highly cost intensive due to the large number of samples that need to be acquired and processed in view of deciding whether a method or product is reliable or not. Besides acquiring said samples a further expensive step may also be the manual annotation of the samples by experts for reasons of comparison. The verification of said methodologies and products is especially crucial if they involve e.g., medical diagnostic techniques, aerospace engineering, food safety, or the like.

[0003] The literature of sequential testing, also called statistics of accumulating data, is extensive. It started with the seminal paper of Wald, in which the base of sequential tests can be found (Wald, 1992). Examples of variations of sequential testing are group sequential testing, and adaptive sequential testing (Armitage et a / ., 1969; Bothwell et a / ., 2018; Pocock, 1977). Gould and Pecore (Gould et a / ., 1982) introduced costs for group sequential testing.

[0004] US2009306933A1 describes methods for sampling sufficiency determination (i.e., sampling sufficiency system) during auditing and testing processes by using statistical analysis and probability calculations to assess whether a population of items meets an acceptable failure rate. The methods allow early termination of testing based on confidence thresholds that correspond to the acceptable failure rates to optimize per-sample cost and time efficiency. The method disclosed in US2009306933A1 relates to testing a predetermined sized sample subset of a larger population. The subset can be e.g., a random sample set of a given size. Said method is only applicable if the entire sample set is available, namely for cases where obtaining a set of samples has no cost. The set size in said method is chosen arbitrarily.HC003P 2

[0005] However, there are no methods relating to adaptive sequential testing with sample size re-estimation where the re-estimation procedure is based on the optimization of the overall cost.

[0006] Therefore, since the costs of verification of methodologies and products are a significant portion of the market price of said methodologies and products, there is still the need for a cost-effective approach for sequential testing methods underpinned by rigorous statistics for verifying the quality of said methodologies and products.Summary of invention

[0007] It is the object of the present invention to provide computer implemented methods to improve and accelerate decisions concerning quality control and / or performance testing of methodologies and / or products in a cost-effective way. The object is solved by the subject matter of the present invention.

[0008]

[0009] According to the invention, there is provided a computer implemented adaptive and iterative method to improve and accelerate a quality control decision for a quality control measurement to be implemented in a specific technical field while reducing the overall cost of the quality control process; the method comprising the consecutive steps:(a) Constructing a sequential test:I. specifying the costs for obtaining a set of samples,II. specifying the costs for evaluating each individual sample, III. specifying the size of the samples set,IV. setting criteria for generating a pass or fail decision;(b) Executing the sequential test:I. obtaining a first set of samples of the size specified in (a)III;II. analyzing the first set of samples;III. terminating the sequential test if a pass or fail decision is generated; or(c) Continuing sequential testing if no pass or fail decision is generated;I. determining in an adaptive way the size of an additional set of samples required for the next iterative step;II. obtaining the additional set of samples;III. analyzing the additional set of samples;HC003P 3IV. terminating the sequential test if the pass or fail criteria are met; or (d) Repeating the sequential testing of (c)(I) – (IV) until a pass or fail decision is generated.

[0010] According to one embodiment of the invention, the quality control decision is taken based on standards set by an organization for standardization, or “good practice” quality guidelines and regulations.

[0011] According to another embodiment of the invention, the specific technical field is any one of medicine, pharmacy, engineering, science, computer science, specifically the technical field relates to medical diagnostic techniques, pharmaceutical production, chemical engineering, process engineering, civil engineering, electrical engineering, computer engineering, mechanical engineering, geological engineering, agricultural engineering, applied engineering, biomedical engineering, biomedical nanoengineering, biological engineering, building services engineering, energy engineering, geomatics engineering, information engineering, industrial engineering, mechatronics engineering, engineering management, military engineering, mining engineering, nanoengineering, quantum engineering, nuclear engineering, petroleum engineering, project engineering, railway engineering, software engineering, supply chain engineering, systems engineering, textile engineering, or cybersecurity engineering.

[0012] According to one embodiment of the invention, the quality control decision is taken in the field of medicine.

[0013] According to another embodiment of the invention, the medical field is medical Al, diagnostic to medical diagnostic techniques, pharmaceutical production tool, diagnostic model, and / or healthcare software.

[0014] According to one embodiment of the invention, the quality control decision is meeting the regulatory requirements or compliance, achieving its intended purpose, satisfying consumer safety, product performance, standards published by an organization for standardization, or “good practice” guidelines and regulations, or providing evidence that the product works for the intended population.

[0015] According to a further embodiment of the invention, the sample data is derived from images, labeled images, product parameters, process parameters, computer program output, measurements, designs, drug testing, training of machine learning algorithms.HC003P 4

[0016] According to one embodiment of the invention, said decision output is deployed to a mobile device or application, printer or labeling system, a live computing environment, a web dashboard, a physical dashboard display, e-mail alerts, SMS notifications, a chatbot, an API (application programming interface), an LLM (large language model), or some historical data storage.

[0017] On embodiment of the invention relates to reducing costs compared to classical methods.

[0018] According to a further embodiment of the invention there is provided a computer implemented adaptive and iterative method to improve and accelerate the quality control decision for a quality control measurement in the field of medical imaging analysis while reducing the overall cost of the imaging analysis method; the method comprising:(a) Constructing a sequential test:I. specifying the costs for obtaining a set of image samples, II. specifying the costs for evaluating each individual image sample,III. specifying the size of the image sample sets, IV. specifying criteria for generating a pass or fail decision;(b) Executing the sequential test:I. obtaining a first set of image samples of the size specified in (a)lll;II. analyzing the first set of image samples;III. terminating the sequential test if a pass or fail decision is generated; or(c) Continuing sequential testing if no pass or fail decision is generated;I. determining in an adaptive way the size of an additional set of image samples required for the next iterative step;II. obtaining an additional set of image samples;III. analyzing the additional set of image samples;IV. terminating the sequential test if the pass or fail criteria are met; or(d) Repeating the sequential testing (c)(I) – (IV) until a pass or fail decision is generated.HC003P 5

[0019] According to one embodiment of the invention, the medical imaging analysis is selected from the group consisting of radiology, sonography, ultrasound, elastography, photoacoustic imaging, tomography, X-ray, X-ray computed tomography (CT), positron emission tomography (PET), magnetic resonance imaging (MRI), PET-CT, PET-MRI, echocardiography, functional near-infrared spectroscopy (FNIR), magnetic particle imaging (MPI), scintigraphy, single-photon emission computed tomography (SPECT), Digital Subtraction Angiography (DSA), Fluoroscopy, Dual-Energy X-ray Absorptiometry (DEXA), Fluorescence Imaging, Electroencephalography (EEG), Magnetoencephalography (MEG), Optical Coherence Tomography (OCT), or electrocardiography (ECG).Brief description of drawings

[0020] Fig. 1: Flow chart providing a brief description of the inventive method. Three main processes are shown. The box titled “Construction” relates to the construction of a sequential test. Related costs, and criteria in terms of certainty are laid out. The box titled “Sequential analysis iteration” relates to the execution of the sequential test. This part of the inventive method contains iterative cycles. A set of samples is obtained, analyzed, and either a decision is made or the next set size is determined. The box titled “Decision” relates to decision taken within the execution of the sequential test. If said criteria is satisfied to make a pass or fail decision with the desired certainty, that decision is made and the procedure is terminated. The output is either pass or fail.

[0021] Fig. 2: Averaged costs with cfix= 100. Offline: the classical method, the entire sample set is tested. Prior art methods: fixed set size sequential method with set size 10. Inventive: our inventive method, in each step the set size is calculated in an adaptive way to optimize the remaining costs.

[0022] Fig. 3: Averaged costs with cfix= 2000. Offline: the classical method, the entire sample set is tested. Prior art methods: fixed set size sequential method with set size 10. Inventive: our inventive method, in each step the set size is calculated in an adaptive way to optimize the remaining costs.Description of Embodiments

[0023] The present invention provides a computer implemented adaptive and iterative method to improve and accelerate a quality control decision in a specific technical field while reducing the overall cost of the quality control process;HC003P 6the method comprising the construction of a sequential test, wherein said construction encompasses giving the cost for obtaining a set of samples, giving the cost for evaluating each individual sample, giving the sizes of sample sets, and setting criteria for generating a pass or fail decision;the method further comprising the execution of said sequential test, wherein said execution comprises the sequential steps of obtaining a first set of samples, analyzing the first set of samples, terminating the sequential test if the pass or fail decision is generated, or continuing the sequential test if no pass or fail decision is generated, wherein the continuing comprises the sequential steps of determining in an adaptive way an additional set of samples required for the next iterative step, obtaining the additional set of samples, analyzing the additional set of samples, terminating the sequential test if the pass or fail criteria are met, or continuing the sequential testing (c) (i) - (iv) until a pass or fail decision is generated.

[0024] In addition to a per-sample cost variable in such a method, the inventors introduce a set cost variable and minimize both variables at the same time to guarantee a cost-effective way to improve and accelerate decisions concerning quality control and / or performance testing of methodologies and / or products. The computer implemented method described herein is set up to work in an adaptive and iterative way. The method comprises sequential steps that are executed in an adaptive and iterative way. Costs for obtaining a set of samples, evaluating each sample, possible sizes of sample sets, and criteria for generating a pass or fail decision are provided upfront. A sequential test is performed. A first sample set is analyzed. After evaluating the first set, a pass or fail decision is generated. If a pass decision has been generated, the method ends. If a fail decision has been generated, the size of the next sample set is optimized using the information from the previous sample set. This is the adaptive aspect of the method described herein. Then, a new sample set is provided at the predefined costs and with the calculated size of the sample set. The newly provided sample set is analyzed, and a new decision is generated. If a pass decision is generated, the method ends. If a fail decision is generated, the size of the next sample set to be analyzed is optimized using the information from the previous sample set. This loop is repeated until a pass decision is generated. This is the iterative aspect of the method described herein. The overall task is to decide on the size number of the new sample set to be acquired in order to minimize the total cost.HC003P 7

[0025] As used herein, “computer-implemented method” may refer to any method executed by one or more processors, a computer system having one or more processors, a mobile device such as a smartphone (which may include one or more processors), a tablet, a laptop computer, a set-top box, a gaming console, and so forth.

[0026] As used herein, the term “iterative” means the process of repeating a sequence of operations.

[0027] As used herein, the term “adaptive” refers to the estimation of the optimal number of samples needed for an iterative process while said process is ongoing.

[0028] As used herein, a “specific technical field” may be selected from any technical field. In a specific embodiment, the specific technical field is selected from the group consisting of medicine, pharmacy, engineering, science, and computer science.In a specific embodiment the specific technical field is selected from the list comprising, but not limited to, chemical engineering, civil engineering, electrical engineering, and mechanical engineering.In a further specific embodiment the specific technical field is selected from, but not limited to, acoustics engineering, aerospace engineering, agricultural engineering, airplanes, applied engineering, aquaculture engineering, architectural engineering, artificial intelligence, automation / control systems / mechatronics / robotics, automotive engineering, autonomous robotics, bioacoustics, biochemical engineering, bioengineering, bioinformatics, bioinstrumentation, biological engineering, biological systems engineering, biomaterial, biomechanical engineering, biomechanics, biomechatronics engineering, biomedical engineering, biomedical nanoengineering, biomedical optics, biomolecular engineering, bioprocess engineering, bioresource engineering, biosignal processing, biosystems engineering, biotechnical engineering, biotechnology, boats, building services engineering, cellular engineering, clinical engineering, coal-fired power plants, coastal engineering, combat engineering, component engineering, computer engineering, computer vision, computer-aided drawing and design (CADD), construction, construction engineering, control engineering, control theory, cryptographic engineering, cybersecurity engineering, data science, diesel engine (ice) power plants, drilling engineering, drones, earthquake engineering, ecological engineering, electrical engineering, electronicHC003P 8engineering, electronic nanoengineering, electronics, energy engineering, environmental engineering, equipment and processes), fabric engineering, fire engineering, fire protection engineering, food and biological process engineering, food engineering, forest engineering, fusion energy, general, genetic engineering, geodesy, geoenvironmental and hydrogeological engineering, geological engineering, geomatics engineering, geophysical engineering, geospatial, geotechnical and rock engineering, geotechnical engineering, geothermal power plants, graphics, groundwater engineering, hardware engineering, health and safety engineering, helicopters, high voltage engineering, highway engineering, hydraulic engineering, hydroelectric power plants, image processing, industrial engineering, industrial plant engineering, information & electrical systems engineering, information engineering, information technology engineering, information theory, instrumentation and control engineering, instrumentation engineering, irrigation and drainage engineering, irrigation engineering, logistics, machine learning, machinery systems engineering, manufacturing engineering, manufacturing engineering (e.g. tools, marine engineering, marine vehicles and structures, materials nanoengineering, mechatronics engineering, medical image computing, medical imaging, medical physics, microbiological engineering, military engineering, mineral and energy resource exploration engineering, mining engineering, mobile robotics, molecular engineering, municipal or urban engineering, nanoengineering, nanotechnology, natural language processing, natural resources engineering, naval architecture, network engineering, neural engineering, nuclear engineering, nuclear fuel, nuclear reactor design, ocean engineering, oceanographic engineering, oil rigs, optical engineering, optomechanical engineering, optomechatronics engineering, petroleum engineering, pharmaceutical engineering, photonics and quantum optics, power engineering, power plant engineering, power system design, power system operations and control, power system planning, pricing, process engineering, production, production engineering, project engineering, protection and control, protein engineering, public health engineering, quantum computing, quantum cryptography, quantum engineering, quantum information systems, quantum properties of nanomaterials, radiation protection, railway engineering, rehabilitation engineering, reliability engineering, renewable energy engineering, reservoir engineering, river engineering, robotics and automation, safety engineering, sanitary engineering, semiconductor engineering, ships, signal processing, softwareHC003P 9engineering, solar engineering, solar power plants, spacecraft, sports engineering, structural engineering, supply chain engineering, survey engineering, synthetic biology, systems biology, systems engineering, telecommunications, telecommunications engineering, teletraffic engineering, textile engineering, thermal engineering, tidal power plants, tissue engineering, traffic engineering, transport engineering, vehicle engineering, water resources engineering, web engineering, wind engineering, and wind turbine power plants.

[0029] As used herein, “medical imaging analysis” refers to the manual and / or software analysis of medical images acquired with a medical product, specifically a medical imaging product. In a specific embodiment, the medical imaging is performed with a medical imaging product. In a specific embodiment the medical imaging is performed with a medical product selected from the group consisting of radiology, sonography, ultrasound, elastography, photoacoustic imaging, tomography, X-ray, X-ray computed tomography (CT), positron emission tomography (PET), magnetic resonance imaging (MRI), PET-CT, PET-MRI, echocardiography, functional near-infrared spectroscopy (FNIR), magnetic particle imaging (MPI), scintigraphy, single-photon emission computed tomography (SPECT), Digital Subtraction Angiography (DSA), Fluoroscopy, Dual-Energy X-ray Absorptiometry (DEXA), Fluorescence Imaging, Electroencephalography (EEG), Magnetoencephalography (MEG), Optical Coherence Tomography (OCT), or electrocardiography (ECG). In a specific embodiment, the analysis is performed using software tools including but not limited to detection algorithms or artificial intelligence (Al) algorithms.

[0030] As used herein, a “product” refers to manufactured and software products. Specifically, a product with a primary medical application is referred to as “medical product”. Non-limiting examples of medical products are medical artificial intelligence (Al) algorithms, diagnostic models, healthcare software, or diagnostic tools such as radiology, sonography, ultrasound, elastography, photoacoustic imaging, tomography, X-ray, X-ray computed tomography (CT), positron emission tomography (PET), magnetic resonance imaging (MRI), PET-CT, PET-MRI, echocardiography, functional near-infrared spectroscopy (FNIR), magnetic particle imaging (MPI), scintigraphy, single-photon emission computed tomography (SPECT), Digital Subtraction Angiography (DSA), Fluoroscopy, Dual-Energy X-ray Absorptiometry (DEXA), Fluorescence Imaging, Electroencephalography (EEG),HC003P 10Magnetoencephalography (MEG), Optical Coherence Tomography (OCT), electrocardiography (ECG), or laboratory equipment.

[0031] As used herein, the term “sample” generally refers to any item that is being processed with the inventive method. In one embodiment the sample is selected from the group consisting of, but not limited to, images, labeled images, product parameters, process parameters, computer program output, measurements, designs, input and / or output of machine learning algorithms, drugs, tissue or organ samples, blood, cell-free blood such as serum and plasma, platelet-poor plasma, lymph, urine, saliva, biopsy probes, water samples, and food samples. In a specific embodiment, samples are labeled images, wherein the labeled images are images annotated, specifically annotated by person trained in medicine. In a further specific embodiment, samples are CT images. In another specific embodiment, samples are subject data from a drug trial, specifically blood samples from a subject from a drug trial, more specifically annotated analysis of blood samples from subjects from a drug trial. In a further specific embodiment, the samples are the output from a medical Al algorithm, specifically from an Al classification algorithm used for medical purposes. According to a specific embodiment, the samples are training data of machine learning algorithms.

[0032] As used herein, the term “obtaining a set of samples” refers to the acquiring of a batch of samples, wherein the process of acquirement can be physical or digital.

[0033] As used herein, the term “costs” includes, but is not limited to, financial, human, and computational costs, specifically financial costs of acquiring samples, financial and / or computational costs evaluating and processing samples, personnel costs for manually evaluating and annotating samples, monetary costs, and resources required to acquire and evaluate samples. The term “overall costs”, as used herein, serves as an umbrella term for all possible costs that arise during the inventive method. “Reducing the overall costs”, as used herein, refers to cost reduction achieved with the inventive method compared to similar methods known in the art, e.g., classical methods based on a Neyman-Pearson type test, or methods is based on Wald’s sequential test (Wald, 1992).

[0034] As used herein, the term “sequential test” refers to the statistical field of sequential analysis or sequential hypothesis testing where the sample size is not fixed in advance. Specifically, samples are evaluated as they are collected, andHC003P 11further sampling is stopped in accordance with a pre-defined stopping rule as soon as significant results are observed. A decision may sometimes be reached at a much earlier stage than would be possible with more classical hypothesis testing or estimation, at consequently lower costs.

[0035] As used herein, the term “evaluating” refers to assessing a sample against one or more standards to determine its quality (e.g., quality control) or its performance. In one embodiment the samples are classified into one of at least two categories during the evaluation. The categories are selected from, but not limited to, pass, fail, true, false, true positive, true negative, false positive, false negative, likely true, likely false, working, not working, broken, functional, and unclear.

[0036] As used herein, the terms “criteria” or “statistical criteria” refers to an acceptance and a rejection number, which are used in the decision making process. Specifically, by using an acceptance and rejection number, and after determining the number of incorrectly classified faulty objects / samples, one can decide whether one needs additional samples or not.

[0037] “Constructing a sequential test” refers to the initial stage of the procedure, during which the relevant costs of obtaining and analyzing samples and the target error rates are specified. At this stage, thresholds derived from the distribution of the test statistics are determined which will be used in subsequent iteration step(s) to ensure the target criteria are met.

[0038] As used herein, the term “quality control” refers to a procedure or set of procedures intended to ensure that a manufactured product or performed methodology adheres to a defined set of quality criteria or meets the requirements of the manufacturer.

[0039] As used herein, the term “quality control decision” refers to the outcome of a quality control procedure. In a specific embodiment the quality control decision is falling into one of at least two categories, specifically the categories are selected from, but not limited to, pass, fail, true, false, true positive, true negative, false positive, false negative, likely true, likely false, working, not working, broken, functional, and unclear. In a further embodiment the quality control decision are binary classifiers.

[0040] As used herein, the term “quality control measurement” refers to the outcome of a quality control procedure.HC003P 12

[0041] As used herein, a “performance test” is a test to determine how a product or sample performs, specifically how a product or sample performs in response to specified conditions.

[0042] As used herein, the term “performance test decision” refers to the outcome of a performance test. A performance test decision may be based on regulatory requirements or compliances, standards published by an organization for standardization, or “good practice” (GxP) guidelines and regulations. A performance test decision may also be based on whether the intended purpose is achieved, consumer safety is ensured, or the specified performance criteria are met. In one embodiment the performance test decision is falling into one of at least two categories. The categories may be selected from, but are not limited to, pass, fail, true, false, true positive, true negative, false positive, false negative, likely true, likely false, working, not working, broken, functional, and unclear. In a further embodiment the quality control decision are binary classifiers.

[0043] “Organizations for standardization”, also known as standards organizations, are organizations whose function is the overseeing of technical standards. Nonlimiting examples of organizations for standardization are the International Organization for Standardization (ISO), the European Committee for Standardization (CEN), the European Committee for Electrotechnical Standardization (CENELEC), the European Telecommunications Standards Institute (ETSI), the Institute for Reference Materials and Measurements (IRMM), the Pacific Area Standards Congress (PASO), the Pan American Standards Commission (COPANT), the African Organisation for Standardisation (ARSO), the Arabic industrial development and mining organization (AIDMO), Deutsches Institut fur Normung (DIN), American National Standards Institute (ANSI), British Standards Institution (BSI), and Association franqaise de normalization (AFNOR).

[0044] “Good practice” (GxP) guidelines and regulations are well known in the art and their intended use is to give guidance on how to ensure a product is safe and meets its intended use. Non-limiting examples of GxPs are agricultural and collection practices (GACP), agricultural practice (GAP), auditing practice (GAP), automated laboratory practice (GALP), automated manufacturing practice (GAMP), business practice (GBP), cell culture practice (GCCP), clinical data management practice (GCDMP), clinical laboratory practice (GCLP), clinical practice (GCP), documentation practice (GDP, or GDocP), distribution practice (GDP), engineeringHC003P 13practice (GEP), financial practice (GFP), guidance practice (GGP), hygiene practice (GHP), hygiene practice (GHP), laboratory practice (GLP), machine learning practice (GMLP), management practice (GMP), manufacturing practice (GMP), microbiological practice (GMiP), participatory practice (GPP), pharmacovigilance practice (GPvP or even GVP), pharmacy practice (GPP), policing practice (GPP), recruitment practice (GRP), research practice (GRP), safety practice (GSP), storage practice (GSP), or tissue practice (GTP).

[0045] The term “historical data storage” generally refers to computer data storage devices that no longer receive active support or that are no longer commercially available. Non-limiting examples are floppy disks, magnetic tape, or carousel memory.

[0046] In a specific embodiment, the decision of the inventive method may be deployed to a mobile device, application, mobile application, printer, labeling system, a live computing environment, a web dashboard, a physical dashboard display, output device, e-mail alerts, SMS notifications, chatbot, API (application programming interface), LLM (large language model), or historical data storage. Examples

[0047] The Examples which follow are set forth to aid in the understanding of the invention but are not intended to and should not be construed to limit the scope of the invention in any way. The Examples do not include detailed descriptions of conventional methods. Such methods are well known to those of ordinary skill in the art.Example 1: Sequential testing methods

[0048] Samples are taken in sets which have associated costs. The costs consist of a price per sample and a fixed price for any occasion new samples are demanded. A cost-effective algorithm (meaning that it typically costs less than the conventional tests) is constructed. Said algorithm can be widely applied, e.g., in quality control, clinical designs or testing for classification algorithms.

[0049] The set of objects is divided into two distinct subsets. Applying a classification procedure is applied which classifies the objects into these two categories. Subsequently running a test is run to show whether this procedure is working with a desired efficiency.

[0050] p denotes the probability that a given object is incorrectly classified. The value of p is unknown. The aim is to test whether p exceeds a given value p'. AHC003P 14tolerance is introduced by taking the values p0and p such that p0< p' < Pi and having the following null (JT0) and alternative hypotheses (j^): / fo p < p.i.n-1 H r

[0051] For testing, samples may be taken in groups, hence a sequential test is needed. Furthermore, it is assumed that the samples have the following costs: a price for each sample (per-samp / e price and a fixed price for any occasion new samples are demanded (set price). With these assumptions the aim is to construct a sequential test with minimal overall cost, for which the probability that ℋ₀ is accepted is at most? (error of the second kind) wheneverholds, and the probability that we reject ℋ₀ is at most a (error of the first kind ) whenever ℋ₀ holds.

[0052] In this section different methods to perform the desired test are presented.Offline method

[0053] The following method relates to a classical method based on a Neyman-Pearson type test, referred to hereafter as the offline approach, because all the necessary samples are taken in a single set. This method could be optimal if the set price is significantly higher than the per-sample price. Using this method, based on a and p, one can determine the number of samples needed to make such a decision.

[0054] Let X, X1..., Xxbe independent random variables with Bernoulli distribution having parameter p, meaning thatI i.\ - ■ 11 — p. 1 i — n i

[0055] Here, X, = 1 represents the case when the i-th classification is incorrect. Letf / ,i ’ / >■ = p.i dU' l / / 1 ■ [>_

[0056] With these hypotheses the assumption is satisfied that the probability we accept ℋ₀ is < (error of the second kind) whenever H ‘1holds, and the probability that we reject ℋ₀ is < a (error of the first kind ) whenever ℋ₀ holds. The aim is to calculate the sample size n for given errors a and p. These hypotheses can be tested with the following test statisticand with critical value xasuch thatHC003P 15

[0057] Under0the sum £”=1i has a binomial distribution with parameters n and p0, so the critical value xacan be approximated due to the de-Moivre-Laplace Theorem by z - a, where z - a is the 1 - a quantile of the standard normal distribution. Hence,

[0058] To determine the sample size n for given errors a and p, needs to be calculated in the function of n, namely

[0059] Underthe sum =i i has binomial distribution with parameters n and p1so the probability above can be approximated due to the de-Moivre-Laplace Theorem, thuswhere is the cumulative distribution function (CDF) of the standard normal distribution. Hence, the following equation needs to be solved,for n, where the approximated critical value z - a is used instead of xa. In this case,

[0060] For example, if p0= 0.2, p = 0.25 and a = = 0.05, then n = 751.

[0061] Note that the critical value xacan be determined exactly using the CDF of the binomial distribution, namely■C -. H ' J ' 1 “ r. P - ‘HC003P 16hence1Bin -rm •!1" ''Wx1J “:where is the inverse CDF or quantile function of the corresponding binomial distribution. However, the normal approximation is good if n is large, e.g. if n = 751, p0= 0.2 and a = 0.05, then FBi^omiai^po)^ ~ a) = 168 and- \ <■! ’ ■ I J'u1• ~so, in this case the same decision using the exact or the approximated critical value is obtained.

[0062] In summary, the offline method requires one large set of samples with the drawback of having unnecessarily many samples generating a large cost which can be avoided with the inventive method.Online method

[0063] The following method is based on Wald’s sequential test (Wald, 1992), referred to hereafter as the online method, because in this method the samples are taken one-by-one. In this method the expected number of samples until decision depends on the true probability p. This method could be optimal if the per-sample price is much greater than the set price.

[0064] Using the previous notations letbe the number of incorrectly classified objects in the first n samples. Based on the test statisticacceptance numberand rejection numberjpr,:=i.ifeC1a _x n_£1... J —!_L 1,1!42_L _. J — Clone can construct the following sequential test. If An< dn< Rn, then no decision is made and a new observation is needed. J-Cois accepted if dn< An, JF0is rejected, and thus accept17if dn> Rn.HC003P 17

[0065] In summary, the online method requires samples one-by-one until a decision can be made with the drawback of paying the set cost every time unnecessarily, generating a large cost which can be avoided with the inventive method.Fixed set size method

[0066] This fixed set size method is similar to the online method, but instead of taking the samples one-by-one, a set size is fixed, and sets of samples are taken of this size. In fixed set size method, the same sequential test as in the online method is applied with n = s,2s, ■■■, where s is the size of the set.

[0067] For any given set size, the expected cost of the procedure can be calculated, given some a priori estimate p (such as p0,or (p0+ p1') / 2') on the true parameter p.

[0068] Namely, a function fp̂: ℕ² → [0,1] is defined such that for every k ∈ ℕ the value is the probability that, assuming that the true parameter is an estimate (p), after the k-th set the number of errors is m if m E (Aks, Rks), and f(k,m) = 0 for all m g (Aks, Rks- Note that a new set of samples is needed, if and only if, the number of errors falls into this interval.

[0069] The valuesof can be calculated recursively. First,j,. I 1 j < > I 1 — / 11 i i '.1 >. I

[0070] Then, for any k = 2, 3,... and m E Aks, Rks) it follows~ Pi <7;.,, ~ M I — ' Pl <h,. — r f P; |_

[0071] Then, considering that a new set is required if and only if the previous one ended between the critical values corresponding to the online procedure, the expected cost of the fixed set size method with a set size of s is the following:HC003P 18

[0072] Then the set size with the minimal expected cost is chosen and it is proceeded with the fixed set size procedure. Note that in a concrete application it is not possible to compute the exact value of E since an infinite summation is required. One possible solution is to stop the sum at a reasonable time by setting a desired accuracy level a, and capwhere N is the first index such thatis at most 1 - a. This means that after N sets the probability of not having made a decision is at most 1 - a. It is suggested to set a for example to 0.99 or 0.9.

[0073] In summary, the fixed set size method requires sets of samples with the drawback of the set size being fixed at the beginning of the procedure based on a priori information. It is unchanged even if accumulated information significantly differs from a priori information. This generates a large cost which can be avoided with our invention that takes into consideration all previously acquired information, so it may demand a smaller set or fewer sets of samples compared to the fixed size method.The inventive method

[0074] During the procedure of the inventive method, after evaluating each set, the size of the next set is optimized using the information from the previous samples.

[0075] First, a list of possible set sizes: it determines how many samples one can ask for each time a new set is needed.

[0076] Suppose that a set of samples is just received during testing. Afterwards, just like during the online method, it is decided whether a decision regarding0or H ‘1can be made. The same critical values as seen above in the section describing the online method are applied. If a rejection or acceptance is made, then the procedure is over. Otherwise, a new set of samples needs to be requested. The task is to decide on the number of new samples to be acquired to minimize the costs. To do so, one calculates the expected remaining cost of a fixed set size procedure until a decision is made for every possible set size. To do that, we apply the calculation detailed in section describing the fixed set size method. The onlyHC003P 19difference being that in the latter method the expected cost of the whole method starting with the first sample is derived. In the inventive method, on the other hand, the remaining cost is calculated with a similar approach, while already having some previous testing. Therefore, after K samples are tested, one knows that the number of incorrectly classified objects is dK, leading to an estimate of the true parameter p̂κ:= dκ / κ.

[0077] Hence the expected remaining cost corresponding to the fixed set size s at time K is the following.where,and for any k = 2,3,... and m E (AK+ks, RK+ks) n (dK,dK+ ks),

[0078] All that remains is to compare the EC^ values for every possible set size s and choose the minimal for the size of the next set. Then repeat this iterative step.

[0079] Note that, just like in the fixed set size method, to have a finite calculation one must cap the summation as seen before.Truncation method

[0080] Any forms of the sequential methodology, meaning all previous methods except for the offline method, comes with the possibility of having to take many samples. Here an alternative is offered, which is to cap the number of samples at some level that is already known at the beginning of a testing. This threshold along with the corresponding critical value is described below.HC003P 20

[0081] Assume that one can take sets of samples of size s e N. In the fixed set size method, a function f was defined such that fp(k,m) is the probability that, having true error parameter p, after k sets the total number of errors is m when m is within the critical values of the online procedure; and 0 otherwise. All that needs to be done is keep track of two extra values: the probability that the number of errors has exceeded the upper level at the end of any previous sets and the same for the lower one. These are denoted by U k) and L(k) for the k-th set, respectively. For example, for k E H,

[0082] The lower function L can be computed similarly.

[0083] Suppose that one would like to stop after the k-th set. Then what should be the critical value? It is known that under the null hypothesis the probability of rejection should be below a. For any p < p0this probability is at mostwhere c E N is the critical value. Note thatPo(k,m) is computable in a recursive manner for any k,m E H. One looks for the biggest c value for which the formula in (2) reaches the value a. This means that the achieved c value satisfies the requirements under ℋ₀. What remains to be checked is whether the power of the procedure is at least1 -?. Therefore, it remains to check ifI — f, 1. I,-,- 1 •

[0084] If it holds, then a suitable critical value for the method has been found. Otherwise, one keeps going and do the same calculation at the end of the (k + 1)-th step, and so on. Eventually, say after the T-th set, a proper critical value is found. Then one knows that the sequential procedure can be truncated after the T-HC003P 21th set and also the corresponding critical value c is known. Then the method goes the following way.1. Based on the values a, (3, p0and p1and the possible set size s the number of samples until the truncation is calculated, Ts. The corresponding critical value c is also determined.2. Before time Ts the sequential procedure (according to the purely online, fixed set size, or inventive method) is performed, but when the number of samples Ts is reached the procedure ends. The procedure ends if the sequential part of the test reaches a decision before time Ts.3. Otherwise, either the number of errors at time Ts(drs) is at least c, then ℋ₁ is accepted, or ℋ₀ is accepted.Truncated method

[0085] There is no fixed set size for the method, so it is not evident what s value to use for the truncated method at the very beginning. However, there is a set of possible set sizes for the inventive method. Therefore, it is proposed to use either the smallest possible set size or the greatest common divisor of the list as s. As opposed to the fixed set size method, the time Ts for the inventive method does not necessarily fall to the end of a set. Therefore, it is proposed to modify the calculation of the expected remaining cost for choosing each new set size. Namely, for a possible set size r,

[0086] It follows that for any r E N we have ECtrr≤ ECr.

[0087] Figure 1 shows a flow chart summarizing the inventive method. Three main processes are shown in Figure 1. The box titled “Construction” relates to the construction of a sequential test. Related costs, and criteria in terms of certainty are laid out. The box titled “Sequential analysis iteration” relates to the execution of the sequential test. This part of the inventive method contains iterative cycles. A set of samples is obtained, analyzed, and either a decision is made, or the next set size is determined. The box titled “Decision” relates to decision taken within the execution of the sequential test. If said criteria are satisfied to make a pass or failHC003P 22decision with the desired certainty, that decision is made and the procedure is terminated. The output is either pass or fail.Example 2: Comparative simulation using the method

[0088] Simulation results for the methods with truncation are shown to illustrate the proposed testing methodology. The following parameters were fixed:Cfix= 100, cobs= 100, and a = / 3 = 0.05.

[0089] The possible set sizes applied for the fixed set size and inventive method are 10, 20, 40, 50. With 1000 repetitions, that is, generating independent trajectories 1000 times for each scenario, the following results were obtained. The term rate refers to the rate of accepting the null hypothesis, which, in this case, was always true. Note that by the nature of the testing procedures, each parameter setup of the alternative hypothesis can be matched to a corresponding setup under the null hypothesis. Namely, for example, if one wishes to see the results when p0= 0.2, Pi = 0.3 and the true parameter is p (soholds), then one can look at those for p0= 0.7, p1= 0.8 when the true parameter is p0. By switching the roles of the binary values, the acceptance rate given in Table 1 needs to be subtracted from 1. Therefore, we only present our results under ℋ₀.

[0090] We can observe that the acceptance rates of the methods are close to the desired 0.95 level and the cost of the inventive method is always the lowest. Also note that the costs peak when the p values are close to 0.5.Table 1 Simulation results, Inventive Inventive true offline online cost fixed cost,,,,Po Pix xmethod method parameter cost ($) ($) ($)wUb I Idle0.1 0.2 0.1 13,400 15,148 8,829 8,708 0.947 0.2 0.3 0.2 20,100 20,695 11,909 11,847 0.95 0.3 0.4 0.3 24,500 24,755 13,948 13,944 0.959 0.4 0.5 0.4 26,700 27,040 15,366 15,012 0.955 0.5 0.6 0.5 26,700 26,004 14,782 14,451 0.951 0.6 0.7 0.6 24,500 24,704 14,094 14,011 0.95 0.7 0.8 0.7 20,100 19,779 11,366 11,323 0.943 0.8 0.9 0.8 13,400 12,474 7,350 7,324 0.949Example 3: Application of the methodA diagnostic tool use case

[0091] Use case: a diagnostic tool that makes a binary classification, such as a software that uses Al to identify cancer cells on pictures. It needs to be decided ifHC003P 23the diagnostic tool can be approved or disapproved by trying it out on some pictures and considering how many times it is successful.

[0092] Each time more results of this diagnostic tool are needed, new pictures need to be acquired along with a medical professional’s guideline on how the tool should categorize said pictures. Whenever new pictures are needed, they need to be bought from e.g. a hospital and also pay for a medical professional’s work. These are fixed costs along with the time it takes us to acquire them. Using the tool to process these pictures could also come with a fixed cost. Examples of individual cost: the source of the pictures may ask for an individual cost as well, and running the diagnostic tool on a specific picture also has a cost.

[0093] The proposed procedure informs at any point whether the new diagnostic tool should be accepted or rejected. If no decision is reached yet, then the proposed procedure determines the optimal number of testing pictures to acquire in order to keep costs as low as possible.A drug trial use case

[0094] Use case: a treatment, possibly a new drug. It has to be decided if the treatment can be approved or disapproved by trying it out on some subjects and considering how many times it is effective.

[0095] To keep costs low, a group of subjects is organized and tested at the same time. A fixed cost needs to be paid for every group while additionally paying for each subject’s test. Examples of fixed cost: choosing a versatile group of subjects, organizing the testing, providing medical staff and testing site. The amount of time that it takes for a group testing from start to finish can also contribute to the fixed cost. Examples of individual cost: paying subject to participate, medical and laboratory expenses per subject.

[0096] The proposed procedure informs at any point if the new treatment is accepted or rejected. If no decision is reached yet, then proposed procedure determines the optimal size of the next testing group in order to keep costs low.A numerical use case

[0097] Use case: Accept a methodology if it is effective in at least 80 % of the cases, and reject it if its effectivity rate is below 75 %. There are 5 % errors of the first and the second kind meaning that the rates of false rejections and false acceptances are kept at 5 %. Let there be a treatment that, in reality, is 80 %HC003P 24effective, thus it should be accepted. Suppose that the fixed and individual costs are the same 100 $.

[0098] Testing a necessary number of subjects at once costs: 75,200 $.

[0099] Testing one subject at a time: 75,661 $.

[0100] The inventive method (where we can have groups of 10, 20, •••,50) costs: 40,976$.Example 4: Comparative analysis with numerical examples

[0101] The following presents a comparative analysis, including a numerical example, between the present method and prior art methods.

[0102] Prior art methods involve using sequential testing methods to make statistically significant pass / fail decision for a population of items based on a sample set. These methods may potentially allow determinations that the testing process may be stopped before the complete testing of the sample set. In the sample set, whose size can be calculated using a classical statistical method (see the offline method), a classical sequential test is performed with an arbitrary set size. The latter can be viewed as a simple generalization of Wald’s sequential method (see the online method). In that sense, prior art methods can be represented as a mixture of online and offline methods. However, prior art methods address only the scenario in which the full sample set is readily available, that is, when acquiring samples incur no costs. Consequently, prior art methods consider only the per-sample costs, and do not apply to the situation where the inventive method is most useful. In prior art methods there are no sophisticated optimizations on the costs because the set size is chosen arbitrarily without taking the costs for acquiring the samples into account.

[0103] The inventive method takes the costs for acquiring samples also into account and performs a joint optimization of the costs per-sample and the costs of the sample set, thus minimizing the total cost of the whole process. When the costs of acquiring samples is comparable to the costs of evaluating each individual sample, prior art methods no longer yields cost-optimal results, making a more sophisticated approach necessary. In the inventive method, the set size is not chosen arbitrarily, instead, it is determined sequentially and adaptively to minimize total costs. While both, the prior art method and the inventive method produce statistically valid decisions, the inventive method achieves these decision in a cost-optimal manner across a wider range of scenarios involving both per-sample and per-set costs.HC003P 25

[0104] As described above, if the set cost is comparable to the per-sample cost, prior art methods may no longer be cost optimal. This is illustrated with the following comparative numerical examples, each based on 100 simulations. In this example, the following input parameters are set:a = 0.05 p = 0.05 (errors of first and second kind) cfix= 100 or 2000 Cobs = 100 (set and per-sample costs) Po = 0.2 Pi = 0.25 p = 0.2 (lower, upper and true values of failure rates)steps = {10, 20,..., 200} (possible set sizes)

[0105] Prior art methods are represented with fixed set size of 10 samples.

[0106] For these parameter settings, the averaged costs as depicted in Fig. 2 and Fig. 3 were obtained.

[0107] For example, as shown in Fig. 2, using a fixed set size of 10 samples (representative for prior methods relying on fixed set sizes), costs of 53,789 $ were obtained on average. This is a cost reduction of 28% of what the classical offline method (i.e. offline method of example 1) provides. However, using the inventive method, in which the set size is optimized in each step, costs of 39,909 $ were reached on average, that is, the cost are reduced by 26% compared to prior art methods of US2009306933A1 and by 47% compared to the classical offline method. The inventive method outperforms prior art methods as it uses significantly less cost. If the set cost is higher, one can see a more dramatic situation as shown in Fig. 3. In this comparative example, prior art methods as described in US2009306933A1 generates cost of 121,350 $ on average, which means that this approach is more costly than the classical offline method, namely the costs are of about 57% higher. However, using the inventive method, costs of 46,788 $ are reached on average, that is, a cost reduction of 39% compared to the classical offline method and of 61% compared to prior art methods as described in US2009306933A1.

[0108] This comparative example highlights that the inventive method is flexible, thus it keeps the overall cost consistently low across all ranges of the input parameters. Although the fixed set size method seems superficially similar to prior art methods, the essence of the inventive method is not the fixed set size method, but optimizing the size of each consecutive set using the inventive method toHC003P 26minimize the total cost (including the per sample cost and the set cost as well, which is completely ignored in prior art methods).HC003P 27References:Armitage et al. (1969). Repeated Significance Tests on Accumulating Data. Journal of the Royal Statistical Society. Series A (General), 132(2): 235-244.Bothwell et al. (2018). Adaptive design clinical trials: a review of the literature and ClinicalTrials.gov. BMJ Open, 8(2): e018320.Gould et al. (1982). Group sequential methods for clinical trials allowing early acceptance of H_0 and incorporating costs. Biometrika, 69(1): 75-80.Pocock. (1977). Group sequential methods in the design and analysis of clinical trials. Biometrika, 64(2): 191-199.Wald. (1992). Sequential Tests of Statistical Hypotheses (pp. 256-298).

Claims

HC003P 28Claims1. A computer implemented adaptive and iterative method to improve and accelerate a quality control decision for a quality control measurement to be implemented in a specific technical field while reducing the overall cost of the quality control process; the method comprising the consecutive steps:(a) Constructing a sequential test:I. specifying the costs for obtaining a set of samples,II. specifying the costs for evaluating each individual sample, III. defining the size of the sample set,IV. setting criteria for generating a pass or fail decision;(b) Executing the sequential test:I. obtaining a first set of samples of the size specified in (a)III;II. analyzing the first set of samples;III. terminating the sequential test if a pass or fail decision is generated; or(c) Continuing sequential testing if no pass or fail decision is generated;I. determining in an adaptive way the size of an additional set of samples required for the next iterative step;II. obtaining the additional set of samples;III. analyzing the additional set of samples;IV. terminating the sequential test if the pass or fail criteria are met; or (d) Repeating the sequential testing of (c)(I) – (IV) until a pass or fail decision is generated.

2. The computer-implemented method of claim 1, wherein the quality control decision is taken based on standards set by an organization for standardization, or “good practice” quality guidelines and regulations.

3. The computer-implemented method of claim 1 or 2, wherein the specific technical field is any one of medicine, pharmacy, engineering, science, computer science, specifically the technical field relates to medical diagnostic techniques, pharmaceutical production, chemical engineering, process engineering, civil engineering, electrical engineering, computer engineering, mechanical engineering, geological engineering, agricultural engineering, applied engineering, biomedical engineering, biomedical nanoengineering, biological engineering, building services engineering, energy engineering,HC003P 29geomatics engineering, information engineering, industrial engineering, mechatronics engineering, engineering management, military engineering, mining engineering, nanoengineering, quantum engineering, nuclear engineering, petroleum engineering, project engineering, railway engineering, software engineering, supply chain engineering, systems engineering, textile engineering, or cybersecurity engineering.

4. The computer-implemented method of any one of claims 1 to 3, wherein the quality control decision is taken in the field of medicine.

5. The computer-implemented method of claim 4, wherein the medical field is medical Al, diagnostic to medical diagnostic techniques, pharmaceutical production tool, diagnostic model, and / or healthcare software.

6. The computer-implemented method of any one of claims 1 to 5, wherein the quality control decision is meeting the regulatory requirements or compliance, achieving its intended purpose, satisfying consumer safety, product performance, standards published by an organization for standardization, or “good practice” guidelines and regulations, or providing evidence that the product works for the intended population.

7. The computer-implemented method of any one of claims 1 to 6, wherein the sample data is derived from images, labeled images, product parameters, process parameters, computer program output, measurements, designs, drug testing, training of machine learning algorithms.

8. The computer-implemented method of any one of claims 1 to 7, wherein said decision output is deployed to a mobile device or application, printer or labeling system, a live computing environment, a web dashboard, a physical dashboard display, e-mail alerts, SMS notifications, a chatbot, an API (application programming interface), an LLM (large language model), or some historical data storage.

9. Use of the computer-implemented method of any one of claims 1 to 8 for reducing costs compared to classical methods.

10. A computer implemented adaptive and iterative method to improve and accelerate the quality control decision for a quality control measurement in the field of a medical imaging analysis while reducing the overall cost of the imaging analysis method; the method comprising:(a) Constructing a sequential test:HC003P 30I. specifying the costs for obtaining a set of image samples, II. specifying the costs for evaluating each individual image sample,III. specifying the size of the image sample sets, IV. specifying criteria for generating a pass or fail decision;(b) Executing the sequential test:I. obtaining a first set of image samples of the size specified in (a)lll;II. analyzing the first set of image samples;III. terminating the sequential test if a pass or fail decision is generated; or(c) Continuing sequential testing if no pass or fail decision is generated;I. determining in an adaptive way the size of an additional set of image samples required for the next iterative step;II. obtaining an additional set of image samples;III. analyzing the additional set of image samples;IV. terminating the sequential test if the pass or fail criteria are met; or(d) Repeating the sequential testing (c)(I) – (IV) until a pass or fail decision is generated.

11. The computer implemented method of claim 10, wherein the medical imaging analysis is selected from the group consisting of radiology, sonography, ultrasound, elastography, photoacoustic imaging, tomography, X-ray, X-ray computed tomography (CT), positron emission tomography (PET), magnetic resonance imaging (MRI), PET-CT, PET-MRI, echocardiography, functional near-infrared spectroscopy (FNIR), magnetic particle imaging (MPI), scintigraphy, single-photon emission computed tomography (SPECT), Digital Subtraction Angiography (DSA), Fluoroscopy, Dual-Energy X-ray Absorptiometry (DEXA), Fluorescence Imaging, Electroencephalography (EEG), Magnetoencephalography (MEG), Optical Coherence Tomography (OCT), or electrocardiography (ECG).

Images