Progress management system, server, and progress management method

The system addresses the challenge of accurately calculating progress rates during manufacturing recipe development by recalculating specification achievement rates and allocating resources effectively, ensuring efficient management of semiconductor manufacturing processes.

WO2026234776A1PCT designated stage Publication Date: 2026-11-12HITACHI HIGH TECH CORP
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
PCT/JP2025/016900
Authority / Receiving Office
WO · WO
Patent Type
Applications
Current Assignee / Owner
Filing Date
2025-05-08
Publication Date
2026-11-12

AI Technical Summary

Technical Problem

Existing progress management systems fail to accurately calculate the progress rate when additional evaluations are required in manufacturing recipe development, particularly in semiconductor manufacturing, due to the impact of re-evaluations and qualitative refinement of conditions.

Method used

A progress management system and method that calculates an evaluation progress rate based on the completion time of all evaluations, adjusts specification achievement rates, and recalculation of progress when additional evaluations are added, using a platform with a visualization tool to manage and analyze data across multiple sites.

Benefits of technology

Enables accurate calculation of progress rates even with additional evaluations, identifies problems, and allocates resources efficiently to resolve delays, enhancing the efficiency of manufacturing recipe development.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure JP2025016900_12112026_PF_FP_ABST
    Figure JP2025016900_12112026_PF_FP_ABST
Patent Text Reader

Abstract

The purpose of the present invention is to provide a technology with which it is possible to calculate an accurate progress rate, even when an evaluation is added. Provided is a progress management system wherein a progress management application executes: a first step in which an evaluation progress degree indicating a progress degree for each evaluation is calculated; a second step in which a first specification achievement rate is calculated using the result of an evaluation; and a third step in which, when no evaluation is added, the state of an evaluation is determined on the basis of the evaluation progress degree and a second specification achievement rate calculated using the first specification achievement rate, and when an evaluation is added, the state of an evaluation is determined on the basis of the recalculated evaluation progress degree and the second specification achievement rate calculated using the recalculated first specification achievement rate.
Need to check novelty before this filing date? Find Prior Art

Description

Progress management system, server, and progress management method

[0001] The present invention relates to a progress management system, a server, and a progress management method in a manufacturing recipe development process.

[0002] In the development of manufacturing recipes for semiconductor manufacturing apparatuses, conditions for obtaining desired performance are searched while actually performing processes (hereinafter also referred to as "demo" or "demonstration"). Since manufacturing recipe development includes evaluation of multiple items, it is required to efficiently advance development by collecting and analyzing data while grasping the progress status of each evaluation, adjusting the next evaluation process, and distributing resources.

[0003] As a progress management technology, for example, Patent Document 1 addresses the problem of "displaying changes in work progress such as delay expansion or delay reduction in work progress display in an easily understandable manner, regardless of the magnitude of the difference between the work amount that should have been achieved as scheduled and the actually achieved work amount", and discloses the following content as an invention of a work progress status display device: "A time axis is displayed from left to right, the work period from the scheduled work start date to the work end date is displayed as a line segment or a rectangle, the ratio of the advanced work amount to the scheduled work amount is displayed by changing to a different thickness or pattern from the left end for a length proportional to the horizontal length of the scheduled line segment or rectangle, the number of days or amount of delay or advance on the date the work status is confirmed is displayed by the size of a figure, and the number of days or amount of delay or advance on the date the work status is confirmed is compared with the number of days or amount of delay or advance on the immediately preceding date the work status was confirmed, and the increase or decrease is displayed with figures of different colors and shapes".

[0004] JP 2002-7654 A

[0005] Generally, in process demonstrations, if additional evaluations are required, the progress of the previous evaluations may be reset. On the other hand, since an evaluation has already been conducted once, qualitative refinement of the conditions has progressed, and it is possible that the overall progress may be affected by this refinement. For this reason, when conducting a re-evaluation, it is necessary to calculate an accurate progress rate, including the return of evaluations and the impact of the refinement of conditions. Patent Document 1 does not describe the case in which a re-evaluation is conducted. Therefore, the present invention aims to provide a technology that can calculate an accurate progress rate even when additional evaluations are required.

[0006] To solve the above problems, one of the representative progress management systems of the present invention is a progress management system comprising a platform on which a progress management application for managing the progress in the evaluation of processing results of semiconductor manufacturing equipment is implemented, the first step is to calculate an evaluation progress rate indicating the progress of all evaluations based on the evaluation completion time, which is the time when all evaluations are completed, or the usage order of the application corresponding to the content of the evaluations; the second step is to calculate a first specification achievement rate using the results of the evaluations; and if no evaluations are added, the status of the evaluations is determined based on the second specification achievement rate calculated using the first specification achievement rate and the evaluation progress rate, and if evaluations are added, the evaluations are recalculated. A third step is performed by the progress management application, in which the status of the evaluation is determined based on a second specification achievement rate calculated using the first specification achievement rate and the recalculated evaluation progress rate, wherein the first specification achievement rate is the achievement rate of the evaluation result with respect to the specifications of the processing result, the second specification achievement rate is the achievement rate of the specifications in all of the evaluations, the recalculated evaluation progress rate is calculated by dividing the evaluation time in the evaluation by the sum of the evaluation times, the sum of the evaluation times is the sum of the evaluation time in the evaluation performed up to the time the evaluation was added, the evaluation time in the added evaluation, and the evaluation time in the evaluation after the added evaluation.

[0007] The present invention makes it possible to calculate the progress rate accurately even when additional evaluations are required. Other issues, configurations, and effects will be clarified by the following description of embodiments for carrying out the invention.

[0008] Figure 1 is a diagram showing an example of the configuration of a progress management system. Figure 2 is a diagram showing an example of the configuration of a visualization tool or data analysis application server. Figure 3 is a flowchart schematically showing the multiple evaluation processes performed in manufacturing recipe development. Figure 4 is a frequency distribution diagram of the multiple evaluations performed in the manufacturing recipe development process. Figure 5 is a diagram showing an example of a progress management graph. Figure 6 is a frequency distribution diagram of each evaluation included in a series of evaluations performed in the manufacturing recipe development process. Figure 7 is a diagram showing an example of the GUI of the progress management system. Figure 8 is a diagram showing an example of the GUI of the progress management system. Figure 9 is a flowchart showing the processing of the visualization tool or data analysis application server 183. Figure 10 is a flowchart showing the processing of the visualization tool or data analysis application server 183. Figure 11 is a diagram showing an example of the GUI of the progress management system.

[0009] The embodiments of the present invention will be described below with reference to the drawings. However, the present invention is not limited to these embodiments. Furthermore, in the drawings, identical parts are denoted by the same reference numerals.

[0010] In this disclosure, "private environment" refers to a network in which devices within a predetermined range can communicate with each other, but cannot communicate with devices outside that range. For example, "the private environment of site A" refers to a network in which communication is not possible with other sites, but communication is possible within site A. "Public environment" refers to a network in which communication is not restricted. For example, "the public environment of site A" refers to a network in which devices within site A can communicate with each other, and can also communicate with devices within other sites. "Manufacturing line equipment group" refers to a group of devices that form a manufacturing line, and in the case of a semiconductor manufacturing line, it includes at least one of the following: semiconductor manufacturing equipment, semiconductor inspection equipment, or semiconductor measurement equipment. "PC" is not limited to personal computers, but also includes servers as so-called computers (processing devices).

[0011] In this disclosure, “application” refers to a program that runs on a PC. In this disclosure, it is also referred to as “app.”

[0012] As described later, the progress management system of this disclosure includes a group of devices including manufacturing equipment, software for managing the progress of multiple evaluations related to manufacturing recipe development, and a server (visualization tool or data analysis application server 183) that stores information on the usage status of human and mechanical resources assigned to the multiple evaluations. A detailed explanation follows below.

[0013] (Configuration of the Progress Management System) The configuration of the progress management system will be explained with reference to Figures 1 and 2. Figure 1 is a diagram showing an example of the configuration of the progress management system. The progress management system 1 manages the progress of manufacturing recipe development when manufacturing recipe development is carried out at multiple locations. Since the progress management system 1 includes a group of equipment including manufacturing equipment and an integrated platform described later, it can also be said to be an important equipment data integration system in manufacturing. Furthermore, although this disclosure describes the case of semiconductor manufacturing equipment as the manufacturing equipment, this disclosure can also be applied to manufacturing equipment other than semiconductor manufacturing equipment.

[0014] (Configuration of the progress management system) The progress management system 1 includes locations A through C, the Internet 154, the Internet 164, the Internet 174, a cloud server 180, a remote monitor 182, a visualization tool or data analysis application server 183 (hereinafter also simply referred to as "server 183"), and a common inter-site cloud environment 184. The common inter-site cloud environment 184 can also be described as a platform on which a progress management application for managing the progress in evaluating the processing results of manufacturing equipment is implemented. The progress management system 1 can also be described as having a platform on which a progress management application for managing the progress in evaluating the processing results of semiconductor manufacturing equipment is implemented. In particular, server 183 of the platform executes the progress management application and manages the progress in evaluating the processing results of semiconductor manufacturing equipment. Note that the platform is not limited to being composed of hardware, but may also be configured as an environment on which the application can be executed by a cloud environment.

[0015] Site A includes a private environment 155 and a public environment 156. The private environment 155 includes server A 150 and a manufacturing line equipment group A 151. The public environment 156 includes server A' 152 and PCA 153. Site A forms a communication network including the equipment group and a processing unit that manages data, and Site A may be referred to as integrated platform A.

[0016] Similarly, site B includes a private environment 165 and a public environment 166. The private environment 165 of site B includes server B 160 and manufacturing line equipment group B 161. The public environment 166 of site B includes server B' 162 and PCB 163. Site B forms a communication network including the equipment group and processing equipment that manages data, and site B may be referred to as integrated platform B.

[0017] Similarly, site C includes a private environment 175 and a public environment 176. The private environment 175 of site C includes a server C 170 and a group of manufacturing line equipment C 171. The public environment 176 of site C includes a server C' 172 and a PCC 176. Site C forms a communication network including the group of equipment and a processing unit that manages data, and site C may be referred to as an integrated platform C.

[0018] Servers A150 and A'152 refer to servers at site A. Servers B160 and B'162 refer to servers at site B, and servers C170 and C'172 refer to servers at site C. The notations "A" or "A'" indicate the site to which they belong; if the sites are not distinguished, they may simply be referred to as "servers." The same notation is used for other PCs and manufacturing line equipment.

[0019] (Processing in the progress management system) In the private environment 155 of site A, the manufacturing line equipment group A151 executes the manufacturing process based on the input manufacturing recipe. For example, in the manufacturing line equipment group A151, the instruction values ​​shown in the manufacturing recipe are transmitted to each piece of equipment within the manufacturing line equipment group A151, and the manufacturing process is carried out. The processed manufacturing recipe is stored in server A150.

[0020] The manufactured objects are transported to inspection, analysis, or analytical devices included in the manufacturing line equipment group A151, where predetermined measurements are performed based on the respective measurement recipes entered into the manufacturing line equipment group A151. The measurement results are stored in server A150.

[0021] PCA153 determines whether data, including recipes and measurement results, entered into the manufacturing line equipment group A151 is accessible or not, and defines the data flow (sets the data route). PCA153 defines publicly accessible data, which is data that is permitted to be communicated (accessible) from locations other than location A, and prohibited publicly accessible data, which is data that is prohibited from being communicated (access is prohibited) from locations other than location A. For example, publicly accessible data that can be sent outside of location A is stored in server A'152. In addition, publicly accessible data and prohibited publicly accessible data are managed separately within server A150. As a specific management method, the data may be divided into categories or levels within the database of server A150, which will be described later. Publicly accessible data is stored in the cloud server 180 via the internet 154.

[0022] Sites B and C have the same configuration as site A. Server B160, manufacturing line equipment group B161, server B'162, and PCB163 at site B correspond to server A150, manufacturing line equipment group A151, server A'152, and PCA153 at site A, respectively. Server C170, manufacturing line equipment group C171, server C'172, and PCC173 at site C correspond to server A150, manufacturing line equipment group A151, server A'152, and PCA153 at site A, respectively.

[0023] The visualization tool or data analysis application server 183 is a processing device such as a server that runs a visualization tool or data analysis application, which is software (hereinafter also simply referred to as "software"). The server 183 is located in the inter-site common cloud environment 184 and communicates with servers, etc., at site A to site C via the Internet 174 from the inter-site common cloud environment 184 and the Internet 154. The server 183 visualizes and analyzes the data acquired at each site. The visualized results can be displayed, for example, on a remote monitor 182.

[0024] The progress management system 1 has a server 183 in the inter-site cloud environment, and the progress status based on the public permission data of each site is visualized by the remote monitor 182. Therefore, the progress management system 1 makes it possible to manage the progress of each site collaborating on development in manufacturing recipe development.

[0025] (Configuration of Visualization Tool or Data Analysis Application Server 183) Figure 2 shows an example of the configuration of a visualization tool or data analysis application server 183. The server 183 includes a bus 1830, a processor 1831, a memory 1832, a storage device 1833, an input / output device 1834, and a communication interface 1835. The processor 1831, memory 1832, storage device 1833, input / output device 1834, and communication interface 1835 are each connected to the bus 1830 and communicate information via the bus 1830. The processor 1831 processes the acquired information. The memory 1832 stores processing instructions to be executed by the processor 1831. The memory 1832 includes, for example, random access memory (RAM) or other dynamic storage device, read-only memory (ROM) or other static storage device, which is also used to store temporary variables or other intermediate information during the execution of instructions by the processor 1831. The processor 1831 can perform functions as a visualization tool or data analysis application server 183, as described later, by executing processing instructions contained in the memory 1832.

[0026] Furthermore, the storage device 1833 is used to store information and instructions and is composed of, for example, a magnetic disk or an optical disk. The storage device 1833 may also be a database (hereinafter also referred to as "DB") that stores instructions. The input / output device 1834 includes a display device that displays information to the user and an input device that transmits the selection of information and commands made by the user to the processor 1831. For example, the display device is a display, and the input device is a mouse and keyboard. The communication interface 1835 enables bidirectional communication over a network. Through the communication interface 1835, the server 183 can obtain information from server A' 152, server B', and server C' 172.

[0027] In this disclosure, memory 1832 stores an application n (where n is an integer) for measuring the progress of evaluation, in addition to the visualization tool and data analysis application. Other applications may also be stored in memory 1832. The input / output device 1834 functions as a GUI (graphical user interface). As described later, it can also present visualized data to the user and accept user instructions. The input / output device 1834 may show GUI functions to the user via the remote monitor 182. The storage device 1833 may also include a database for acquiring information (public permission data) obtained from servers A'152, B', and C'172, as described later. The database may include an application database (app DB) that stores data for each application.

[0028] Although the configuration of server 183 has been described, this disclosure is not limited to this configuration. Server 183 may be composed of other hardware circuits, or of a combination of hardware circuits and software. Furthermore, servers A150, A'152, PCA153, B160, B'162, PCB163, C170, C'172, and PCC173 may have a configuration similar to that of server 183.

[0029] Embodiment 1 of the present invention will be described with reference to Figure 3. Figure 3 is a flowchart schematically showing the multiple evaluation processes performed in manufacturing recipe development. In Figure 3, the specification achievement rate, which is one of the progress management indicators used on a system across multiple locations, is explained. When the manufacturing recipe development is a manufacturing recipe for a semiconductor device, the multiple evaluations are, for example, evaluations to confirm the performance of semiconductor manufacturing equipment, such as etching rate measurement, CD measurement, and cross-sectional observation of the etching shape. Note that multiple evaluations are sometimes collectively referred to as a "series of evaluations".

[0030] In a demonstration of manufacturing recipe development, there may be multiple evaluations running in series, and an application may be used for each evaluation. Furthermore, if customer-defined specifications are set for each evaluation, the achievement rate against those specifications (hereinafter also referred to as the "specification achievement rate") can be calculated. In this disclosure, an achievement rate of 0% is defined as a state where the specifications are not met at all, and an achievement rate of 100% is defined as a state where the specifications are fully met.

[0031] To give a specific example, in a series of evaluations, as shown in Figure 3(a), evaluation 1 is performed first, followed by evaluation 2, then evaluation 3, evaluation 4, and evaluation 5. Although Figure 3 shows a case where there are five evaluation steps from evaluation 1 to evaluation 5, the number of evaluations included in a series of evaluations is not limited to five. Furthermore, in the following explanation, evaluation n will be denoted as evaluation n (where n is 1 to Z (Z is a positive integer, and in this disclosure, the case where Z is 5 is used as an example)). Furthermore, as shown in Figure 3(b), for example, the evaluation status of evaluation n is detected using application n. Furthermore, as shown in Figure 3(c), if there are specifications provided by the customer for each evaluation (for example, specifications for evaluation 1, specifications for evaluation 2, etc.), the achievement rate (specification achievement rate) against the specifications is calculated for each evaluation. The application can be executed on, for example, server 183.

[0032] The specification achievement rate for evaluation n (hereinafter also referred to as the "specification n achievement rate") is calculated according to the following formula (1).

[0033] Assuming that all evaluations have the same weighting, the evaluation progress is calculated using the following formula (2). In formula (2), the evaluation progress up to 50% of evaluation n is calculated as "50% evaluation progress of app n". The total number of evaluations (the number of evaluations included in a series of evaluations) is Z, and the evaluation progress when all evaluations are completed is defined as 100%.

[0034] The 50% evaluation progress of app n, as shown by formula (2), represents the evaluation progress up to 50% in evaluation n. If the specification n achievement rate has not reached 100%, the value from formula (2) is used as the evaluation progress of evaluation n. Furthermore, if the specification n achievement rate has reached 100%, but the specification n+1 achievement rate has not reached 100%, the following formula (3) is used as the evaluation progress of evaluation n.

[0035] Next, Embodiment 2 of the present invention will be described with reference to Figure 4. Figure 4 is a frequency distribution diagram of each evaluation included in a series of evaluations performed in the manufacturing recipe development process. Embodiment 1 showed a case where each evaluation flows in series with the same specific gravity, but Embodiment 2 shows a case where each evaluation has a different specific gravity, and another evaluation is performed at the same time. In Embodiment 2, as in Embodiment 1, the achievement rate of the specifications is detected using an application for each evaluation. In Figure 4, the frequency distribution of each evaluation is shown as a curve. The frequency on the vertical axis indicates the number of items processed when the specifications set for the evaluation are subdivided into multiple items. The time on the horizontal axis indicates the period during which the evaluation is performed. For example, in the frequency on the vertical axis, if the maximum value of the frequency of evaluation n is fn, the maximum value of the frequency of evaluation 1 is f1, and the maximum value of the frequency of evaluation 2 is f2, indicating that the frequency values ​​differ among the evaluations, and that each evaluation has a different specific gravity. Also, in the time on the horizontal axis, the curves for evaluation 1 and evaluation 2 overlap, indicating that evaluations may be performed at the same time. One possible scenario is when a previous evaluation has been completed and the next evaluation has begun, but the results of the next evaluation may influence the specifications of the previous evaluation.

[0036] Let the start time of evaluation n be tns, the end time of evaluation n be tne, and the evaluation end time tE indicating the time when all evaluations are completed (in the present disclosure, this is the same as the end time t5e of evaluation 5), the evaluation progress is calculated as shown in the following formula (4).

[0037] The 50% evaluation progress of app n represented by formula (4) is the evaluation progress up to 50% in each evaluation. If the specification n achievement rate has not reached 100%, the value of formula (4) is used as the evaluation progress of evaluation n. Further, when the specification n achievement rate has reached 100 and the specification n+1 achievement rate has not reached 100%, the following formula (5) is taken as the evaluation progress of evaluation n.

[0038] Note that when calculating the evaluation progress in more detail, a relational expression between the usage frequency of the application and the evaluation time for each evaluation may be acquired, the evaluation time tr may be calculated from the usage frequency of the application, and a more accurate evaluation progress of app n may be obtained by the following formula (6).

[0039] Note that in the case of FIG. 4, the evaluation progress of the entire series of evaluations including evaluation 1 to evaluation 5 may be calculated using the score Pn of evaluation n shown in the following formula (7) and the evaluation progress P(t) shown in formula (8). The evaluation progress of the series of evaluations can be represented by the evaluation progress P(t).

[0040] Next, Embodiment 3 of the present invention will be described with reference to FIG. 5. FIG. 5 is a diagram illustrating an example of a progress management graph. While Embodiment 1 and Embodiment 2 show the case where the specification n achievement rate of each evaluation is evaluated in a series of evaluations, Embodiment 3 shows the case where progress is determined from the specification achievement rate of the entire evaluation. Here, the calculation formula for the specification achievement rate of the entire evaluation is represented by the following formula (9).

[0041] FIG. 5(a) is a graph plotting the correspondence between the evaluation progress degree and the spec achievement rate calculated in Example 1 and Example 2. FIG. 5(a) may be based on estimated values or on actual measurements. The estimated values will be described with reference to FIG. 5(b). In FIG. 5(b), a straight line connecting 100% spec achievement rate at the required deadline G and 0% spec achievement rate at time 0 is shown. When calculating the estimated values, it is assumed that time and the spec achievement rate are in a proportional relationship. Although the horizontal axis in FIG. 5(b) represents time, a graph based on the estimated values shown in FIG. 5(a) can be obtained by replacing the deadline G with 100% evaluation progress degree.

[0042] As an example, in FIG. 5(b), the spec achievement rate b at time ta is shown as an estimated value, the horizontal axis is changed from time to evaluation progress degree, and the evaluation progress degree a and the spec achievement rate b calculated (actually measured) at the current time (time ta) are respectively plotted, whereby the graph of FIG. 5(a) is created. For example, FIG. 5(b) can also be said to be practical from the viewpoint that a time delay relative to the deadline G can be calculated. Hereinafter, with reference to FIG. 5, the calculation formulas for the spec achievement rate delay and the time delay are expressed as the following formulas (10) and (11).

[0043] Example 4 of the present invention will be described with reference to FIG. When a new evaluation is added before or after a certain evaluation in the series of evaluations described in Example 1, it is necessary to recalculate the spec achievement rate and the evaluation progress degree for the entire series of evaluations. In Example 4, a case where addition of an evaluation occurs will be described.

[0044] Figure 6 is a frequency distribution diagram of each evaluation included in a series of evaluations performed in the manufacturing recipe development process. Figure 6(a) shows an example of a series of evaluations. The series of evaluations includes Z evaluations, from evaluation 1 to evaluation Z. Although the maximum values ​​of the frequencies differ, all evaluations are performed in series, and the next evaluation is performed only after one evaluation is completed. It is planned that it will take time t to complete evaluations 1 and 2, and that it will take time tE1 to complete evaluations 3 through Z. In the series of evaluations from evaluation 1 to evaluation Z, let x be the evaluation progress when evaluations 1 and 2 are completed, and let y (= 100 - x) be the evaluation progress for evaluations 3 through Z in the series if the remaining evaluations 3 through Z are completed. y can be called the planned value.

[0045] Figure 6(b) shows the case where there is an additional evaluation. Time t indicates the point at which the evaluation return occurs. In other words, at time t, when evaluations 1 and 2 are completed, evaluation 2' is added. Here, the formula for calculating the evaluation progress at time t is given by the following formula (12). z indicates the evaluation progress shown in the evaluation progress of the existing evaluations (evaluation 1 to evaluation Z) when evaluation 2', which is added at t, is performed.

[0046] The evaluation progress indicator shown in formula (12) allows for the calculation of evaluation progress even if the evaluation needs to be redone. Although Figure 6 shows a case where an additional evaluation is performed once, this disclosure is not limited to this case. Additional evaluations may be performed two or more times, or the same evaluation may be performed again, for example, by repeating Evaluation 1.

[0047] Next, the visualization of the analyzed data will be explained with reference to Figures 7 and 8. Figures 7 and 8 show examples of the GUI of the progress management system. Figures 7 and 8 specifically show the display screen of the GUI, which can be displayed on the input / output device 1834 of the server 183 and the remote monitor 182.

[0048] On server 183, the specification n achievement rate for each evaluation calculated in Examples 1 to 4, a graph with the specification achievement rate on the vertical axis and time or evaluation progress on the horizontal axis, and a graph with frequency on the vertical axis and time on the horizontal axis are visualized. This data is output from server 183 and is accessed in real time from the application DB of server 183 and reflected in the data on the graphs, so the graphs are updated in real time. It is also possible to link with parts management, equipment management, and other application DBs to graph the shortage status of parts and the availability status of equipment and visualize them in real time as well.

[0049] If, as a result of analyzing the evaluation progress and specification achievement rate, it is found that the evaluation progress or specification achievement rate has not been met, as shown in Figure 5, the server 183 identifies the problem and displays it on the GUI shown in Figure 7. To determine the cause of the problem, the server 183 determines the evaluation stage based on the current time, compares it with the progress history and application log data to pinpoint the cause of the problem, and displays it on the GUI. Furthermore, to resolve the cause, the server 183 compares it with the availability of personnel, parts, and equipment, reviews resource allocation, and displays it on the GUI to allocate resources to the parts that are behind schedule.

[0050] The graph display unit 200 includes, for example, a graph that visualizes the public permission data. The server 183 displays data stored in the application DB of the server 183 or data obtained from the cloud server 180 in the form of graphs, etc. The evaluation stage display unit 201 displays information related to the evaluation stage, specifically the evaluation progress, the specification achievement rate, and the project status. The cause display unit 202 displays information related to resolving the cause, showing the content of the problem, the cause of the problem, and instructions for resolving the problem.

[0051] Furthermore, the process from identifying the cause of a problem to issuing instructions is stored in the DB of the storage device 1833. Once the DB has become sufficiently developed, machine learning can be used to analyze combinations of data within this DB, enabling AI-powered progress management, problem identification, and the display of instructions on the GUI.

[0052] In the progress management system 1, it is conceivable that a user, such as an administrator responsible for progress management, can monitor the GUI and issue instructions according to the GUI. Furthermore, it is also possible to forward the instruction portion to a PC at the site where the problem is occurring via server 183, and issue instructions directly to that site.

[0053] Furthermore, let's assume that three manufacturing recipe development projects are being carried out in parallel. By plotting the current status of each project on the graph in Figure 5 using server 183, it becomes possible to compare the progress of the projects, as shown in Figure 8. This allows us to understand in detail what is hindering the progress of each project by looking at the differences between projects that are behind schedule and those that are ahead.

[0054] The corresponding graph display unit 203 shows a corresponding graph plotting the evaluation progress and specification achievement rate for each project. For example, it is possible to display Project A as a black circle, Project B as a triangle, and Project C as a white circle on the corresponding graph. The cause display unit 204 shows the results of progress management for each project and includes information on evaluation progress, specification achievement rate, project status, problems, causes, and instructions based on the cause analysis.

[0055] Next, the processing of the server 183 will be described with reference to Figures 9 and 10. Figures 9 and 10 are flowcharts showing the processing of the visualization tool or data analysis application server 183. In Figure 9, steps S1 to S8 are shown, and in Figure 10, steps S8 to S15 are shown.

[0056] In step S1, the server 183 converts the evaluation end time, which is the time when the final step of the evaluation among multiple evaluations related to manufacturing recipe development is completed, into a reference value. In this disclosure, as in the derivation of formula (4), the start time tns, the end time tne, and the evaluation end time tE of evaluation n are set, setting the end time tne for each evaluation n included in multiple evaluations corresponds to the conversion of the reference value. In addition, in the case of Figure 6(a), the end time tE1 of evaluation X is set, and in the case of Figure 6(b), the end time tE2 of evaluation X is converted into a reference value.

[0057] Next, in step S2, the server 183 calculates a desired evaluation progress time for each evaluation item as an evaluation progress value, based on the application usage order and reference values ​​corresponding to the evaluation items of the multiple evaluations. In other words, the server 183 calculates an evaluation progress value that indicates the progress of the evaluation for all evaluations, based on the evaluation completion time, which is the time when all evaluations are completed, or the application usage order corresponding to the evaluation items (first step). The evaluation progress value is calculated using the start time of each evaluation, the end time of each evaluation, and the evaluation completion time. The evaluation progress value is also the value obtained by dividing the time of each evaluation, calculated using the application usage frequency, by the evaluation completion time. In this disclosure, the evaluation progress value corresponds to the 50% evaluation progress value of application n shown in formula (2) or the evaluation progress value of application n shown in formula (3).

[0058] Next, in step S3, the server 183 calculates the evaluation progress rate as the ratio of the evaluation progress value to the reference value. In other words, the server 183 uses the results of each evaluation to calculate the achievement rate of the evaluation result against the specifications of the processing result (first specification achievement rate) (second step). In this disclosure, the 50% evaluation achievement rate of application n shown in formula (4) and the evaluation progress rate of application n shown in formula (5) are calculated. It is also possible to calculate the evaluation progress rate by creating a relational expression between evaluation time and application usage frequency in advance, calculating the evaluation time inversely from the application frequency, and then calculating the evaluation progress rate from the relationship between the reference value and evaluation time. In this disclosure, the relational expression for application usage frequency can be as shown in formula (6). It is also possible to calculate the evaluation progress rate from the application usage frequency, in which case it can be as shown in formula (8).

[0059] Next, in step S4, the server 183 determines whether or not an evaluation interruption has occurred. If there is an additional evaluation (Yes in step S4), in step S5, the server 183 calculates the sum of the time taken for the evaluations performed up to the present + the time taken for the evaluations to be performed from now on + the time taken for the evaluations to be re-performed. In other words, the server 183 calculates the sum of the time taken for the evaluations performed up to the present (the time it takes for the evaluations to be completed), the time taken for the additional evaluations, and the time taken for the evaluations after the additional evaluations. In the case of Figure 6(b), an evaluation interruption has occurred at time t, and the sum of the time taken for the evaluations performed up to time t (evaluation 1 and evaluation 2), the time taken for the evaluations to be performed from now on (evaluation 2' and evaluations 3 to evaluation Z), and the time taken for the evaluations to be re-performed (in the case of Figure 6(b), there are no evaluations to be re-performed, so this is 0) is calculated. In the case of Figure 6(b), the sum calculated is tE2.

[0060] Next, in step S6, the server 183 converts the sum of the evaluation times into a new evaluation time reference value. Specifically, in the case of Figure 6, the server 183 calculates the time corresponding to time tE2, the time corresponding to time t, and the time corresponding to the time t' when evaluation 2' ends.

[0061] Next, in step S7, the server 183 recalculates the first specification achievement rate by dividing the time required for each evaluation by the sum of the time required for all evaluations. In the case of Figure 6, for the series of evaluations including evaluation 2', the 50% evaluation achievement rate of app n shown in formula (4), the evaluation progress of app n shown in formula (5), and the figures are calculated.

[0062] If no interruption to the evaluation occurs (No in step S4) or if the evaluation progress is recalculated in step S7, the process proceeds to step S8. In step S8, the server 183 calculates the specification achievement rate by dividing the current specification calculated by the application by the required specification. In this disclosure, the specification n achievement rate shown in formula (1) is calculated.

[0063] Next, in step S9, the server 183 calculates a revised evaluation progress rate (second specification achievement rate) based on the evaluation progress rate and the specification achievement rate. The second specification achievement rate is calculated using the first specification achievement rate and is the specification achievement rate for all evaluations (a series of evaluations). The second specification achievement rate is also the value obtained by dividing the sum of the first specification achievement rates by the total number of evaluations. In this disclosure, the specification achievement rate shown in formula (9) is calculated.

[0064] Next, in step S10, the server 183 determines the evaluation status based on the second specification achievement rate and the evaluation progress. In this disclosure, the server 183 determines the evaluation status based on the revised evaluation progress and specification achievement rate. In other words, if no evaluation is added in step S10, the server 183 determines the evaluation status based on the second specification achievement rate calculated using the first specification achievement rate and the evaluation progress. If an evaluation is added, the server 183 determines the evaluation status based on the second specification achievement rate calculated using the recalculated first specification achievement rate and the recalculated evaluation progress (third step). The evaluation progress is calculated by dividing the evaluation time in the evaluation by the sum of the evaluation times. The sum of the evaluation times is the sum of the evaluation time in the evaluation performed up to the point in time when the evaluation was added, the evaluation time in the added evaluation, and the evaluation time in the evaluation after the added evaluation.

[0065] Next, in step S11, the server 183 detects the end of the demo. When one evaluation (demo) included in the series of evaluations is completed, the server 183 performs the next evaluation. Steps S4 to S11 are repeated until the series of evaluations is completed. In the case of Figure 6(b), there are no additions other than the addition of evaluation 2', so steps S4 to S11 are performed only once.

[0066] Next, in step S12, the server 183 calculates the actual time required for each evaluation after all demonstrations are completed, and calculates the progress rate for each evaluation by dividing the time required for each evaluation by the sum of the time required for all evaluations (calculating the third specification achievement rate). The third specification achievement rate is the achievement rate of the evaluation results against the specifications of the processing results, and is calculated based on the evaluation results after all evaluations have been completed. In the case of Figure 6(b), after time tE2 is completed, the time taken for each evaluation from evaluation 1 to evaluation Z (required time) is calculated. The ratio of the time required for each evaluation to the total time taken from evaluation 1 to evaluation Z (sum of the time required for each evaluation) is calculated, and the progress rate for each evaluation (for example, the revised evaluation progress rate and specification achievement rate) is calculated.

[0067] Next, in step S13, the server 183 determines the evaluation status based on the newly revised evaluation progress and specification achievement rate (determining the evaluation status based on the second specification achievement rate calculated using the third specification achievement rate and the measured evaluation progress) (fourth step). The measured evaluation progress indicates the progress of the evaluation for all evaluations and is calculated by dividing the evaluation time for all evaluations after all evaluations have been completed by the sum of the evaluation times for all evaluations after all evaluations have been completed.

[0068] Next, in step S14, the server 183 identifies delays in progress as problems based on the evaluation status. In this disclosure, formula (10) calculates the current delay in the specification achievement rate and formula (11) calculates the current delay from the deadline. The server 183 also determines the cause from the correlation between the specification history for each evaluation item, the usage history of the equipment used in the evaluation, and the problems.

[0069] Next, in step S15, the server 183 calculates the surplus of human resources and mechanical resources, identifies tasks that can resolve the cause by allocating the surplus to human resources and mechanical resources, and displays these tasks in the GUI. In other words, the server 183 displays a graph in the GUI with the second specification achievement rate and evaluation progress as two axes, and a graph with the second specification achievement rate and evaluation time as two axes. Furthermore, if a delay in evaluation progress is identified as a problem from the evaluation status, the server 183 displays in the GUI the tasks that resolve the problem by allocating the calculated surplus of human resources (resources of people allocated to the evaluation) and the calculated surplus of mechanical resources (resources of equipment used for evaluation) to human resources and mechanical resources, respectively. In this disclosure, as shown in Figures 7 and 8, along with the cause identified in step S12, the user is shown instructions regarding the allocation of the surplus.

[0070] Furthermore, the server 183 may store data in a database that combines evaluation progress issues, the causes of the issues, and the instructions corresponding to the causes of the issues, and use machine learning with this combined data as a dataset. By doing so, it is possible to improve the accuracy of surplus allocation and instructions to users.

[0071] Furthermore, in step S15, progress management may be performed by having means to visualize the evaluation progress, the specification achievement rate, and the status of the project related to the manufacturing recipe development. In this disclosure, the graph display unit 200 in Figure 7 may include a graph of the specification achievement rate and the evaluation progress, and a plot of the current position of the project on the graph.

[0072] Furthermore, in step S15, it is also possible to graph the specification achievement rate and evaluation time, and plot the current location of the project and the evaluation deadline on the graph. In addition, the GUI may have a display function that allows comparison of the specification achievement rate and evaluation progress, project status, progress problems, and causes of problems for multiple projects at multiple locations. In this disclosure, as shown in the corresponding graph display unit 203 of Figure 8, the deadline G is plotted, and the current location of projects A through C is plotted.

[0073] Furthermore, in step S15, if a new evaluation is added to the series of evaluations, it is also possible to plot the current location and evaluation deadline of the project to which the new evaluation has been added on the graph. Figure 11 shows an example of the GUI of the progress management system. Project A' shows the case where a new evaluation has been added to Project A. The corresponding graph display unit 203 shows the deadline and current location of Project A' by diamond-shaped marks. In addition, the cause display unit 204 shows information regarding the progress management of Project A'.

[0074] (Effects and Benefits) As described above, according to this disclosure, even if additional evaluations are required, the progress rate can be calculated accurately. In this disclosure, the evaluation progress rate and the specification achievement rate are used as indicators for progress management, and if additional evaluations are required, the evaluation progress rate and the specification achievement rate are recalculated to track progress. Furthermore, it becomes possible to identify problems and their causes derived from the evaluation status, and to issue work instructions corresponding to the identified causes of the problems, thereby enabling efficient demo evaluation and development.

[0075] In the above embodiments, we have described a case where a PC (PCA153, PCB163, PCB173) is used in the progress management system 1, but a virtual area, server, or mobile terminal may be used instead of a PC terminal. Also, the communication method may be FTP communication, NFS communication, etc., instead of SMB communication. Furthermore, a PC may be used instead of the remote monitor 182. In addition, in the above embodiments, we have described a case where a manufacturing device is used in the progress management system 1, but the progress management system 1 may also be used in a device that performs work involving evaluation.

[0076] Furthermore, the present invention is not limited to the embodiments described above, and various modifications are possible without departing from the spirit of the invention.

[0077] The following describes, but is not limited to, embodiments that may constitute the present invention. (Aspect 1) A progress management system comprising a platform on which a progress management application for managing the progress in the evaluation of processing results of semiconductor manufacturing equipment is implemented, wherein the progress management application performs the following steps: first step, which is the time when all the evaluations are completed, an evaluation progress rate indicating the progress of the evaluation for all the evaluations is calculated based on the evaluation completion time, which is the time when all the evaluations are completed, or the order in which the application corresponding to the content of the evaluation is used; second step, which is the calculation of a first specification achievement rate using the results of the evaluations; third step, which is the determination of the evaluation status based on a second specification achievement rate calculated using the first specification achievement rate and the evaluation progress rate if no evaluations are added, and third step, which is the determination of the evaluation status based on a second specification achievement rate calculated using the recalculated first specification achievement rate and the recalculated evaluation progress rate, the first specification achievement rate is the achievement rate of the evaluation results for the specifications of the processing results; the second specification achievement rate is the achievement rate of the specifications in all the evaluations; and the recalculated evaluation progress rate is calculated by dividing the evaluation time in the evaluation by the sum of the evaluation times. A progress management system characterized in that the sum of the evaluation times is the sum of the evaluation time in the evaluation performed up to the point in time when the evaluation was added, the evaluation time in the added evaluation, and the evaluation time in the evaluation after the added evaluation.(Aspect 2) A progress management system according to the progress management system described in Aspect 1, wherein the progress management application further executes a fourth step in which the evaluation status is determined based on the second specification achievement rate calculated using the third specification achievement rate and the measured evaluation progress, wherein the measured evaluation progress indicates the progress of the evaluation for all the evaluations, and is calculated by dividing the evaluation time after all the evaluations have been completed by the sum of the evaluation times after all the evaluations have been completed, and the third specification achievement rate is the achievement rate of the evaluation result for the specification of the processing result and is calculated based on the evaluation result after all the evaluations have been completed. (Aspect 3) A progress management system according to the progress management system described in Aspect 1 or Aspect 2, wherein the evaluation progress is calculated using the start time of the evaluation, the end time of the evaluation and the evaluation completion time. (Aspect 4) A progress management system according to any one of Aspects 1 to 3, wherein the evaluation progress is a value obtained by dividing the evaluation time calculated using the usage frequency of the application by the evaluation completion time. (Aspect 5) A progress management system according to any one of aspects 1 to 4, characterized in that a graph with the second specification achievement rate and the evaluation progress rate as two axes, or a graph with the second specification achievement rate and the evaluation time as two axes, is displayed on the GUI. (Aspect 6) A progress management system according to any one of aspects 1 to 5, characterized in that, when a delay in the progress of the evaluation is extracted as a problem from the evaluation status, the GUI displays the work to resolve the problem by allocating the calculated surplus of human resources and the calculated surplus of mechanical resources to the human resources and the mechanical resources, respectively, the human resources are the human resources allocated to the evaluation, and the mechanical resources are the resources of the equipment used for the evaluation.(Aspect 7) A progress management system according to any one of aspects 1 to 6, characterized in that machine learning is performed using a dataset of data consisting of combinations of problems in the progress of the evaluation, the causes of the problems, and the instructions corresponding to the causes of the problems. (Aspect 8) A progress management system according to any one of aspects 1 to 7, characterized in that the second specification achievement rate is the value obtained by dividing the sum of the first specification achievement rates by the total number of evaluations. (Aspect 9) A progress management system according to any one of aspects 1 to 8, characterized in that the platform is a server. (Aspect 10) A progress management system according to any one of aspects 1 to 9, characterized in that the evaluation is etching rate measurement.(Aspect 11) In a server on which a progress management application for managing the progress in the evaluation of processing results of semiconductor manufacturing equipment is implemented, the progress management application performs the following steps: first step, which is the time when all the evaluations are completed, an evaluation progress rate indicating the progress of the evaluation for all the evaluations is calculated based on the evaluation completion time, which is the time when all the evaluations are completed, or the order in which the application corresponding to the content of the evaluation is used; second step, which is the calculation of a first specification achievement rate using the results of the evaluations; third step, which is the determination of the evaluation status based on the second specification achievement rate calculated using the first specification achievement rate and the evaluation progress rate if no evaluations are added, and third step, which is the determination of the evaluation status based on the second specification achievement rate calculated using the recalculated first specification achievement rate and the recalculated evaluation progress rate, the first specification achievement rate is the achievement rate of the evaluation results for the specifications of the processing results; the second specification achievement rate is the achievement rate of the specifications in all the evaluations; the recalculated evaluation progress rate is calculated by dividing the evaluation time in the evaluation by the sum of the evaluation times. The server is characterized in that the sum of the evaluation times is the sum of the evaluation time in the evaluation performed up to the time the evaluation was added, the evaluation time in the added evaluation, and the evaluation time in the evaluation after the added evaluation.(Aspect 12) A progress management method for managing the progress in the evaluation of processing results of a semiconductor manufacturing apparatus, comprising: a step of calculating an evaluation progress rate indicating the progress of the evaluation for all the evaluations based on an evaluation completion time which is the time when all the evaluations are completed or the usage order of an application corresponding to the content of the evaluations; a step of calculating a first specification achievement rate using the results of the evaluations; if no evaluations are added, determining the state of the evaluation based on a second specification achievement rate calculated using the first specification achievement rate and the evaluation progress rate; if an evaluation is added, determining the state of the evaluation based on a second specification achievement rate calculated using the recalculated first specification achievement rate and the recalculated evaluation progress rate, wherein the first specification achievement rate is the achievement rate of the evaluation results for the specifications of the processing results; the second specification achievement rate is the achievement rate of the specifications in all the evaluations; and the recalculated evaluation progress rate is calculated by dividing the evaluation time in the evaluation by the sum of the evaluation times. A progress management method characterized in that the sum of the evaluation times is the sum of the evaluation time in the evaluation performed up to the time the evaluation was added, the evaluation time in the added evaluation, and the evaluation time in the evaluation after the added evaluation.

[0078] 150: Server A, 151: Manufacturing line equipment group A, 152: Server A', 153: PCA, 154: Internet, 155: Private environment of site A, 156: Public environment of site A, 160: Server B, 161: Manufacturing line equipment group B, 162: Server B', 163: PCB, 164: Internet, 165: Private environment of site B, 166: Public environment of site B, 170: Server C, 171: Manufacturing line equipment group C, 172: Server C', 173: PCC', 174: Internet 175: Private environment of site C, 176: Public environment of site C, 180: Cloud server, 182: Remote monitor, 183: Visualization tool or data analysis application server, 184: Inter-site common cloud environment, 200: Graph display unit, 201: Evaluation stage display unit, 202: Cause display unit, 203: Corresponding graph display unit, 204: Cause display unit, 1830: Bus, 1831: Processor, 1832: Memory, 1833: Storage device, 1834: Input / output device, 1835: Communication interface

Claims

1. In a progress management system comprising a platform on which a progress management application for managing the progress in the evaluation of processing results of semiconductor manufacturing equipment is implemented, the progress management application performs the following steps: first step, which is the time when all the evaluations are completed, an evaluation progress rate indicating the progress of all the evaluations is calculated based on the evaluation completion time, which is the time when all the evaluations are completed, or the order in which the application corresponding to the content of the evaluations is used; second step, which is the calculation of a first specification achievement rate using the results of the evaluations; third step, which is the determination of the evaluation status based on a second specification achievement rate calculated using the first specification achievement rate and the evaluation progress rate if no evaluations are added, and third step, which is the determination of the evaluation status based on a second specification achievement rate calculated using the recalculated first specification achievement rate and the recalculated evaluation progress rate, the first specification achievement rate is the achievement rate of the evaluation results with respect to the specifications of the processing results; the second specification achievement rate is the achievement rate of the specifications in all the evaluations; and the recalculated evaluation progress rate is calculated by dividing the evaluation time in the evaluation by the sum of the evaluation times. A progress management system characterized in that the sum of the evaluation times is the sum of the evaluation time in the evaluation performed up to the point in time when the evaluation was added, the evaluation time in the added evaluation, and the evaluation time in the evaluation after the added evaluation.

2. A progress management system according to claim 1, wherein a fourth step is further performed by the progress management application in which the evaluation status is determined based on the second specification achievement rate calculated using a third specification achievement rate and the measured evaluation progress rate, the measured evaluation progress rate indicates the progress of the evaluation for all the evaluations, is calculated by dividing the evaluation time after all the evaluations have been completed by the sum of the evaluation times after all the evaluations have been completed, and the third specification achievement rate is the achievement rate of the evaluation result for the specification of the processing result and is calculated based on the evaluation result after all the evaluations have been completed.

3. The progress management system according to claim 1, characterized in that the evaluation progress is calculated using the start time of the evaluation, the end time of the evaluation, and the end time of the evaluation.

4. The progress management system according to claim 1, characterized in that the evaluation progress is a value obtained by dividing the evaluation time calculated using the usage frequency of the application by the evaluation completion time.

5. A progress management system according to claim 1, characterized in that a graph with the second specification achievement rate and the evaluation progress rate as two axes, or a graph with the second specification achievement rate and the evaluation time as two axes, is displayed on the GUI.

6. The progress management system according to claim 1, wherein if a delay in the progress of the evaluation is identified as a problem from the evaluation status, the calculated surplus of human resources and the calculated surplus of mechanical resources are allocated to the human resources and mechanical resources, respectively, and the work to resolve the problem is displayed on the GUI, wherein the human resources are the human resources allocated to the evaluation, and the mechanical resources are the resources of the equipment used in the evaluation.

7. A progress management system according to claim 1, characterized in that machine learning is performed using a dataset of data consisting of combinations of problems in the progress of the evaluation, the causes of the problems, and the instructions corresponding to the causes of the problems.

8. The progress management system according to claim 1, characterized in that the second specification achievement rate is the value obtained by dividing the sum of the first specification achievement rates by the total number of evaluations.

9. The progress management system according to claim 1, wherein the platform is a server.

10. The progress management system according to claim 1, wherein the evaluation is etching rate measurement.

11. In a server on which a progress management application for managing the progress in the evaluation of processing results of semiconductor manufacturing equipment is implemented, the progress management application performs the following steps: first step, which is the time when all the evaluations are completed, an evaluation progress rate indicating the progress of all the evaluations is calculated based on the evaluation completion time, which is the time when all the evaluations are completed, or the order in which the applications corresponding to the content of the evaluations are used; second step, which is the calculation of a first specification achievement rate using the results of the evaluations; third step, which is the determination of the evaluation status based on a second specification achievement rate calculated using the first specification achievement rate and the evaluation progress rate if no evaluations are added, and third step, which is the determination of the evaluation status based on a second specification achievement rate calculated using the recalculated first specification achievement rate and the recalculated evaluation progress rate, the first specification achievement rate is the achievement rate of the evaluation results with respect to the specifications of the processing results; the second specification achievement rate is the achievement rate of the specifications in all the evaluations; and the recalculated evaluation progress rate is calculated by dividing the evaluation time in the evaluation by the sum of the evaluation times. The server is characterized in that the sum of the evaluation times is the sum of the evaluation time in the evaluation performed up to the time the evaluation was added, the evaluation time in the added evaluation, and the evaluation time in the evaluation after the added evaluation.

12. A progress management method for managing the progress in the evaluation of processing results of semiconductor manufacturing equipment, comprising: a step of calculating an evaluation progress rate indicating the progress of the evaluation for all the evaluations based on the evaluation completion time, which is the time when all the evaluations are completed, or the usage order of the application corresponding to the content of the evaluations; a step of calculating a first specification achievement rate using the results of the evaluations; if no evaluations are added, determining the status of the evaluation based on a second specification achievement rate calculated using the first specification achievement rate and the evaluation progress rate; if an evaluation is added, determining the status of the evaluation based on a second specification achievement rate calculated using the recalculated first specification achievement rate and the recalculated evaluation progress rate, wherein the first specification achievement rate is the achievement rate of the evaluation results for the specifications of the processing results; the second specification achievement rate is the achievement rate of the specifications in all the evaluations; and the recalculated evaluation progress rate is calculated by dividing the evaluation time in the evaluation by the sum of the evaluation times. A progress management method characterized in that the sum of the evaluation times is the sum of the evaluation time in the evaluation performed up to the time the evaluation was added, the evaluation time in the added evaluation, and the evaluation time in the evaluation after the added evaluation.