The invention provides a method for evaluating the degree of integration of industry knowledge into a
large model, and relates to the technical field of AIs, and the method comprises the steps: obtaining massive industry supervised corpora through the AI to establish an industry supervised
data set; the method comprises the following steps:
mining industry matching chunks through two times of aggregation driven by 2-
gram based on an industry supervised
data set to obtain a high-frequency chunk
library SensCommon; on the basis of a high-frequency chunk
library SensCommon, a maximum reverse matching word segmentation
algorithm is adopted to evaluate the
training effect of the industry fine-tuning
large model, and whether the model continues to be trained or whether the stored model is deleted or not is judged according to the
evaluation result; according to the method, two times of aggregation is driven by 2-
gram to mine long matching language chunks commonly used in the industry supervised
data set, the degree of integration of industry knowledge after industry great model training is effectively evaluated, whether the model is continuously trained or not and whether the stored model is deleted or not is judged, and therefore GPU computing power resources and hard disk resources are effectively saved.